When creating questions, we like to follow these guidelines. We have two types of questions, public and premium. Our public questions are examples of our premium questions therefore the same set of guidelines apply to all submissions:
- The question should be written to evaluate the candidate first. The question should separate the good candidates from the bad.
- The question should not have ready-made solutions available or be easily answerable using search engines.
- The question should be a sample of actual work. It should be easy to imagine running into the problem during a normal work day.
A sample of work
Rather than asking candidates to reproduce facts and equations from a textbook or simply implement a well-known (or obscure) algorithm, our questions are samples of the type of work and higher-order thinking skills (apply and higher) that would be expected of an employee.
A coding example:
Write a function to traverse a tree data structure with N child nodes.
This is not a very good question. There's very little motivation or story behind the question, no employee outside of academia would be asked to write a function in this way. Here’s an example that more closely represents a real world unit of work:
Shipping Inc.'s organizational chart represents the hierarchy and relationship between each person and role. You have been asked to write a function to collect all employees that ultimately report to a specific manager.
In the above example, the problem has an additional layer of abstraction. Rather than being told that the data is a tree, the candidate needs to first identify that it's a tree. Rather than being told that they need to write a function to traverse a data structure, they need to first work out what the question is asking of them. This additional layer of abstraction is often what separates academia from the private sector.
Here's a multiple choice project management question example. A bad question may ask:
Which of the following definitions best defines "Closing" a project?
This question is bad as it asks for an easily googleable textbook definition that has only one correct answer. An improvement might be:
Select all the statements regarding closing a project that are correct.
This sets up a question that has multiple correct answers, which are not easily googleable. While we do have some questions like this, it misses the story element. A further improvement could be:
An engineering project that you have been managing has been delivered and is coming to an end. The clients have taken delivery and are happy with the product. Select all of the activities that need to be carried out to complete the project.
This makes no mention of "closing" and, thus, is more difficult to google. It requires the candidate to, first, identify and categorize the problem, rather than being able to easily read it, and, then, use their own creative skills to choose the answers that are most appropriate.
We aim for our questions to be either easy or hard. Easy questions should be solvable by good candidates for a junior position, and hard questions should be solvable by good candidates for more experienced or senior position.
Question type and length
Question length can be difficult to evaluate, when we release a question we usually want to give candidates more time than we think should be required, then gradually reduce it based on candidates submissions.
Live coding questions:
- < 7 min questions should expect no more than a 1-3 line change. The template should be no more than 20 lines.
- 10-15 min questions should expect no more than a 1-10 line change. The template should be no more than 40 lines.
- 15 min questions should have a template of no more than 40 lines.
- Where possible, the question description should contain an example case of the code.
- Live coding questions should have no more than 4 test cases. The first of which should always be an example case (the same as the description).
- Each test case should be a common bug in the solution.
- Test cases should have simple descriptive names. For example, “Each owner has a single file” or "Various files".
- Algorithmic thinking questions often have performance test cases, for more details see the section 'Algorithmic Thinking Questions' below.
- Should give time for reading the question, and an appropriate time for any calculations required.
- Have 4-6 options. In rare cases, there can be more options, but 8 is absolute maximum.
- They should preferably be multiple-correct-answer questions where at least 2 options are correct and at least 2 are incorrect.
- Make sure it is clear from the question description that there are multiple correct answers.
- Where possible, avoid similar/confusing answers. For a coding question, this would involve reducing syntax-like difference answers, and, for a numerical question, ensuring that all answers are based on possible mistakes, rather than consecutive values.
The questions should use simple American English, and common words and phrases. Descriptions and titles should follow the Chicago Manual of Style guidelines where possible. We use the following style guides for our coding questions:
Questions should have 1-2 topics (rare cases may use several more, if a good rationale can be provided). A question should focus on testing a single topic well. When writing a new question, it's often useful to decide on the topic(s) first. Once a topic is decided, a story and question can be written around that topic.
General/broad topics should be avoided where possible. It's much more useful for users to be able to see the specific technologies, language constructs, and concepts that the question is testing.
Focus the question on the topic as much as possible. Remove any additional constructs or concepts that might confuse or distract a candidate from solving the problem.
Each description should state why the developer should know this topic. For example, "Every programmer should be familiar with data-searching methods, as they are very common in data-analysis processes.".
The public questions should all contain hints that allow our public candidates to solve a question more easily if they are struggling. Short questions (under 5 mins) should have 1 hint, longer questions should have at least 2 hints. The hints should provide some insight that points the candidate in the correct direction, rather than providing part of the answer.