Using a Code Review Assessment to Assess Software Engineering Talent

Introduction

Building a practical interview process for software engineers is hard. It especially gets harder when you are trying to hire senior engineers. Senior engineers have to do a variety of tasks on the job from designing large-scale applications to mentoring junior engineers. When evaluating their technical skills, they can often get offended if you give them a screening quiz involving data structure and algorithm questions. How relevant is that to the job?

Enter the "Code Review Assessment". This is a new interview type that is gaining popularity fast, especially among hiring and vetting senior engineers.

What is a Code Review Assessment?

In this assessment, candidates would receive a pull request (or merge request) and would be asked to perform a code review. Instead of asking the candidate to write code, the candidate will be required to read the source code and provide detailed comments on issues and areas of improvement. This type of assessment does not only measure technical skills, but important soft skills like communication, ability to give feedback, and attention to detail.

Benefits of Code Review Assessments?

Code review assessments have a variety of benefits - mainly that they can be completed in a short amount of time and provide a lot of signals about a candidate. In the interview process, you often want to optimize the "time-to-signal" ratio - how do you get the most amount of signal in the shortest amount of time? The code review assessment can help with that. Here are a few of their benefits.

Code Reviewing is Faster Than Writing Code

Most technical assessments take too much time. A code review assessment can be confined to less than one hour (on a call or through a take-home fashion) compared to other types of assessments that require candidates to write code.

Code Reviewing is a Relevant Skill

Unlike traditional data structure and algorithm assessments, providing feedback on code is a relevant on-the-job skill. Code review is a common responsibility for an engineer of all levels. It measures not only coding skills but also evaluates communication and other crucial soft skills.

It Fits Well in the Interview Process

The code review assessment can feed nicely into subsequent stages of the interview process. For example, following a code review assessment, you might ask candidates to write code to fix mistakes they identified. This makes the entire process seem more cohesive as opposed to a disjointed series of steps that interview processes often feel like.

Challenges of Code Review Assessments

Like all interview styles, there are some drawbacks to this interview format.

Cost of Implementation

A code review assessment can be difficult to create and maintain. Often times you want to make the assessment relevant, so you'll need to refresh it once in and while so the content stays relevant. Furthermore, if you want to offer candidates a choice of programming language, you'll need to create multiple versions of the same assessment in different languages. This can be hard to accommodate.

Difficulty in Maintaining Consistent Standards

Since this assessment involves assessing soft skills as well as hard skills, it can be difficult to maintain consistent standards when reviewing these assessments. You'll want to be sure that you create an effective scorecard to evaluate solutions.

Writing Code is not Evaluated

One of the biggest drawbacks of this assessment style is that you are not assessing how well candidates can write code. You would have to combine this assessment with another interview step if you'd want to see the candidate's proficiency in producing code.

Addressing these Challenges

In the rest of this post, we will share how to create a code review assessment that addresses some of these challenges. Firstly, we share methods on how you can create a consistent standard by sharing how we use feedback forms to evaluate solutions. Secondly, we share one example of a code review assessment to help you get started and reduce the cost of implementation. If you are looking for solutions to help you create and maintain code review assessments be sure to check out Hatchways.

Creating a Code Review Assessment

As mentioned in our creating a React assessment blog post, to create an assessment, you don't start with the assessment. Instead, you need to start by identifying the key skills you'd like to test. Here is a rough list of the steps required to create a code review assessment:

1. Identify the evaluation criteria: The first step is to determine the key topic areas and skills you'd like to assess. You'd want to consider the role you are hiring, the seniority level you are targeting, and the type of day-to-day work they would be doing.
2. Gather inspiration: The next step is to get some inspiration for the assessment. Your own code base can be one area of inspiration. Look at past pull requests you had in your organization and filter by those with lots of comments (you can use a query like `is:pr sort:comments-desc`). Alternatively, you can find inspiration from existing assessments online (like those on hatchways.io) or the example we will provide below.
3. Build the assessment: Based on your inspiration, it is time to create the assessment. The nice thing about a code review assessment is you don't need the code to be perfect!
4. Develop the feedback form: Before sending this off to other candidates to complete, it is important to build a feedback form that you'll be using to evaluate the solution. Make sure to focus on the evaluation criteria you determined in Step 1. In this section, you'll want to create a code review checklist containing all the code review comments you'd expect a candidate to find. This feedback form can be iterated on as you test and iterate on the assessment. The quality of your assessment is based both on the assessment itself as well as the feedback form used to evaluate the solution.
5. Test and iterate: Your first attempt at the assessment will not be perfect! No need to worry, it is important to track candidate experience and the signal you are receiving from the assessment to improve it over time. Again, you want to optimize the "time-to-signal" ratio.

An Example Code Review Assessment

In this section, we go over an example code review assessment. We will be sharing snippets of the assessment itself. However, if you want to see the full assessment, you do so here (you'll need a Hatchways account first to view this link).

The Challenge

In this assessment, candidates are asked to review the implementation of a cron job for a payroll management application. The cron job is supposed to run daily emails to notify administrators of each company that they must run payroll for a specific employee today. Here is a snippet of some of the code a candidate would have to review:

As you can see from the code, the employees are paid on different schedules (`bi-weekly`, `bi-monthly`, and `monthly`). The pull request also contains automated tests (where two tests are actually failing due to some bugs in the code).

The Feedback Form

As mentioned earlier, it is important to develop both a good assessment as well as a feedback form that will be used to evaluate the solution. In this section, we will share a glimpse of the feedback form we use to evaluate this assessment to help demonstrate how to create an effective feedback form.

Code Review Checklist

The first part of the feedback form is a checklist containing a specific list of items that we'd expect an engineer to comment on during the code review process of this assessment. Each line item in the checklist should be specific and you are just looking for the presence of comments in this part of the feedback form. Here are a couple of examples of checklist items (not specific to this assessment, as we wouldn't want to give candidates an edge on what we are looking for!):

Notice that the checklist has a subheader ("Comments on Code Quality"). This allows you to group the comments found into different categories that can match the next part of the feedback form.

The Scorecard

The next part of the feedback form is the scorecard, which allows our reviewer to give a rating on how the candidate performed in a variety of different categories. Some of these categories will correspond to the subheaders in the previous checklist question and some of these categories will be about the overall performance of the code review. When developing a scorecard, it is important to be explicit about what each category and level means.

In this assessment, we use the following categories in our scorecard:

• Comments on Code Quality
• Identifying Bugs and Suggesting Improvements
• Comments on Efficiency / Optimization
• Tone & Clarity of Feedback
• Comments on Product and User Experience

The Decision Question

The last question in the feedback form is the "Decision Question", which summarizes the reviewer's opinion on the candidate's submission. It is essentially the decision if the candidate should move forward in the interview or not.

It is important that this question is not answered based on just gut feeling. Instead, the other questions in the form should be used to determine this overall rating combined with the expectations for the candidate's seniority level. You will want to create a marking guidelines document that outlines what each level means for different seniority levels as shown in this blog post.

Conclusion

From our experience, code review assessments have received great feedback from candidates completing the assessment as well as interviewers using them in their interview process. The image below is from a Hatchways customer that used this specific code review assessment in their interview process.

We hope this post helped demonstrate some of the benefits of a code review assessment and provided you with an example so you know what is involved in creating one. If you are looking for some more examples, be sure to check out the Hatchways assessment catalogue.

‍