How to iterate on a take-home assessment

Take-home assessments need iteration all the time. When you first launch a new assessment, it is rarely perfect - you cannot predict what is going to happen in reality.  For example, one of the most common problems is the take-home assessment is too long - candidate experience suffers and people take too long to move to the next stage of the interview process. Subsequently, as your organization grows, the skills you are looking to hire for also change, making iteration of your take-home assessment crucial to hiring the right time.

In this article, we will go over common problems we see with take-home assessments and how to iterate on your assessment to get to a point that really works for your team.

Problem Area #1: Candidate Experience Suffers

One of the most common issues with take-home assessments is you notice that candidate experience starts to suffer. There are a couple of ways to detect this as a problem:

1. A low completion rate for the assessment. The benchmark for completion rates vary based on the role's seniority and the assessment's placement in the hiring process (whether it's an initial screening tool or an alternative to an onsite interview).
2. Getting poor feedback on candidate NPS surveys or in discussions in subsequent interviews with the candidate.
3. Candidates are just taking too long to complete the assessment, and the momentum to hire slows

‍

When candidate experience starts to suffer, its important to dig into why the problem exists. You will always get the candidate or internal team member that says candidates just don’t like doing take-homes. However the reality is this is rarely the full truth of the matter. You need to dig deeper into why - as this sentiment was certainly developed based on a poor experience of a take-home assessment in the pass. Take-home assessments are superior in theory to live interviews - as they replicate the real-world better and allow candidates to work in a less-anxiety environment and on their own time. There must be a deeper reason on why candidates are not completing the assessment. Here are a few of them.

The assessment takes too long to complete

A recurring problem with many take-home assessments is that they are too long. Companies often craft lengthy evaluations, not out of necessity, but because they aren't sure about what they really want from candidates. So they end up creating an assessment that evaluates too many skills.

Below are 3 simple steps to follow on how to shorten your take-home assessment.

1. Determine the core skills you want to assess

Often, the root issue is a lack of clarity about the desired qualifications in a new hire. This results in a wide net being cast, evaluating far more skills than necessary. This not only overwhelms candidates but also skews hiring decisions. Begin by narrowing down the essential skills required for the role. Knowing these specifics will provide clarity on where to make necessary cuts in the assessment.

2. Review past candidate performance

To fine-tune your assessment, revisit the rubric or scorecard you've been using to judge candidates. An insightful action would be to review past candidate scores. If there's a section where candidates consistently earn similar scores, consider its relevance. Lack of score variability could suggest that the section isn't differentiating candidates effectively. If it doesn't yield insightful information, it might be redundant.

3. Trim the assessment

in this step, the idea is to cut out anything that is essentially not providing you signal or is providing the wrong signal.

A common time-drain for candidates is unnecessary setup, especially writing boilerplate code that doesn't directly relate to the skills being assessed. Instead of having candidates start from scratch, provide them with a basic codebase. This can significantly reduce the assessment's length, allowing candidates to focus on the core tasks. For instance, when designing a stellar React assessment, starting codebases can be an invaluable asset.

Candidates never find the time to start the assessment

You many notice that candidates never end up starting your assessment. One possible reason for this is they have difficulty finding the time to start the assessment. Here are a few workarounds to this challenge.

1. Ask the candidate to commit to a specific deadline

We have noticed that you can improve the number of candidates that start (and complete!) the assessment by asking for them to provide a deadline on a recruiter phone screen prior to the take-home challenge. People generally like to hit deadlines they provide and if they do not, they usually reach out with a reason why which gives you more visibility on their situation and the interest they have in your role. This also leads to a good candidate experience because you are allowing the candidate to set expectations and you get a good understanding of any constraints they might have.

2. Ask the candidate to schedule a time to work on it

Another effective strategy is to actually turn your take-home assessment into a scheduled async interview. The idea is that candidates can pick a time and date to do your assessment, and they receive the assessment at that time/date. You can even timebox the assessment if that is a signal you want to have. We have seen companies use this strategy to increase their completion rates to 90-95%.

Candidates are looking for a more personal experience

Sometimes, the impersonal nature of take-home assignments can discourage candidates. They might feel that the company resorts to such tests simply to avoid investing time in potential hires.

To alleviate this, consider making the process more engaging. For instance, adding candidates to dedicated Slack channels or other communication platforms can foster a sense of community and support. They can use these channels to ask questions, or even just to feel more connected to the hiring process.

Furthermore, design the assessment to be a true reflection of the job. When candidates realize that the tasks they're working on mirror real-world scenarios they'd encounter on the job, they're more likely to appreciate the thoughtfulness behind the process.

Lastly, consider positioning the take-home assignment later in the hiring sequence. Meeting the team or having a few initial discussions before diving into the assessment can make candidates feel more valued.

Candidates feel like this is an irrelevant assessment

There are times when candidates might feel that the take-home test doesn't truly align with the skills required for the role. This sentiment could be closely tied to the nature of the assessment itself. If candidates perceive the tasks as disconnected from real-world job responsibilities, they might question its relevance. In such cases, it's vital to revisit and possibly redesign the assessment. Ensure it's a genuine representation of the tasks they'd undertake if hired.

Problem Area #2: Capturing the wrong Signal

The other very common problem area when it comes to assessing candidates with a take-home, is the assessment is not capturing the right signal. Here are a couple of ways to detect this problem:

1. A disproportionate number of candidates fail the assessment. For instance, if only 5% of candidates are passing, despite the recruiter's confidence in their quality, it suggests a misalignment.
2. Many candidates sail through the assessment but struggle in subsequent interview stages. This not only induces interview fatigue among your team but also diminishes the perceived value of the take-home assessment.
3. There's a disconnect between performance in the assessment and actual job performance for hired candidates.

Each of these problems have different possible solutions. We go through each of them below.

‍

Not enough candidates are passing the assessment

When a noticeable majority of candidates fail the assessment, it's evident that the test might be too challenging. Rather than brushing it off, it's crucial to investigate the underlying reasons. Are candidates consistently stumbling over the same sections due to ambiguities? Would clarifying the instructions mitigate these misunderstandings?

There are multiple strategies to enhance the assessment's efficacy:

Modify the assessment itself

Firstly, you can modify the assessment requirements themselves. Often this involves digging deeper into why candidates are failing. Some times the reason can be as simple as the instructions are too vague and you are looking for a specific solution to a problem, while the candidate is solving a different problem. A simple solution to this is to just modify the assessment instructions to be clearer on what you are looking for.

Modify the marking guidelines

Secondly, you might be happy with the assessment, but the marking criteria may be misaligned with the role you are hiring. In this case, you might want to keep the assessment requirements the same and just revise the evaluation rubric to better align with what you're aiming to gauge.

Modify the way the assessment is done

Lastly, you might want to even change the way the assessment is administered. For example, integrating automated tests that candidates can view their results to know if they are on the right track might be beneficial. This not only provides immediate feedback but also offers an opportunity for resubmission, which can be insightful for both the candidate and the evaluator. Another strategy might be to allow candidates to ask clarifying questions about the assessment if the instructions are purposely vague.

Candidates are doing poorly in subsequent rounds

If candidates ace the take-home but falter in subsequent interview stages, there's a misalignment somewhere. Either the assessment isn't gauging the right skills, or the subsequent rounds are deviating from what's truly essential. One best practice when using a take-home assessment is to always supplement it with a synchronous call post-submission. Such calls can validate the authenticity of the submission and, more importantly, provide insights into the candidate's thought process. It also serves as a checkpoint to identify and rectify any gaps in the evaluation process.

To solve this particular issue, you can follow the same steps as the previous topic area, however you might need to learn more about why by looking at feedback given to candidates in the subsequent rounds.

Candidates performing poorly on the job

The most pressing concern is when candidates who perform well in assessments underperform once they're on the job. Such discrepancies necessitate a thorough review of the entire interview process (not just the take-home assessment. This involves scrutinizing the competencies the assessment intends to measure and verifying if they're indeed being accurately evaluated.

A practical way to validate the assessment's effectiveness is by having internal employees, from various levels, attempt it. This internal testing can be illuminating, as the results can then be juxtaposed with the known competencies of these employees, thus highlighting any areas of the assessment that may need recalibration.

Conclusion

Creating a good take-home assessment is no easy feat. If you are having any of the challenges above, consider using Hatchways to assist you in the process. Our platform offers robust analytics, as illustrated by the screenshots highlighted earlier. Not only can we help refine and enhance your assessments, but we also provide valuable benchmarks, ensuring your evaluations remain effective.

‍