How ChatGPT will change how we vet software engineers

This was originally posted on Linkedin, by the CTO of Hatchways.

After experimenting with using ChatGPT to solve LeetCode questions, I was blown away by its capabilities.

I asked it to solve the maximum subarray problem (a medium LeetCode problem) in Rust. Not only did it give me an efficient working solution, but it also explained the solution:

Here is where it gets more interesting. The tool can act like your own personal tutor. It remembers the context of the conversation, and you can ask it to improve the solution. I asked ChatGPT to make the variable names more meaningful and add more documentation. Not only did it adapt the solution to my request, but it also explained how it improved the solution:

I even asked it to "change it up" and it modified the way the array was iterated in the solution.

What does this mean for how we vet software engineers?

As companies look for ways to vet software engineers, the use of AI-powered tools like ChatGPT can present challenges. In the past, companies could use an online vetting tool like HackerRank and ask candidates to complete a programming problem from the tool's library. If candidates just copied a solution from the internet, the vetting tool would notice the similarity of the answer to previous submissions and flag it for plagiarism.

However, with a tool like ChatGPT, candidates can use generative AI to create unique answers. For problems that are commonly seen (like existing LeetCode problems or those part of a vetting tool's library), the tool would easily be able to solve the problem for you. It would even explain to you the solution and thought process, while also allowing candidates to modify the solution to make it unique and hard to be flagged for plagiarism.

This means that companies must consider the possibility that job applicants may be using AI tools like ChatGPT to impress during the vetting process. As AI technology continues to advance, it's important for companies to stay up-to-date and find new ways to accurately assess the skills of potential hires.

Companies will start creating more custom questions

As a result of ChatGPT, answering commonly-seen LeetCode questions will be trivial for candidates. In response, companies will start moving away from these types of questions and instead ask more sophisticated problems during the interview process, particularly if they are using a take-home assessment.

Here are some potential changes to the interview process that companies may implement to better assess the skills of potential hires:

• More closely replicate a real-world work environment by providing candidates with a codebase and asking them to fix bugs, write features, perform a code review. See here how we create more practical real world type assessments at Hatchways
• Asking candidates to solve open-ended, real-world problems that are not commonly found online
• Conducting pair-programming sessions where the candidate works on a problem with one of the company's engineers

Although ChatGPT can still help candidates with the tasks above (such as debugging, writing, and analyzing code), when you provide candidates with a more practical real-world problem embedded in more context, ChatGPT will become more of a tool to help candidates as opposed to the problem solver.

Companies will also need to benchmark and test their questions assuming candidates have access to ChatGPT. They will need to see how the tool can help candidates solve the problem and that the problem is sufficiently hard enough to differentiate candidates.

What does the future of interviewing look like?

Not only can ChatGPT be a tool to help candidates solve interview problems, but it'll also eventually become a tool to help companies build more sophisticated asynchronous assessments.

Imagine being able to use ChatGPT to create a take-home assessment where candidates can leverage the technology to solve a problem together. With this approach, you can address the challenge of candidates using ChatGPT to solve take-home assessments by actively monitoring and allowing for its use. By embracing this approach, companies can gain valuable insights into how candidates interact with the technology and use it to solve problems.

With ChatGPT, take-home assessments have the potential to become pair programming sessions. This approach can provide valuable insights into a candidate's ability to communicate and collaborate, allowing you to better understand their thought process. This is often the goal of technical interview processes. By combining take-home assessments and pair programming, ChatGPT has the potential to create a new type of interview process that offers the benefits of both. This approach would allow candidates to complete a pair programming interview on their own time and in a low-stress environment.

‍