Tech Interviews in the Age of AI
Blueprint on How to Interview When Candidates Can Easily Cheat with AI
👋 Hello Technocrats!
I’m Bobby, 4x CTO in Private Equity, Enterprise and Startups. '
For any leader running technical interviews the game has certainly changed in the last 2 years. Today I deep-dive into what companies can do to win at these interviews in the age of ChatGPT. 🤖
Cheers & let’s dive in!
Bobby
To a lot of execs, tech leaders & hiring managers it certainly feels like AI has completely destroyed the traditional tech interview.
When a candidate can just look up answers easily on ChatGPT how are you supposed to assess someone?
And hiring was already super difficult!
Here are some complaints I’ve been hearing from CTOs & Product Leaders in the last couple of years:
“Cheating is rampant in interviews and people don’t even seem embarrassed doing it”
“Candidates will just type my questions straight into ChatGPT”
“Resumes are all AI generated; I keep seeing the same descriptions over and over”
“Asking candidates to walk me through their reasoning results in 0 good answers…”
“I had candidates eyes jump from me to ChatGPT and then waiting 30 seconds, it was silly”
AI definitely IS changing tech interviewing & a lot of it is for the worse. And of course, working from home made it even easier for some candidates to work around the traditional interview process.
But companies also have their role to play in this (worsening) game. In fact, here’s where most companies mess up the technical interview:
Asking the same, lame, formulaic technical questions
Hiring managers that aren’t technical enough & have no idea what they’re talking about 😂
Relying too much on questions that don’t reflect real-world scenarios
Don’t bother to learn how AI tools work
Focus on tech trivia instead of real problem solving
Treat hiring like a checklist exercise instead of having deep conversations
So technical interviews are changing and if you look at it, both sides are to blame for the current mess—candidates who rely too much on AI to fake their expertise, and companies that cling to outdated hiring practices.
Let’s look at how companies can handle things better.
Adjust Your Tech Interviewing Strategy 🎯
Rather than focusing on tasks that can be easily outsourced to ChatGPT, hiring managers should pivot towards evaluating creativity, teamwork, and the candidate’s overall problem-solving skills.
So less standardized coding tests that can easily be gamed and more focus on past challenges and how they worked through them.
You really have to switch to probing for strengths and weaknesses and figuring out the candidates mindset — things ChatGPT can’t tell you.
More In-Person Interviews 🧑
When you can, bring people in to the office for interviews and do them in person. Much less cheating with ChatGPT happens this way.
It’s more time consuming but so is wading through dozens of bad Zoom interviews or hiring the wrong candidate (ouch).
Ensure that when candidates do come in person, they are given real, practical tasks that reflect the job, not just whiteboard puzzles.
Stringent Zoom Rules 🖥️
If you’re going to run interviews on Zoom you have to stick to a workable formula that mitigates some of the risks of AI-based cheating.
For example, the candidate has to share their screen with you, they have to download the coding exercise in front of you, and they have to solve the problem live with you.
In fact, live coding sessions can be super-helpful to assess a candidate.
That doesn’t mean they can’t use ChatGPT or some other tool. The only stipulation is they have to share their screen so you can evaluate their thinking not just pure code quality.
Know the Specific Requirements of the Role ✅
Every role is a bit different from company to company and so the requirements & questions should vary — unfortunately a lot of companies don’t get specific enough.
You should think through and include some role-specific questions in the interview. Let them be unique to your company & its culture.
Here’s an example for a QA interview: “At our company QA is not embedded with engineering, have you worked in this environment before and what are the pro’s and con’s as you see them?”
Don’t Ban GPT (Necessarily) 🤖
I’m not sure banning ChatGPT is the answer. After all, even though it’s powerful it’s still just a tool that you would probably expect your developers to make use of in their day-to-day.
But during the interview things are a bit different: what you’re trying to figure out is whether the candidate is a good thinker & problem-solver.
My recommendation is to allow ChatGPT if you can visually see the person using it on Zoom or if they are with you in person — that way you can still assess how they think about the code.
Of course, there are companies that ban it (just for the interview process or even all-together) and I think right now that’s OK too.
Train Your Recruiters on AI 👩
You have to get better at detecting the use of AI tools by candidates in terms of live during the interview and also in resumes.
You and your recruiters should know the popular AI tools. You should get logins and use them in your day-to-day in order to understand how candidates might use them.
Over time you’ll become better at spotting candidates who use AI to either grossly embellish their capabilities or outright cheat on a live interview.
Take Home Tests Can Still Work 📖
Take home tests can still work but not like they used to (even when you add “DO NOT USE AI” to the test like some companies do 😆).
You can’t actually use a take-home as a true test of coding abilities anymore because of GPTs, but you can use it as a jumping off point to have a conversation.
If the candidate completes a take-home test and can walk you through the steps they took in-depth and their reasoning behind their approach then the take-home can be useful.
Use Recruiters More Than Before 💰
As a tech leader, you don’t want to be wading through thousands of AI-generated resumes — instead leave that up to recruiters (internal or external) who have the tools & skills to detect this.
AI is used so prolifically in resume writing now that it might just kill resumes all together (good thing?). For sure it has ruined resumes for technical hiring.
Recruiters are good at spotting these resumes.
In fact, my suggestion is that if a candidate has produced their entire resume using AI and it’s just a bunch of generic garbage you should pass on them.
Art of the Conversation 👂
Finally, to win in the new world of AI-based “gaming the system” in interviews, companies must learn the art of the conversation with their candidates.
Interviews can’t be as “cookie cutter” as they were before. You have to be adept at digging deeper, adapting to each candidate, and steering the discussion in real time.
Instead of just running through a script, great interviewers follow up, probe for details, role play problems, and ask candidates to explain their thought process in unexpected ways.
The live coding session you did over Zoom or in-person is a perfect opportunity to discuss the candidates approach to problem-solving.
Final Thoughts
Tech hiring has always been difficult for many reasons…the least of which is assessing anyone in an hour is just straight-up hard.
You have to have some pretty great instincts as a leader to get a high percentage of your new hires to be successful, long-term employees.
But with the rise of AI (and specifically LLMs) interviewing has gotten even more difficult.
I believe resumes are mostly dying but good interviewing is not. In fact, smart interviewing is more important than ever for making good technical hires.
But you have to adjust your strategies to focus not just on formulaic technical questions, and instead more on creative problem-solving and teamwork questions.
(That’s probably how it should’ve been anyway!)
As a leader, you and your hiring managers also have to get better at the art of the conversation.
That means probing about past projects, how the candidate learns, their method of collaboration, do they give up at the first sign of trouble, and so forth.
Maybe you and your managers are good at conversation already, but if not you better start developing your skills.
(Btw, interviewers HAVE to know what they are talking about technically to have an in-depth discussion, so don’t just throw anyone at the candidate.)
Of course all this will take more time in terms of hiring, but it’ll be cheaper than hiring the wrong person.
Just keep in mind there is no perfect template for hiring — you have to use these tips and customize them for your organization.
If you make these adjustments you’ll be able to identify high-aptitude, trustworthy and motivated engineers again via your interview process.
Reach out for help if you need it: bobby@technocratic.io.
And in the meantime, keep the shark swimming! 🦈