The same way we did before. Very very simple code submission (most people using AI use it even though we call out we're going to ask them later to modify later without AI tools writing code for them) then pairing interview where we ask some basic "are you actually at the level you say you are" question, then ask them to extend their program submission with:
- engineers on the call as pairing assistants
- google, ai tools, whatever for libraries, syntax, etc. we tell the candidate directly that it's impossible for us to gsther signal on how they think about problems if they ask Claude to just whip them up a solution
- the expected output
- their own unit test suite
Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go. I've interviewed dozens at this point. I'm not saying "they're unfamiliar with the structure", I'm saying "they cannot actually break down the problem even verbally".
It doesn't matter their pedigree or past experience on their resume, if they used AI to generate they don't seem to be able to resurrect the skills that actually matter for the thing, engineering and product work.
Note because I know folks hate code submissions. It's not hard. We give a CSV with 3 columns, 10 lines. Do some basic mapping and some structuring, some basic data modeling. We only expect about 1 actual class or struct. Then unit tests and it should run in the terminal. Max submission length with verbosity has been a java program at something like a hundred lines total if that, most folks complete the submission in an hour or two. Extension is that we modify one of the rules and extend the CSV by 5 lines.
I seem to still be getting good signal from this, since the engineers that I've hired off of this have been fantastic with or without AI tooling immediately in their hands during the day.
We were hiring for senior devs java/spring boot so I coded a tiny "Todo app" rest API, basically a single crud controller, and filled the code with many mistakes. Some obvious, some less obvious. Missing authentication, no tests, logging with System.out etc.. then we ask the candidate to review the code (we provide a laptop with a few IDEs but no AI tools).
When they point out errors, I get them to explain why they are errors by playing the junior engineer to see how they explain technical issues to someone that might not have the same understanding
I am not looking for them to find all issues but if they can't find at least some it's a red flag.
It takes about 20-25 minutes and gives really good insights. The other 30 minutes we asked some questions about past projects and some technical questions
To me AI hasn't had any impact on interviewing, only on filtering the CVs as I immediately flag obviously LLM-generated ones that don't have any substance, while those that are LLM-assisted with tasteful judgement are fine.
On the interview side, I have gone through a phase which required do an architectural design task on a whiteboard. I realized how stressful and not very helpful at evaluating it might be when I myself had to sit on the interviewee's side several times. It's difficult to get into this play-design mode.
Currently, I just make natural conversations, ask some technical questions on the level where I estimate the candidate is, try to understand the level of their knowledge and whether I would like to work with them in one team as a person. Maybe it wouldn't work for highly specialized roles, but for a general backend work I have been happy with the results.
Leetcode doesn’t tell you anything apart from someone memorised set of questions and their solution.
For your question, I assume you want people who can solve problems, can explain their thought process and reasons of decisions. I did interview devs recently, and my questions were related to their experience in their CV. Something like, tell me about project X in Y company, what did you do, what did you use, why, what did you learn from it.
My personal opinion is that LLMs are no difference from developers who over engineer and over complicate things. You can get them to do good work with the right steering and right understanding of the whole system within the context of the company.
Measure them on merit, their body of work and how well they will fit into the team culture.
As for the questions you’re asking, instead of asking old-school coding interview questions ask them stuff that is actually relevant to agentic programming.
Ask them to build a harness, have them explain what skills are, have them build a self-made implementation of Claude Code as take-home assignment.
The times have changed, you need to update interviewing questions to meet them. Start with agentic questions and ramp them up into more complex scenarios. This is how you gauge your candidates, see how they think and solve their way out of it.
Try to determine how creative, adaptable, motivated the candidate is. Figure out what there actual skill-sets are that they can bring to you, not whether they know some CS pedantry
After doing agenetic engineering in anger for about a year now, I'm starting to sour on it quite a bit. If I got this as a take-home, I think I would decline to go further. Watching the atrophy of people's skills and judgement has taught me that I value working with humans who can think for themselves.
Give them a time-constrained challenging problem that is wide in scope and see how well a candidate can decompose it before feeding it to AI. You can get pretty high signal within the first 2 prompts. Are they able to effectively steer the model or do they just ride with the flow and accept every AI suggestion?
How well do they know their models and their limitations? Are they able to switch tools/models effectively on the fly depending on the task? Or are they just using cursor auto mode and copy-pasting the task description into their IDE? Do they have a custom harness/workflow? What skills are they using, if any?
Judge them on quality of the output first. And pay attention to their taste.
So I actually had a take home challenge for an interview a few months ago, which didn't ask for any code at all. Instead they gave us a prompt for something to build (a tool that could break down an image into it's component colours), and we basically built whatever we wanted.
Honestly one of the most fun interviews I've ever done - it involved really understanding the problem, and building an interesting solution. AI was totally allowed and expected, but you had to prove you understood the problem and what you'd made in the interview. I felt it was actually way better than traditional interviews, because it gave a chance to show the actual quality and thought of my work rather than a glorified logic puzzle.
The org I am at we have build an internal harness for coding, and we only hire devs that do work "AI-first". We did start hiring juniors as well as seniors; the process for either of them is the same. We drop them into any of our repos without any prior knowledge of what the project is about; and ask them to interact with the harness to build a larger feature that has been specified in project management. For the task, they can only use AI; there is no room for any manual coding at this particular org. We grade them more or less on their interaction and problem solving patterns, as well as asking us questions; and how they write prompts. The coding interview last 2 hours, and there are only 3 rounds of interviews. Before the coding interview, we do send out a brief of what it will entail.
Maybe this sounds a little strange, but this has worked exceptionally well, as we have now several juniors working with us, that do still get checked by seniors. The harness itself which holds several levels of standards, rules as well as quality gates, allows these juniors to ship out a huge amount of high quality code. We still reject about 80% of the people who interview with us, because even though they know how to code with AI, their AI interaction patterns just aren't on par with what we need. I really started to like this process. Also to clarify, the harness isn't some Coding agent setup, but rather a whole setup that works well with Codex, Claude or Cursor; its well maintained versioned and constantly improved upon, with a huge amount of automations, rules and hooks. The moment someone starts working on anything at all, this is immediately connected and tracked with project management.
That sounds like a really great idea. Would you be open to sharing more about how the tool is designed? Are there any open-source or commercial tools like that out there right now?
Ive been trialing different types of interviews for over 2 years now and here’s what I’ve found.
I tried 2 flavors of a technical interview and this round was 1 hour in total.
The first was a traditional interview problem where we give the candidate a codebase and a docker image to run against. The problem itself involves using the docker image to post and get responses from but the candidate is expected to write a new helper/service in any language they choose. We ask for no agentic workflows for this but any other resource is fair game. This has been the most successful for us to find candidates and generally provides the best experience.
Our full agentic interview involves an existing fullstack typescript codebase that we give to the candidate a week or two before. The candidate is free to use any agentic workflows they want during the interview and we give them a ticket that we went to implement during the interview. I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping. This is lead to the biggest disparity in results of candidates so we stopped doing it.
> The first was a traditional interview problem where we give the candidate a codebase and a docker image to run against
Is that also on a company provided computer?
There are a lot more scams these days where a job-seeker compromises their computer by running provided code, and often docker isn't much against malice.
> I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping. This is lead to the biggest disparity in results of candidates so we stopped doing it.
Agentic engineering is the new microservices craze. Everyone's doing it, and some will do it exceptionally badly. This is always how these trends go. The anecdote you give here of some people just one shotting it in Claude Code without thinking about it and this producing extreme variability matches my experience. Basically, the longer you go down this road you realise some people will still actually think about things and some people will just cede cognitive control and stop thinking much at all. If you're the former, working with the latter is exceptionally draining over the long-term. I have in recent times become pretty bearish on agentic engineering for this reason.
I am very curious about this as well, as an IC. I have not written code by hand for almost two years now and have produced a mountain of Python code over the past year (well architected, using all of my experience in other languages to support good judgement). However, Python is new to me -- I've never hand written a line.
This is all fine and good at my current job, but were I to ever go back on the market. What an uncomfortable conversation!
Are they interesting to talk to? Are they passionate, curious, authentic, opinionated? Do they have a product mindset? Do they need to be told what to do? Would you hang out with them after work? Can they explain themselves in a way that’s easy to understand?
In my experience these are much better predictors of success than leetcode.
i think it would make more sense to let developers whatever coding agent they want to use to build something complex and then have them explain every bits and pieces of what was built, why, alternatives, design, scalability considerations etc.. this would result in much better technical discussion instead of having someone code manually. every developer needs to use a coding assistant today to build what they need to with the best quality in short time. so interviews need to be way to see if the developer can do that in real work.
Recently, I was screen-interviewed for coding speed with a fairly easy project. The company used a 3rd party platform which gave me 90m to complete 4 levels, each one building upon the other.
I think that was a nice test, but the platform is too buggy to recommend.
Basically the same as always--an extended interactive Turing test. Can they speak coherently and in-depth about the things they claim to have done on their CV/resume?
how many applicants are you getting? this plays a big role.. how many submissions do you have to sift through will dictate what ai agents to use. if you value the job: multiple of interview rounds are needed. Then its up to your hiring crew. Take this time to create your system of hiring, Of course a simpler approach is to ask what code editing software they use and hire the person who says, " I don't use a code editor. I use terminal!" hire those people.
Were the people you were interviewing already unemployed? Work trials are harder when people are already employed. There can be legal issues as well as scheduling ones.
Not sure it benefits the hiring company when they are artificially reducing the pool of applicants to only people who don't already have jobs.
Not that there's shame in not having a job, especially in today's market (shit happens, I'm sure lots of very qualified people are currently looking), but there are solid statistical reasons why social proof of already being employed is seen as a positive.
Net benefits perhaps. It absolutely lets the hiring company get more information by increasing the amount of time spent by the candidate before a commitment occurs.
How do you deal with onboarding, both technically, and legally if your eval period is 1-2 days? I mean unless you are a one-man team, and can just give all permissions yourself, and ignore legalities, it sounds way too short
The same way we did before. Very very simple code submission (most people using AI use it even though we call out we're going to ask them later to modify later without AI tools writing code for them) then pairing interview where we ask some basic "are you actually at the level you say you are" question, then ask them to extend their program submission with:
- engineers on the call as pairing assistants - google, ai tools, whatever for libraries, syntax, etc. we tell the candidate directly that it's impossible for us to gsther signal on how they think about problems if they ask Claude to just whip them up a solution - the expected output - their own unit test suite
Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go. I've interviewed dozens at this point. I'm not saying "they're unfamiliar with the structure", I'm saying "they cannot actually break down the problem even verbally".
It doesn't matter their pedigree or past experience on their resume, if they used AI to generate they don't seem to be able to resurrect the skills that actually matter for the thing, engineering and product work.
Note because I know folks hate code submissions. It's not hard. We give a CSV with 3 columns, 10 lines. Do some basic mapping and some structuring, some basic data modeling. We only expect about 1 actual class or struct. Then unit tests and it should run in the terminal. Max submission length with verbosity has been a java program at something like a hundred lines total if that, most folks complete the submission in an hour or two. Extension is that we modify one of the rules and extend the CSV by 5 lines.
I seem to still be getting good signal from this, since the engineers that I've hired off of this have been fantastic with or without AI tooling immediately in their hands during the day.
We were hiring for senior devs java/spring boot so I coded a tiny "Todo app" rest API, basically a single crud controller, and filled the code with many mistakes. Some obvious, some less obvious. Missing authentication, no tests, logging with System.out etc.. then we ask the candidate to review the code (we provide a laptop with a few IDEs but no AI tools). When they point out errors, I get them to explain why they are errors by playing the junior engineer to see how they explain technical issues to someone that might not have the same understanding I am not looking for them to find all issues but if they can't find at least some it's a red flag. It takes about 20-25 minutes and gives really good insights. The other 30 minutes we asked some questions about past projects and some technical questions
So far it provided good signals
Love this. +1 to the explaining part
To me AI hasn't had any impact on interviewing, only on filtering the CVs as I immediately flag obviously LLM-generated ones that don't have any substance, while those that are LLM-assisted with tasteful judgement are fine.
On the interview side, I have gone through a phase which required do an architectural design task on a whiteboard. I realized how stressful and not very helpful at evaluating it might be when I myself had to sit on the interviewee's side several times. It's difficult to get into this play-design mode.
Currently, I just make natural conversations, ask some technical questions on the level where I estimate the candidate is, try to understand the level of their knowledge and whether I would like to work with them in one team as a person. Maybe it wouldn't work for highly specialized roles, but for a general backend work I have been happy with the results.
Leetcode doesn’t tell you anything apart from someone memorised set of questions and their solution.
For your question, I assume you want people who can solve problems, can explain their thought process and reasons of decisions. I did interview devs recently, and my questions were related to their experience in their CV. Something like, tell me about project X in Y company, what did you do, what did you use, why, what did you learn from it.
My personal opinion is that LLMs are no difference from developers who over engineer and over complicate things. You can get them to do good work with the right steering and right understanding of the whole system within the context of the company.
Measure them on merit, their body of work and how well they will fit into the team culture.
As for the questions you’re asking, instead of asking old-school coding interview questions ask them stuff that is actually relevant to agentic programming.
Ask them to build a harness, have them explain what skills are, have them build a self-made implementation of Claude Code as take-home assignment.
The times have changed, you need to update interviewing questions to meet them. Start with agentic questions and ramp them up into more complex scenarios. This is how you gauge your candidates, see how they think and solve their way out of it.
Try to determine how creative, adaptable, motivated the candidate is. Figure out what there actual skill-sets are that they can bring to you, not whether they know some CS pedantry
After doing agenetic engineering in anger for about a year now, I'm starting to sour on it quite a bit. If I got this as a take-home, I think I would decline to go further. Watching the atrophy of people's skills and judgement has taught me that I value working with humans who can think for themselves.
Prompts, prompts, prompts.
Give them a time-constrained challenging problem that is wide in scope and see how well a candidate can decompose it before feeding it to AI. You can get pretty high signal within the first 2 prompts. Are they able to effectively steer the model or do they just ride with the flow and accept every AI suggestion?
How well do they know their models and their limitations? Are they able to switch tools/models effectively on the fly depending on the task? Or are they just using cursor auto mode and copy-pasting the task description into their IDE? Do they have a custom harness/workflow? What skills are they using, if any?
Judge them on quality of the output first. And pay attention to their taste.
So I actually had a take home challenge for an interview a few months ago, which didn't ask for any code at all. Instead they gave us a prompt for something to build (a tool that could break down an image into it's component colours), and we basically built whatever we wanted.
Honestly one of the most fun interviews I've ever done - it involved really understanding the problem, and building an interesting solution. AI was totally allowed and expected, but you had to prove you understood the problem and what you'd made in the interview. I felt it was actually way better than traditional interviews, because it gave a chance to show the actual quality and thought of my work rather than a glorified logic puzzle.
The org I am at we have build an internal harness for coding, and we only hire devs that do work "AI-first". We did start hiring juniors as well as seniors; the process for either of them is the same. We drop them into any of our repos without any prior knowledge of what the project is about; and ask them to interact with the harness to build a larger feature that has been specified in project management. For the task, they can only use AI; there is no room for any manual coding at this particular org. We grade them more or less on their interaction and problem solving patterns, as well as asking us questions; and how they write prompts. The coding interview last 2 hours, and there are only 3 rounds of interviews. Before the coding interview, we do send out a brief of what it will entail.
Maybe this sounds a little strange, but this has worked exceptionally well, as we have now several juniors working with us, that do still get checked by seniors. The harness itself which holds several levels of standards, rules as well as quality gates, allows these juniors to ship out a huge amount of high quality code. We still reject about 80% of the people who interview with us, because even though they know how to code with AI, their AI interaction patterns just aren't on par with what we need. I really started to like this process. Also to clarify, the harness isn't some Coding agent setup, but rather a whole setup that works well with Codex, Claude or Cursor; its well maintained versioned and constantly improved upon, with a huge amount of automations, rules and hooks. The moment someone starts working on anything at all, this is immediately connected and tracked with project management.
That sounds like a really great idea. Would you be open to sharing more about how the tool is designed? Are there any open-source or commercial tools like that out there right now?
Ive been trialing different types of interviews for over 2 years now and here’s what I’ve found.
I tried 2 flavors of a technical interview and this round was 1 hour in total.
The first was a traditional interview problem where we give the candidate a codebase and a docker image to run against. The problem itself involves using the docker image to post and get responses from but the candidate is expected to write a new helper/service in any language they choose. We ask for no agentic workflows for this but any other resource is fair game. This has been the most successful for us to find candidates and generally provides the best experience.
Our full agentic interview involves an existing fullstack typescript codebase that we give to the candidate a week or two before. The candidate is free to use any agentic workflows they want during the interview and we give them a ticket that we went to implement during the interview. I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping. This is lead to the biggest disparity in results of candidates so we stopped doing it.
> The first was a traditional interview problem where we give the candidate a codebase and a docker image to run against
Is that also on a company provided computer?
There are a lot more scams these days where a job-seeker compromises their computer by running provided code, and often docker isn't much against malice.
> I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping. This is lead to the biggest disparity in results of candidates so we stopped doing it.
Agentic engineering is the new microservices craze. Everyone's doing it, and some will do it exceptionally badly. This is always how these trends go. The anecdote you give here of some people just one shotting it in Claude Code without thinking about it and this producing extreme variability matches my experience. Basically, the longer you go down this road you realise some people will still actually think about things and some people will just cede cognitive control and stop thinking much at all. If you're the former, working with the latter is exceptionally draining over the long-term. I have in recent times become pretty bearish on agentic engineering for this reason.
Ask them how many "R"s are in the word "strawberry"
Trick question, there are no “R”s and only “r”s!
How many NFKC_Casefold('R')s are in NFKC_Casefold("strawberry")?
I am very curious about this as well, as an IC. I have not written code by hand for almost two years now and have produced a mountain of Python code over the past year (well architected, using all of my experience in other languages to support good judgement). However, Python is new to me -- I've never hand written a line.
This is all fine and good at my current job, but were I to ever go back on the market. What an uncomfortable conversation!
How does one approach this weird state we're in?
Same way as I did before.
Are they interesting to talk to? Are they passionate, curious, authentic, opinionated? Do they have a product mindset? Do they need to be told what to do? Would you hang out with them after work? Can they explain themselves in a way that’s easy to understand?
In my experience these are much better predictors of success than leetcode.
i think it would make more sense to let developers whatever coding agent they want to use to build something complex and then have them explain every bits and pieces of what was built, why, alternatives, design, scalability considerations etc.. this would result in much better technical discussion instead of having someone code manually. every developer needs to use a coding assistant today to build what they need to with the best quality in short time. so interviews need to be way to see if the developer can do that in real work.
Recently, I was screen-interviewed for coding speed with a fairly easy project. The company used a 3rd party platform which gave me 90m to complete 4 levels, each one building upon the other.
I think that was a nice test, but the platform is too buggy to recommend.
Basically the same as always--an extended interactive Turing test. Can they speak coherently and in-depth about the things they claim to have done on their CV/resume?
it feels surreal to see that people still do interviews, when I've been applying for 8 months, not even getting to an interview.
how many applicants are you getting? this plays a big role.. how many submissions do you have to sift through will dictate what ai agents to use. if you value the job: multiple of interview rounds are needed. Then its up to your hiring crew. Take this time to create your system of hiring, Of course a simpler approach is to ask what code editing software they use and hire the person who says, " I don't use a code editor. I use terminal!" hire those people.
Conventional technical interview (DSA, whiteboard architecture).
Then 1-2 day paid work trial.
Interestingly…no one has said no to the trial.
It turns out that people want to derisk their decision just as much, if not more.
Were the people you were interviewing already unemployed? Work trials are harder when people are already employed. There can be legal issues as well as scheduling ones.
This will always be the problem with trials. Anyone with a job, often underemployed between software roles these days, will be excluded.
Still, it does benefit the hiring company, just not the applicant in these situations.
> Still, it does benefit the hiring company
Not sure it benefits the hiring company when they are artificially reducing the pool of applicants to only people who don't already have jobs.
Not that there's shame in not having a job, especially in today's market (shit happens, I'm sure lots of very qualified people are currently looking), but there are solid statistical reasons why social proof of already being employed is seen as a positive.
Net benefits perhaps. It absolutely lets the hiring company get more information by increasing the amount of time spent by the candidate before a commitment occurs.
If you’re doing traditional on-site interviews, you’re already asking for multiple days (unless they live nearby).
It can be a challenge sometimes, but not as often as you might think.
I’m sure there are heavy-handed employment contracts that prohibit side work, but hasn’t been a problem in practice.
How do you deal with onboarding, both technically, and legally if your eval period is 1-2 days? I mean unless you are a one-man team, and can just give all permissions yourself, and ignore legalities, it sounds way too short
I assume the paid work is issued via a private repo and not full access?
Same as for other contractors.
Yes, there is friction, but if you do it enough, it gets reasonably small.