I’ve been doing this with voice mode in agent harnesses. In Codex for example I often speak orally and I find that (1) I discover something new myself, (2) agents seem to understand much better than pure text prompts. The only downside is it costs lot of tokens.
In almost every interview I gave at the last several companies I worked for, I'd start with "This may feel a bit odd, but with this coding problem, I'd appreciate if you spoke through your problem solving. If it helps, pretend like I am your intern. Good ideas, bad ideas, pros and cons of whatever you're considering, it's all valuable information for me to learn a bit about how you approach problems."
I feel like being explicit like this put candidates at a bit more ease than having them try to suss out what the interviewer is expecting from them implicitly (and I know there are many interviewers that will auto-veto any candidate with the audacity to suggest a possibly imperfect approach).
I always hated when interviewers did this. My chain of thought is often backwards, very abstract, or otherwise wildly different from how it seems most others think, and it's very difficult if not impossible for me to articulate my thinking on the spot. In school for writing assignments I would have to spend alot of time reversing or reordering how I arrived at a conclusion before even writing my first sentence. And even just arriving at a conclusion involves taking many dead-ends; the process can be seemingly instantaneous in places, or require laboriously following rabbit holes (often with throwaway code) that I know are dead ends but nonetheless help me to explore context. I need to have the entire problem in my head and able to explore it to at least some extent before even starting to formulate a linear chain of thought, let alone commit something comprehensible to code.
IOW, I can't explain in real-time how I'm working things out. I have to work them out to completion first, quietly, or at least have a good idea about what the full path looks, end-to-end, in detail and only after the fact could I ever hope to explain my thinking. Asking me to explain my thinking while I'm thinking is easily the most stressful and impossible task anyone could ask of me. I'd rather you break my fingers as part of the interview process.
I understand as an interviewer that makes for quite a dilemma. And I suppose if you find this approach reasonable then you may not like working closely with someone like me (though if you're just a manager, then it's a bias to be aware of).
Similarly, practices like pair coding just don't work for me. I'm not not a team player. I like and benefit from design and ad hoc technical discussions, just don't ask me to sit down and write code with someone else, or even with someone looking over my shoulder. All parties involved, including myself, will just end up pulling their hair out. Though I do work best with people who think similar to me. We instinctively know when and how to set our boundaries and expectations.
To interview someone like me it's probably best to focus on having the interviewee identify waypoints to confirm they have the experience and understanding to navigate difficult problems. Just don't ask them to actually navigate on the spot. If you want to see how they ultimately connect the dots in practice, look at code they've already written. Unlike in most other fields, software developers often have a corpus of work that is easy to share and often already published.
I really wanna ask, because... I'm 46. Been writing code for a living about 30 years, never on a salary, always contractor. I wrote and maintain enough code for non-tech companies that aren't going to take a risk on AI rewrites anytime soon that I can probably coast until retirement. But I haven't interviewed or even needed to sell myself for a long time.
I always thought I would easily pass an interview, but I may have been wrong before all this on the basis that I don't work that well on a team.
OTOH ...is the question now really valuable whether a candidate can explain their thought process rather than deliver? Why wasn't that a question 20 years ago when it actually mattered? Or does it matter more now because the bar needs to be higher, and there's no cache of more experienced devs to train them up?
I think delivering is still a requirement either way, but the value is in hearing a candidate has the ability to problem solve in an unknown problem space vs regurgitate a memorized solution. Even better is witnessing some has the ability to try, fail, then self-correct, and rebound.
So, my policy has always been that if I sit through a meeting where a client asks for new feature "X" and I figure out a way to avoid building the new feature, and either use existing business logic / day-to-day processes, or else find some new way that doesn't involve code, I don't bill for that time. I'd say that I probably give up about 35-40% of potential coding work just by helping them work through that. Which is fine, because it leaves me free time to do other things.
On the other hand, I don't have the option to fail even slightly. I deployed a patch yesterday for an old, rare, logic bug in one reservation system, and within an hour I was being bombarded with edge cases of people seeing their reservations as cancelled during the 15 minute switch-over period.
Sometimes I feel like the job of a programmer for a company is just very different from what I do. If anything goes wrong with anything I deploy, I'm awake for however long it takes to make sure it's right. On the other hand: No PRs, no teamwork. Just a click to deploy once I'm relatively sure. And a lot of responsibility.
In some cases, it can certainly help. However, it isn't helpful for deep thoughts or when innovative, creative ideas are needed.
I have found that speaking the steps out loud helps, especially with new, confusing working steps. For example, it helps when tying a complex knot or following the steps/work of others without explicit instructions. In such situations, speaking out loud helps me to remember the steps much faster (simple repetition through multiple pathways) and also helps me to detect and correct spatial and sequential mistakes. It also helps enormously when working with or teaching someone else.
I have a funny memory from my youth: one time, I had to solve a particularly difficult maths problem on the blackboard. I don't remember exactly why, but while I talked out loud about solution A, I wrote a slightly different step (B) on the board. Looking back, I realised I had made mechanical mistakes, so I had to correct them to get the final valid solution. My teacher laughed and joked, 'A sign of genius is to write A, say B, but only C is the true solution.' :) (no, he was just too kind).
I have often experienced my handwriting being much slower than my speaking, so sometimes they don't sync and funny mistakes are made (one line contains the words of the next line). But I fondly remember that particular moment.
I do the same. My impression is that very few people do this, or take proper notes at all. This is based on my observation that most people I work with seem to have zero recollection about things that happened more than a few months ago.
Witnessing a candidate’s thought process is probably the most valuable part of the interview process for me. Nothing else comes close. Just let me listen/watch you solve problems.
I wonder if this may be related to anendophasia, the absence of an inner voice. Apparently it has an influence on self-reflection and verbal working memory.
I have aphantasia so anendophasia sounds like such a nightmare to me, though I imagine those so have it would likely feel the opposite of me. I've met some folks in the past who I'd suspect have anendophasia, but I'm not sure how I'd ever bring that up without coming across as uncouth.
Anyone with anendophasia care to opine on their experience?
I'm not the OP, but I think everyone might have a different definition of what "audible inner monologue" mean.
For example, I don't talk to myself in "monologue". But I can close my eyes or stare at the screen and visualize the solutions. The words that I usually tell myself during that are "what?", "why?", "strange!", "really",... not a monologue or sentences.
That's my impression too, that my thoughts are a hybrid, where I'll think in a rapid sequence of whole concepts and visual flashes, and then summarize that in my inner voice "for the record", as a jumping-off point for the next batch of thoughts. But I don't have much control over this, as my inner voice is hard to shut up for any length of time, always giving a running narrative.
The author says they could narrate their own thought process out loud.
I couldn’t do that. I have no insight into my thought processes whatsoever. There are no semantic (or any other) processes to witness. It’s complete crickets at all times…
My wife is the opposite - constant chatter in her head. And she can trace every thought exactly. (No idea how people like her can focus or even function!)
We focus/function via the inner monologue. It doesn't just do its own thing while we're trying to focus or get something done, it walks through the issue and comes up with solutions and then criticizes them and iterates, and as far as we're concerned, that's us solving the problem.
I am such a person and I've always done 'chain of thought' in a notepad in order to think through complex things, well before LLMs were a thing. There are certainly tasks I can 'one-shot' but I've learned from experience that for some problems that just results in me focusing on irrelevant things.
Curious whether the lack of audible inner monologue is a benefit for you?
I feel like I'm the opposite, where most of my thinking is done in inner English, which feels slow. I.e. as opposed to instantly realizing the result of deep subconscious processing or something, I usually go the slow language way, if you get what I mean.
"Free association" in psychoanalysis just asks people to let themselves speak, to fall into their speech, let it unfold - but it is very, very hard to do, because what you say tends to not be what you want to say. You want to avoid certain things, you can see your chain of speech leading up ahead to somewhere you don't want to go. So you stop yourself, or resist, and then what you are saying is not at all free but strained, hedging, tense and anxious.
We use this in my science education platform: we make students talk out loud about what they're thinking. Not a chatbot, but guided inquiry that's designed to make them actually mentally engage with what they're seeing
I’ve been doing this with voice mode in agent harnesses. In Codex for example I often speak orally and I find that (1) I discover something new myself, (2) agents seem to understand much better than pure text prompts. The only downside is it costs lot of tokens.
In almost every interview I gave at the last several companies I worked for, I'd start with "This may feel a bit odd, but with this coding problem, I'd appreciate if you spoke through your problem solving. If it helps, pretend like I am your intern. Good ideas, bad ideas, pros and cons of whatever you're considering, it's all valuable information for me to learn a bit about how you approach problems."
I feel like being explicit like this put candidates at a bit more ease than having them try to suss out what the interviewer is expecting from them implicitly (and I know there are many interviewers that will auto-veto any candidate with the audacity to suggest a possibly imperfect approach).
I always hated when interviewers did this. My chain of thought is often backwards, very abstract, or otherwise wildly different from how it seems most others think, and it's very difficult if not impossible for me to articulate my thinking on the spot. In school for writing assignments I would have to spend alot of time reversing or reordering how I arrived at a conclusion before even writing my first sentence. And even just arriving at a conclusion involves taking many dead-ends; the process can be seemingly instantaneous in places, or require laboriously following rabbit holes (often with throwaway code) that I know are dead ends but nonetheless help me to explore context. I need to have the entire problem in my head and able to explore it to at least some extent before even starting to formulate a linear chain of thought, let alone commit something comprehensible to code.
IOW, I can't explain in real-time how I'm working things out. I have to work them out to completion first, quietly, or at least have a good idea about what the full path looks, end-to-end, in detail and only after the fact could I ever hope to explain my thinking. Asking me to explain my thinking while I'm thinking is easily the most stressful and impossible task anyone could ask of me. I'd rather you break my fingers as part of the interview process.
I understand as an interviewer that makes for quite a dilemma. And I suppose if you find this approach reasonable then you may not like working closely with someone like me (though if you're just a manager, then it's a bias to be aware of).
Similarly, practices like pair coding just don't work for me. I'm not not a team player. I like and benefit from design and ad hoc technical discussions, just don't ask me to sit down and write code with someone else, or even with someone looking over my shoulder. All parties involved, including myself, will just end up pulling their hair out. Though I do work best with people who think similar to me. We instinctively know when and how to set our boundaries and expectations.
To interview someone like me it's probably best to focus on having the interviewee identify waypoints to confirm they have the experience and understanding to navigate difficult problems. Just don't ask them to actually navigate on the spot. If you want to see how they ultimately connect the dots in practice, look at code they've already written. Unlike in most other fields, software developers often have a corpus of work that is easy to share and often already published.
I really wanna ask, because... I'm 46. Been writing code for a living about 30 years, never on a salary, always contractor. I wrote and maintain enough code for non-tech companies that aren't going to take a risk on AI rewrites anytime soon that I can probably coast until retirement. But I haven't interviewed or even needed to sell myself for a long time.
I always thought I would easily pass an interview, but I may have been wrong before all this on the basis that I don't work that well on a team.
OTOH ...is the question now really valuable whether a candidate can explain their thought process rather than deliver? Why wasn't that a question 20 years ago when it actually mattered? Or does it matter more now because the bar needs to be higher, and there's no cache of more experienced devs to train them up?
I think delivering is still a requirement either way, but the value is in hearing a candidate has the ability to problem solve in an unknown problem space vs regurgitate a memorized solution. Even better is witnessing some has the ability to try, fail, then self-correct, and rebound.
So, my policy has always been that if I sit through a meeting where a client asks for new feature "X" and I figure out a way to avoid building the new feature, and either use existing business logic / day-to-day processes, or else find some new way that doesn't involve code, I don't bill for that time. I'd say that I probably give up about 35-40% of potential coding work just by helping them work through that. Which is fine, because it leaves me free time to do other things.
On the other hand, I don't have the option to fail even slightly. I deployed a patch yesterday for an old, rare, logic bug in one reservation system, and within an hour I was being bombarded with edge cases of people seeing their reservations as cancelled during the 15 minute switch-over period.
Sometimes I feel like the job of a programmer for a company is just very different from what I do. If anything goes wrong with anything I deploy, I'm awake for however long it takes to make sure it's right. On the other hand: No PRs, no teamwork. Just a click to deploy once I'm relatively sure. And a lot of responsibility.
In some cases, it can certainly help. However, it isn't helpful for deep thoughts or when innovative, creative ideas are needed.
I have found that speaking the steps out loud helps, especially with new, confusing working steps. For example, it helps when tying a complex knot or following the steps/work of others without explicit instructions. In such situations, speaking out loud helps me to remember the steps much faster (simple repetition through multiple pathways) and also helps me to detect and correct spatial and sequential mistakes. It also helps enormously when working with or teaching someone else.
I have a funny memory from my youth: one time, I had to solve a particularly difficult maths problem on the blackboard. I don't remember exactly why, but while I talked out loud about solution A, I wrote a slightly different step (B) on the board. Looking back, I realised I had made mechanical mistakes, so I had to correct them to get the final valid solution. My teacher laughed and joked, 'A sign of genius is to write A, say B, but only C is the true solution.' :) (no, he was just too kind).
I have often experienced my handwriting being much slower than my speaking, so sometimes they don't sync and funny mistakes are made (one line contains the words of the next line). But I fondly remember that particular moment.
I basically "chain of thought" to myself in my own notes when working through a hard task. Mayne we can call it "live tweeting in private".
Aometimes it feels like a super power because it takes away the "starting anxiety" and makes everything a trivial sequence of "following simple steps"
I do the same. My impression is that very few people do this, or take proper notes at all. This is based on my observation that most people I work with seem to have zero recollection about things that happened more than a few months ago.
Witnessing a candidate’s thought process is probably the most valuable part of the interview process for me. Nothing else comes close. Just let me listen/watch you solve problems.
As someone with zero exposure to my own thought processes, I find this always so discriminatory! ;)
I wonder if this may be related to anendophasia, the absence of an inner voice. Apparently it has an influence on self-reflection and verbal working memory.
Not Everybody Has an Inner Voice: Behavioral Consequences of Anendophasia - https://doi.org/10.1177/09567976241243004
I have aphantasia so anendophasia sounds like such a nightmare to me, though I imagine those so have it would likely feel the opposite of me. I've met some folks in the past who I'd suspect have anendophasia, but I'm not sure how I'd ever bring that up without coming across as uncouth.
Anyone with anendophasia care to opine on their experience?
Verbalizing my thoughts as I try to solve a problem drops my IQ down by like 20 points at least.
Exactly, specially not in your native language
This is basically https://www.christopherroosen.com/blog/2020/4/20/how-the-rit...
After one to many miss-rides, I now do this often even when just riding the train.
I wonder if the author is one of those people that doesn't have an audible inner monologue?
I'm not the OP, but I think everyone might have a different definition of what "audible inner monologue" mean.
For example, I don't talk to myself in "monologue". But I can close my eyes or stare at the screen and visualize the solutions. The words that I usually tell myself during that are "what?", "why?", "strange!", "really",... not a monologue or sentences.
That's my impression too, that my thoughts are a hybrid, where I'll think in a rapid sequence of whole concepts and visual flashes, and then summarize that in my inner voice "for the record", as a jumping-off point for the next batch of thoughts. But I don't have much control over this, as my inner voice is hard to shut up for any length of time, always giving a running narrative.
Yeah I’m the exact opposite, thought is almost entirely auditory.
I'm essentially thinking out loud in my head constantly.
The author says they could narrate their own thought process out loud.
I couldn’t do that. I have no insight into my thought processes whatsoever. There are no semantic (or any other) processes to witness. It’s complete crickets at all times…
My wife is the opposite - constant chatter in her head. And she can trace every thought exactly. (No idea how people like her can focus or even function!)
We focus/function via the inner monologue. It doesn't just do its own thing while we're trying to focus or get something done, it walks through the issue and comes up with solutions and then criticizes them and iterates, and as far as we're concerned, that's us solving the problem.
Anendophasia? Fair number of people have this.
I am such a person and I've always done 'chain of thought' in a notepad in order to think through complex things, well before LLMs were a thing. There are certainly tasks I can 'one-shot' but I've learned from experience that for some problems that just results in me focusing on irrelevant things.
Curious whether the lack of audible inner monologue is a benefit for you?
I feel like I'm the opposite, where most of my thinking is done in inner English, which feels slow. I.e. as opposed to instantly realizing the result of deep subconscious processing or something, I usually go the slow language way, if you get what I mean.
i think everyone has the same inner monologue and it's entirely a matter of how we define "monologue"
"Free association" in psychoanalysis just asks people to let themselves speak, to fall into their speech, let it unfold - but it is very, very hard to do, because what you say tends to not be what you want to say. You want to avoid certain things, you can see your chain of speech leading up ahead to somewhere you don't want to go. So you stop yourself, or resist, and then what you are saying is not at all free but strained, hedging, tense and anxious.
We use this in my science education platform: we make students talk out loud about what they're thinking. Not a chatbot, but guided inquiry that's designed to make them actually mentally engage with what they're seeing
A Mathematicians Lament mentions students explaining rather than reciting the equations, too, to correctly learn them
> eh, "Chain of Thought for Humans" sounds stupid
> but maybe it has some nuance?
> eh, I think I still don't care enough to click
> but maybe I could post a funny comment
> it doesn't sound that funny, and besides, HN doesn't usually like these sorts of comments
> okay, let's just read the next story
> unless
Just make a theoretical minimum type test like the landau physicist
holy shit this would break my ability to think completely.
I think and chase alternate chains of thought way faster than i can speak.