1. It is what you do after the prompt: scan diffs in real time and hit the escape key at the slightest sign of trouble.
2. Deep questioning. Constantly probing the assistant: what does it think it is trying to achieve, why did it just make decision X, is there a better way, what does it think the current constraint is?
3. Fighting drift. Knowing that the model will always try to regress to the mean of the training corpus, and constantly being on guard against that drift.
4. Keeping state in your head, because the model cannot. It is up to the programmer to remeber what connects to what else in what way and why.
in general the context goes bad if it gets long enough. if it seems to be losing the thread and you are fighting with it and going in circles: STOP! You might think it would be useful to keep going in one conversation but unless your next task is "do what we just did in this other part of the code" it makes sense to start new conversations for new tasks.
Are there any artifacts that you could recommend for your fourth point to document the state? An Architecture Decision Record (ADR) or its equivalent perhaps?
I can't say that resonates with me, but I'd say this in short.
I often use coding agents when I'm feeling emotionally tired and finding it hard to get into something and could use the support. I think of it as pair programming with a junior who is really smart in some ways but is prone to lapses of bad judgement. Alternately I think of myself as a foreman on a construction site.
I will start out with a paragraph of two about what i think about the situation, what i want to get done, concerns i have, examples where something similar is done in the code that it should look at. I always ask
"Does this make sense? Do you have any questions for me before we proceed?"
and go back and forth until it says it makes sense and it's questions are resolved. Usually i will wind up saying something like
"That's really good, but I have this little problem, what do you think we can do about it?"
If I am not worried about the cost of tokens I am inclined, when debugging, to write up what I understand about the problem to the agent and ask it what it thinks the problem is and then go race it trying to understand the code in the IDE and/or the use the debugger. Sometimes when the problem is simple but in my intellectual blind spot it really wins!
Another thing I think people forget is that agents are good for code understanding so if don't know how to do something or don't understand how it works, just open it in the IDE and start asking questions.
A lot of times people will complain that agents 'aren't listening to them' and I'm going to say often this is really a symptom of "doesn't know how to code", "not good at thinking systematically", "isn't competent as a subject matter expert". My own take is that sycophancy means that sometimes instead of just confronting you and saying "take this job and shove it", models will just act confused. I know often when that is going on I eventually come to realize that I didn't understand the problem and my prompts were garbage so GIGO. When I talked /w Microsoft Copilot about it, it said that LLMs learn to mimic the patterns we use so if you talk like you are confused... it takes like it is confused.
Yeah, the mimicking pattern thing is very interesting, didn't know that but definitely relevant. Love the way you use it as a pair programmer, definitely best if you want to learn! I would love to know why the idea doesn't resonate with you though. Also, thanks a bunch for the detailed feedback!
Well I think one reason I've enjoyed this career for as long as I have is that I want to learn. Like, I had a summer job programming Java applets when Java was still in beta and one thing I love about LLM-assisted programming is that I am still learning new things about Java.
So far as your article goes it seemed poorly organized to me. Like on a site like HN a lot of articles go by and people cannot give every article the attention the author might want, so you have to write things in an "inverted pyramid" style where a few paragraphs at the top lay out the most important points, then you might have a list of principles or heuristics and then you could go through and describe them in details and break out tables about their characteristics. I can't entirely agree with the mindset of Greene's 48 Laws of Power but I think that book is a masterclass in good organization.
1. It is what you do after the prompt: scan diffs in real time and hit the escape key at the slightest sign of trouble.
2. Deep questioning. Constantly probing the assistant: what does it think it is trying to achieve, why did it just make decision X, is there a better way, what does it think the current constraint is?
3. Fighting drift. Knowing that the model will always try to regress to the mean of the training corpus, and constantly being on guard against that drift.
4. Keeping state in your head, because the model cannot. It is up to the programmer to remeber what connects to what else in what way and why.
Thanks a bunch Adam! I think the fighting drift point is super interesting, haven't heard that often!
in general the context goes bad if it gets long enough. if it seems to be losing the thread and you are fighting with it and going in circles: STOP! You might think it would be useful to keep going in one conversation but unless your next task is "do what we just did in this other part of the code" it makes sense to start new conversations for new tasks.
Are there any artifacts that you could recommend for your fourth point to document the state? An Architecture Decision Record (ADR) or its equivalent perhaps?
https://news.ycombinator.com/item?id=35308838
I can't say that resonates with me, but I'd say this in short.
I often use coding agents when I'm feeling emotionally tired and finding it hard to get into something and could use the support. I think of it as pair programming with a junior who is really smart in some ways but is prone to lapses of bad judgement. Alternately I think of myself as a foreman on a construction site.
I will start out with a paragraph of two about what i think about the situation, what i want to get done, concerns i have, examples where something similar is done in the code that it should look at. I always ask
"Does this make sense? Do you have any questions for me before we proceed?"
and go back and forth until it says it makes sense and it's questions are resolved. Usually i will wind up saying something like
"That's really good, but I have this little problem, what do you think we can do about it?"
If I am not worried about the cost of tokens I am inclined, when debugging, to write up what I understand about the problem to the agent and ask it what it thinks the problem is and then go race it trying to understand the code in the IDE and/or the use the debugger. Sometimes when the problem is simple but in my intellectual blind spot it really wins!
Another thing I think people forget is that agents are good for code understanding so if don't know how to do something or don't understand how it works, just open it in the IDE and start asking questions.
A lot of times people will complain that agents 'aren't listening to them' and I'm going to say often this is really a symptom of "doesn't know how to code", "not good at thinking systematically", "isn't competent as a subject matter expert". My own take is that sycophancy means that sometimes instead of just confronting you and saying "take this job and shove it", models will just act confused. I know often when that is going on I eventually come to realize that I didn't understand the problem and my prompts were garbage so GIGO. When I talked /w Microsoft Copilot about it, it said that LLMs learn to mimic the patterns we use so if you talk like you are confused... it takes like it is confused.
Yeah, the mimicking pattern thing is very interesting, didn't know that but definitely relevant. Love the way you use it as a pair programmer, definitely best if you want to learn! I would love to know why the idea doesn't resonate with you though. Also, thanks a bunch for the detailed feedback!
Well I think one reason I've enjoyed this career for as long as I have is that I want to learn. Like, I had a summer job programming Java applets when Java was still in beta and one thing I love about LLM-assisted programming is that I am still learning new things about Java.
So far as your article goes it seemed poorly organized to me. Like on a site like HN a lot of articles go by and people cannot give every article the attention the author might want, so you have to write things in an "inverted pyramid" style where a few paragraphs at the top lay out the most important points, then you might have a list of principles or heuristics and then you could go through and describe them in details and break out tables about their characteristics. I can't entirely agree with the mindset of Greene's 48 Laws of Power but I think that book is a masterclass in good organization.