> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes.
I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircraft).
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
You stop relying on the agents following instructions exactly.
You need two pieces:
a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.
b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.
The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are
Code analysis can’t detect a sizable fraction of all possible errors because for any Turing complete language the number of possible errors is infinite.
Each of these are just layers of control at different lifecycles of agent code generation. Analyzers are nice because it gives targeted, static analysis that can prevent certain classes of errors very early and at lower iterative cost (e.g. a build)
I don't know if that's necessarily true in practice the way it is in theory. If you knock down 95% of the issues they create over a certain period of time, in practice you'll have better code than a human would write, and it's all upside from there.
I've been using languages with stronger type systems and that's also a huge boon.
Why would that be the case? You can run human written code through the same “linters, compilers, static analysis, fuzzing, testing” as you do agent produced code.
You would be surprised. Humans will do human things like be extremely inconsistent, ignore warnings (if they are not enforced as errors), skip steps because they are lazy (devs often chose to skip our pre-push hooks and preferred to run in CI and babysit the PR).
Agents can also do all of those things, but they are generally more compliant to instruction.
yes, this is a very helpful approach. I take a similar approach, partly because I find bringing up negative directions seem to bring the undesired action/direction into the agent's view and makes it more likely that they will go there
Why couldn't you express all those as test cases rather than instructions?
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases.
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
> Why couldn't you express all those as test cases rather than instructions?
How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
This is a question of context management and I suppose some would classify this as "harness engineering" as the trend of the moment.
One approach, for example, might be to have the a standalone code reviewer agent that is solely responsible for interfacing with the CAPA system (e.g. via a tool, via MCP) and acts as a back stop. When it finds a new type of CAPA, it stores it (and the backend indexes it with enough metadata to support broad types of retrieval). When it reviews a piece of code, it finds past CAPAs. By file locality. By business domain in the application. By keywords.
Same tool and repository available to both building agents and review agents, but use the review agent as a dedicated back stop as part of the verification process.
That has been a part of post mortems everywhere I work, there was never a name for it though. Just someone would say: “how do we prevent this in the future?” which usually spawns a few tasks that the person who caused the issue or others will pick up.
> Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I think there's a pretty clear and wide gulf between "nobody could have seen that coming", or "I just wasn't on my A game", and frank negligence.
I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
I've had a similar experience, also at a very large organization. In my case this person chose that kind of environment because it allowed them to work on side stuff while waiting for approvals, etc. A lot of people in very large companies don't view the grind of organizational friction inhibiting their actions as problematic.
Every place I've worked at has positioned their SEV review process as 'blameless' with respect to the engineers who created the code, processes, etc that led to the incident. On paper, it's not about pointing fingers (though I've found this to be somewhat idealistic), but about finding faults in the processes that let bad code slip through the cracks.
It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree?
This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
This is a general principal in the eng shop I work for, its written into every step, including the SCRUM etc.. What when well, why did it fail, what could be done better. Its all part of root cause analysis.
I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
I don't see how this could work, right now, since every failing I've had was just generic stupidity of the AI, which is brilliant one minute, and a complete idiot the next.
I think correction actions still require the ability to execute them, which (in all the cases I've had) would require more capable models.
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
Honestly I feel like I'm taking crazy pills these days.
Write a program! If it has a bug, debug it and update the program. Use an LLM to debug it if you must. If you're doing something repeatable, then use a traditional automation scheme.
If an LLM has to handle a subsection of the program, so be it. Just wrap that part with clean inputs and outputs.
Why does it feel like we've forgotten how to automate things?
> Why does it feel like we've forgotten how to automate things?
everybody is too busy everyday fighting fires to garden all those standard automations; its much "easier" to just throw it at an llm and move on (just saying what i see)
the history of assembly lines would be instructive. I don't think we have achieved a fully automated assembly line yet, but certainly they are much more automated than they were when Ford set up the first one.
Sorry to get hung up on what is probably just flavor text, but the first sentence.. "Agents can oneshot games that are actually fun", uhhh such as?
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
Yeah, that goes against my experience. Agents can absolutely one-shot games, but the gap between something that looks like a game and something that is fun to play and worth trying for more than a few minutes is gaping wide.
As someone who builds games, particularly games meant to be played in the browser or apps, it can 100% one-shot a game-like experience, but it will not be very good.
You need to playtest the hell out of games. Play a bit, iterate, play a bit more, iterate, etc. You will get way better results if you start small and don’t try to one-shot it. Start small, vertical slice, playtest, and go fro there.
Probably fun defined in the ephemeral context. As-in it was "fun" to play in the context of the short form video attention span, social media brain rotted individual. Then it was thrown away and never played again.
It's definitely not fun if you consider building an actual community around it and have a long term plan for it.
Honestly agents can't one shot shit except cool demos. I mean in a way they can but I let an Astra/Fable 5.1 duo build an android app and figured I'd really just loosely vibe code this for once instead of supervising.
It's a working pile of trash that has quadratic growth in the remote postgres database resulting in 10GB stored in a few days for what should have been like 5 MB of data.
It's not a skill issue, I know how to get good results but I figured maybe I don't need to be as cautious as I am all the time.
Not that I don't believe we can get there eventually.
Agreed, I read this and almost instinctively decided to close the page, agents can be used to build games for sure, but that "one shot" and "fun" in the same sentence is really misleading
FYI, it looks like you've been banned for the last ~8-9 months, for no obvious reason I can see. I vouched for your comment to make it visible, but they appears to show up as [dead] by default.
My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
I’ve definitely had a hunch about a software factory harness kind of thing.
Your thoughts on the matter seem clearer than mine.
I’m kind of relieved to see so many stages; that’s something I’ve grappled with conceptually. The volume of (sometimes optional) stages has made it hard to resist the temptation to hack something together with a GHA workflow.
I salute your choice to support distributed; that’s a requirement in my book.
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I think that dissolves the "self driving" distinction, though? The mechanisms for driving the codebase must be present in the codebase itself. Otherwise its just a regular out-of-band development process.
If you have a logo in the top left corner of your company blog the logo should route to the home page so I can see what product you're actually building.
Makes sense to me - I see it going down two separate routes in parallel:
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
What would you prefer: 99% of user requests to succeed but w/ 5% risk of getting O-day'd, or have 90% of user requests succeed but w/ 0.1% risk of getting 0-day'd.
The first and it isn't even close. Normal operation of the system is more important than preventing attacks, and trading it away at a 2:1 ratio is a bad trade.
The point of the example is that with the first choice you have 5% legitimate customers and 5% malicious "customers" and by letting all of them in aka all 10%, you get maximum customer satisfaction including satisfaction of the malicious customers.
Alternatively you reject the malicious customers and as collateral damage accept that some customers do not get their request approved.
You're also confusing "normal operation" with what a user request is. A user request is a request to change something, so by definition it is not the continuation of the existing system. You're thinking that a user request is a HTTP request which is not what's being discussed.
Personally - for banking - I'd be plenty happy dealing w/ 90% uptime if it gave me an order of magnitude more security against getting robbed. This tradeoff may not have been realistic a decade ago, but it seems to be where things are trending.
It also is de-facto the case today when logging in as per usual sometimes requires a captcha or additional verification.
This would be more convincing if there were a simple stat on the home page of the dispositions of the reported bugs (fixed/no-repro/won't-fix).
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
We should publish some stats. The fix rates on bugs are around 45% baseline, increasing over time as we learn from behavior within a given codebase. The merge rates for the codebase health work we do (e.g. deleting dead code) are very high, at least 90%.
Is our fate to become more and more product designers and less and less software engineers?
I feel that way.
But I'm now able to actually create and release things. Before, I used to obsess over code architecture, optimizing every single line. I could and can craft beautiful code. And then I always burned out and lost interest (gemini). Now I don't touch code at all and have been actually creating and releasing the ideas I've always had in mind. Then I realized that's what I always wanted. The code was always a means to an end. But that's me.
In the end, we already have processes that work for humans and we know the types of testing, verification and review that makes a codebase grow healthily. We just need to adapt these designs the best we can to AIs.
AI have a lot of advantages over humans like speed, controllability, good defaults (e.g. Claude will write helpful comments unprompted which is better than most engineers), and the ability to implement best practices like comprehensive testing with much lower cost than a human team. Using AI in my personal projects, I am able to write software at a much lower bug rate than manually, because now I can make an automated test suite whereas previously I had no time to do so. However, AI also has downsides like rigidity, limited context window, and lack of creativity by default which needs to be resolved with harnessing (e.g. different agents with separate, well-compressed contexts) or worked around with human review layers.
I suspect the best core pattern for coding automation will be to create an engineering manager agent layer that can flexibly apply a set of engineering practices to the concrete problems at hand. It would handle all coding agent pings before it reaches a human, with prompting so that it reliably considers the big picture when making decisions. Very rarely, it would escalate an issue or plan review directly to humans when it is truly blocking, but most likely it would organize the information it obtains from the lower layer coding agents and surface them as reports for humans to review.
Kind of feels like we're in a similar place with AI programming as we are with self driving vehicles. Good enough for the typical case, maybe even better than most humans, but bad when dealing with novel situations or situations which require a really cohesive model of the system (code, business requirements, physical world, whatever).
My answer to this is adding an engineering assurance module to my specification and verification toolkit. Every project has a different definition of success and quality. My goal is to provide a consistent way to define, analyze, and report quality metrics tailored to each project.
LLMs also do not have anything like memory, by design. Everything bolted on that smells like memory or is being called memory is a crutch, at best. Not being pedantic, I just think this aspect is lost on a lot of people.
Also, when we humans are tricked into perceiving a remote mind, we automatically assume it has a memory like us, or at least like large animals.
In other words, there's a story-document between a SherlockHolmesBot and UserPlaceholder that's growing like a crystal formation. Some software sees "Sherlock Homes Says" and then "performs" the dialogue at us, and we then assume Sherlock Holmes exists with a mind and memory, rather than being a facet of a text-story.
Exactly, this is why we see such minimalistic memory systems being used. Information retrieval is bad, but improving information availability by pre-computing information retrieval is not the obvious fix people think it is.
Sometimes, but I think there are a few problems with this:
1. Some context doesn't have an obvious place to write it in the code. If you're explaining why a tricky function is implemented a certain way, you can leave the explanation above the function or within the function. If you're explaining something more general, there might not be a natural place to put it.
2. Relatedly, some context doesn't have an obvious location to read in the code. E.g. you can comment on a schema that a table is intended to be append-only, but an agent could easily miss this comment if it doesn't gather context all the way down to the raw schema. Any research process has unknown unknowns. There may not be a canonical place to look.
3. Adding comments for every single human intent might be a bit noisy. E.g., if a code reviewer flags something that I know isn't a bug, I want the system to learn from this, and I want that knowledge to take effect outside of just code review, but I might not want every single code review thread to yield a codebase comment.
1: There are corner cases, no doubt, but the main source of information should remain near the code.
2 & 3: How to be sure that the agent will pick the "append only hint" from the memory? Especially when it starts growing and cannot be part of the context?
I think the bigger need is for outside concerns, like: what is the infrastructure like? How much volume does this feature handle? How should changes to live code be rolled out to prevent running processes from failing?
In addition to all the product details that aren't in the codebase or docs, like "keep this logical path because it is used by our one big client."
Mostly what I think is needed is a richer worldview available to the LLM so it understands not just the code but can understand the product and its real world usage and constraints.
But how you are going to link the "keep this logical path ..." with the code? You cannot feed all the memory to the agent because of the context rot, and you are not sure the agent will always pick it when needed.
The things in the 'what should be self-driving' section are the exact things i give to juniors so they can build their mental model of the codebase.
Now, I agree it'd be fantastic to have that automated, but how do we then share that context with the humans that will drive the non-self-driving actions?
I have integrated AI into my application. The source code is embedded into the binary. When someone questions the numbers in my app it can debug itself. If it truly finds a bug in itself it will tell the user and open a ticket. Soon we are going to let the application fix itself.
How do you protect intellectual property? Or is this a case of the value being somewhere else, such as in your backend? If so, how does the agent debug frontend and backend? I presume it stops at frontend
I recently built a calculator to try to quantify the gains an org could expect as they become more "AI-native." I tried to account for a bunch of things including env, which I call "AI roadway," but I'm definitely missing codebase readiness (thank you, author), and also bottleneck analysis. Even without those considerations, though, gains are usually modest. It's hard to get to 2x.
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This happens way too often in my experience and I tried a lot of things. If the agent has access to an existing codebase it will always try to stay as close to it as possible. It doesn’t diverge from the initial direction the project/code went into, even if it’s already clear that this direction goes nowhere. Somehow agents aren’t able to detach themselves from existing code to come up with completely new solutions.
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
> Put another way: Detail’s secret plan is to create a product that bootstraps its own work, and then use that work to help engineering teams know what investments to make and how well they’re working.
Sorry, guys, I still don't understand what are you selling. Prompts? Design manual for building testable system? In which way agent-friendly dev env differs from human-friendly dev env?
It's a losing proposition. Self-driving cars go from point A to point B, and it's really easy to specify point B. My experience so far using agentic AI 24/7 for almost a year has been the AI can drive almost anywhere, so the whole problem is specifying where to drive. This doesn't get solved with faster, better AI, because all that will do is allow it to autonomously drive your codebase to random places you didn't want it to go. Then you're still left with the problem of specifying the problem.
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Well put. I've found that agents are great at adding features to areas of the code that I already understand, where I can phrase my goals coherently and keep them on the rails.
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.
I'm hoping the "blogging about doing" format will fade away, now that doing is cheaper than blogging. It used to be that the doing part was expensive, and risky. Now you can just do 5 things and throw away 4. So I'd rather see the finished product than talk about it.
The missing piece in most versions of this is the feedback loop, not the harness. A codebase can only drive itself as far as it can tell whether it's still on the road: tests that fail for the right reason, error budgets, a way to notice that a change made something slower rather than broken.
Without that, "self-driving" degrades into applying diffs that compile. With it, the interesting question becomes what the system is allowed to do when the signal goes red - revert, retry, or stop and ask - and that's a policy decision nobody has good defaults for yet.
I'm playing around with ecosystem driven development (hear me out ;)):
So i want claude to build a small game for me so of course i create one big file in which i write everything then i tell claude to analyse it and grill me (grill me skill) to clarify all smaller details.
This alone might lead to really good small sfotware but I still have to push it sometimes.
Now instead of doing this, i tell claude to build a small tool which generates a dashboard and memory and which can save specs and ask me choicses (do you prefer this color over the other, what do you think about problem A? What solution would you prefer? A, b, c or something else).
This does a few things:
1. claude doesn't has to save/store everything in a context
2. claude can now talk to that tool to ask it stuff
3. claude can now use the tool as a todo list
4. claude now can more easily spin up more agents in parallel
5. i have a nice interface and i can solve issues while claude works on unblocked tasks
6. I actually can follow the progress a lot easier
The only problem with this is: with the next update, you have to reinvestigate how claude was finetuned and adjust. A few month ago /goal was really good, now you need it a lot less because claude will do something for an hour without /goal
And the spec file only started to work after November/Opus moment but it got so good, that i can pack A LOT of stuff in a half structured markdown file and let it code what I need.
The progress is still too fast for the whole ecosystem
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
The more important question - how would you actually get LLM agents to follow the instructions in your ever-growing CAPA reliably?
It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.
You stop relying on the agents following instructions exactly.
You need two pieces:
a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.
b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.
The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
This may be platform dependent.
C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are
[0] https://learn.microsoft.com/en-us/visualstudio/code-quality/...
Code analysis can’t detect a sizable fraction of all possible errors because for any Turing complete language the number of possible errors is infinite.
I don't think I implied that they could.
Each of these are just layers of control at different lifecycles of agent code generation. Analyzers are nice because it gives targeted, static analysis that can prevent certain classes of errors very early and at lower iterative cost (e.g. a build)
I said
“You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.”
And you replied
“This may be platform dependent.”
I’m unsure how else to read that other than an implication that this might be possible on some platforms.
I don't know if that's necessarily true in practice the way it is in theory. If you knock down 95% of the issues they create over a certain period of time, in practice you'll have better code than a human would write, and it's all upside from there.
I've been using languages with stronger type systems and that's also a huge boon.
> better code than a human would write,
Why would that be the case? You can run human written code through the same “linters, compilers, static analysis, fuzzing, testing” as you do agent produced code.
You would be surprised. Humans will do human things like be extremely inconsistent, ignore warnings (if they are not enforced as errors), skip steps because they are lazy (devs often chose to skip our pre-push hooks and preferred to run in CI and babysit the PR).
Agents can also do all of those things, but they are generally more compliant to instruction.
That’s an assertion with no evidence that doesn’t match my experience.
Agents require far stricter guardrails than humans. Without linters, tests, static analysis, oracles etc… no agent can create a large program.
Even if you’re correct, you just build those checks into CI so that neither humans nor agents can skip them.
yes, this is a very helpful approach. I take a similar approach, partly because I find bringing up negative directions seem to bring the undesired action/direction into the agent's view and makes it more likely that they will go there
Why couldn't you express all those as test cases rather than instructions?
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases.
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
> Why couldn't you express all those as test cases rather than instructions?
How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
This is a question of context management and I suppose some would classify this as "harness engineering" as the trend of the moment.
One approach, for example, might be to have the a standalone code reviewer agent that is solely responsible for interfacing with the CAPA system (e.g. via a tool, via MCP) and acts as a back stop. When it finds a new type of CAPA, it stores it (and the backend indexes it with enough metadata to support broad types of retrieval). When it reviews a piece of code, it finds past CAPAs. By file locality. By business domain in the application. By keywords.
Same tool and repository available to both building agents and review agents, but use the review agent as a dedicated back stop as part of the verification process.
I think the only way for now it through distillation of your own models, which can get expensive fast.
The same way you do for humans, regular training and audits.
They don’t behave like humans.
On the contrary, they behave exactly like moderately autistic humans
That has been a part of post mortems everywhere I work, there was never a name for it though. Just someone would say: “how do we prevent this in the future?” which usually spawns a few tasks that the person who caused the issue or others will pick up.
> Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I think there's a pretty clear and wide gulf between "nobody could have seen that coming", or "I just wasn't on my A game", and frank negligence.
I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
I've had a similar experience, also at a very large organization. In my case this person chose that kind of environment because it allowed them to work on side stuff while waiting for approvals, etc. A lot of people in very large companies don't view the grind of organizational friction inhibiting their actions as problematic.
[flagged]
Every place I've worked at has positioned their SEV review process as 'blameless' with respect to the engineers who created the code, processes, etc that led to the incident. On paper, it's not about pointing fingers (though I've found this to be somewhat idealistic), but about finding faults in the processes that let bad code slip through the cracks.
It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree?
This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
This is a general principal in the eng shop I work for, its written into every step, including the SCRUM etc.. What when well, why did it fail, what could be done better. Its all part of root cause analysis.
I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
I don't see how this could work, right now, since every failing I've had was just generic stupidity of the AI, which is brilliant one minute, and a complete idiot the next.
I think correction actions still require the ability to execute them, which (in all the cases I've had) would require more capable models.
Long term, I think you're probably correct.
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
Honestly I feel like I'm taking crazy pills these days.
Write a program! If it has a bug, debug it and update the program. Use an LLM to debug it if you must. If you're doing something repeatable, then use a traditional automation scheme.
If an LLM has to handle a subsection of the program, so be it. Just wrap that part with clean inputs and outputs.
Why does it feel like we've forgotten how to automate things?
the history of assembly lines would be instructive. I don't think we have achieved a fully automated assembly line yet, but certainly they are much more automated than they were when Ford set up the first one.
Sorry to get hung up on what is probably just flavor text, but the first sentence.. "Agents can oneshot games that are actually fun", uhhh such as?
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
Yeah, that goes against my experience. Agents can absolutely one-shot games, but the gap between something that looks like a game and something that is fun to play and worth trying for more than a few minutes is gaping wide.
As someone who builds games, particularly games meant to be played in the browser or apps, it can 100% one-shot a game-like experience, but it will not be very good.
You need to playtest the hell out of games. Play a bit, iterate, play a bit more, iterate, etc. You will get way better results if you start small and don’t try to one-shot it. Start small, vertical slice, playtest, and go fro there.
I built this in one shot cube-sweeper.com
(If that doesn't load try cube-sweeper.onrender.com some isps block it)
I did make a bunch of changes after the initial "one shot" but the bulk of the game was done after a single prompt
Even very basic games can be fun (e.g. tetris). Though maybe the funness of a game is more an art than a science.
If AI coding now is all about taste, when the first sentence shows off just how little taste the author has, I can't trust the rest of what they say.
Probably fun defined in the ephemeral context. As-in it was "fun" to play in the context of the short form video attention span, social media brain rotted individual. Then it was thrown away and never played again.
It's definitely not fun if you consider building an actual community around it and have a long term plan for it.
I wouldn’t call these one-shot “games” fun either, but your definition of “fun” sounds more like “work” to me.
That's a weird definition of fun.
Honestly agents can't one shot shit except cool demos. I mean in a way they can but I let an Astra/Fable 5.1 duo build an android app and figured I'd really just loosely vibe code this for once instead of supervising.
It's a working pile of trash that has quadratic growth in the remote postgres database resulting in 10GB stored in a few days for what should have been like 5 MB of data.
It's not a skill issue, I know how to get good results but I figured maybe I don't need to be as cautious as I am all the time.
Not that I don't believe we can get there eventually.
Agreed, I read this and almost instinctively decided to close the page, agents can be used to build games for sure, but that "one shot" and "fun" in the same sentence is really misleading
FYI, it looks like you've been banned for the last ~8-9 months, for no obvious reason I can see. I vouched for your comment to make it visible, but they appears to show up as [dead] by default.
i can see the comment now, so this seems undone?
[dead]
Author here, hello! Happy to expand on how we're thinking about this if any of it is unclear.
We wrote this post as part of a launch, which you can check out here: https://x.com/danlovesproofs/status/2095182189499711759
My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
Anyway, here's mine, still wip:
https://hale-lang.org/docs/dna/
https://github.com/hale-lang/hale/issues/690
I’ve definitely had a hunch about a software factory harness kind of thing.
Your thoughts on the matter seem clearer than mine.
I’m kind of relieved to see so many stages; that’s something I’ve grappled with conceptually. The volume of (sometimes optional) stages has made it hard to resist the temptation to hack something together with a GHA workflow.
I salute your choice to support distributed; that’s a requirement in my book.
I look forward to trying it out.
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I think that dissolves the "self driving" distinction, though? The mechanisms for driving the codebase must be present in the codebase itself. Otherwise its just a regular out-of-band development process.
fine-tuning may be a more scalable approach to LLM personalization than sending all the same context to two LLMs
I'm working towards both in my homelab to see which works better with little qwen
[dead]
If you have a logo in the top left corner of your company blog the logo should route to the home page so I can see what product you're actually building.
Makes sense to me - I see it going down two separate routes in parallel:
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
> fluid, self-modifying things w/ contracts
This smells like worm food. Those contracts had better be airtight.
What would you prefer: 99% of user requests to succeed but w/ 5% risk of getting O-day'd, or have 90% of user requests succeed but w/ 0.1% risk of getting 0-day'd.
I could imagine it going so far as to have an LLM re-impl core libs every time a request is processed. Lightning never strikes the same place twice.
The first and it isn't even close. Normal operation of the system is more important than preventing attacks, and trading it away at a 2:1 ratio is a bad trade.
You're trading it away at a 50:1 ratio though.
The point of the example is that with the first choice you have 5% legitimate customers and 5% malicious "customers" and by letting all of them in aka all 10%, you get maximum customer satisfaction including satisfaction of the malicious customers.
Alternatively you reject the malicious customers and as collateral damage accept that some customers do not get their request approved.
You're also confusing "normal operation" with what a user request is. A user request is a request to change something, so by definition it is not the continuation of the existing system. You're thinking that a user request is a HTTP request which is not what's being discussed.
Personally - for banking - I'd be plenty happy dealing w/ 90% uptime if it gave me an order of magnitude more security against getting robbed. This tradeoff may not have been realistic a decade ago, but it seems to be where things are trending.
It also is de-facto the case today when logging in as per usual sometimes requires a captcha or additional verification.
This would be more convincing if there were a simple stat on the home page of the dispositions of the reported bugs (fixed/no-repro/won't-fix).
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
[0] https://github.com/tailscale/tailscale/pull/17843
We should publish some stats. The fix rates on bugs are around 45% baseline, increasing over time as we learn from behavior within a given codebase. The merge rates for the codebase health work we do (e.g. deleting dead code) are very high, at least 90%.
Is our fate to become more and more product designers and less and less software engineers?
I feel that way.
But I'm now able to actually create and release things. Before, I used to obsess over code architecture, optimizing every single line. I could and can craft beautiful code. And then I always burned out and lost interest (gemini). Now I don't touch code at all and have been actually creating and releasing the ideas I've always had in mind. Then I realized that's what I always wanted. The code was always a means to an end. But that's me.
In the end, we already have processes that work for humans and we know the types of testing, verification and review that makes a codebase grow healthily. We just need to adapt these designs the best we can to AIs.
AI have a lot of advantages over humans like speed, controllability, good defaults (e.g. Claude will write helpful comments unprompted which is better than most engineers), and the ability to implement best practices like comprehensive testing with much lower cost than a human team. Using AI in my personal projects, I am able to write software at a much lower bug rate than manually, because now I can make an automated test suite whereas previously I had no time to do so. However, AI also has downsides like rigidity, limited context window, and lack of creativity by default which needs to be resolved with harnessing (e.g. different agents with separate, well-compressed contexts) or worked around with human review layers.
I suspect the best core pattern for coding automation will be to create an engineering manager agent layer that can flexibly apply a set of engineering practices to the concrete problems at hand. It would handle all coding agent pings before it reaches a human, with prompting so that it reliably considers the big picture when making decisions. Very rarely, it would escalate an issue or plan review directly to humans when it is truly blocking, but most likely it would organize the information it obtains from the lower layer coding agents and surface them as reports for humans to review.
Kind of feels like we're in a similar place with AI programming as we are with self driving vehicles. Good enough for the typical case, maybe even better than most humans, but bad when dealing with novel situations or situations which require a really cohesive model of the system (code, business requirements, physical world, whatever).
My answer to this is adding an engineering assurance module to my specification and verification toolkit. Every project has a different definition of success and quality. My goal is to provide a consistent way to define, analyze, and report quality metrics tailored to each project.
https://github.com/agent-ix/engineering-assurance https://github.com/agent-ix/quoin
> Global memory
I don't get why we need global memory for code? Aren't code comments (even if invented for humans) the ideal place where to put "memories"?
LLMs also do not have anything like memory, by design. Everything bolted on that smells like memory or is being called memory is a crutch, at best. Not being pedantic, I just think this aspect is lost on a lot of people.
Also, when we humans are tricked into perceiving a remote mind, we automatically assume it has a memory like us, or at least like large animals.
In other words, there's a story-document between a SherlockHolmesBot and UserPlaceholder that's growing like a crystal formation. Some software sees "Sherlock Homes Says" and then "performs" the dialogue at us, and we then assume Sherlock Holmes exists with a mind and memory, rather than being a facet of a text-story.
Exactly, this is why we see such minimalistic memory systems being used. Information retrieval is bad, but improving information availability by pre-computing information retrieval is not the obvious fix people think it is.
Sometimes, but I think there are a few problems with this:
1. Some context doesn't have an obvious place to write it in the code. If you're explaining why a tricky function is implemented a certain way, you can leave the explanation above the function or within the function. If you're explaining something more general, there might not be a natural place to put it.
2. Relatedly, some context doesn't have an obvious location to read in the code. E.g. you can comment on a schema that a table is intended to be append-only, but an agent could easily miss this comment if it doesn't gather context all the way down to the raw schema. Any research process has unknown unknowns. There may not be a canonical place to look.
3. Adding comments for every single human intent might be a bit noisy. E.g., if a code reviewer flags something that I know isn't a bug, I want the system to learn from this, and I want that knowledge to take effect outside of just code review, but I might not want every single code review thread to yield a codebase comment.
1: There are corner cases, no doubt, but the main source of information should remain near the code.
2 & 3: How to be sure that the agent will pick the "append only hint" from the memory? Especially when it starts growing and cannot be part of the context?
I think the bigger need is for outside concerns, like: what is the infrastructure like? How much volume does this feature handle? How should changes to live code be rolled out to prevent running processes from failing?
In addition to all the product details that aren't in the codebase or docs, like "keep this logical path because it is used by our one big client."
Mostly what I think is needed is a richer worldview available to the LLM so it understands not just the code but can understand the product and its real world usage and constraints.
But how you are going to link the "keep this logical path ..." with the code? You cannot feed all the memory to the agent because of the context rot, and you are not sure the agent will always pick it when needed.
The things in the 'what should be self-driving' section are the exact things i give to juniors so they can build their mental model of the codebase.
Now, I agree it'd be fantastic to have that automated, but how do we then share that context with the humans that will drive the non-self-driving actions?
I have integrated AI into my application. The source code is embedded into the binary. When someone questions the numbers in my app it can debug itself. If it truly finds a bug in itself it will tell the user and open a ticket. Soon we are going to let the application fix itself.
How do you protect intellectual property? Or is this a case of the value being somewhere else, such as in your backend? If so, how does the agent debug frontend and backend? I presume it stops at frontend
This article hints at it, but I think the blocker is environment.
How can you self drive an app on windows? There's no clear UI framework, design pattern, nothing which can bullet proof your app.
I bet you'd have better luck on Plan 9.
Remove the choice. Make frameworks which have limited options which are usable by default.
Spot on about environment.
I recently built a calculator to try to quantify the gains an org could expect as they become more "AI-native." I tried to account for a bunch of things including env, which I call "AI roadway," but I'm definitely missing codebase readiness (thank you, author), and also bottleneck analysis. Even without those considerations, though, gains are usually modest. It's hard to get to 2x.
Tool is here if useful: https://timvasil.com/ai-native
Not sure about you, but I've been writing real world software using agents for a while now. https://github.com/jaksa76/microfactory
Is this the "real world software" or a tool that creates "real world software"?
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This happens way too often in my experience and I tried a lot of things. If the agent has access to an existing codebase it will always try to stay as close to it as possible. It doesn’t diverge from the initial direction the project/code went into, even if it’s already clear that this direction goes nowhere. Somehow agents aren’t able to detach themselves from existing code to come up with completely new solutions.
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
Super exciting, and absolute killers on the customer testimonial page.
Sounds like the next shitty Vercel ad at re:Invent.
It's a losing proposition. Self-driving cars go from point A to point B, and it's really easy to specify point B. My experience so far using agentic AI 24/7 for almost a year has been the AI can drive almost anywhere, so the whole problem is specifying where to drive. This doesn't get solved with faster, better AI, because all that will do is allow it to autonomously drive your codebase to random places you didn't want it to go. Then you're still left with the problem of specifying the problem.
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Well put. I've found that agents are great at adding features to areas of the code that I already understand, where I can phrase my goals coherently and keep them on the rails.
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.
I'm hoping the "blogging about doing" format will fade away, now that doing is cheaper than blogging. It used to be that the doing part was expensive, and risky. Now you can just do 5 things and throw away 4. So I'd rather see the finished product than talk about it.
> Agents can oneshot games that are actually fun.
Stopped reading right there.
> Agents can oneshot games that are actually fun.
HAHAHAHAHAHAHAHAAH
here we go again
Joust isn’t fun?
“how can we make ourselves even MORE replaceable?”
Reminds me of that Richard Scarry children's book illustration of a fox busily sawing off a tree branch he's sitting on.
I currently see a shit ton of work involved with "scaling" (aka cleaning up) the AI generated slop that was output in the past 2-3 years.
The missing piece in most versions of this is the feedback loop, not the harness. A codebase can only drive itself as far as it can tell whether it's still on the road: tests that fail for the right reason, error budgets, a way to notice that a change made something slower rather than broken.
Without that, "self-driving" degrades into applying diffs that compile. With it, the interesting question becomes what the system is allowed to do when the signal goes red - revert, retry, or stop and ask - and that's a policy decision nobody has good defaults for yet.
[flagged]
[flagged]
[dead]
I'm playing around with ecosystem driven development (hear me out ;)):
So i want claude to build a small game for me so of course i create one big file in which i write everything then i tell claude to analyse it and grill me (grill me skill) to clarify all smaller details.
This alone might lead to really good small sfotware but I still have to push it sometimes.
Now instead of doing this, i tell claude to build a small tool which generates a dashboard and memory and which can save specs and ask me choicses (do you prefer this color over the other, what do you think about problem A? What solution would you prefer? A, b, c or something else).
This does a few things:
1. claude doesn't has to save/store everything in a context 2. claude can now talk to that tool to ask it stuff 3. claude can now use the tool as a todo list 4. claude now can more easily spin up more agents in parallel 5. i have a nice interface and i can solve issues while claude works on unblocked tasks 6. I actually can follow the progress a lot easier
The only problem with this is: with the next update, you have to reinvestigate how claude was finetuned and adjust. A few month ago /goal was really good, now you need it a lot less because claude will do something for an hour without /goal
And the spec file only started to work after November/Opus moment but it got so good, that i can pack A LOT of stuff in a half structured markdown file and let it code what I need.
The progress is still too fast for the whole ecosystem