This is from March 2026, but it does match what I've encountered professionally. Code review was always a significant bottleneck and is even more so with agentic coding. We've had discussions about leaning more on automated review, but it hasn't translated into any practical improvements so far. I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature. I'm sure that's happening, but it's likely invisible to most studies and surveys at the moment. If it proves successful, and such companies start shipping significantly faster, the rest of the industry will be forced to adopt.
And test creation and maintenance. AI-generated tests will often embed existing bugs as testable patterns that fail when fixed or rely on empty asserts to increase coverage. A human author generally knows better. Coverage is increasingly becoming theater and you have runtime testing eating up CI resources and time with bloated, unnecessary, untargeted testslop that haven't been mutation tested or a human eye assessing if the business logic is actually represented.
It's really the smaller firms which are able to learn quick adaptations that will win out in the long-run of this thing. Legacy firms with institutionalized bureaucracies are relying on gobbling up competitors and creating vertical monopolies while pushing out the worst slop - it's why GH goes down, why Meta products are buggy as hell now, and everything feels like it's unpolished MVPs these days.
This. The ability to know that a vibe coded PR is all right to merge, is a function of both the thoughtfulness and simplicity of the underlying architecture, and the level of ownership that contributors have of the underlying products. Both benefit more nimble teams.
I agree. The latest models are getting quite good at validation, as long as they are clear on what the requirements are. Recently I got frustrated that while we have a huge test suite I noticed the LLM was hardly ever breaking a test with its changes (PRs only added tests, never changed old ones) which seemed suspicious.
I had it do a full audit. There were about 100k lines of tests, of which it determined about 60k were near useless. Many were overly mocked, some mocked to the point that they really weren't testing anything at all. It deleted those and wrote more expansive scenario tests to replace them, caught a lot of bugs on the way and fixed those too. Total time was about 2 hours.
I think humans need to review requirements, specifications, and test scenarios. And for some industries and uses they're still going to have to review every detail. But for a typical CRUD app, I think we're at a point where we can already set the process and quality bar higher than ever, in addition to moving faster.
I chuckle a little bit at this specification thing. We used to do this. We used to write big binders of specifications, and then we'd set about implementing them. We stopped doing it because it didn't really work. We'd learn things in the process of building which negated the value of the big specs we wrote. That's why we stopped doing it. This is where continuous improvement, CI/CD, tighter release cycles, etc. were all born.
But why? On reason is that implementation was a bottleneck. That made it expensive to change a system once it was entirely implemented. It was better to get the feedback sooner so you didn't waste time implementing the wrong thing.
The calculus on a lot of these old adages is changing.
I don't think this was only about implementation being the bottleneck. It was also because implementation forces you to think about the thing you're building at a very grannular level of detail, at which point you might discover flaws or opportunities that weren't obvious at the higher level.
And I find that, when it reasons through and encounters some of those, it just makes assumptions rather than asking the user.
It is _still_ better to get the feedback sooner. Writing up some big specification, spending a bunch of tokens on implementation, and then shipping a massive PR to match the spec only to then find out it's wrong is a big waste of time. Nothing about writing grand plans in the face of unknown unknowns has changed.
The more things change, the more they stay the same.
Glad to see someone else notice this. I dread looking at or dealing with AI written unit tests because of the incredible amount of whiteboxing that goes on, though usually for the opposite reason because AI testing is almost entirely whitebox testing which fails if some internal details change
Funnily enough I've found that LLMs can be effective at mitigating many of the drawbacks of black-box testing and i've significantly expanded our usage of it over the past year
If before AI you couldn't deploy to production without a code review, what makes you think you will be able to with AI?
To be fair, when I was a kid we'd connect the editor to the server using an unencrypted ftp connection and modify php files in production.
This worked perfectly fine, and phpmyadmin made schema changes super easy.
These days we do tests, schema migrations, terraform, staging sites, traffic migration, monitoring, logging, revision control, etc.
... it's a lot of boring work, and for most CRUD apps, its entirely unnecessary :)
That said: I doubt we'll abandon these practices for things that matter.
Because while modifying php files in production can increase velocity, it can also get very "exciting" if you have a few co-workers on the same project :)
Just have AI write requirements then another AI write specs based on the requirements then another AI write tests based on the specs and another AI write the implementation based on the tests then another AI write the PR based on the implementation then another AI approve the PR and another AI deploy approved PRs to production!
I just don't understand the hype for this. "AI can do $x" quickly needs softening to "AI can try to do $x" which then becomes "humans need to verify $x has been done." If humans still have to oversee every part of the processes then what exactly is being automated!?
This would be like if we had our first factory producing cars but each car was like a die roll in terms of getting a car, a washing machine, a pile of scrap metal, or a baby. And people just asserting over and over that this was the future of making cars, no matter how much scrap metal keeps coming out.
IME, the biggest source of rework cost that happens at (or after) code review is when someone notices that the code, correctly or not, implements the wrong thing. For example, the feature, as implemented, doesn’t sufficiently solve the original business problem. Or it solves it in a bad way, like because of overengineering or taking on unnecessary technical debt. I haven’t really seen much evidence of AI code review catching either of these problems.
I’ve heard people complain that human code review doesn’t reliably catch them, either. And I have seen that with my own eyes. Just not universally. Teams that have a strong culture of shared ownership and collaborating on designs regularly catch these kinds of problems in code review. If they’re really good at it, they catch them even earlier. e.g., I know planning poker isn’t fashionable anymore, but when I’ve been on teams that take it seriously they tend to be pretty good at finding minimally expensive, correct designs pretty early in the process. Which makes both coding snd code review go faster because simpler designs are easier to manage. And design flaws that do make it to code review tend to be smaller and less expensive to fix. So those teams didn't tend to bottleneck on code review in the first place.
I think that this is the kind of thing that led to folks like DORA observing that AI coding is something of a multiplier for the effects of both good and bad software development practice. If you’ve got a cowboy coding culture, throwing AI into the mix will probably just lead to agentic cowboy coding.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
I actually dislike traditional code review. In part this is because most of our code review tools are very cumbersome to use (GitHub's and GitLab's web interfaces in particular).
But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
The problem is that purely agentic code review has orthogonal problems, and typically remains subpar when it comes to judging architecture and design considerations, compared to human reviewers. Both false positives and false negatives also remain a concern.
My best results so far have been with agent-assisted code review in a proper IDE, helping you to quickly construct a mental model and to analyze the underlying logic (in addition to having agents do a first pass for correctness issues).
> Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).
The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.
> I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).
I don't think we actually disagree here. I also think it is valuable, but it's also what often makes it difficult.
> The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.
Agree very much with that (with the slight caveat that it is not always the biggest value, just an important one).
> exactly the fact that a reviewer had to develop a mental model
But do they? In my experience, at most workplaces it's simply a meat linter phase with the occasional useless comment about style (which should probably be handled by proper CI/CD).
Like if you don't even check out the code (generalized 'you'), then they can hardly build up that model. And even checked out, viewed in its proper context, Naur's famous paper comes to mind - the author who wrote it is the only one who has properly built the theory (which might have included alternative implementations that failed but have no visible artifact).
I'm not claiming that there are no places that go absolutely above and beyond with their review sessions, but I would argue it's really rare - and AIs are great at figuring out some of the context the PR description lacks.
People say one thing and do another. It’s tiring. Long before LLMs we had the joke that a PR with 10 lines changes got 10 comments (all nitpicking), a PR with 1000 lines got just one, LGTM. We’ve had several bugs where PR review would almost certainly never catch the issue, people are terrible at reviewing things thoroughly. Guess what can do that much much better?
What you describe sounds like little more than rubber stamping a PR target than reviewing it.. I see it too, but I don’t thing we should double down on it just because it’s common
> But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
Code review is not really about correctness and security. We have testing and other verification process for these. Code review is about idea discussion mostly and correcting a few stuff that may have slipped through the author's initial review.
Your PR wants to change the software in some way. My review is to comment on how you've implemented that change, like did you miss some areas where you needed to change, does your description is actually what is happening (either a cursory reading or through testing), or are you doing some unnecessary change.
IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed. If it's an important area, a more careful reading should be done about the tests and relevant stuff.
But generally, if a review takes more than 10 minutes, there's some failure of communication going on (reviewer not apt to understand the change, big diffs that should have been discussed outside of the PR, obtuse code,...)
All good points. Maybe having worked a lot with concurrent and distributed system makes me distrustful of relying on unit/integration tests for correctness; I've seen too many concurrency issues pass CI validation. But unit tests also sometimes miss edge cases in more straightforward code, so I'm still a bit more cautious.
I also agree that merge requests should be short and understandable in 10 minutes, but IME this happy state often does not obtain in practice. It shouldn't, but it often does, whether it's mission creep, whether it's turning out that a change is more complex than originally imagined, or whether it is unavoidably complex and cannot be broken down further.
> IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed
Part of the code review process is to verify that the writer actually did build the agreed design too though.
We shouldn't give up on line-by-line review for serious applications.
My company's leadership is oscillating every few weeks on AI goals. Their most recent story is "we're expecting developer velocity to approximately double over the new 2-3 years as people get better at using AI". But of course that's completely non sequitur. People on my team spend about 10% of their time writing code and 90% doing design and analysis. So if AI doubles our code writing velocity, then it will make us 5% faster overall.
It's interesting. I remember around COVID, there were a bunch of startups building 3d printers for concrete housing. Then I looked up the actual cost of construction for a typical house, and only 14% of it was the most visible part (the balloon framing and physical house). Most of the cost is in foundations, plumbing, wiring, and things like insulation, flooring, cabinets, etc.
I guess at Microsoft coding is similar to building a house in that sense. Either that or all big orgs devolve into large amounts of time spent on politics.
90% of my time these days is trying to understand what the wall of claudeslop jira ticket is actually asking for. It's like archeology digging for the original customer issue to work out what is actually wanted underneath paragraphs of root cause load bearing seams.
The point is that while good to remember, and specifically Fred Brook’s “No Silver Bullet” article about it, there are second order effects.
If coding becomes 100x cheaper, what’s going to happen next? At least to some degree, design and planning was there since coding is expensive. (Not just coding, debugging also got squashed by AI and was an expensive part of the job)
This is why the original comment seems pretty spot on, organizations will learn how to skip more phases, and startups are probably a place where this will be happening faster. (Not clearly though, a lot of big tech are happy tokenmaxxing to try and not stay behind those startups when they come)
Brooks' point was more about accidental vs. essential complexity. You seem to be implying that design was only ever an artifact of "expensive" typing, but design exists to prevent systems from collapsing under different failure states.
If the cost of emitting syntax collapses without adequately resolving the essential complexity then you have not eliminated the design phase and you're just externalizing your failure modes onto downstream teams and systems.
Google had many engineers working on a) failed projects b) people being paid to sit around.
There seems to be this strange belief on here that making something more accessible to everyone will always result in better outcomes. I don’t see this being true of software.
I’m very pro-tech. But nobody can point me to a product built by someone outside of the large tech firms, made by by llm’s that is a great value add to one’s life. Many of those were already created - they’re on the apple App Store for example.
I’m absolutely confident coding speed is irrelevant. A person who has a real vision + strategy and hand written code will beat the team with no vision, strategy and code blasted out by llm’s.
Sure llm’s could augment the former - but you’re missing the point!!
There's a difference between vibe coded slop, vibe coded kinda slop built by an engineer, AI assisted engineers, and AI planning. I enjoy writing code by hand but it's quickly becoming apparent that LLMs are now a tool that one should be using in some capacity.
Nobody writes code in notepad anymore, even though you could and might even like all the black text.
Their very cheerful "high" revenue increase from AI adoption (that few companies in the index manage to achieve) sits at 4.5%. Most of those in the "value generating" minority are seeing sub-1% returns.
Which, of course, is non-zero, real, and all that, and matters when we're talking about a 4-5% increase of a big company's revenue, assuming the expenses are not catching up with that.
Which they absolutely are. The AI is subsidized by venture capital, and the subsidies are drying up (hence OpenAI upcoming funding round). None of the AI companies are profitable (much less self-sustaining), and AI prices are inevitably increasing.
> I can have an agent mass-produce security bug fixes
I can't figure out what you're trying to say here. Are you saying that this is acceptable? Because I very much don't think it is. You'll have twice the bugs after you're done, and they'll be inhuman ones that you have little chance of catching.
> But how many changes are actually serious?
I mean, "security bug fixes" are not on your list of serious changes? What is?
...and they are not using LLMs to help speed up design and analysis?
I'm no booster but this is most of what I currently spend tokens on so it's maybe not as absurd as it sounds. Maybe those gains are possible and without having to sacrifice serious code reviews too.
You don't give up on line-by-line review just because the human is getting better either, and senior engineers want other people to do line by line review on their own code. It's not a matter of how good or bad the person or program generating the code is.
As a senior engineer that is struggling to give up line-by-line reviews, I agree with your take.
I've spent literal decades where one of the most valuable practices has been code reviews, for many dimensions. It is one of the best ways to ensure quality production releases. It is one of the best ways to transfer skill to junior engineers. It increases tribal knowledge throughout the engineering org by exposing others to systems they may not be working in. The number of benefits to a traditional human-centric engineering team is hard to overstate.
But I'm starting to feel that instead of starting at one extreme (heavy handed reviews) and moving towards less review, it probably makes more sense to start at the other extreme (no review at all) and move towards a brand new practice tailored for the new agentic reality.
It is looking increasingly likely that we don't need to modify code review for the agentic era, we need some completely new practice that hasn't yet been invented.
It's really about hamstringing the AIs so they can't write crap. Writing code is not a problem anymore, so anyone with an idea can write a script to do what they want. But if they try to connect two ideas together, now they have a data modeling problem. This is where everything seems to fail with AI coding (largely because the original author had the idea piecemeal).
The problem I have is non technical people now started submitting PRs. They are like the brain damage module on GLaDOS and somehow make opus generate actual broken slop. So I still have to be on my toes to catch that, but they generate such volumes of broken PRs that it takes time away from me doing real work.
Perhaps spec review in the custom of code review - it can even be via PR - so there is a bottleneck before anything is generated, instead of after. Cheap-to-produce code maybe isn't worth understanding line-by-line anymore.
I think spec review, which is definitely valuable, solves a different problem. One question to ask is "are we building the right thing", which a spec review definitely helps to answer.
But another question is "did we build what we planned, and did we build it to a high quality". Code review partially answered that question. Adversarial review is the closest we have to that in the automated LLM world, but I feel we need something better.
In the sense that ai-engineering specs (ie an elaborate prompt with skills etc.) can be said to be compiled into code language x maybe.....the spec is the new code?
It's not code review that's slowing things down. It takes time to get the thing you want, and then you have to be reasonably sure that it's right anyway, or you risk shipping a piece of crap to production and tanking your own reputation. Even on teams that rubber stamp PRs I see people pushing drafts and waiting to merge until they feel comfortable with what they've vibed out.
I lean towards a hybrid review process where humans accept their limitations, focus what they're good on(higher level concepts and architecture of the proposed change), and let the LLMs do the rest.
Doing a line by line review as a human is silly now.
If you're right (and you probably are), we're in for a wild ride further down the quality death spiral.
The era of user telemetry and conversion tracking already decoupled product design from customer experience. No need to wait for customer's to articulate their experience or desires or for discerning analysts to amalgamate and ponder their opinions. The numbers tell you everything you need to know! Just throw 3000 pounds of spaghetti at the wall every month and wait to see what sticks.
But now you really were bottlenecked on engineering.
Only 3000 pounds of spaghetti per month? And throwing more bodies at it barely even makes a difference? Oof.
Of course the solution is to just flush pureed spaghetti-slop through the hose as fast as we can. Brilliant!
What could go wrong? The users are sure to love it. We'll know because they keep clicking. That's how they tell us the truth, even as every forum fills with louder and louder complaints!
> I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature.
I've been hearing this for more than a year. But if these startups are all moving at superhuman speed, a month instead of a year from idea to a shipping product, where are the products? Every time I ask that, people talk about the amazing velocity gains they experience for the thing they are about to ship. It never does, unless we count throwaway hobby projects at GitHub - by that metric, AI is adding a lot of software. And I guess never-downloaded slop-games on mobile app stores.
But if you actually go shopping for consumer or enterprise software that's camera-ready, the landscape is basically the same as in 2022.
Codex/CC and other OpenAI and Anthropic products like Dots, are examples of how quickly real products can be built today. But to me, they feel more like mediocre products bloated with feature slop. Nothing extraordinary or technologically innovative.
Pi.dev and Cursor's Tab completion are two of the most innovative and extraordinary products I know in AI era, and both were built by humans, at least in their "breakthrough" stages.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Why do you feel like that? Agents have got better but better how? Looking at the frontier model output, I’d say the same myopic approach that causes long term issues without frequent course correction is still there.
You have to classify each code review based on what it changes and decide how to handle it, agent review and/or human review. You can have an agent do that.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Our industry has spent the better part of the last two decades interviewing with brain teasers and leet code questions. Reading code is a skill that we just weren't assessing for, and lots of people aren't really good at it.
I would argue that code review is mostly theater, politics, bike shedding and pedantry.
There is a lot of work to get away from "coding" and "programing" and get down to the actual engineering that really needs to happen in the industry.
I think non-senior engineers don’t realize just how much of a burden is barely reviewed ai generated code.
I’ve seen a few projects that heavily rely on ai with not very good (though senior) devs driving it - and its a mess. Not the code. The resulting product. Code is actually sound as ai is pretty good now.
All because ai does code reviews and dump decisions go into prod
My experience with prototyping games is that you need very strong human driven guidance and review in the beginning, then you can increasingly take your hands off the wheel.
If you take it off too early you end up with shit and will spend weeks fixing all these baked bad assumptions your project is built on top of. I spent weeks fixing some of this shit then the LLM revived a not insignificant portion of it in a later change because it got "confused" (its words) because some of the comments it previously made were left around.
> I think senior engineers find it very difficult to give up on line-by-line review
i am pretty senior. I hate code reviews so much. They are so boring, easily the most tedious part of my job. Its such a chore. Who is clinging to line-by-line review. Ppl dont really give that much about their employers.
Code review isn’t a quality gate, it’s a way to maintain shared understanding of a codebase.
Throwing out code review is equivalent to throwing your hands up and saying you don’t need to understand anything about what your code is doing in its black box.
I don’t think anyone on this forum is half as smart as they think they are, especially in terms of systems engineering a black box solution. Myself included. Someone needs to understand what’s going on inside the Widget Factory until we have full Artificial General Intelligence.
>Throwing out code review is equivalent to throwing your hands up and saying you don’t need to understand anything about what your code is doing in its black box.
I think that is what the C-suite is currently wanting out of our engineers. Prompt, don't write, maybe read a bit, don't stop, ship, ship, ship.
Is the C-suite ready to sack the engineers and prompt Claude themselves with their business requirements? Why not cut out the middleman if that's how they now perceive engineers?
This will vary by company focus and size, but sometimes the C-suite just doesn't want to believe that software-engineering has important overlap with business-analysis or product-design.
I initially pivoted to fully automated code reviews, sometimes with a brief skim of things. I've been shifting back to mostly manual review. In the background I'll also run some agentic scans, and once I'm done I'll look to see what the agents had to say. And then I go back in by hand armed with that knowledge and verify it & come to my own conclusion.
Why? Because I started to run into way too many AI generated messes in codebases. The agents are really good at some things and complete ass at others.
It really depends on the context. A react button background color - meh - the critical part of the backend where a new 300ms delay will cost the company millions then yeah.
> the critical part of the backend where a new 300ms delay will cost the company millions then yeah.
If it is so critical, why don't we have tests and benchmarks for it?
Oh right, because "Greg the super-coder" never wrote any of this down and it lives in his head so he can be the arbiter of change.
It was toxic before agentic coding, it's still toxic now.
Document your systems. Add tests and benchmarks for what is important.
We've always known this, we have a dozen or more textbooks written about it. But we always gave it a pass because "Greg the super-coder" was producing more value than he was hurting.
That's no longer the case. Your best coder is now Opus or Astra
I hate code reviews, but I owe it to my coworkers that I know will read my PRs to review my own generated slop line by line. Most of these agents are still overly verbose and I'm in a situation where no, not every single edge case is relevant enough to warrant the additional loc or complexity introduced.
Both Claud and Codex would run circles around 80% of the developers I’ve worked with through my 20 year long career. This whole code review thing is a hold over that probably won’t last six months.
That and they do not necessarily generate better code. They tend to generate code that is repetitive. It misses opportunities for extraction and refactoring, the result is bloated and hard to read code (because those blocks are there literally instead of with a usable name) and they tend to over-comment where it doesn't add but is 'safe' because it is obvious (effectively re-writing code as comments) and fail to document where documentation would be essential. There also typically isn't any real plan to what is written, as long as 'the tests pass' it is all good.
You can fix a lot of this stuff by shepherding the whole process but it can get quite tedious. I'm impressed by the number of people that are letting Jesus (one of these days an AI provider is going to name their AI that way :) ) take the wheel and I wonder how long they will manage to support the output. So far the score isn't very good. I've gotten good mileage out of these tools by letting them loose on very narrowly scoped pieces of code that are written in isolation. As soon as the codebase gets larger than 100K LOC or so managing the whole thing becomes an exercise in putting up barriers to avoid spillover from a tiny issue into a much larger one.
This is not unlike hardware: making prototypes, assuming the principles are valid to begin with is relatively easy. Making a prototype that is a good precursor for a production run that's not going to result in a large number of rejects is stupendously hard. You can't fake your way through that. This is one of the reason many Kickstarters are doomed to failure, it's one thing to have a device working on your table in the lab, a completely different thing to produce that device reliably over and over again with negligible QA failures.
That's why some of these projects really impress me (for instance: that phased array antenna on a Raspberry Pi features here a short while ago: https://scalerf.com/). The amount of skill required to pull something like that off is off the scale, and they make it look so easy.
Well, imagine racing car manufacturers competing on making better, faster engines. At some point engines are so strong that other parts become the limiting factors, such as tires for traction. Now someone invents superfuel, but cars are still only marginally faster because the additional engine power didn't change the bottom line much.
That's what's happening in software. We knew that coding is only a fraction of the overall work, and even completely automating it away won't increase productivity by an order of magnitude.
Yes, the job is changing, but only parts of it. High-level engineering best practices still apply even without the coding part. And there is no free lunch.
IMO it's a lot more like a new more powerful engine is now available. Sure, faster drivers/teams will run faster lap times, no doubt, and it'd be a bad idea to not use it. But, less mature teams/drivers aren't going to catch those faster teams by just slapping the new engine in and going. They are still largely held back by the myriad of other things that go into being fast on track consistently IE (driver + datalog review, and just good ol seat time), vehicle [other than engine] setup, aero tuning, pitting practice (esp fueling for endurance stuff), etc etc. All those same things that were making them slower than the fastest teams before. And even still, if you give only the slower teams the new engine, they probably still won't be competitive with the faster teams and, quite honestly, the not great drivers will probably just make the same mistakes they were making before that slowed them down, except faster.
But, even if the new engine is the golden egg, now what? You're still competing against every other team that has the golden egg... And it's right back to the driver mod, vehicle setup, etc.
That's not to say that the slower teams have any choice but to run the engine too to not be even less competitive. It's just to say that having the engine isn't gonna fix the underlying problem, no matter how much of the team you lay off to fuel the new, more power, engine.
"The overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used" - Amdahl's Law
Except the problem with this is an engine designer doesnt want to pivot to designing a stronger driveshaft, or strengthening the gears of a gearbox, or designing more efficient brakes. They're an engine builder.
Same goes for software. 20 years ago I got into software development for the love and passion of solving problems and writing code. Now I've been forced to pivot to a glorified BA/Products Manager. It sucks.
The flip side of that is as someone who got into software development 20 years ago because I love to design and make products... Well this is pretty great. I feel for my colleagues, though. And it's not like the "coding problem" is solved, much to the dismay of the hype machine.
Bide your time. When the eventual hangover arrives your talents will be in short supply. AI providers are essentially making a long bet: that they will be able to improve faster than the messes that they make are growing. But that can't keep going forever. AI is an accelerator: it will accelerate you if you were careful before towards something better and it will accelerate you towards something much worse if you were not. Just like any other power tool.
I've recently talked about how it makes almost zero sense to ever delegate anything again at work. 3-6 months ago, that was not a thing. 12 months ago, that would have gotten me laughed out of the room. Wherever we are headed, we are not even close to the steady state
> Wherever we are headed, we are not even close to the steady state
That I agree with, but the messes are already catching up with some early adopters and I've been watching a couple of these up close with the roughly same level of interest that you'd bring to a road accident if you were part of the technical investigation team. It is more than a little interesting to me how the difference between the initial conditions seems to be a fairly good predictor of the outcome. Great teams are doing quite well with this new tech, others, not so much. And great teams were rare to begin with.
Exactly. For many of us, designing and writing code has always been the fun part; reviewing and whatnot, the parts that are still left for us meatbags, was more of a chore.
Or we could just recognize these llms crank out poorly architected bug infested slop. The coding work of these llms is just as garbage as all the other bland, flat, wtf writing they crank out with prose. Even worse because there's no logical mind behind the garbage. With bad human written code there was at least one human there thinking about it.
As is standard for academic papers, this is an attempt to describe the state of the world as of when they did the research. But then it takes time to write it up and get it out, after which newspapers report it as current events.
In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.
The data is still mostly 2025 and the 2026 data is mostly from the first few months, and most production deployments (at least in, say, the Fortune 500) lag the frontier by at least several more months (or much more). I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
> I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
> I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
Given my own conclusion that it's not possible for conventional academic research to accurately snapshot industrial adoption of this technology, I would expect this pattern to continue for a while until either academia becomes much faster or technology progress slows down (or both). This just isn't yet a situation where you can rely on published papers telling you what to think -- you still need to mix in a bit of your own instinct and experience and extrapolation to get an accurate up-to-date read, even if there's some prediction error as a result.
I have my own experience to go off of, but this can also be inferred easily. There are many large corporations still working on "digital transformation" from 10+ years ago. There's no way such companies have fully onboarded and integrated frontier AI agents throughout their workforces with high adoption. The paper itself reports low adoption even after conditioning on access.
Everything did change after late 2025, but there are tiers of usefulness. Late 2025/early 2026 was the first time agents became clearly useful for personal projects. That's distinct from immediately having economically valuable end-to-end autonomous workflows in teams of median talent, which many would suggest is the threshold we've just recently crossed.
I don't, actually. I can simply point out that its data is out of date, and then the numbers themselves are called into question without other numbers to compare them to. And that's what I did: I shared two arguments in two comments that converge on the shared conclusion that the paper's data is not up to date (or, more precisely, cannot be up to date).
> I can simply point out that its data is out of date
You can, but it doesn't make it correct or, evidently, yield a logical argument. Do you notice that your logic is circular? The date doesn't matter if the passage of time has not affected the outcome and that's basically what I questioned, which you refuted without any evidence. So the question still stands. Are there any comparable stats you have access to that change the picture?
They've updated it for the agentic code era, and have found basically the same thing. Code can be created faster but that has moved the bottleneck to review.
Today I can write what I'm looking for, as if I'm writing a design doc, and the models can do the rest. A year ago it was faster for me to just write the code myself. Models today are enormously better than what we had in 2025.
A dimension not much discussed here is that code review is not only about quality but accountability and security. An automated review is going to help to ensure the code behaves as intended, but it is not going to ensure the intent was correct. Employees inserting back doors, bypassing tests, "cheating" on design decisions to get the work done faster etc are not going to be identified if the person driving the change tells the AI it is the intended outcome.
For this reason alone, I don't expect code review to go away entirely in most contexts where the code "matters" - in the sense that there is substantial liability to it being mis-specified at the implementation level, not just technically wrong in implementation. I do expect it to evolve a lot and become dramatically AI-assisted.
The main protection ultimately is a human being willing to put their reputation on the line by saying "I understand this change and I approve it". That, inherently, means review of some kind.
Has someone presented an alternative to code Review , i mean, instead of just not doing it? Formal verification sounds expensive, asking an llm to Review another llm otput does not seem bulletproof. Is there another approach out there ? Can it exist? I've been thinking on this latelly but i can't find a solution
Maybe I'm missing something, but it seems like the obvious solution is to have human accountability on the API design and tests, not on line-by-line source code. If the human writes an appropriately-scoped API and the code passes human-reviewed tests, then the implementation details don't matter so much (depending, of course, on project-specific requirements around performance and security).
I think there's a bit of a false premise around "line by line".
I don't need to syntactically parse every line to scan it for the the type of risks that matter here. You can look high level and see the pattern and shape of the code, the type of things it is doing. If it is doing stupid things ("Wait is it making a new connection to the database inside the loop here?"). It does take an experienced eye, but with that experienced eye it is a very rapid process.
Man, I'm probably going to expose my own inexperience here, but like, did anybody actually expect code reviews to catch and prevent bugs on a line by line basis? Like ever?
I never have operated this way. Code reviews were a basic sanity check, a chance to implement best practices, and a chance to teach these things to juniors.
If I was concerned about something breaking, I'd duscuss test coverage.
I review code line by line and build a mental model of the software as I review it. I find plenty of important bugs that way. PRs are an opportunity to learn (both committer and reviewer), communicate, and improve the quality of code a team ships. Sadly, many people decline that opportunity for the sake of short-term velocity.
I’ve caught bugs in code reviews before, but I would agree that it’s probably not the most crucial thing you should strive to do with code reviews.
That said, I think I’ve done a tremendous amount of PREVENTING bugs from being introduced via code review. Caught all kinds of weird code patterns, duplication, wrong levels of abstraction, etc. Essentially, the kinds of places that bugs like to hide and grow in.
I still think that’s valuable, and it’s the kind of thing that models generally aren’t great at, to this day. It generally relies on having a large amount of context about the project and about what’s being solved, so if you wanted to do this via automated review, you maybe COULD, but it would probably get pretty expensive, especially on larger projects.
At another level, code reviews also give you the time to really think about what is being built and why. I've often come across looming architectural or strategic failures simply by having to stare at a bunch of code and wonder why it works a certain way. Same goes for writing the code by hand too, though it's harder to see past your own biases and ideas sometimes. I find code reviews deeply valuable for myself and my team.
> Essentially, the kinds of places that bugs like to hide and grow in.
I like an ecological-framing, especially as systems get more complex.
Yes, I could hunt cockroaches all day, but it would be even better to have caught how the new wall was missing a metal screen over a low ventilation opening.
No. You are on to something here. The utopia I see programmers lamenting the loss of, the one where human code review was perfect and never let any bugs slip through, never existed. I see people on this board and others complaining all the time about LLM generated spaghetti code and bug-ridden codebases. Well, I've been working those code bases for 20 years. We didn't agents to make them. LLMs are a new tool, but the problems that come with them are nothing new.
Yes I have often caught stuff like "not using DB transactions in the right way" or "this is not going to be race safe" for instance from juniors just by glancing on a PR.
I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.
It was more important for giant C or fortran code bases. You will legit catch pointer errors or memory mgmt stuff, going line by line. Once Sun busted in with Java is when I really saw the shift to skim reviews. So much stuff is caught by compilers now, or SAST, that using dev time to find code smells is mostly a waste of their time. And it's not like the old way caught everything, it's just that the stuff we take for granted being caught by compilers or Sonarqube now, just used to get shipped to prod and cause abends and production support calls at 3am.
I think a code review should be just that. A review of the optics of code itself not the solution or anything like that. If that is important it should be done earlier in the process before coding. Even more important when coding with AI. The reviewer is not a tester. I've heard horrible stories from friends who mainly works with fixing AI generated code that junior devs pushes at a rapid pace.
It doesn't match my experience. We are producing more software, and being a lot more ambitious about the things we develop. The trick is mostly to do with getting good guidance in place and then you can spend a lot more time iterating the product itself. I spend a lot more time play testing the product now and trying different things / refining. However, there is a lot of previous work that's been done to create a huge amount of technical guidance on how to do things. Lots of things on Specification, Design/Architecture/Coding, Performance, Testing, QA testing, etc. I think getting all this guidance/tooling/process in place is critical to producing software.
I think it highly depends on what field you're in. Personal projects that I find useful in my everyday life? Churning them out like crazy. Enterprise software? It has slowed our productivity down to a halt--part of this is due to people not knowing how to use these tools, and the other part of it is the collective AI psychosis of our leadership (I work at a Fortune 500 software company).
Do you think its enterprise software? or the team? I work on a product ( https://www.mywildeye.com/ ) but we do a lot of back end enterprisey type stuff to manage our logistics / manufacturing as well. We are a small dev team with a lot of latitude to make decisions. For us, AI allows a dev to deal with a lot more things. We also churn out internal tools like crazy. Our Hardware, Firmware, Accounts, and Support people are all creating tools. Our CEO (former developer) is also getting back to making things, we recently released a free ios app https://epicfields.app/ so farmers can monitor water demand using free satellite data.
Very Large Enterprise is very hard to change, or rewrite, especially old systems. Our company mostly sells contracts to large corporations / government institutions (1500+ employees), with an insane amount of transactions, functionality, etc. Even with the latest models you cannot automate everything, the world is far too complex, and there is quite a bit of pushback when anything changes.
In order to keep some of these extremely large contracts (using Samsung as an example here) for example, we had to revert certain changes that were global but now we have a "Samsung version" because we want to maintain a large presence in Asia and Samsung leaving would be a huge blow.
At this scale, code isn't even the bottleneck, it's customer wants, politics, etc.
Note that they define “more software” by “more tickets closed.” Make of that what you will. They argue that this is a good measure because the scope of tickets did not increase. Worth considering whether you believe this captures what they claim it does. Are the extra tests you write thanks to coding agents represented in the tickets for your work?
“But all that extra code doesn’t translate directly into improved software output on the firm level. On the contrary, the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira did not change in a statistically significant way after AI tools were introduced (the researchers also found no “compositional shift” in the size or complexity of those Jira-tracked issues across the AI introduction).”
The bottleneck is, and has been, the whole team evolving the product. Having a clear, actionable, vision and being good at biting off the right next step and adjusting for the step after that is always the issue. Before AI teams took the wrong step size in the wrong direction or were so wrapped up in process that they couldn't take a step or so distracted by 'things' that they never even took steps. After AI the same is still true. Our process sucks in this industry. We love to make process because we love to make machines but the processes we love to make rarely help us and that is a fundamental problem that AI isn't changing.
I'm going to make a potentially bold claim here. This depends _heavily_ on what the threshold of "rarely" is, and whether we actually _love_ to make these processes.
In the case of people before process, as stated in the Agile Manifesto, the process is begat _from_ the people in many cases. That doesn't mean it shouldn't be questioned, but when the chimney sweeps that built the process are saying "I don't want to go back to things that are similar to the way that gave me mesothelioma", I don't think it's fair to write them off as the minority.
I’d strongly disagree with one aspect. Homemade personal use software has Definately skyrocketed. Projects I never thought I’d even have the time for are now possible.
Yes, I am literally inundated by cool new tools I've built that I never would have before.
I'm literally building a black-box client + server emulator for one of my favorite old MMOs, remotely, from a coffe shop in SF as I type this, on my laptop at home via Claude remote-control.
I read this paper several days ago, when my director shared it. Several things jumped out at me:
- the data is mostly from 2025 and earlier
- the baseline mean lines of code per month was 5017, the estimated increase was 1494 lines per worker-month
- the baseline is 5.34 PRs per month, and increased by 1.22
- the baseline mean for code review is 7.03 days
The second and third bullets are confirmation that this is old data. In late 2026, people aren't delivering 30% more lines of code. If people were just producing 30% more lines, they'd still be reading every line. My point isn't that "more lines is good". I'm worried about a few too many lines of code that no one reads. My point is that this shows the data is old.
For the latter, I'm a bit annoyed by using mean, median would be more valuable, but I think it's safe to say that the engineering culture was already pretty bad. If your average pull request sits for 7 days, you evidently don't care about shipping anything, and agents aren't going to change that.
Mostly, meaning “more than half”, or perhaps “noticeably more than half”, both of which are literally true.
Do we speak different versions of English or something? I’m genuinely past being annoyed, I don’t actually know why you think what I’m saying is false.
I'm noticing AI catching many bugs in code that previously went unnoticed. Some of them are important, and it's great to catch them early.
The other 11 items in the list of 12 would probably never be hit in practice, but now that you know, you have to sort out how to handle each (even if that just means documenting, filing, or ignoring: you have to understand each of the disparate 12 issues first.)
Before it was all coding on its own, I asked a model for bugs in a new project, saved me the 15 minutes of new web project I waste every time trying to remember obscure first time setup things you dont touch for years in under a minute.
I suspect theres a lot of hours saved that are otherwise wasted by devs when some simple thing throws off their implementation completely. Basically we cant measure the saved developer hours by LLMs because it could be literally anything.
> We study the impacts of AI coding assistants and agents on software engineering work... across 718 firms.
> Our primary source of data is Jellyfish (“JF”), an analytics platform used by firms to understand the activity of their software engineering teams.
What about all the software being built by everyone else not using Jellyfish? What about the small companies that don't have the same operational bottlenecks and truly are building more than their larger competitors? What about the solo founders starting companies and building software which they never would have built without AI?
I don't think my team has delivered any more usable and needed features than before. If anything, we're running faster in the wrong direction, or things just gets never-ending scope creep or whatever.
I'm completely uninterested in the opinion of a video game journalist on what does and doesn't work well in writing software. The author knows about as much about writing code as he knows about performing heart surgery
Ars Technica in general has also been catering to their knee-jerk anti-AI audience. I used to find their comments section somewhat interesting but now it's the same old tired, outdated dunks on AI.
Agree. There's more code, but what's the trade-off? Fewer people on the team understand what's actually in there, so there are more and more bugs. Of course the agent will happily fix each one, and break three other things on the way, because nobody stopped to ask why the code looked the way it did.
Feels like we're building legacy code at a record speed.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.
The first method is far more difficult." - T. Hoare (1981)
Any study that wants to say anything about the impact of coding agents on whether more software is being created or not can't limit itself to what firms are doing that create a footprint of "work events — including GitHub coding activity, Jira issues, and Google Calendar events".
This is software that would not existed without coding agents: https://fluximp.org
Sure its a coding agent itself, but I would never have tried to create it manually as I would have expected it to be too much work to see it through as solo dev. There were no issues created anywhere, no calendar events, and there is very little github activity (I primarily used Mercurial on a private server; git is just for distribution).
I started it with Kilo Code and after a couple of months it was purely self-modified and has been since then. I was a bit worried that it wouldn't work on Linux and I'd either have to use another agent or use it running on my Windows box to modify the code on my Linux box to get it working, but it just worked so the streak of being self-modified has continue.
I want to make an indie game and I'm working towards that with the project I'm using to benchmark FluxImp. My first job was as game dev and I took a shot at indie game dev 20 years ago (failed; not fun). Its something I had long written off as too time consuming to even start, but with coding agents I no longer feel that's true.
All of the stories about non-software engineers making small software that is useful to them also seems unlikely to generate the kind of footprint that lends itself to being easily studied.
I don't see how AI coding agents are going to create 'more software' until the PO's and PM's of the world step back and let developer's communicate directly with the stakeholders. Or the PO's and PM's could cross-train on AI coding agents.
We are still being stressed to create all stories with such fidelity that 'any' developer can pick it up and implement it in the timeframe allowed for the size the story was given. When the story is written in that detail, we may as well copy it into an AI agent and have it implement it. But we can't do that, so we have people who have SME write up story's that sit in the backlog for 2 weeks until a developer can bring it in, so that they can spend a day implementing it (really 2 hours with 2 hours of meetings and 4 hours of video games) because the smallest pointing size we can give is 1 point and people aren't allowed to take on more than 8 points a sprint.
How can we expect things to get done faster when the team that reports work up to senior management put's such strict limitations on our throughput?
Yes, instead of reviewing 1,000 lines a day I now "review" 10,000 lines a day. My brain is not 10x faster, so either I spend 10x more time reviewing code, or I just opt to accept that AI is probably right based on the complexity of the problem. However I realize that if we just let the agents loose on the codebase, everything would speed up like 100x.
Invariably once certain AI models in certain domains reach "full automation" quality this will happen, until now the human eyeball bottleneck is holding a leash on millions of tireless superhuman coders who have just a few narrow but egregious contextual awareness issues.
Given what I've been able to accomplish with even just local models at home, and my experience at my day job, I imagine this has almost everything to do with the organizations themselves and less to do with using AI to build software.
Wrong thing to index on. Always bet on the slope, not on points in the slope. Things will only get better and better and factor in that downward slope thought while taking actions and pick ones that are favorable. The folks who focus on a point in time will be left behind, period.
The PR->merge->deploy SDLC is probably not what we want anymore.
Something like continuously delivery and validation through the whole stack so you can develop an incremental change, validate it as much as you can locally, then validate it as much as you can in the cloud using integration tests with synthetic, then with replayed production, and then finally allow it become production. If stuff starts breaking, you have your AI swam go in, identify the culprit, bounce it, and the train never stops.
There are many ways to say the same thing, that code throughput wasn't the only bottleneck, communication is by far the hardest, most brittle and time consuming part of software production.
We need to start becoming better writers, capable of expressing complex thoughts concisely and with technical precision. Software production has become a sibling field to linguistics, more than it already was.
I have a pet theory. LLMs are trained on many successful open source projects. Due to their success, the projects are used very widely and they cover extreme edge cases which have accumulated over many years and which are usually not covered, and don't need to be covered, in majority of regular software. So, when LLMs write code, they try to mimic the code they have been trained on and incorporate these edge cases into each fragment they write. It results in more code which is also harder to read and comprehend.
Aside from other comments mentioning flaws and cutoff of study, anyone in the industry or even working on hobby products can say this conclusion is wrong.
But the longer the codebase survives the more hair and cruft it accumulates. My once beautiful logical paths get carved up by compounding requirements and edge cases.
That's what the AI was trained on. The end result - the hairy ball of wax that noone can look at sideways.
I often think of that whenever AI comes up with what I consider to be an irrational but plausible edge case.
My experience is that agents are decent in coding a feature, solving a bug - doing specific well defined task. They are very bad at judging UI/UX and user flows through app. Like really, really bad. They can quickly put together vibe coded app but make it usable takes a lot of extra manual work.
Did anyone solve this problem? Any good approaches?
I confirm this diagnosis. The UIs are always clunky, disorganized, and kinda ugly. But they work well enough. I guess UIs are opinionated enough that we probably all customize them anyway. Maybe the agents know this so don't bother putting too much effort in? (Deep thought of the day: acquired)
I've spent most of my time working on tools to keep my agents on track, and keep code legible. Where I'm probably most strict in the tooling is on documentation, which makes the code review much more manageable.
They generate more software too. It's just organizations haven't adapted to remove the bottlenecks yet. I replaced so much software for my self and its better than all the alternatives I used to use. And there are tons of other people doing the same thing.
one thing i noticed, there are a lot of new things, but the software has not changed at all, we are building the same things again and again, but no new things.
Along these lines: agents will write mountains of super repetitive test code, seemingly uninterested in refactoring it (without explicit instruction) to reduce redundancies.
In any non-trivial integration test scenario, it's easy to end up several times the LoC in tests compared to actual shipped code.
Second problem: different models, agents, and harnesses seem to have different tendencies and they tend to avoid touching code that is not their own unless explicitly instructed. What I mean by this: it will happily duplicate some helper method from another test file rather than pulling it out and reusing it, seemingly avoiding coupling (good kind in this case since model level changes should be propagated atomically).
Tests are where I see agents just spew absolute mountains of code.
When people have gotten excited about having so many tests, I've asked them what their plan is when 500 of their 1000 test suddenly stop passing. I've always favored 50 tests I know are meaningful, where any failure is something to respond to, over 500 tests with a bunch of overlap and edge cases I'm not sure matter. But now people aren't even thinking about that, they're just letting the agents make all those decisions.
I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.
> I've asked them what their plan is when 500 of their 1000 test suddenly stop passing
That's not a terrible thing; would indicate that some shared behavior is broken and -- assuming well structured code -- hopefully easy to isolate the behavior.
Bigger problems to me are more practical: 1) large tests waste context both in and out, 2) large volume of tests waste CI cycles which are becoming the bottleneck.
IMO the solution to too many tests is to encode more things in the type system. If you can encode constraints like the length of an array in the type system you know it is correct if it compiles, and that is one less run-time test to run.
Agree, but this is not a pattern that is emergent from any coding agent I've used; it is only the result of explicit guidance to structure code in specific ways + relying on platform level analyzers.
But more specifically, I usually see the explosion of test code in integration tests where there is heavy setup of entity graphs.
Every senior I've ever worked with at some point embraced the idea of minimalism. Produce minimal changes with a minimal surface. Worked well at startups and especially well on large spaghetti legacy projects. Seeing all of a sudden now how suddenly number of kLocs, PRs, token maxxing, number of tasks in progress is again a metric with which non-engineers brag about on engineering conferences feels like the industry has reverted to some braindead pre-infant state. I guess all mistakes need to be relearned at every hype cycle.
This doesn't solve the issue with "too many tests", but one of my teams addressed this to some extent by writing "blueprints" for test scaffolding that specifically aims to surface the scenario under test.
Was initially hand written as part of our codebase before AI, just extracted it as MIT, OSS since there's no secret sauce here. But really cuts down on test verbosity and makes it much, much easier to read, review, and evaluate tests.
This headline being in the present tense, while reporting on a data from 2025, makes it basically dishonest. 2026 AI coding agents bear little resemblance to 2025 AI coding agents, and early-2026 coding agents are not like like-2026 coding agents.
Creating code is free now. Maintaining it is not. At least not at this point. The amount of slop we need to maintain is becoming very big. There is no other way than to use ai to deal with this. But that creates even more slop. Hopefully slop will be dealt with at some point.
Maybe but I’d argue the frontier corps will self destruct via green annd product management enough to make this impractical. So far here’s how I see that going
1) can’t easily autopilot due to security. Ganna keep on prompting you into running weird shell commands
2) it’ll launch your code on weird Anthropic or ChatGPT sites behind a paywall
3) between the two of them it’ll
Misalign the solution so it’s unusable
The link is a piece of crap wrapped in candy packaging, able to fool anyone who couldn't be bothered to unwrap the candy. DOA slop. It has nothing to do with either art or craft. This is actually a perfect example why agents' code cannot be trusted.
Way back in the day, everyone wrote assembly. Then things like C came around, which had to be compiled to assembly, and most people objected: "I don't want to be that far from what the CPU is doing!".
Each time we move up an abstraction layer, people object, saying that nothing will work right if we move up that high in the stack. Give it time. I'll bet in 5-10 years, almost no one will look at code anymore, except in special cases.
We still have people who write in assembly, we still have people who write in C, mostly for performance reasons. Because you still need to be close to the CPU to get that last bit of performance. But in most cases, it doesn't matter. It doesn't matter if the LLM makes bad code if it works and passes all the tests.
Granted, this is the first time we're going from deterministic to non-deterministic, but is it really? Large scale distributed systems are already non-deterministic. They create inputs and situations that programs aren't designed to handle all the time. What's a little more chaos? :)
This is from March 2026, but it does match what I've encountered professionally. Code review was always a significant bottleneck and is even more so with agentic coding. We've had discussions about leaning more on automated review, but it hasn't translated into any practical improvements so far. I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature. I'm sure that's happening, but it's likely invisible to most studies and surveys at the moment. If it proves successful, and such companies start shipping significantly faster, the rest of the industry will be forced to adopt.
I would reframe this as: Quality was always a significant bottleneck and is even more so with agentic coding.
Code review is where many of the design & code quality problems are observed/highlighted.
And test creation and maintenance. AI-generated tests will often embed existing bugs as testable patterns that fail when fixed or rely on empty asserts to increase coverage. A human author generally knows better. Coverage is increasingly becoming theater and you have runtime testing eating up CI resources and time with bloated, unnecessary, untargeted testslop that haven't been mutation tested or a human eye assessing if the business logic is actually represented.
It's really the smaller firms which are able to learn quick adaptations that will win out in the long-run of this thing. Legacy firms with institutionalized bureaucracies are relying on gobbling up competitors and creating vertical monopolies while pushing out the worst slop - it's why GH goes down, why Meta products are buggy as hell now, and everything feels like it's unpolished MVPs these days.
This. The ability to know that a vibe coded PR is all right to merge, is a function of both the thoughtfulness and simplicity of the underlying architecture, and the level of ownership that contributors have of the underlying products. Both benefit more nimble teams.
I agree. The latest models are getting quite good at validation, as long as they are clear on what the requirements are. Recently I got frustrated that while we have a huge test suite I noticed the LLM was hardly ever breaking a test with its changes (PRs only added tests, never changed old ones) which seemed suspicious.
I had it do a full audit. There were about 100k lines of tests, of which it determined about 60k were near useless. Many were overly mocked, some mocked to the point that they really weren't testing anything at all. It deleted those and wrote more expansive scenario tests to replace them, caught a lot of bugs on the way and fixed those too. Total time was about 2 hours.
I think humans need to review requirements, specifications, and test scenarios. And for some industries and uses they're still going to have to review every detail. But for a typical CRUD app, I think we're at a point where we can already set the process and quality bar higher than ever, in addition to moving faster.
I chuckle a little bit at this specification thing. We used to do this. We used to write big binders of specifications, and then we'd set about implementing them. We stopped doing it because it didn't really work. We'd learn things in the process of building which negated the value of the big specs we wrote. That's why we stopped doing it. This is where continuous improvement, CI/CD, tighter release cycles, etc. were all born.
But why? On reason is that implementation was a bottleneck. That made it expensive to change a system once it was entirely implemented. It was better to get the feedback sooner so you didn't waste time implementing the wrong thing.
The calculus on a lot of these old adages is changing.
I don't think this was only about implementation being the bottleneck. It was also because implementation forces you to think about the thing you're building at a very grannular level of detail, at which point you might discover flaws or opportunities that weren't obvious at the higher level.
And I find that, when it reasons through and encounters some of those, it just makes assumptions rather than asking the user.
Well, kind of. I think this is an interesting opinion on that: https://www.mountaingoatsoftware.com/agile/the-cost-of-chang...
Which could be summarised by a single line: "In other words, the primary cost of change increasingly becomes feedback delay, not development effort."
It is _still_ better to get the feedback sooner. Writing up some big specification, spending a bunch of tokens on implementation, and then shipping a massive PR to match the spec only to then find out it's wrong is a big waste of time. Nothing about writing grand plans in the face of unknown unknowns has changed.
The more things change, the more they stay the same.
Yet software quality has been in deep dive for a while, long before AI.
It is funny. At some point the code is the specification.
Remember when it was time to fire all the engineers because your business-analysts could just make UML diagrams?
Glad to see someone else notice this. I dread looking at or dealing with AI written unit tests because of the incredible amount of whiteboxing that goes on, though usually for the opposite reason because AI testing is almost entirely whitebox testing which fails if some internal details change
Funnily enough I've found that LLMs can be effective at mitigating many of the drawbacks of black-box testing and i've significantly expanded our usage of it over the past year
If before AI you couldn't deploy to production without a code review, what makes you think you will be able to with AI?
To be fair, when I was a kid we'd connect the editor to the server using an unencrypted ftp connection and modify php files in production.
This worked perfectly fine, and phpmyadmin made schema changes super easy.
These days we do tests, schema migrations, terraform, staging sites, traffic migration, monitoring, logging, revision control, etc.
... it's a lot of boring work, and for most CRUD apps, its entirely unnecessary :)
That said: I doubt we'll abandon these practices for things that matter.
Because while modifying php files in production can increase velocity, it can also get very "exciting" if you have a few co-workers on the same project :)
Yes, unfortunately a lot of requirements and specifications are now AI slop, thus developers stop reading them, and agent produce even more slop.
We need to take back writing requirements and specifications to humans.
Adopting high level BDD style testing of human written user stories (cucumber) is probably a good idea.
Just have AI write requirements then another AI write specs based on the requirements then another AI write tests based on the specs and another AI write the implementation based on the tests then another AI write the PR based on the implementation then another AI approve the PR and another AI deploy approved PRs to production!
Don't forget: let the user's AI use the product, find flaws, submit AI-generated bug-reports, which get turned into AI-generated tickets, etc.
And meanwhile, did any human find the whole process useful - no, not all...
Also, the entire thing is fully funded with AI tokens, which are paid for with money from investments into AI companies, made with AI pension funds.
Wait, no, that's ACTUAL HUMAN pensions, because plenty of AI companies are still "investment grade"
https://www.forbes.com/sites/brandonkochkodin/2026/10/12/ai-...
The Vile Offspring has to be born somehow.
FWIW, it's a great little "trick" to always ask for red-green tests. In my experience it's enough to ask for that and LLMs know what to do.
Same, although I use "TDD red phase"
https://simonwillison.net/guides/agentic-engineering-pattern...
I just don't understand the hype for this. "AI can do $x" quickly needs softening to "AI can try to do $x" which then becomes "humans need to verify $x has been done." If humans still have to oversee every part of the processes then what exactly is being automated!?
This would be like if we had our first factory producing cars but each car was like a die roll in terms of getting a car, a washing machine, a pile of scrap metal, or a baby. And people just asserting over and over that this was the future of making cars, no matter how much scrap metal keeps coming out.
IME, the biggest source of rework cost that happens at (or after) code review is when someone notices that the code, correctly or not, implements the wrong thing. For example, the feature, as implemented, doesn’t sufficiently solve the original business problem. Or it solves it in a bad way, like because of overengineering or taking on unnecessary technical debt. I haven’t really seen much evidence of AI code review catching either of these problems.
I’ve heard people complain that human code review doesn’t reliably catch them, either. And I have seen that with my own eyes. Just not universally. Teams that have a strong culture of shared ownership and collaborating on designs regularly catch these kinds of problems in code review. If they’re really good at it, they catch them even earlier. e.g., I know planning poker isn’t fashionable anymore, but when I’ve been on teams that take it seriously they tend to be pretty good at finding minimally expensive, correct designs pretty early in the process. Which makes both coding snd code review go faster because simpler designs are easier to manage. And design flaws that do make it to code review tend to be smaller and less expensive to fix. So those teams didn't tend to bottleneck on code review in the first place.
I think that this is the kind of thing that led to folks like DORA observing that AI coding is something of a multiplier for the effects of both good and bad software development practice. If you’ve got a cowboy coding culture, throwing AI into the mix will probably just lead to agentic cowboy coding.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
I actually dislike traditional code review. In part this is because most of our code review tools are very cumbersome to use (GitHub's and GitLab's web interfaces in particular).
But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
The problem is that purely agentic code review has orthogonal problems, and typically remains subpar when it comes to judging architecture and design considerations, compared to human reviewers. Both false positives and false negatives also remain a concern.
My best results so far have been with agent-assisted code review in a proper IDE, helping you to quickly construct a mental model and to analyze the underlying logic (in addition to having agents do a first pass for correctness issues).
> Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).
The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.
> I actually believe the opposite: exactly the fact that a reviewer had to develop a mental model independently is why code review is and remains valuable (and why people use different models when doing agentic reviews).
I don't think we actually disagree here. I also think it is valuable, but it's also what often makes it difficult.
> The biggest value of doing code reviews is in ensuring maintainability of the code (1. someone else can understand it, 2. it is readable and 3. can be evolved in the future) — everything else is downstream from that.
Agree very much with that (with the slight caveat that it is not always the biggest value, just an important one).
> exactly the fact that a reviewer had to develop a mental model
But do they? In my experience, at most workplaces it's simply a meat linter phase with the occasional useless comment about style (which should probably be handled by proper CI/CD).
Like if you don't even check out the code (generalized 'you'), then they can hardly build up that model. And even checked out, viewed in its proper context, Naur's famous paper comes to mind - the author who wrote it is the only one who has properly built the theory (which might have included alternative implementations that failed but have no visible artifact).
I'm not claiming that there are no places that go absolutely above and beyond with their review sessions, but I would argue it's really rare - and AIs are great at figuring out some of the context the PR description lacks.
People say one thing and do another. It’s tiring. Long before LLMs we had the joke that a PR with 10 lines changes got 10 comments (all nitpicking), a PR with 1000 lines got just one, LGTM. We’ve had several bugs where PR review would almost certainly never catch the issue, people are terrible at reviewing things thoroughly. Guess what can do that much much better?
What you describe sounds like little more than rubber stamping a PR target than reviewing it.. I see it too, but I don’t thing we should double down on it just because it’s common
> But also because humans aren't all that great at code review when it comes to correctness and security. Part of the problem is that you don't have access to the author's mental model for non-trivial changes and have to reconstruct it, otherwise your understanding of the program logic will be inadequate.
Code review is not really about correctness and security. We have testing and other verification process for these. Code review is about idea discussion mostly and correcting a few stuff that may have slipped through the author's initial review.
Your PR wants to change the software in some way. My review is to comment on how you've implemented that change, like did you miss some areas where you needed to change, does your description is actually what is happening (either a cursory reading or through testing), or are you doing some unnecessary change.
IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed. If it's an important area, a more careful reading should be done about the tests and relevant stuff.
But generally, if a review takes more than 10 minutes, there's some failure of communication going on (reviewer not apt to understand the change, big diffs that should have been discussed outside of the PR, obtuse code,...)
All good points. Maybe having worked a lot with concurrent and distributed system makes me distrustful of relying on unit/integration tests for correctness; I've seen too many concurrency issues pass CI validation. But unit tests also sometimes miss edge cases in more straightforward code, so I'm still a bit more cautious.
I also agree that merge requests should be short and understandable in 10 minutes, but IME this happy state often does not obtain in practice. It shouldn't, but it often does, whether it's mission creep, whether it's turning out that a change is more complex than originally imagined, or whether it is unavoidably complex and cannot be broken down further.
> IMO, the initial reading shouldn't take more than 10 minutes. More than that (so a big diff) it should have been discussed upfront, and the design agreed
Part of the code review process is to verify that the writer actually did build the agreed design too though.
We shouldn't give up on line-by-line review for serious applications.
My company's leadership is oscillating every few weeks on AI goals. Their most recent story is "we're expecting developer velocity to approximately double over the new 2-3 years as people get better at using AI". But of course that's completely non sequitur. People on my team spend about 10% of their time writing code and 90% doing design and analysis. So if AI doubles our code writing velocity, then it will make us 5% faster overall.
Just to back you up most surveys say only 10% to 20% of programmers time is coding.
Microsoft study of 484 developers, published in 2025 says 11%.
Another Microsoft study where programmers had telemetry installed showed 21%
https://gwern.net/doc/psychology/writing/2017-meyer.pdf
It's interesting. I remember around COVID, there were a bunch of startups building 3d printers for concrete housing. Then I looked up the actual cost of construction for a typical house, and only 14% of it was the most visible part (the balloon framing and physical house). Most of the cost is in foundations, plumbing, wiring, and things like insulation, flooring, cabinets, etc.
I guess at Microsoft coding is similar to building a house in that sense. Either that or all big orgs devolve into large amounts of time spent on politics.
90% of my time these days is trying to understand what the wall of claudeslop jira ticket is actually asking for. It's like archeology digging for the original customer issue to work out what is actually wanted underneath paragraphs of root cause load bearing seams.
I wish they taught Amdahl's law in business school. Or at the very least the simple concept of "bottleneck".
"The Goal" is a key business book for a reason, but leadership seems to not get the lesson. Or care?
I’m sure they learn it by other names.
The point is that while good to remember, and specifically Fred Brook’s “No Silver Bullet” article about it, there are second order effects.
If coding becomes 100x cheaper, what’s going to happen next? At least to some degree, design and planning was there since coding is expensive. (Not just coding, debugging also got squashed by AI and was an expensive part of the job)
This is why the original comment seems pretty spot on, organizations will learn how to skip more phases, and startups are probably a place where this will be happening faster. (Not clearly though, a lot of big tech are happy tokenmaxxing to try and not stay behind those startups when they come)
Brooks' point was more about accidental vs. essential complexity. You seem to be implying that design was only ever an artifact of "expensive" typing, but design exists to prevent systems from collapsing under different failure states.
If the cost of emitting syntax collapses without adequately resolving the essential complexity then you have not eliminated the design phase and you're just externalizing your failure modes onto downstream teams and systems.
I like the "9 women can't birth 1 baby in one month"
Coding was never the bottleneck...
And how many billions or trillions have we wasted on people who didn't understand that (or did, and are just busy generating hype).
Ultimately, this (llm) technology (as it is being developed/used), is not useful to us...
Yup we already knew this.
Google had many engineers working on a) failed projects b) people being paid to sit around.
There seems to be this strange belief on here that making something more accessible to everyone will always result in better outcomes. I don’t see this being true of software.
I’m very pro-tech. But nobody can point me to a product built by someone outside of the large tech firms, made by by llm’s that is a great value add to one’s life. Many of those were already created - they’re on the apple App Store for example.
I’m absolutely confident coding speed is irrelevant. A person who has a real vision + strategy and hand written code will beat the team with no vision, strategy and code blasted out by llm’s.
Sure llm’s could augment the former - but you’re missing the point!!
> A person who has a real vision + strategy and hand written code will beat the team with no vision, strategy and code blasted out by llm’s.
But you have to beat the team with vision + strategy who is also blasting out code with LLMs.
Certainly, the time it takes to figure out that an idea is bad had been shortened :)
There's a difference between vibe coded slop, vibe coded kinda slop built by an engineer, AI assisted engineers, and AI planning. I enjoy writing code by hand but it's quickly becoming apparent that LLMs are now a tool that one should be using in some capacity.
Nobody writes code in notepad anymore, even though you could and might even like all the black text.
Which is basically what this study is showing.
> So if AI doubles our code writing velocity, then it will make us 5% faster overall.
This tracks with the actual revenue impact of AI adoption:
https://www.bcg.com/publications/2026/the-formula-for-agenti...
Their very cheerful "high" revenue increase from AI adoption (that few companies in the index manage to achieve) sits at 4.5%. Most of those in the "value generating" minority are seeing sub-1% returns.
Which, of course, is non-zero, real, and all that, and matters when we're talking about a 4-5% increase of a big company's revenue, assuming the expenses are not catching up with that.
Which they absolutely are. The AI is subsidized by venture capital, and the subsidies are drying up (hence OpenAI upcoming funding round). None of the AI companies are profitable (much less self-sustaining), and AI prices are inevitably increasing.
Is your company striving to automate design and analysis, then?
> We shouldn't give up on line-by-line review for serious applications.
But how many changes are actually serious?
I can have an agent mass-produce security bug fixes, zip through our low-priority backlog, etc.
IMO - we need to get out of our own way.
Building is cheap now. Agents are also really good at following checklists and running tests.
So instead of having individual engineers reviewing code, we need to
* Leave human review for the most impactful changes
* Get better at defining the quality bar
* Continually codify human concerns into rules or systems that can be checked against
A lot of this was touched on a few months ago with Spec-driven development becoming a hot topic when agentic coding started.
We should continue that with spec-driven review, testing, etc.
> the most impactful changes
Quite a lot of our bugs in production have been in seemingly less impactful changes. How do you know what's impactful a priori?
> I can have an agent mass-produce security bug fixes
I can't figure out what you're trying to say here. Are you saying that this is acceptable? Because I very much don't think it is. You'll have twice the bugs after you're done, and they'll be inhuman ones that you have little chance of catching.
> But how many changes are actually serious?
I mean, "security bug fixes" are not on your list of serious changes? What is?
...and they are not using LLMs to help speed up design and analysis?
I'm no booster but this is most of what I currently spend tokens on so it's maybe not as absurd as it sounds. Maybe those gains are possible and without having to sacrifice serious code reviews too.
For me reading ai code feels a lot like that time I tried reading Joyce. I don’t understand but I’m sure it’s beautiful.
You don't give up on line-by-line review just because the human is getting better either, and senior engineers want other people to do line by line review on their own code. It's not a matter of how good or bad the person or program generating the code is.
As a senior engineer that is struggling to give up line-by-line reviews, I agree with your take.
I've spent literal decades where one of the most valuable practices has been code reviews, for many dimensions. It is one of the best ways to ensure quality production releases. It is one of the best ways to transfer skill to junior engineers. It increases tribal knowledge throughout the engineering org by exposing others to systems they may not be working in. The number of benefits to a traditional human-centric engineering team is hard to overstate.
But I'm starting to feel that instead of starting at one extreme (heavy handed reviews) and moving towards less review, it probably makes more sense to start at the other extreme (no review at all) and move towards a brand new practice tailored for the new agentic reality.
It is looking increasingly likely that we don't need to modify code review for the agentic era, we need some completely new practice that hasn't yet been invented.
It's really about hamstringing the AIs so they can't write crap. Writing code is not a problem anymore, so anyone with an idea can write a script to do what they want. But if they try to connect two ideas together, now they have a data modeling problem. This is where everything seems to fail with AI coding (largely because the original author had the idea piecemeal).
The problem I have is non technical people now started submitting PRs. They are like the brain damage module on GLaDOS and somehow make opus generate actual broken slop. So I still have to be on my toes to catch that, but they generate such volumes of broken PRs that it takes time away from me doing real work.
Perhaps spec review in the custom of code review - it can even be via PR - so there is a bottleneck before anything is generated, instead of after. Cheap-to-produce code maybe isn't worth understanding line-by-line anymore.
I think spec review, which is definitely valuable, solves a different problem. One question to ask is "are we building the right thing", which a spec review definitely helps to answer.
But another question is "did we build what we planned, and did we build it to a high quality". Code review partially answered that question. Adversarial review is the closest we have to that in the automated LLM world, but I feel we need something better.
In the sense that ai-engineering specs (ie an elaborate prompt with skills etc.) can be said to be compiled into code language x maybe.....the spec is the new code?
It's not code review that's slowing things down. It takes time to get the thing you want, and then you have to be reasonably sure that it's right anyway, or you risk shipping a piece of crap to production and tanking your own reputation. Even on teams that rubber stamp PRs I see people pushing drafts and waiting to merge until they feel comfortable with what they've vibed out.
I lean towards a hybrid review process where humans accept their limitations, focus what they're good on(higher level concepts and architecture of the proposed change), and let the LLMs do the rest.
Doing a line by line review as a human is silly now.
> shipping significantly faster
If you're right (and you probably are), we're in for a wild ride further down the quality death spiral.
The era of user telemetry and conversion tracking already decoupled product design from customer experience. No need to wait for customer's to articulate their experience or desires or for discerning analysts to amalgamate and ponder their opinions. The numbers tell you everything you need to know! Just throw 3000 pounds of spaghetti at the wall every month and wait to see what sticks.
But now you really were bottlenecked on engineering.
Only 3000 pounds of spaghetti per month? And throwing more bodies at it barely even makes a difference? Oof.
Of course the solution is to just flush pureed spaghetti-slop through the hose as fast as we can. Brilliant!
What could go wrong? The users are sure to love it. We'll know because they keep clicking. That's how they tell us the truth, even as every forum fills with louder and louder complaints!
Correct people need to go slower but this doesn’t benefit frontier labs. Go go go max max max is what they want.
"The users-only forum for complaints about authentication errors is completely empty, therefore everything is working perfectly."
> I think if there is a major change in code review culture, it's not going to happen first in established engineering teams. It'll happen in startups that adopt a no-human-review policy from jump and then maintain it as they mature.
I've been hearing this for more than a year. But if these startups are all moving at superhuman speed, a month instead of a year from idea to a shipping product, where are the products? Every time I ask that, people talk about the amazing velocity gains they experience for the thing they are about to ship. It never does, unless we count throwaway hobby projects at GitHub - by that metric, AI is adding a lot of software. And I guess never-downloaded slop-games on mobile app stores.
But if you actually go shopping for consumer or enterprise software that's camera-ready, the landscape is basically the same as in 2022.
Codex/CC and other OpenAI and Anthropic products like Dots, are examples of how quickly real products can be built today. But to me, they feel more like mediocre products bloated with feature slop. Nothing extraordinary or technologically innovative.
Pi.dev and Cursor's Tab completion are two of the most innovative and extraordinary products I know in AI era, and both were built by humans, at least in their "breakthrough" stages.
It's as if building something was not a bottleneck for startups. Go figure.
I want the boosters to address this head on - chime in?
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Why do you feel like that? Agents have got better but better how? Looking at the frontier model output, I’d say the same myopic approach that causes long term issues without frequent course correction is still there.
We're going to need to start putting the month in brackets after the article title, not just the year!
The study was updated in August 2026.
You have to classify each code review based on what it changes and decide how to handle it, agent review and/or human review. You can have an agent do that.
> I think senior engineers find it very difficult to give up on line-by-line review, no matter how much better the agents have gotten.
Our industry has spent the better part of the last two decades interviewing with brain teasers and leet code questions. Reading code is a skill that we just weren't assessing for, and lots of people aren't really good at it.
I would argue that code review is mostly theater, politics, bike shedding and pedantry.
There is a lot of work to get away from "coding" and "programing" and get down to the actual engineering that really needs to happen in the industry.
I think non-senior engineers don’t realize just how much of a burden is barely reviewed ai generated code.
I’ve seen a few projects that heavily rely on ai with not very good (though senior) devs driving it - and its a mess. Not the code. The resulting product. Code is actually sound as ai is pretty good now.
All because ai does code reviews and dump decisions go into prod
can you be more specific. If the code is not a mess, how is the resulting product a mess?
What could possibly go wrong?
My experience with prototyping games is that you need very strong human driven guidance and review in the beginning, then you can increasingly take your hands off the wheel.
If you take it off too early you end up with shit and will spend weeks fixing all these baked bad assumptions your project is built on top of. I spent weeks fixing some of this shit then the LLM revived a not insignificant portion of it in a later change because it got "confused" (its words) because some of the comments it previously made were left around.
> I think senior engineers find it very difficult to give up on line-by-line review
i am pretty senior. I hate code reviews so much. They are so boring, easily the most tedious part of my job. Its such a chore. Who is clinging to line-by-line review. Ppl dont really give that much about their employers.
You don’t sound very senior.
Code review isn’t a quality gate, it’s a way to maintain shared understanding of a codebase.
Throwing out code review is equivalent to throwing your hands up and saying you don’t need to understand anything about what your code is doing in its black box.
I don’t think anyone on this forum is half as smart as they think they are, especially in terms of systems engineering a black box solution. Myself included. Someone needs to understand what’s going on inside the Widget Factory until we have full Artificial General Intelligence.
>Throwing out code review is equivalent to throwing your hands up and saying you don’t need to understand anything about what your code is doing in its black box.
I think that is what the C-suite is currently wanting out of our engineers. Prompt, don't write, maybe read a bit, don't stop, ship, ship, ship.
Is the C-suite ready to sack the engineers and prompt Claude themselves with their business requirements? Why not cut out the middleman if that's how they now perceive engineers?
This will vary by company focus and size, but sometimes the C-suite just doesn't want to believe that software-engineering has important overlap with business-analysis or product-design.
So you’re not writing the code, and you’re not reviewing the code…
What exactly would you say it is you do around here?
I initially pivoted to fully automated code reviews, sometimes with a brief skim of things. I've been shifting back to mostly manual review. In the background I'll also run some agentic scans, and once I'm done I'll look to see what the agents had to say. And then I go back in by hand armed with that knowledge and verify it & come to my own conclusion.
Why? Because I started to run into way too many AI generated messes in codebases. The agents are really good at some things and complete ass at others.
> Who is clinging to line-by-line review.
It really depends on the context. A react button background color - meh - the critical part of the backend where a new 300ms delay will cost the company millions then yeah.
> the critical part of the backend where a new 300ms delay will cost the company millions then yeah.
If it is so critical, why don't we have tests and benchmarks for it?
Oh right, because "Greg the super-coder" never wrote any of this down and it lives in his head so he can be the arbiter of change.
It was toxic before agentic coding, it's still toxic now.
Document your systems. Add tests and benchmarks for what is important.
We've always known this, we have a dozen or more textbooks written about it. But we always gave it a pass because "Greg the super-coder" was producing more value than he was hurting.
That's no longer the case. Your best coder is now Opus or Astra
Good thing you reviewed the PR so you could catch the gap in Greg’s documentation and force him to write it before approving his PR!
> Your best coder is now Opus or Astra
No, that's our _fastest_ coder, which is a different thing than _best_
> he critical part of the backend where a new 300ms delay will cost the company millions then yeah.
If a performance regression like this isn't caught by an automated testing suite, you have made a horrible mistake.
Reviewing the PR gives you a chance to identify this gap in the performance regression suite.
The agent could also do that part while its still in draft. If the rigorous tests already exist of course.
> new 300ms delay will cost the company millions
these might be 1% typically
Absolutely. And also the idea that this is somehow the last bastion against introducing bugs? Maybe run a test
> Maybe run a test
One of those things made of code that we should neither be writing nor reviewing? Shouldn't I just assume that the LLM did that or whatever?
I hate code reviews, but I owe it to my coworkers that I know will read my PRs to review my own generated slop line by line. Most of these agents are still overly verbose and I'm in a situation where no, not every single edge case is relevant enough to warrant the additional loc or complexity introduced.
Both Claud and Codex would run circles around 80% of the developers I’ve worked with through my 20 year long career. This whole code review thing is a hold over that probably won’t last six months.
That and they do not necessarily generate better code. They tend to generate code that is repetitive. It misses opportunities for extraction and refactoring, the result is bloated and hard to read code (because those blocks are there literally instead of with a usable name) and they tend to over-comment where it doesn't add but is 'safe' because it is obvious (effectively re-writing code as comments) and fail to document where documentation would be essential. There also typically isn't any real plan to what is written, as long as 'the tests pass' it is all good.
You can fix a lot of this stuff by shepherding the whole process but it can get quite tedious. I'm impressed by the number of people that are letting Jesus (one of these days an AI provider is going to name their AI that way :) ) take the wheel and I wonder how long they will manage to support the output. So far the score isn't very good. I've gotten good mileage out of these tools by letting them loose on very narrowly scoped pieces of code that are written in isolation. As soon as the codebase gets larger than 100K LOC or so managing the whole thing becomes an exercise in putting up barriers to avoid spillover from a tiny issue into a much larger one.
I agree, I do like them for creating believable interaction prototypes though.
I agree too, the problem arrives later when the prototype must scale
This is not unlike hardware: making prototypes, assuming the principles are valid to begin with is relatively easy. Making a prototype that is a good precursor for a production run that's not going to result in a large number of rejects is stupendously hard. You can't fake your way through that. This is one of the reason many Kickstarters are doomed to failure, it's one thing to have a device working on your table in the lab, a completely different thing to produce that device reliably over and over again with negligible QA failures.
That's why some of these projects really impress me (for instance: that phased array antenna on a Raspberry Pi features here a short while ago: https://scalerf.com/). The amount of skill required to pull something like that off is off the scale, and they make it look so easy.
Well, imagine racing car manufacturers competing on making better, faster engines. At some point engines are so strong that other parts become the limiting factors, such as tires for traction. Now someone invents superfuel, but cars are still only marginally faster because the additional engine power didn't change the bottom line much.
That's what's happening in software. We knew that coding is only a fraction of the overall work, and even completely automating it away won't increase productivity by an order of magnitude.
Yes, the job is changing, but only parts of it. High-level engineering best practices still apply even without the coding part. And there is no free lunch.
IMO it's a lot more like a new more powerful engine is now available. Sure, faster drivers/teams will run faster lap times, no doubt, and it'd be a bad idea to not use it. But, less mature teams/drivers aren't going to catch those faster teams by just slapping the new engine in and going. They are still largely held back by the myriad of other things that go into being fast on track consistently IE (driver + datalog review, and just good ol seat time), vehicle [other than engine] setup, aero tuning, pitting practice (esp fueling for endurance stuff), etc etc. All those same things that were making them slower than the fastest teams before. And even still, if you give only the slower teams the new engine, they probably still won't be competitive with the faster teams and, quite honestly, the not great drivers will probably just make the same mistakes they were making before that slowed them down, except faster.
But, even if the new engine is the golden egg, now what? You're still competing against every other team that has the golden egg... And it's right back to the driver mod, vehicle setup, etc.
That's not to say that the slower teams have any choice but to run the engine too to not be even less competitive. It's just to say that having the engine isn't gonna fix the underlying problem, no matter how much of the team you lay off to fuel the new, more power, engine.
"The overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used" - Amdahl's Law
The hardest part always has been and always will be - a simplified vision and strategy.
Everything else is a case of project decomposition, planning and scaling up - aka this is when having code written faster is great.
The SV approach of the past decade has been about experimentation, A/B testing and data science - it has its limits.
Except the problem with this is an engine designer doesnt want to pivot to designing a stronger driveshaft, or strengthening the gears of a gearbox, or designing more efficient brakes. They're an engine builder.
Same goes for software. 20 years ago I got into software development for the love and passion of solving problems and writing code. Now I've been forced to pivot to a glorified BA/Products Manager. It sucks.
The flip side of that is as someone who got into software development 20 years ago because I love to design and make products... Well this is pretty great. I feel for my colleagues, though. And it's not like the "coding problem" is solved, much to the dismay of the hype machine.
Bide your time. When the eventual hangover arrives your talents will be in short supply. AI providers are essentially making a long bet: that they will be able to improve faster than the messes that they make are growing. But that can't keep going forever. AI is an accelerator: it will accelerate you if you were careful before towards something better and it will accelerate you towards something much worse if you were not. Just like any other power tool.
I've recently talked about how it makes almost zero sense to ever delegate anything again at work. 3-6 months ago, that was not a thing. 12 months ago, that would have gotten me laughed out of the room. Wherever we are headed, we are not even close to the steady state
> Wherever we are headed, we are not even close to the steady state
That I agree with, but the messes are already catching up with some early adopters and I've been watching a couple of these up close with the roughly same level of interest that you'd bring to a road accident if you were part of the technical investigation team. It is more than a little interesting to me how the difference between the initial conditions seems to be a fairly good predictor of the outcome. Great teams are doing quite well with this new tech, others, not so much. And great teams were rare to begin with.
Exactly. For many of us, designing and writing code has always been the fun part; reviewing and whatnot, the parts that are still left for us meatbags, was more of a chore.
Or we could just recognize these llms crank out poorly architected bug infested slop. The coding work of these llms is just as garbage as all the other bland, flat, wtf writing they crank out with prose. Even worse because there's no logical mind behind the garbage. With bad human written code there was at least one human there thinking about it.
this is not my experience, in oct2026. Claude does for me what I ask better than any human I've delegated to in my career
As is standard for academic papers, this is an attempt to describe the state of the world as of when they did the research. But then it takes time to write it up and get it out, after which newspapers report it as current events.
In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.
The study was updated twice with the last time on August 2026.
The data is still mostly 2025 and the 2026 data is mostly from the first few months, and most production deployments (at least in, say, the Fortune 500) lag the frontier by at least several more months (or much more). I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
> I suspect it's just not really possible for academic research at its conventional pace to offer an up-to-date perspective on this topic right now.
If we weren't still in a massive hype-and-investment bubble, the burden of proof would be on the LLM manufacturers to demonstrate what deficiencies that studies like this have revealed have been fixed in their latest products.
I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
> I mention this because I've been hearing "Yeah, well, that opinion/study is completely invalid because it used a six month old tool. Today's release of SOMNOS 6.9 (Pinnacle Snoozytime) has totally changed the game!" every six months for what feels like the past five years now.
Given my own conclusion that it's not possible for conventional academic research to accurately snapshot industrial adoption of this technology, I would expect this pattern to continue for a while until either academia becomes much faster or technology progress slows down (or both). This just isn't yet a situation where you can rely on published papers telling you what to think -- you still need to mix in a bit of your own instinct and experience and extrapolation to get an accurate up-to-date read, even if there's some prediction error as a result.
Do you have any references for the models deployed in any of these organizations?
> I suspect it's just not really possible for academic research
But we were told everything changed in fall-winter of 2025 and that when we got “real capable” agents/models. Whatever happened to that?
I have my own experience to go off of, but this can also be inferred easily. There are many large corporations still working on "digital transformation" from 10+ years ago. There's no way such companies have fully onboarded and integrated frontier AI agents throughout their workforces with high adoption. The paper itself reports low adoption even after conditioning on access.
Everything did change after late 2025, but there are tiers of usefulness. Late 2025/early 2026 was the first time agents became clearly useful for personal projects. That's distinct from immediately having economically valuable end-to-end autonomous workflows in teams of median talent, which many would suggest is the threshold we've just recently crossed.
If you want to refute a study which presents numbers and statistics, you have to provide it in the form of numbers and statistics.
I don't, actually. I can simply point out that its data is out of date, and then the numbers themselves are called into question without other numbers to compare them to. And that's what I did: I shared two arguments in two comments that converge on the shared conclusion that the paper's data is not up to date (or, more precisely, cannot be up to date).
> I can simply point out that its data is out of date
You can, but it doesn't make it correct or, evidently, yield a logical argument. Do you notice that your logic is circular? The date doesn't matter if the passage of time has not affected the outcome and that's basically what I questioned, which you refuted without any evidence. So the question still stands. Are there any comparable stats you have access to that change the picture?
They've updated it for the agentic code era, and have found basically the same thing. Code can be created faster but that has moved the bottleneck to review.
By that logic, it is not ever possible to do research on LLM-assisted work.
> The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5.
What we have today is not meaningfully better than what we had a year ago, so that isn't relevant.
Today I can write what I'm looking for, as if I'm writing a design doc, and the models can do the rest. A year ago it was faster for me to just write the code myself. Models today are enormously better than what we had in 2025.
Lol
A dimension not much discussed here is that code review is not only about quality but accountability and security. An automated review is going to help to ensure the code behaves as intended, but it is not going to ensure the intent was correct. Employees inserting back doors, bypassing tests, "cheating" on design decisions to get the work done faster etc are not going to be identified if the person driving the change tells the AI it is the intended outcome.
For this reason alone, I don't expect code review to go away entirely in most contexts where the code "matters" - in the sense that there is substantial liability to it being mis-specified at the implementation level, not just technically wrong in implementation. I do expect it to evolve a lot and become dramatically AI-assisted.
The main protection ultimately is a human being willing to put their reputation on the line by saying "I understand this change and I approve it". That, inherently, means review of some kind.
Has someone presented an alternative to code Review , i mean, instead of just not doing it? Formal verification sounds expensive, asking an llm to Review another llm otput does not seem bulletproof. Is there another approach out there ? Can it exist? I've been thinking on this latelly but i can't find a solution
Maybe I'm missing something, but it seems like the obvious solution is to have human accountability on the API design and tests, not on line-by-line source code. If the human writes an appropriately-scoped API and the code passes human-reviewed tests, then the implementation details don't matter so much (depending, of course, on project-specific requirements around performance and security).
> not on line-by-line source code
I think there's a bit of a false premise around "line by line".
I don't need to syntactically parse every line to scan it for the the type of risks that matter here. You can look high level and see the pattern and shape of the code, the type of things it is doing. If it is doing stupid things ("Wait is it making a new connection to the database inside the loop here?"). It does take an experienced eye, but with that experienced eye it is a very rapid process.
Man, I'm probably going to expose my own inexperience here, but like, did anybody actually expect code reviews to catch and prevent bugs on a line by line basis? Like ever?
I never have operated this way. Code reviews were a basic sanity check, a chance to implement best practices, and a chance to teach these things to juniors.
If I was concerned about something breaking, I'd duscuss test coverage.
Was I doing it wrong this whole time??
I review code line by line and build a mental model of the software as I review it. I find plenty of important bugs that way. PRs are an opportunity to learn (both committer and reviewer), communicate, and improve the quality of code a team ships. Sadly, many people decline that opportunity for the sake of short-term velocity.
I’ve caught bugs in code reviews before, but I would agree that it’s probably not the most crucial thing you should strive to do with code reviews.
That said, I think I’ve done a tremendous amount of PREVENTING bugs from being introduced via code review. Caught all kinds of weird code patterns, duplication, wrong levels of abstraction, etc. Essentially, the kinds of places that bugs like to hide and grow in.
I still think that’s valuable, and it’s the kind of thing that models generally aren’t great at, to this day. It generally relies on having a large amount of context about the project and about what’s being solved, so if you wanted to do this via automated review, you maybe COULD, but it would probably get pretty expensive, especially on larger projects.
At another level, code reviews also give you the time to really think about what is being built and why. I've often come across looming architectural or strategic failures simply by having to stare at a bunch of code and wonder why it works a certain way. Same goes for writing the code by hand too, though it's harder to see past your own biases and ideas sometimes. I find code reviews deeply valuable for myself and my team.
> Essentially, the kinds of places that bugs like to hide and grow in.
I like an ecological-framing, especially as systems get more complex.
Yes, I could hunt cockroaches all day, but it would be even better to have caught how the new wall was missing a metal screen over a low ventilation opening.
No. You are on to something here. The utopia I see programmers lamenting the loss of, the one where human code review was perfect and never let any bugs slip through, never existed. I see people on this board and others complaining all the time about LLM generated spaghetti code and bug-ridden codebases. Well, I've been working those code bases for 20 years. We didn't agents to make them. LLMs are a new tool, but the problems that come with them are nothing new.
Yes I have often caught stuff like "not using DB transactions in the right way" or "this is not going to be race safe" for instance from juniors just by glancing on a PR.
I find recent LLMs don't do such mistakes though, in my experience. And also LLMs will find such mistakes if asked to review, if not perfect then more predictable than humans.
It was more important for giant C or fortran code bases. You will legit catch pointer errors or memory mgmt stuff, going line by line. Once Sun busted in with Java is when I really saw the shift to skim reviews. So much stuff is caught by compilers now, or SAST, that using dev time to find code smells is mostly a waste of their time. And it's not like the old way caught everything, it's just that the stuff we take for granted being caught by compilers or Sonarqube now, just used to get shipped to prod and cause abends and production support calls at 3am.
The best code review process I've done are more like oral exams ie: the person walks through the changes and we listen as they describe it.
It's a good way to catch thinking errors, which are the insidious errors.
Code review is like a dam holding back a huge body of water. Nobody does it perfectly and almost all software turns to shit given enough time.
I think a code review should be just that. A review of the optics of code itself not the solution or anything like that. If that is important it should be done earlier in the process before coding. Even more important when coding with AI. The reviewer is not a tester. I've heard horrible stories from friends who mainly works with fixing AI generated code that junior devs pushes at a rapid pace.
It doesn't match my experience. We are producing more software, and being a lot more ambitious about the things we develop. The trick is mostly to do with getting good guidance in place and then you can spend a lot more time iterating the product itself. I spend a lot more time play testing the product now and trying different things / refining. However, there is a lot of previous work that's been done to create a huge amount of technical guidance on how to do things. Lots of things on Specification, Design/Architecture/Coding, Performance, Testing, QA testing, etc. I think getting all this guidance/tooling/process in place is critical to producing software.
I think it highly depends on what field you're in. Personal projects that I find useful in my everyday life? Churning them out like crazy. Enterprise software? It has slowed our productivity down to a halt--part of this is due to people not knowing how to use these tools, and the other part of it is the collective AI psychosis of our leadership (I work at a Fortune 500 software company).
Do you think its enterprise software? or the team? I work on a product ( https://www.mywildeye.com/ ) but we do a lot of back end enterprisey type stuff to manage our logistics / manufacturing as well. We are a small dev team with a lot of latitude to make decisions. For us, AI allows a dev to deal with a lot more things. We also churn out internal tools like crazy. Our Hardware, Firmware, Accounts, and Support people are all creating tools. Our CEO (former developer) is also getting back to making things, we recently released a free ios app https://epicfields.app/ so farmers can monitor water demand using free satellite data.
Very Large Enterprise is very hard to change, or rewrite, especially old systems. Our company mostly sells contracts to large corporations / government institutions (1500+ employees), with an insane amount of transactions, functionality, etc. Even with the latest models you cannot automate everything, the world is far too complex, and there is quite a bit of pushback when anything changes.
In order to keep some of these extremely large contracts (using Samsung as an example here) for example, we had to revert certain changes that were global but now we have a "Samsung version" because we want to maintain a large presence in Asia and Samsung leaving would be a huge blow.
At this scale, code isn't even the bottleneck, it's customer wants, politics, etc.
Note that they define “more software” by “more tickets closed.” Make of that what you will. They argue that this is a good measure because the scope of tickets did not increase. Worth considering whether you believe this captures what they claim it does. Are the extra tests you write thanks to coding agents represented in the tickets for your work?
“But all that extra code doesn’t translate directly into improved software output on the firm level. On the contrary, the resolution rate for Issues and Epics (i.e. wholesale software features) tracked by tools like Jira did not change in a statistically significant way after AI tools were introduced (the researchers also found no “compositional shift” in the size or complexity of those Jira-tracked issues across the AI introduction).”
The bottleneck is, and has been, the whole team evolving the product. Having a clear, actionable, vision and being good at biting off the right next step and adjusting for the step after that is always the issue. Before AI teams took the wrong step size in the wrong direction or were so wrapped up in process that they couldn't take a step or so distracted by 'things' that they never even took steps. After AI the same is still true. Our process sucks in this industry. We love to make process because we love to make machines but the processes we love to make rarely help us and that is a fundamental problem that AI isn't changing.
> the processes we love to make rarely help us
I'm going to make a potentially bold claim here. This depends _heavily_ on what the threshold of "rarely" is, and whether we actually _love_ to make these processes.
In the case of people before process, as stated in the Agile Manifesto, the process is begat _from_ the people in many cases. That doesn't mean it shouldn't be questioned, but when the chimney sweeps that built the process are saying "I don't want to go back to things that are similar to the way that gave me mesothelioma", I don't think it's fair to write them off as the minority.
I’d strongly disagree with one aspect. Homemade personal use software has Definately skyrocketed. Projects I never thought I’d even have the time for are now possible.
Yes, I am literally inundated by cool new tools I've built that I never would have before.
I'm literally building a black-box client + server emulator for one of my favorite old MMOs, remotely, from a coffe shop in SF as I type this, on my laptop at home via Claude remote-control.
Insane.
Well, Claude is building it. You are building it in the same sense that my manager was building the software I worked on at my last job.
Organizations that use Jira have not reaped efficiency gains from AI, news at 11
Where’s your data?
Based on a sample size of one or…?
I read this paper several days ago, when my director shared it. Several things jumped out at me:
- the data is mostly from 2025 and earlier
- the baseline mean lines of code per month was 5017, the estimated increase was 1494 lines per worker-month
- the baseline is 5.34 PRs per month, and increased by 1.22
- the baseline mean for code review is 7.03 days
The second and third bullets are confirmation that this is old data. In late 2026, people aren't delivering 30% more lines of code. If people were just producing 30% more lines, they'd still be reading every line. My point isn't that "more lines is good". I'm worried about a few too many lines of code that no one reads. My point is that this shows the data is old.
For the latter, I'm a bit annoyed by using mean, median would be more valuable, but I think it's safe to say that the engineering culture was already pretty bad. If your average pull request sits for 7 days, you evidently don't care about shipping anything, and agents aren't going to change that.
You read the paper, saw that the stated collection dates ranged from January 2021 to March 2026, and are claiming the authors are lying?
If you’d asked reasonably, I might have noted the slight correction (I misremembered that data collection ended January 2026, not March).
But since you’re such a charmer, why did you read the word “mostly” then lie about me lying?
It’s unambiguously true that most of the data (including that on coding agents) is from 2025 and earlier.
Mostly 2025 and before means mostly 2025 and before…
Mostly, meaning “more than half”, or perhaps “noticeably more than half”, both of which are literally true.
Do we speak different versions of English or something? I’m genuinely past being annoyed, I don’t actually know why you think what I’m saying is false.
I'm noticing AI catching many bugs in code that previously went unnoticed. Some of them are important, and it's great to catch them early.
The other 11 items in the list of 12 would probably never be hit in practice, but now that you know, you have to sort out how to handle each (even if that just means documenting, filing, or ignoring: you have to understand each of the disparate 12 issues first.)
Before it was all coding on its own, I asked a model for bugs in a new project, saved me the 15 minutes of new web project I waste every time trying to remember obscure first time setup things you dont touch for years in under a minute.
I suspect theres a lot of hours saved that are otherwise wasted by devs when some simple thing throws off their implementation completely. Basically we cant measure the saved developer hours by LLMs because it could be literally anything.
This stuff is still 90% garbage hype. See also:
https://www.youtube.com/watch?v=NnV_cWeoo5Q&list=LL&index=3
From the study:
> We study the impacts of AI coding assistants and agents on software engineering work... across 718 firms.
> Our primary source of data is Jellyfish (“JF”), an analytics platform used by firms to understand the activity of their software engineering teams.
What about all the software being built by everyone else not using Jellyfish? What about the small companies that don't have the same operational bottlenecks and truly are building more than their larger competitors? What about the solo founders starting companies and building software which they never would have built without AI?
I don't think my team has delivered any more usable and needed features than before. If anything, we're running faster in the wrong direction, or things just gets never-ending scope creep or whatever.
I'm completely uninterested in the opinion of a video game journalist on what does and doesn't work well in writing software. The author knows about as much about writing code as he knows about performing heart surgery
You are free to not comment in the future!
Ars Technica in general has also been catering to their knee-jerk anti-AI audience. I used to find their comments section somewhat interesting but now it's the same old tired, outdated dunks on AI.
Agree. There's more code, but what's the trade-off? Fewer people on the team understand what's actually in there, so there are more and more bugs. Of course the agent will happily fix each one, and break three other things on the way, because nobody stopped to ask why the code looked the way it did.
Feels like we're building legacy code at a record speed.
"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - T. Hoare (1981)
Any study that wants to say anything about the impact of coding agents on whether more software is being created or not can't limit itself to what firms are doing that create a footprint of "work events — including GitHub coding activity, Jira issues, and Google Calendar events".
This is software that would not existed without coding agents: https://fluximp.org
Sure its a coding agent itself, but I would never have tried to create it manually as I would have expected it to be too much work to see it through as solo dev. There were no issues created anywhere, no calendar events, and there is very little github activity (I primarily used Mercurial on a private server; git is just for distribution).
I started it with Kilo Code and after a couple of months it was purely self-modified and has been since then. I was a bit worried that it wouldn't work on Linux and I'd either have to use another agent or use it running on my Windows box to modify the code on my Linux box to get it working, but it just worked so the streak of being self-modified has continue.
I want to make an indie game and I'm working towards that with the project I'm using to benchmark FluxImp. My first job was as game dev and I took a shot at indie game dev 20 years ago (failed; not fun). Its something I had long written off as too time consuming to even start, but with coding agents I no longer feel that's true.
All of the stories about non-software engineers making small software that is useful to them also seems unlikely to generate the kind of footprint that lends itself to being easily studied.
I don't see how AI coding agents are going to create 'more software' until the PO's and PM's of the world step back and let developer's communicate directly with the stakeholders. Or the PO's and PM's could cross-train on AI coding agents.
We are still being stressed to create all stories with such fidelity that 'any' developer can pick it up and implement it in the timeframe allowed for the size the story was given. When the story is written in that detail, we may as well copy it into an AI agent and have it implement it. But we can't do that, so we have people who have SME write up story's that sit in the backlog for 2 weeks until a developer can bring it in, so that they can spend a day implementing it (really 2 hours with 2 hours of meetings and 4 hours of video games) because the smallest pointing size we can give is 1 point and people aren't allowed to take on more than 8 points a sprint.
How can we expect things to get done faster when the team that reports work up to senior management put's such strict limitations on our throughput?
Listen man. Let me give you a tip.
Don't take it seriously.
Efficiency is not your job, until you can 1) solve it, 2) present the solved and implemented problem to senior leadership in one fell swoop.
Corporate America is a giant slush fund and this is how it keeps going.
Be grateful!
What kind of professional would spend 4 hours of their time per day playing video games?
The true professionals spend those 4 hours making snarky Hacker News comments.
I would be more concerned with playing video games half the day than whether the PMs are slowing you down by figuring out the requirements
Yes, instead of reviewing 1,000 lines a day I now "review" 10,000 lines a day. My brain is not 10x faster, so either I spend 10x more time reviewing code, or I just opt to accept that AI is probably right based on the complexity of the problem. However I realize that if we just let the agents loose on the codebase, everything would speed up like 100x.
Invariably once certain AI models in certain domains reach "full automation" quality this will happen, until now the human eyeball bottleneck is holding a leash on millions of tireless superhuman coders who have just a few narrow but egregious contextual awareness issues.
A rocketship pointed in the wrong direction is not going to get to mars. Steering is probably the next evolution of code review
Given what I've been able to accomplish with even just local models at home, and my experience at my day job, I imagine this has almost everything to do with the organizations themselves and less to do with using AI to build software.
Wrong thing to index on. Always bet on the slope, not on points in the slope. Things will only get better and better and factor in that downward slope thought while taking actions and pick ones that are favorable. The folks who focus on a point in time will be left behind, period.
The PR->merge->deploy SDLC is probably not what we want anymore.
Something like continuously delivery and validation through the whole stack so you can develop an incremental change, validate it as much as you can locally, then validate it as much as you can in the cloud using integration tests with synthetic, then with replayed production, and then finally allow it become production. If stuff starts breaking, you have your AI swam go in, identify the culprit, bounce it, and the train never stops.
My conclusion from all this is that Douglas Adams joined the writing team some time ago and has been responsible for the whole LLM arc.
There are many ways to say the same thing, that code throughput wasn't the only bottleneck, communication is by far the hardest, most brittle and time consuming part of software production.
We need to start becoming better writers, capable of expressing complex thoughts concisely and with technical precision. Software production has become a sibling field to linguistics, more than it already was.
I have a pet theory. LLMs are trained on many successful open source projects. Due to their success, the projects are used very widely and they cover extreme edge cases which have accumulated over many years and which are usually not covered, and don't need to be covered, in majority of regular software. So, when LLMs write code, they try to mimic the code they have been trained on and incorporate these edge cases into each fragment they write. It results in more code which is also harder to read and comprehend.
Aside from other comments mentioning flaws and cutoff of study, anyone in the industry or even working on hobby products can say this conclusion is wrong.
LLMs generate more code yes.
I generate more beautiful code, yes.
But the longer the codebase survives the more hair and cruft it accumulates. My once beautiful logical paths get carved up by compounding requirements and edge cases.
That's what the AI was trained on. The end result - the hairy ball of wax that noone can look at sideways.
I often think of that whenever AI comes up with what I consider to be an irrational but plausible edge case.
My experience is that agents are decent in coding a feature, solving a bug - doing specific well defined task. They are very bad at judging UI/UX and user flows through app. Like really, really bad. They can quickly put together vibe coded app but make it usable takes a lot of extra manual work.
Did anyone solve this problem? Any good approaches?
I confirm this diagnosis. The UIs are always clunky, disorganized, and kinda ugly. But they work well enough. I guess UIs are opinionated enough that we probably all customize them anyway. Maybe the agents know this so don't bother putting too much effort in? (Deep thought of the day: acquired)
I've spent most of my time working on tools to keep my agents on track, and keep code legible. Where I'm probably most strict in the tooling is on documentation, which makes the code review much more manageable.
They generate more software too. It's just organizations haven't adapted to remove the bottlenecks yet. I replaced so much software for my self and its better than all the alternatives I used to use. And there are tons of other people doing the same thing.
Maybe for some orgs, I know theres orgs out there where there is no such review only QA and unit testing.
We got rid of end to end QA as head of engineering decided to "shift left" on automation.
one thing i noticed, there are a lot of new things, but the software has not changed at all, we are building the same things again and again, but no new things.
The title should include “IF you’re still tied to a traditional human-reviewed dev process” at the end.
The reality is that new apps and new companies are being built without the bottleneck, constantly now.
Look at the shipping speed of the frontier labs themselves. It’s all the evidence you need to know that this isn’t true.
> Look at the shipping speed of the frontier labs themselves. It’s all the evidence you need to know that this isn’t true.
I certainly look at how their software sucks, and wonder why their fans think the rest of us should adopt such a bad approach to making software.
The paper is from January, so the data is probably from 2025. Very unlikely anything in the paper is relevant now (if it ever was).
The paper was updated in August this year.
Along these lines: agents will write mountains of super repetitive test code, seemingly uninterested in refactoring it (without explicit instruction) to reduce redundancies.
In any non-trivial integration test scenario, it's easy to end up several times the LoC in tests compared to actual shipped code.
Second problem: different models, agents, and harnesses seem to have different tendencies and they tend to avoid touching code that is not their own unless explicitly instructed. What I mean by this: it will happily duplicate some helper method from another test file rather than pulling it out and reusing it, seemingly avoiding coupling (good kind in this case since model level changes should be propagated atomically).
Tests are where I see agents just spew absolute mountains of code.
When people have gotten excited about having so many tests, I've asked them what their plan is when 500 of their 1000 test suddenly stop passing. I've always favored 50 tests I know are meaningful, where any failure is something to respond to, over 500 tests with a bunch of overlap and edge cases I'm not sure matter. But now people aren't even thinking about that, they're just letting the agents make all those decisions.
I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.
Bigger problems to me are more practical: 1) large tests waste context both in and out, 2) large volume of tests waste CI cycles which are becoming the bottleneck.
IMO the solution to too many tests is to encode more things in the type system. If you can encode constraints like the length of an array in the type system you know it is correct if it compiles, and that is one less run-time test to run.
Agree, but this is not a pattern that is emergent from any coding agent I've used; it is only the result of explicit guidance to structure code in specific ways + relying on platform level analyzers.
But more specifically, I usually see the explosion of test code in integration tests where there is heavy setup of entity graphs.
Back up to the point where all the tests were passing, of course? Then debug.
or git bisect until you get the exact commit that introduced it
Every senior I've ever worked with at some point embraced the idea of minimalism. Produce minimal changes with a minimal surface. Worked well at startups and especially well on large spaghetti legacy projects. Seeing all of a sudden now how suddenly number of kLocs, PRs, token maxxing, number of tasks in progress is again a metric with which non-engineers brag about on engineering conferences feels like the industry has reverted to some braindead pre-infant state. I guess all mistakes need to be relearned at every hype cycle.
This doesn't solve the issue with "too many tests", but one of my teams addressed this to some extent by writing "blueprints" for test scaffolding that specifically aims to surface the scenario under test.
https://github.com/zeeq-ai/zeeq-biome
Was initially hand written as part of our codebase before AI, just extracted it as MIT, OSS since there's no secret sauce here. But really cuts down on test verbosity and makes it much, much easier to read, review, and evaluate tests.
How come? For example, "Brandon Thomas released Artcraft, a free, open-source suite of apps ... alternatives to the entire Adobe Creative Suite".
I checked and it looks and works great!
That's pretty sick.
This headline being in the present tense, while reporting on a data from 2025, makes it basically dishonest. 2026 AI coding agents bear little resemblance to 2025 AI coding agents, and early-2026 coding agents are not like like-2026 coding agents.
I take both approaches. just /yolo or /goal or measure twice and cut once.
yolo works when I need to focus on higher priority, and make sure things just work (though not "great").
For the latter, I use it for more like spec-driven fashion with HITL. Basically everytime it ffinishes a task, I review and then move on.
Both appraoches work. just make sure your yolo slop isn't bad and passes stuff.
LoC don't mean jack shit.
Creating code is free now. Maintaining it is not. At least not at this point. The amount of slop we need to maintain is becoming very big. There is no other way than to use ai to deal with this. But that creates even more slop. Hopefully slop will be dealt with at some point.
Maybe but I’d argue the frontier corps will self destruct via green annd product management enough to make this impractical. So far here’s how I see that going 1) can’t easily autopilot due to security. Ganna keep on prompting you into running weird shell commands 2) it’ll launch your code on weird Anthropic or ChatGPT sites behind a paywall 3) between the two of them it’ll Misalign the solution so it’s unusable
Counterpoint: https://getartcraft.com
I'd say that's "more software."
The paradigm shift is trusting the agents' code, and not reviewing it, the same way that an engineering manager does not read every line of every PR.
The link is a piece of crap wrapped in candy packaging, able to fool anyone who couldn't be bothered to unwrap the candy. DOA slop. It has nothing to do with either art or craft. This is actually a perfect example why agents' code cannot be trusted.
Have you downloaded and used any of the apps?
If you reject any software that used AI in its development as slop, I have bad news about the rest of your life.
Way back in the day, everyone wrote assembly. Then things like C came around, which had to be compiled to assembly, and most people objected: "I don't want to be that far from what the CPU is doing!".
Each time we move up an abstraction layer, people object, saying that nothing will work right if we move up that high in the stack. Give it time. I'll bet in 5-10 years, almost no one will look at code anymore, except in special cases.
We still have people who write in assembly, we still have people who write in C, mostly for performance reasons. Because you still need to be close to the CPU to get that last bit of performance. But in most cases, it doesn't matter. It doesn't matter if the LLM makes bad code if it works and passes all the tests.
Granted, this is the first time we're going from deterministic to non-deterministic, but is it really? Large scale distributed systems are already non-deterministic. They create inputs and situations that programs aren't designed to handle all the time. What's a little more chaos? :)