Something I've had good progress with using local models and simple open-source harnesses is to repeat, in a new context, simple verification prompts.
I'd run the following 5-10 times with one model, then again with a 2nd model.
"Verify the correctness and completeness of all security configs/rules in SETUP.md. Consider if anything is missing, and if anything is not needed. Do not modify any files; only write potential findings to report.txt"
"Verify all findings and claims in report.txt."
Replace "SETUP.md" with whatever you're working on.
It's both terrifying and incredible watching what the models get correct and what they get completely wrong.
However, after enough runs they tend to settle on a state they claim does not need any more edits. And that result is generally useful with much fewer errors/hallucinations compared to a single run.
Don't you think "consider if anything is missing" leads them into adding something with sycophancy RL training and "if anything is not needed" making it remove something?
Or does "verify all claims in report" counteract that?
Working in satellite simulation, where we end up with complex harnesses (cabling) and where I don't really know the nitty-gritty of how the sausage is made by the engineers creating them, I was really curious for a moment; but alas, it's an AI thing.
FWIW, my quick impression is that takes reasonable concepts and tries to formalize them into a framework; I can see potential benefits, I've certainly asked in a claude code session for it to have a look at pipeline so and so and figure out the issue, but I'm not really convinced by this at first glance either. Both setup-cost and token cost seem like downsides.
A good harness should not only make agents more capable at completing tasks, but also make their outputs much easier to review.
For example:
A good harness constrains the action surface, context, and task boundaries.
An agent’s failure isn’t always due to “writing incorrect code” — it can also result from “doing things it wasn’t supposed to do.”
Tests and lints can verify part of the correctness, but they often fail to validate task scope.
A well-designed harness should shift the review process from “reading the entire diff” to “verifying whether the changes stay within the defined task boundaries.”
I fully agree with the idea: above a model capability threshold, the power comes from the harness far more than the model. Engineers can get tremendous power from learning how to do CICD and automation. If you view the models and agentic code pipelines as a natural evolution of this, you see the benefit.
I did a quick look at the content, and it seems verbose and AI generated but conceptually OK. I learn by tinkering, not a good fit for me, but if you learn by reading, maybe this is for you.
My view is that human time is more precious than computer time. If something can be automated, then automate it. I don't lint code by hand, I get the linter to do it. Similarly, LLMs expand the list of things that computers can do. That's what you get from the harness, however you learn to do it.
I would love a resource to get more into the details of bend radius and vibrational modes in harnesses, specifically as they're used on different types of vehicles. Marine wiring endures very different motion than road-vehicle wiring, for instance.
I wonder about failure modes and fault identification. I've heard stories of things like screws or brackets wearing through the insulation and causing an intermittent fault that defies diagnosis. One of those things I think about when I am procrastinating.
My first impression, based upon the title was that this was about Wire Harness Engineering, which is a thing. Apparently it's not about that at all, and is (surprise!) AI related.
Something I've had good progress with using local models and simple open-source harnesses is to repeat, in a new context, simple verification prompts.
I'd run the following 5-10 times with one model, then again with a 2nd model.
"Verify the correctness and completeness of all security configs/rules in SETUP.md. Consider if anything is missing, and if anything is not needed. Do not modify any files; only write potential findings to report.txt"
"Verify all findings and claims in report.txt."
Replace "SETUP.md" with whatever you're working on.
It's both terrifying and incredible watching what the models get correct and what they get completely wrong.
However, after enough runs they tend to settle on a state they claim does not need any more edits. And that result is generally useful with much fewer errors/hallucinations compared to a single run.
I have also had positive experience with doing this multiple times via multiple model families, and then to recursively have the fixes reviewed too.
It's called review-anvil and does find significant amount of problems that might pop up:
https://github.com/mrshu/agent-skills/#review-anvil
Don't you think "consider if anything is missing" leads them into adding something with sycophancy RL training and "if anything is not needed" making it remove something?
Or does "verify all claims in report" counteract that?
It can indeed cause some models to try too hard to come up stuff, but the next verification prompt does counteract it.
E.g. some findings first classified as moderate priority often get reclassified as low priority even if the finding itself is correct.
The exact phrasing doesn't seem to matter as much as keeping the prompts short, simple and to the point.
However some models seem to do a bit better when adding ", if any" to prompts such as "List potential improvements".
Working in satellite simulation, where we end up with complex harnesses (cabling) and where I don't really know the nitty-gritty of how the sausage is made by the engineers creating them, I was really curious for a moment; but alas, it's an AI thing.
FWIW, my quick impression is that takes reasonable concepts and tries to formalize them into a framework; I can see potential benefits, I've certainly asked in a claude code session for it to have a look at pipeline so and so and figure out the issue, but I'm not really convinced by this at first glance either. Both setup-cost and token cost seem like downsides.
ai generated by this guy: https://www.aispacewalk.cn/about
Why not just look through the actual Claude code codebase and use your own AI to deconstruct it
https://github.com/codeaashu/claude-code
Couldn't tolerate the content; it's too structured; I'm open to something more chaotic.
The content is clearly AI-generated, which is really ironic.
A good harness should not only make agents more capable at completing tasks, but also make their outputs much easier to review. For example:
A good harness constrains the action surface, context, and task boundaries. An agent’s failure isn’t always due to “writing incorrect code” — it can also result from “doing things it wasn’t supposed to do.” Tests and lints can verify part of the correctness, but they often fail to validate task scope. A well-designed harness should shift the review process from “reading the entire diff” to “verifying whether the changes stay within the defined task boundaries.”
Someone know of any other recommended resource similar to this one?
I fully agree with the idea: above a model capability threshold, the power comes from the harness far more than the model. Engineers can get tremendous power from learning how to do CICD and automation. If you view the models and agentic code pipelines as a natural evolution of this, you see the benefit.
I did a quick look at the content, and it seems verbose and AI generated but conceptually OK. I learn by tinkering, not a good fit for me, but if you learn by reading, maybe this is for you.
My view is that human time is more precious than computer time. If something can be automated, then automate it. I don't lint code by hand, I get the linter to do it. Similarly, LLMs expand the list of things that computers can do. That's what you get from the harness, however you learn to do it.
I would love a resource to get more into the details of bend radius and vibrational modes in harnesses, specifically as they're used on different types of vehicles. Marine wiring endures very different motion than road-vehicle wiring, for instance.
I wonder about failure modes and fault identification. I've heard stories of things like screws or brackets wearing through the insulation and causing an intermittent fault that defies diagnosis. One of those things I think about when I am procrastinating.
This is AI slop written primarily with the intent to promote the authors, not to actually educate. So sick of this kind of content.
[dead]
My first impression, based upon the title was that this was about Wire Harness Engineering, which is a thing. Apparently it's not about that at all, and is (surprise!) AI related.
slop
[flagged]
[flagged]
[dead]
AI slop aside, looks like this website is built with Vitpress [0] which looks interesting.
[0] https://vitepress.dev/