Yup, this thing is basically useless. I just did /goal optimize the cuda kernel, and ai agent just proposed and tested a bunch of ideas by itself, and it profiled them using nsight ncu etc. by itself, which lead to one order of magnitude faster kernel.
respect for shipping something this technical solo, this is the kind of project that usually needs a team to even validate correctness. how are you handling regression testing across kernel variants, feels like the hardest part of an agentic optimizer isn't finding a faster kernel, it's proving the faster one didn't quietly break something
That is the funny part actually; we can either provide a reference kernel + input cases for correctness check. In this case at first it will use test harness to run reference kernel with reference inputs ad save the outputs as ground truth.
Or we can let AI to create a very basic reference implementation and input cases :D for my own experiments I used second one.
Can someone explain why this isn't better accomplished through a single prompt to Claude code or codex? I don't think I understand.
Yup, this thing is basically useless. I just did /goal optimize the cuda kernel, and ai agent just proposed and tested a bunch of ideas by itself, and it profiled them using nsight ncu etc. by itself, which lead to one order of magnitude faster kernel.
Very cool. Did you also try using the karpathy autoresearch? How do you think this compares?
honestly I know it exists but I never used it so can't compare
respect for shipping something this technical solo, this is the kind of project that usually needs a team to even validate correctness. how are you handling regression testing across kernel variants, feels like the hardest part of an agentic optimizer isn't finding a faster kernel, it's proving the faster one didn't quietly break something
That is the funny part actually; we can either provide a reference kernel + input cases for correctness check. In this case at first it will use test harness to run reference kernel with reference inputs ad save the outputs as ground truth. Or we can let AI to create a very basic reference implementation and input cases :D for my own experiments I used second one.
How are you checking that the AI is not just gaming your correctness tests? It's very easy to write incorrect synchronization for example.