Jev and friends need to show their results on coin flips and dice rolls. They really seem to miss the mark here, undermining any trust I have in their "confidence" for other decisions
I assume this will end up like counting 'r' chars, so we'll need some private evaluation cases that are obvious, but where the models faceplant
That's true. I would definitely smoke test your use case before any serious work. And if it faceplants, you can always fine tune to your taste. If the false negative/positive is cheap, I think it's a mighty tool.
Jev and friends need to show their results on coin flips and dice rolls. They really seem to miss the mark here, undermining any trust I have in their "confidence" for other decisions
I assume this will end up like counting 'r' chars, so we'll need some private evaluation cases that are obvious, but where the models faceplant
That's true. I would definitely smoke test your use case before any serious work. And if it faceplants, you can always fine tune to your taste. If the false negative/positive is cheap, I think it's a mighty tool.