Improving AI-generated tests using mutation testing(blog.senko.net)

1 pointby senko5 hours ago1 comment

derrak4 hours ago
First, I have some advice if you are open to it. Apply a spell/grammar checker to your post before sharing it with other people. Your post has several typos and many people will stop reading after the first or second typo. “The author didn’t care enough to proofread, why should I bother reading.”
> Remember, this is important: do not look at the tests. If you let them into your context
This is a bad idea. You are trusting the LLM’s ability to follow instructions. Worse, depending on your harness the LLM might not even be able to follow these instructions. The harness may indiscriminately place code into the context in a way that is uncontrolled by the LLM.
A better idea is to modify your harness so that certain files are excluded from the context.
> I asked another AI to carefully review the tests and identify those that don't make sense.
Test validation is an entire area of research and I’m yet to be convinced that this is a task for LLMs.
- senko3 hours ago
  > Apply a spell/grammar checker to your post before sharing it with other people.
  Hey, at least you know it isn't LLM generated! :) Thank you - I usually do, here I obviously didn't. Appreciate the callout.
  >> do not look at the tests. If you let them into your context
  > This is a bad idea. You are trusting the LLM’s ability to follow instructions
  Oh I'm bot, I manually checked what it was doing. I might set up the ignore explicitly if I turn this into a repeatable procedure.