Curated Semantic Mutants: Multi-Purpose Artifacts for Grading and Hinting Student Test Suites
Mutation testing can evaluate a student’s test suite, but the hard part is \emph{curation}: deciding which faults are worth a student’s points and grading them fairly. Token-level operators (e.g., swapping $==$ for $!=$) are easily applied but cannot say how many points to assign for each mutant, and a surviving mutant does not tell a student what \emph{semantic gap} their tests leave. We address both with one instructor-curated artifact: a semantic mutant that grades the test suite, paired with a short phrase that seeds an LLM-generated hint the student can request. Deployed across two semesters of an introductory software-engineering course ($N=1{,}893$ submissions), the curated mutant gives every student automated per-submission feedback on what their test suite fails to detect—something the prior practice of reactively pinning discussion threads to firefight common issues never did. In the deployed term, $46%$ of students requested an LLM expansion of the seed phrase, and within that cohort requesting one predicted improvement on the next submission (about $1.6$ times as often as students who did not request one), at roughly three cents a hint. The same submissions show why an operator engine cannot grade these suites on its own. It emits several times as many mutants as the curated set, a fifth to a half on lines no student test reaches, with no way to flag equivalents or to say which faults are worth a student’s points—the selection curation supplies.