Curated Semantic Mutants: Multi-Purpose Artifacts for Grading and Hinting Student Test Suites
Mutation testing can evaluate a student’s test suite, but the hard part is \emph{curation}: deciding which faults are worth a student’s points and grading them fairly. Token-level operators (e.g., swapping $==$ for $!=$) are easily applied but cannot say how many points to assign for each mutant, and a surviving mutant does not tell a student what \emph{semantic gap} their tests leave. We address both with one instructor-curated artifact: a semantic mutant that grades the test suite, paired with a short phrase that seeds an LLM-generated hint the student can request. Deployed across two semesters of an introductory software-engineering course ($N=1{,}893$ submissions), the curated mutant gives every student automated per-submission feedback on what their test suite fails to detect—something the prior practice of reactively pinning discussion threads to firefight common issues never did. In the deployed term, $46%$ of students requested an LLM expansion of the seed phrase, and within that cohort requesting one predicted improvement on the next submission (about $1.6$ times as often as students who did not request one), at roughly three cents a hint. The same submissions show why an operator engine cannot grade these suites on its own. It emits several times as many mutants as the curated set, a fifth to a half on lines no student test reaches, with no way to flag equivalents or to say which faults are worth a student’s points—the selection curation supplies.
Sun 4 OctDisplayed time zone: Pacific Time (US & Canada) change
15:30 - 17:00 | Observing, Exploring and Assessing Student WorkSPLASH-E at Grand Ballroom Salons A+B Chair(s): Pierre Donat-Bouillud Czech Technical University | ||
15:30 30mTalk | Curated Semantic Mutants: Multi-Purpose Artifacts for Grading and Hinting Student Test Suites SPLASH-E DOI | ||
16:00 30mTalk | Observations on Tracing Program Design through Natural Language DialoguePresentation only SPLASH-E Kouta Kumamoto Institute of Science Tokyo, Youyou Cong Institute of Science Tokyo, Hidehiko Masuhara Institute of Science Tokyo File Attached | ||
16:30 30mTalk | Proof Interfaces for Exploratory MathematicsPresentation only SPLASH-E Nishant Kheterpal University of Michigan, Matthew Keenan University of Michigan, Cyrus Omar University of Michigan, Jean-Baptiste Jeannin University of Michigan File Attached | ||