While the ultimate goal of interactive theorem proving is to prove
theorems, it can really help to test them first. Testing theorems,
specifically using property-based testing, helps users identify
incorrect definitions and theorem statements before they waste
time on a proof that could never succeed. Unfortunately, the testing
infrastructure provided by modern theorem provers has yet to reach its full
potential. Even QuickChick, the state-of-the-art property-based testing framework for Rocq,
which offers random generation for data satisfying inductively defined relations,
often requires substantial effort and expertise to be used effectively.
This is in part because this effectiveness is heavily sensitive
to both the order that hypotheses appear within a theorem, and to the order that
inductive constraints appear within the inductive relations involved.
In this paper, we present a novel strategy for testing theorems that is
highly effective, fully automatic, and robust to equivalent formulations of
theorem and definition statements. To do so, we characterize the exponentially large space of possible
QuickChick-style properties and generators as solutions to a constrained
scheduling problem. To find the best property or generator in this space, we estimate effectiveness by introducing a notion of ``density'' for inductive relations, which approximates the tendency for a generator to succeed given arbitrary inputs.
We implement our algorithm on top of the QuickChick framework for
Rocq and evaluate it in a number of case studies from the literature,
demonstrating that our push-button automation is on par with and in some cases even more effective at finding bugs than
expertly handcrafted tests.
Wed 7 OctDisplayed time zone: Pacific Time (US & Canada) change
13:30 - 15:00 | Property-Based Testing and Test QualityOOPSLA at East Hall 1 Chair(s): Jonathan Bell Northeastern University | ||
13:30 18mTalk | Block Tests OOPSLA Kevin Guan Cornell University, Pengyue Jiang Cornell University, Milos Gligoric University of Texas at Austin, Owolabi Legunsen Cornell University DOI | ||
13:48 18mTalk | Detecting Flaky Tests by Controlling Nondeterministic API Behavior OOPSLA Hengchen Yuan University of Texas at Austin, Jiefang Lin University of Texas at Austin, August Shi University of Texas at Austin DOI | ||
14:06 18mTalk | Random Testing via Runtime Abstract Interpretation OOPSLA DOI | ||
14:24 18mTalk | Testing Theorems, Fully Automatically OOPSLA Segev Elazar Mittelman University of Maryland, Harrison Goldstein University at Buffalo, Leonidas Lampropoulos University of Maryland DOI | ||
14:42 18mTalk | Fail Faster: Staging and Fast Randomness for High-Performance PBT OOPSLA Cynthia Richey University of Pennsylvania, Joseph W. Cutler University of Pennsylvania, Harrison Goldstein University at Buffalo, Benjamin C. Pierce University of Pennsylvania DOI | ||