This program is tentative and subject to change.
Inline tests validate single program statements and were shown to find single-statement bugs or kill mutants that unit tests miss. Inline tests complement unit tests by enabling testing at a finer program granularity level than methods. So, inline tests can more easily find faults in target statements that unit tests do not reach, or where errors do not propagate to unit tests’ oracles. But, the limitation to single statements means inline tests cannot validate data or control flow across code fragments—sequences of multiple statements in a method. We motivate the need for testing arbitrary fragments and propose block tests, which generalize inline tests and validate fragments. To motivate, we discuss six software testing needs (e.g., due to increased usage of lambdas in imperative code) for which unit tests are too coarse grained and for which inline tests are too fine grained. We also give real-world examples where block tests could have found bugs that unit tests missed. To bridge this gap, we propose syntax and semantics for specifying inputs, outputs, test oracles, and scope of block tests in Java. We also implement a block-test development kit (BDK) for writing and running block tests. We evaluate block tests and BDK in two ways. First, we write 1,012 block tests for 346 fragments in 146 open-source projects. Developer written unit tests do not cover 58.67% of these fragments, and automated unit-test generation does not reach 45.95% of them after 30.81 CPU days. But, each block test takes 2.23 minutes to write and 0.934 seconds to run on average. Second, we use mutation testing to evaluate the fault-finding effectiveness of block tests. In fragments that unit tests cover, block tests kill many mutants that survived unit tests. These results provide initial but strong evidence on block tests’ feasibility and utility. We outline an agenda for future research on block testing.
This program is tentative and subject to change.
Wed 7 OctDisplayed time zone: Pacific Time (US & Canada) change
13:30 - 15:00 | Property-Based Testing and Test QualityOOPSLA at East Hall 1 Chair(s): Jonathan Bell Northeastern University | ||
13:30 18mTalk | Block Tests OOPSLA Kevin Guan Cornell University, Pengyue Jiang Cornell University, Milos Gligoric The University of Texas at Austin, Owolabi Legunsen Cornell University | ||
13:48 18mTalk | Detecting Flaky Tests by Controlling Nondeterministic API Behavior OOPSLA Hengchen Yuan The University of Texas at Austin, Jiefang Lin The University of Texas at Austin, August Shi The University of Texas at Austin | ||
14:06 18mTalk | Random Testing via Runtime Abstract Interpretation OOPSLA | ||
14:24 18mTalk | Testing Theorems, Fully Automatically OOPSLA Segev Elazar Mittelman University of Maryland, College Park, Harrison Goldstein University at Buffalo, SUNY, Leonidas Lampropoulos University of Maryland at College Park | ||
14:42 18mTalk | Fail Faster: Staging and Fast Randomness for High-Performance PBT OOPSLA Cynthia Richey University of Pennsylvania, Joseph W. Cutler Jane Street, Harrison Goldstein University at Buffalo, SUNY, Benjamin C. Pierce University of Pennsylvania | ||