Metamorphic Testing for Infrastructure-as-Code Engines
This program is tentative and subject to change.
Infrastructure-as-Code (IaC) engines, such as Terraform, OpenTofu, and Pulumi, automate the provisioning and management of cloud resources. They parse IaC specifications and orchestrate the required actions, making them the backbone of modern clouds, and critical to the reliability of both the underlying infrastructure and the software that depends on it. Despite this importance, this class of systems has received little attention: prior work largely targets the correctness of IaC programs rather than the IaC engines themselves. Existing test suites rely on manually written oracles and struggle to expose faults that manifest across multiple executions, leaving a significant reliability gap.
We present EMIaC, a metamorphic testing framework for IaC engines. EMIaC defines metamorphic relations as graph-based transformations of IaC programs and checks invariants across executions of the original and transformed programs. A central novelty is our use of \emph{e-graphs} in software testing, as both a test-input generator and an equivalence oracle. E-graphs compactly represent program equivalences, enabling the systematic generation of large spaces of equivalent IaC programs. To ground these relations, we analyze 43,593 real-world Terraform programs and show that IaC dependency graphs are typically small and sparse, making e-graphs a natural fit.
Evaluating EMIaC on Pulumi, Terraform, and OpenTofu, we show that it complements existing test suites by exercising engine-critical code paths and covering 98 previously untested statements in Terraform and 1,313 in Pulumi. EMIaC also uncovers previously unknown issues in all three test suites, improving their adequacy. Three test cases have been merged into Terraform’s main branch, and Pulumi has merged a specification fix.
This program is tentative and subject to change.
Tue 6 OctDisplayed time zone: Pacific Time (US & Canada) change
10:30 - 12:00 | Fuzzing and Test GenerationOOPSLA at East Hall 1 Chair(s): Michael Pradel CISPA Helmholtz Center for Information Security | ||
10:30 18mTalk | Hunting CUDA Bugs at Scale with cuFuzz OOPSLA Link to publication DOI Pre-print Media Attached | ||
10:48 18mTalk | RandSet: Randomized Corpus Reduction for Fuzzing Seed Scheduling OOPSLA Yuchong Xie Hong Kong University of Science and Technology, Kaikai Zhang Hong Kong University of Science and Technology, Yu Liu Fudan University, Rundong Yang Fudan University, Ping Chen Fudan University, Shuai Wang Hong Kong University of Science and Technology, Dongdong She HKUST (The Hong Kong University of Science and Technology) | ||
11:06 18mTalk | Metamorphic Testing for Infrastructure-as-Code Engines OOPSLA David Spielmann University of St. Gallen, George Zakhour University of St. Gallen, Dominik Arnold University of Zurich, Matteo Biagiola University of St. Gallen and Università della Svizzera italiana, Roland Meier armasuisse, Guido Salvaneschi University of St. Gallen Pre-print | ||
11:24 18mTalk | Prunario: Testing Autonomous Driving Systems by Pruning Likely Redundant Scenarios OOPSLA | ||
11:42 18mTalk | OBsmith: LLM-powered JavaScript Obfuscator Testing OOPSLA | ||