Metamorphic Testing for Infrastructure-as-Code Engines
Infrastructure-as-Code (IaC) engines, such as Terraform, OpenTofu, and Pulumi, automate the provisioning and management of cloud resources. They parse IaC specifications and orchestrate the required actions, making them the backbone of modern clouds, and critical to the reliability of both the underlying infrastructure and the software that depends on it. Despite this importance, this class of systems has received little attention: prior work largely targets the correctness of IaC programs rather than the IaC engines themselves. Existing test suites rely on manually written oracles and struggle to expose faults that manifest across multiple executions, leaving a significant reliability gap.
We present EMIaC, a metamorphic testing framework for IaC engines. EMIaC defines metamorphic relations as graph-based transformations of IaC programs and checks invariants across executions of the original and transformed programs. A central novelty is our use of e-graphs in software testing, as both a test-input generator and an equivalence oracle. E-graphs compactly represent program equivalences, enabling the systematic generation of large spaces of equivalent IaC programs. To ground these relations, we analyze 43,593 real-world Terraform programs and show that IaC dependency graphs are typically small and sparse, making e-graphs a natural fit.
Evaluating EMIaC on Pulumi, Terraform, and OpenTofu, we show that it complements existing test suites by exercising engine-critical code paths and covering 98 previously untested statements in Terraform and 1,313 in Pulumi. EMIaC also uncovers previously unknown issues in all three test suites, improving their adequacy. Three test cases have been merged into Terraform's main branch, and Pulumi has merged a specification fix.
Tue 6 OctDisplayed time zone: Pacific Time (US & Canada) change
10:30 - 12:00 | Fuzzing and Test GenerationOOPSLA at East Hall 1 Chair(s): Michael Pradel CISPA Helmholtz Center for Information Security | ||
10:30 18mTalk | Hunting CUDA Bugs at Scale with cuFuzz OOPSLA Link to publication DOI Pre-print Media Attached | ||
10:48 18mTalk | RandSet: Randomized Corpus Reduction for Fuzzing Seed Scheduling OOPSLA Yuchong Xie Hong Kong University of Science and Technology, Kaikai Zhang Hong Kong University of Science and Technology, Yu Liu Fudan University, Rundong Yang Fudan University, Ping Chen Fudan University, Shuai Wang Hong Kong University of Science and Technology, Dongdong She Hong Kong University of Science and Technology DOI | ||
11:06 18mTalk | Metamorphic Testing for Infrastructure-as-Code Engines OOPSLA David Spielmann University of St. Gallen, George Zakhour University of St. Gallen, Dominik Arnold University of Zurich, Matteo Biagiola USI Lugano; University of St. Gallen, Roland Meier armasuisse, Guido Salvaneschi University of St. Gallen Link to publication DOI Pre-print | ||
11:24 18mTalk | Prunario: Testing Autonomous Driving Systems by Pruning Likely Redundant Scenarios OOPSLA DOI | ||
11:42 18mTalk | OBsmith: LLM-Powered JavaScript Obfuscator Testing OOPSLA Shan Jiang University of Texas at Austin, Chenguang Zhu University of Texas at Austin, Sarfraz Khurshid University of Texas at Austin DOI File Attached | ||