Call for Artifacts

Authors of Accepted and Minor Revision papers at OOPSLA are invited to submit an artifact that supports the claims in their papers. Artifacts are the software, mechanized proofs, test suites, benchmarks, or anything else that bolsters the claims of the paper–except paper proofs–which the AEC lacks the time and expertise to carefully review.

Authors of papers may submit an artifact once their revisions reach Accepted or Minor Revision status. Per the ACM guidelines for Artifact Review and Badging, OOPSLA provides three types of validation for artifacts as badges that appear on the first page of the paper:

1. 

   Artifact Available: This badge is for artifacts that are published in a permanent location (with a DOI). Artifacts do not need to be evaluated to receive this badge.

2. Artifact Evaluated: This badge is for artifacts that have been approved by the OOPSLA Artifact Evaluation Committee (AEC). There are two levels for the badge; papers can receive at most one of them:

   1. 

      Functional, for artifacts which are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation

   2. 

      Reusable, for artifacts that are Functional and facilitate reuse through careful documentation and clear organization.

3. 

   Results Validated: Results Reproduced, for artifacts that can be used to replicate the main scientific claims of the paper.

See Badges for more details on the badges that may be awarded.

Artifact evaluation is optional, but highly encouraged. Artifact evaluation is a service provided by the community to help authors of accepted papers extend the reach of their work and encourage future researchers to build on it. Our goal is to work with the authors to identify issues early on, improve the artifact, and get it accepted wherever possible. Typically, almost all artifacts are accepted for at least one badge.

See Submission guidelines for how to prepare and submit your artifact.

For any questions or concerns, please contact the chairs Eric Atkinson, Satyajit Gokhale, and Simon Oddershede Gregersen.

Call Artifact Evaluation Comittee Nominations

We seek nominations for members of our community to serve on the OOPSLA 2026 Artifact Evaluation Committee. We encourage nominations and self-nominations across all levels of seniority, including graduate students and postdocs. Please submit nominations via this online form by December 15, 2025:

Artifact Available

Artifacts that are publicly available in an archival location can earn the Available badge from the publisher. This badge is not controlled by the AEC, which has some important consequences:

• Artifacts that were not submitted for evaluation can be Available,
• Artifacts that did not pass evaluation can be Available, and
• Artifacts that passed evaluation need not be Available to accommodate rare situations in which the authors must keep the artifact private.

The requirements for this badge are set by the publisher and will be provided with the camera-ready instructions for OOPSLA papers. In the past, there have been two primary options for earning the Available badge:

• Option 1: Authors upload a snapshot of the artifact to Zenodo to receive a DOI. Uploads can be done manually, or through GitHub.
• Option 2: Authors work with Conference Publishing to send their artifact to the ACM for hosting on the ACM DL.

Every artifact that passes evaluation (Functional or Reusable) is strongly encouraged to be Available unless there are licensing or privacy concerns about sharing it.

Artifact Evaluated

There are two levels for the Artifact Evaluated badge: Functional and Reusable.

Functional: This is the basic “accepted” outcome for an artifact. An artifact can be awarded a Functional badge if the artifact is:

• Documented: At minimum, an inventory of artifacts is included, and sufficient description provided to enable the artifacts to be exercised.
• Consistent: The artifacts are relevant to the associated paper, and contribute in some inherent way to the generation of its main results.
• Complete: To the extent possible, all components relevant to the paper in question are included.
• Exercisable: Included scripts and/or software used to generate the results in the associated paper can be successfully executed, and included data can be accessed and appropriately manipulated.

Reusable: The goal of the Reusable badge is to recognize high-quality artifacts which support other people in understanding, applying, and extending the artifact. The criteria for the Reusable badge are less cut-and-dry than for the Functional badge. Reusability depends on the type of artifact and the community’s quality standards for research code. In addition to the Functional badge requirements, Reusable artifacts should ideally:

• clearly explain the capabilities of the artifact (e.g., with examples),
• contain high-quality documented code (e.g., READMEs, comments, tests),
• document how to adapt the artifact to new inputs, and
• be packaged to enable their reuse as a component in another project.

Not all parts of an artifact must be evaluated for reusability. For example, a shell script that generates graphs for the paper may not be intended for reuse, only for reproducing the results.

Results Reproduced

The Results Reproduced badge is awarded if the AEC can replicate all claims presented in the paper using the artifact, possibly excluding some minor claims if there are very good reasons why they cannot be supported.

In the ideal case, an artifact with this designation includes all relevant code, dependencies, input data (e.g., benchmarks), and the artifact’s documentation is sufficient for reviewers to reproduce the exact results described in the paper. If the artifact claims to outperform a related system in some way (in time, accuracy, etc.) and the other system was used to generate new numbers for the paper (e.g., an existing tool was run on new benchmarks not considered by the corresponding publication), artifacts should include a version of that related system, and instructions for reproducing the numbers used for comparison as well. If the alternative tool crashes on a subset of the inputs, simply note this as the expected behavior.

Deviations from the ideal must be for good reason. A non-exclusive list of justifiable deviations follows:

• Some benchmark code is subject to licensing or intellectual property restrictions and cannot legally be shared with reviewers (e.g., licensed benchmark suites like SPEC, or when a tool is applied to private proprietary code). In such cases, the public benchmarks should be included. If all benchmark data for a major claim is private, alternative data should be supplied. Providing a tool with no meaningful inputs to evaluate on is not sufficient to justify claims that the artifact works.
• Some of the results are performance data, and therefore exact numbers depend on the particular hardware. In this case, artifacts should explain how to recognize when experiments on other hardware reproduce the high-level results. For example, certain optimizations might exhibit a particular trend, or one tool might outperform another in a certain class of inputs.
• Repeating the evaluation takes a very long time. If so, provide small and representative inputs to demonstrate the behavior. Reviewers may or may not reproduce the full results in such cases.
• The evaluation requires specialized hardware (e.g., a CPU with a particular new feature, or a specific class of GPU, or a cluster of GPUs). Authors should provide instructions on how to gain access to the hardware or contact the chairs as soon as possible to work out how to make these possible to evaluate. In past years, one outcome was that an artifact requiring specialized hardware paid for a cloud instance with the hardware, which reviewers could access anonymously.

Artifacts are evaluated in relation to the expectations set by the paper. For an artifact to be accepted, it must support the main claims made in the paper. Thus, in addition to just running the artifact, the evaluators will read the paper and may try to tweak provided inputs or otherwise slightly generalize the use of the artifact from the paper in order to test the artifact’s limits. In general, artifacts should be:

• consistent with the paper,
• as complete as possible,
• well-documented, and
• easy to reuse, facilitating further research.

The AEC strives to place itself in the shoes of such future researchers and then to ask: how would this artifact help me to reproduce the results and build on them?

Submit your artifact through the AE Hotcrp submission link: Link Forthcoming

Data Availability Statement

Authors should prepare their artifacts in compliance with their paper’s Data-Availability Statement. Please refer to the OOPSLA CfP for more details.

Submission Process

All Accepted and Minor Revision papers at OOPSLA are eligible to submit artifacts.

Artifact evaluation begins with authors submitting artifacts on the artifact evaluation HotCRP. Artifact evaluation is optional. Authors are strongly encouraged to submit an artifact, but not doing so will not impact paper acceptance. Authors may, but are not required to, provide their artifact to paper reviewers as supplemental material.

Artifacts are submitted as a stable URL with a DOI. We recommend using a service that you can update in response to reviewer comments, to fix issues that come up. We recommend using Zenodo to create the stable URL for your artifact at submission time. Not only can you upload multiple versions in response to reviewer comments, you can use the same stable URL when publishing your paper.

Artifact evaluation is single blind: Reviewers will know the authors for each artifact, so there is no need to expend effort to anonymize the artifact. If you have any questions not answered by the guidelines below about how best to package your artifact, please don’t hesitate to contact the AEC chairs.

Packaging

We recommend (but do not strictly require) packaging your artifact with a virtual machine or Docker image. Images avoid issues with differing operating systems, versions, or dependencies and protect reviewers from malfunctioning artifacts damaging their computer. An image also offer a consistent snapshot of your artiact that does not rely on third-party sources such as package managers. However, other options for artifacts (such as plain source code, binary installers, web versions, or screencasts) can also be acceptable.

• Plain source code: It can be appropriate not to use an image in cases where the software has very few dependencies and requires only a working installation of a single programming language or package manager – e.g., Cargo or OPAM. In these cases, please be extra careful to document the installation of your artifact and required version(s) of everything via clear, step-by-step instructions. Additionally, make sure to test the instructions yourself on a fresh machine without any software installed. If the reviewers are unable to replicate your setup during the kick-the-tires phase, they may ask you to package your artifact in a different way.
• Virtual machine: We recommend VirtualBox 7.2, a free and actively maintained virtual machine host software. As of fall 2023, VirtualBox had some issues with running on newer ARM-based Macs (M1/M2/M3). For this reason, please ensure you are using at least version 7.2, which now supports both macOS/Arm and Windows/Arm. Recent graphical Linux releases, such as Ubuntu 20.04 LTS, are good choices for the guest OS: the reviewer can easily navigate the image or install additional tools, and the resulting images are not too large to download.
• Docker: If you use Docker, be warned that Docker images are not fully cross-platform out-of-the-box. We recommend building a multi-platform image, see this (slightly more complicated) process to bundle multi-platform images. Please document how your image was built.
• Remote access (recommended for resource-intensive artifacts): Particularly for artifacts requiring a large amount of computing resources (e.g. more than 8 vCPUs or 8GB of RAM), authors may consider providing remote access to an appropriate machine. Reviewers must be able to access the machine anonymously. The AEC may also be able to provide a remote environment using Cloudlab; please contact the Artifact Evaluation Chairs if you are interested in this option.

Virtual machines and containers should contain everything necessary for artifact evaluation: your source code, any compiler and build system your software needs, any platforms your artifact runs on (a proof assistant, Cabal, the Android emulator, …), any data or benchmarks, and all the tools you use to summarize the results of experiments such as plotting software. Execute all the steps you expect the reviewer to do in the container; it should already have dependencies pre-downloaded, the software pre-compiled, and output files pre-generated.

Please use a widely available compressed archive format such as ZIP (.zip), tar and gzip (.tgz), or tar and bzip2 (.tbz2). Please use open formats for documents (such as .html, .md, and .pdf).

Based on the outcome of the previous editions, the strongest recommendation we can give for ensuring quality packaging is to test your own directions on a fresh machine (or VM) and follow exactly the directions you have prepared.

Do not include anything in the artifact that would compromise reviewer anonymity, such as telemetry or analytics; if that is impossible to ensure, logging data should not be accessed by authors. If there is anything the reviewers should be worried about, please provide a clear disclaimer at the top of your artifact.

Documentation

Artifacts should provide top-level documentation in a clearly marked location; typically, as a single README file. The HotCRP submission will also provide a field where you can enter any additional instructions to access the documentation.

Besides the artifact itself, we recommend your documentation contains:

• Download, installation, and sanity-testing instructions
• Evaluation instructions
• Reusability guidelines
• Reproducibility guidelines (a set of steps to reproduce the results from the paper), in particular, include a list of all significant claims made by the paper and how they manifest and can be reproduced with the artifact.
• Additional artifact description (file structure, guidelines for extending the tool or adding new examples, etc.)

Mechanized proof artifacts (recommended): If your artifact is a mechanized proof, we recommend that you follow the guidelines on this page: Proof Artifacts. Be sure to explain how the mechanization realizes or encodes concepts and theorems from the paper.

Download, installation, and sanity-testing

The download, installation, and sanity-testing instructions should contain complete instructions for obtaining a copy of the artifact and ensuring that it works. List any software the reviewer will need (such as virtual machine host software) along with version numbers and platforms that are known to work. Then list all files the reviewer will need to download (such as the virtual machine image) before beginning. Downloads take time, and reviewers prefer to complete all downloads before beginning evaluation.

Note the guest OS used in the virtual machine, and any unusual modifications made to it. Explain its directory layout. It’s a good idea to put your artifact on the desktop of a graphical guest OS or in the home directory of a terminal-only guest OS.

Installation and sanity-testing instructions should list all steps necessary to set up the artifact and ensure that it works. This includes explaining how to invoke the build system; how to run the artifact on small test cases, benchmarks, or proofs; and the expected output. Your instructions should make clear which directory to run each command from, what output files it generates, and how to compare those output files to the paper. If your artifact generates plots, the sanity testing instructions should check that the plotting software works and the plots can be viewed.

Helper scripts that automate building the artifact, running it, and viewing the results can help reviewers out. Test those scripts carefully.

Aim for the installation and testing instructions to be completable in about a half hour. Remember that reviewers will not necessarily know what error messages mean or how to circumvent errors. The more foolproof the artifact, the easier evaluation will be for them and for you.

Evaluation Instructions

The evaluation instructions should describe how to run the complete artifact, end to end, and then evaluate each claim in the paper that the artifact supports. For experimental artifacts, this often takes the form of a series of commands that generate evaluation data, and then a claim-by-claim list of how to check that the evaluation data agrees with the claims in the paper.

For each command, note the output files it writes to, so the reviewer knows where to find the results. If possible, generate data in the same format and organization as in the paper: for a table, include a script that generates a similar table, and for a plot, generate a similar plot.

Indicate how similar you expect the artifact results to be. Program speed usually differs in a virtual machine, and this may lead to, for example, more timeouts. Indicate how many you expect. You might write, for example:

    “The paper claims 970/1031 benchmarks pass (claim 5). Because the program runs slower in a virtual machine, more benchmarks time out, so as few as 930 may pass.”

Reviewers must use their judgement to check if the suggested comparison is reasonable, but authors can provide expert guidance to set expectations.

Explicitly include commands that check soundness, such as counting admits in a Rocq code base. Explain any checks that fail.

Aim for the evaluation instructions to take no more than a few hours. Clearly note steps that take more than a few minutes to complete. If the artifact cannot be evaluated in a few hours (experiments that require days to run, for example) consider an alternative artifact format, like a screencast.

Reusability guidelines

Reusable artifacts should be released under an open-source license (e.g., OSI list).

In the reusability guidelines, explain which parts of your artifact constitute the core pieces which should be evaluated for reusability. If relevant, explain how to adapt the artifact to new inputs or new use cases. Provide instructions for how to find/generate/read documentation about the core artifact. Articulate any limitations to the artifact’s reusability.

Reproducibility guidelines

In the reproducibility guidelines, explain how to reproduce all significant claims made by the paper using the artifact. Articulate any limitations or expected deviation.

List of Claims

The list of claims should list all claims made in the paper. For each claim, provide a reference to the claim in the paper, the portion of the artifact evaluating that claim, and the evaluation instructions for evaluating that claim. The artifact need not support every claim in the paper; when evaluating the completeness of an artifact, reviewers will weigh the centrality and importance of the supported claims. Listing each claim individually provides the reviewer with a checklist to follow during the second, evaluation phase of the process. Organize the list of claims by section and subsection of the paper. A claim might read,

    “Theorem 12 from Section 5.2 of the paper corresponds to the theorem “foo” in the Rocq file “src/Blah.v” and is evaluated in Step 7 of the evaluation instructions.”

Some artifacts may attempt to perform malicious operations by design. Boldly and explicitly flag this in the instructions so AEC members can take appropriate precautions before installing and running these artifacts.

Reviewers expect artifacts may be buggy, immature, and have obscure error messages. Explicitly listing all claims allows the author to delineate which bugs invalidate the paper’s results and which are simply a normal part of the software engineering process.

Additional artifact description

Expect reviewers to examine such a section if something goes wrong (an unexpected error, for example) or if they are satisfied with the artifact and want to explore it further. Reviewers expect that new inputs can trigger bugs, flag warnings, or behave oddly. However, describing the artifact’s organization lends credence to claims of reusability. Reviewers may also want to examine components of the artifact that interest them.

Remember that the AEC is attempting to determine whether the artifact meets the expectations set by the paper. Package your artifact to help the committee easily evaluate this.

Evaluation

The artifact evaluation process will proceed in two phases.

Kick-the-Tires

In the first “kick the tires” phase reviewers download and install the artifact (if relevant) and exercise the basic functionality of the artifact to ensure that it works. We recommend authors include explicit instructions for this step. Failing the first phase—so that reviewers are unable to download and install the artifact—will prevent the artifact from being accepted. The preliminary reviews will be shared with authors immediately, who may make modest updates and corrections in order to resolve any issues the reviewers encountered.

Additional rounds of interaction are allowed via comments throughout the initial kick-the-tires period. Our goal here is twofold: we want to give authors the opportunity to resolve issues early (before other reviewers rediscover them), and we want authors to have as much time as possible for debugging (more than the typical 3-day response window).

Evaluation phase

In the second evaluation phase reviewers systematically evaluate all claims in the paper via procedures included in the artifact to ensure consistency, completeness, documentation, and reusability. We recommend authors list all claims in the paper as part of the artifact documentation and indicate how to evaluate each claim.

During the full review period, comments are closed by default but may be reopened at reviewers’ discretion to debug small issues. The purpose of re-opening communication is to maximize the number of Functional submissions. After the two-phase evaluation process, the AEC will discuss each artifact and notify authors of the final decision.

Common Issues

In the kick-the-tires phase

• Overstating platform support. Several artifacts claiming the need for only UNIX-like systems failed severely under macOS — in particular those requiring 32-bit compilers, which are no longer present in newer macOS versions. We recommend future artifacts scope their claimed support more narrowly.
• Missing dependencies, or poor documentation of dependencies. The most effective way to avoid these sorts of issues ahead of time is to run the instructions independently on a fresh machine, VM, or in a Docker container.

In the evaluation phase

• Comparing against existing tools on new benchmarks, but not including ways to reproduce the other tools’ executions.
• Not explaining how to interpret results. Several artifacts ran successfully and produced the output that was the basis for the paper, but without any way for reviewers to compare these for consistency with the paper. Examples included generating a list of warnings without documenting which were true vs. false positives, and generating large tables of numbers that were presented graphically in the paper without providing a way to generate analogous visualizations. We recommend using scripts to generate data and experimental figures. This fully automated approach may be a bit more costly to set up, but you won’t have any copy/pasting issue for your paper, and regenerating data is heavily simplified.

Conflicts of Interest

Conflicts of interest for AEC members are handled by the chairs. Conflicts of interest involving one of the chairs are handled by the other chairs, or the PC of the conference if all chairs are conflicted. Artifacts involving an AEC chair must be unambiguously accepted (they may not be borderline), and they may not be considered for the distinguished artifact award.

Resources

Further advice for packaging particular types of artifacts can be found below:

• SIGPLAN Empirical Evaluation Guidelines
• Proof Artifact Guidelines
• Checking Machine-Checked Proofs

Artifacts are an important product of the scientific process, and it is your goal, as members of the AEC, to study these artifacts and determine whether they meet the expectations laid out in the paper and adequately support its central claims.

These guidelines are adapted from the POPL AE reviewer guidelines.

General guidelines and advice

Please keep in mind the following general guidelines and advice:

• Artifact evaluation is a collaborative process with the authors. Our goal is not necessarily to find flaws with the artifact, but to try to the best of our ability to validate the artifact’s claims. This means that back-and-forth discussion is always a good idea if there are issues, particularly with installation.
• Artifact evaluation is not an examination of the scientific merits of the paper. In other words, it is within scope to evaluate the artifact (towards the badges), but not to evaluate the paper itself.

In cooperation with the authors, and without sacrificing scientific rigor, we aim to accept all artifacts which support the claims laid out in the paper.

Badges and Acceptance Criteria

We will be awarding four different kinds of badges: Available, Functional, Reusable, and Results reproduced. See the Badges tab for the formal requirements and how they relate.

Overview of the Process

Milestone 0: Bidding

During the initial bidding phase, you will examine the list of papers, and place bids for the artifacts that most closely match your research background, interests, and experience. Based on your bids, and depending on which paper authors choose to submit artifacts, we will assign you 3-5 artifacts to review.

After artifacts are assigned, the evaluation process is organized as the following milestones:

Milestone 1: Kick-the-Tires

Research software is delicate and needs careful setup. In order to ease this process, in the first phase of artifact evaluation, you will be expected to at least install the artifact and run a minimum set of commands (usually provided in the README by the authors) to sanity check that the artifact is correctly installed.

Here is a suggested process with some questions you can try to answer.

After reading the paper:

• Q1: What is the central contribution of the paper?
• Q2: What claims do the authors make of the artifact, and how does it connect to Q1 above?
• Q3: Can you locate the specific, significant claims made in the paper (such as figures, tables, etc.)?

After installing the artifact:

• Q4: Are you able to install and test the artifact as indicated by the authors in their “kick the tires” instructions?
• Q5: Are there any significant modifications you needed to make to the artifact while answering Q4?
• Q6: For each claim highlighted in Q3 above, do you know how to reproduce the result, using the artifact?
• Q7: Is there anything else that the authors or other reviewers should be aware of?

During the process, you can leave a comment on HotCRP indicating success, or ask the authors questions. These questions can concern unclear commands, or error messages that you encounter. The authors will have a chance to respond, fix bugs in their artifact or distribution, or make additional clarifications. Errors at this stage will not be counted against the artifact. Remember, the evaluation process is cooperative!

Milestone 2: Evaluating functionality

After the kick-the-tires phase, you will perform an actual review of the artifact.

During this phase, here is a suggested list of questions to answer:

• Q8: Do the results of running/examining the artifact meet your expectations after having read the paper? This corresponds to the criterion of consistency between the paper and the artifact.
• Q9: Is the artifact well-documented, to the extent that answering questions Q4–Q8 is straightforward? (Note: by well-documented, for this stage, we are generally considering only the README and instructions – we don’t mean that the code itself needs to be documented. That would matter only for reusability if the intention would be to modify the code in some way.)

In unusual cases, depending on the type of artifact and its specific claims, questions Q1–Q9 may be inappropriate or irrelevant. In these cases, we encourage you to disregard our suggestions and review the artifact as you think is most appropriate.

Milestone 3: Evaluating reusability

You will evaluate artifacts for reusability in new settings. To evaluate reusability, the following three initial questions are suggested for all artifacts:

• Q10: If you were doing follow-up research in this area, do you think you would be able to reuse the paper as a baseline in your own work?
• Q11: Is the code released via an open source license (e.g., released with an OSI approved license)? Is it made publicly available on a platform such as GitHub, GitLab, or BitBucket?
• Q12: Does the artifact have clear installation instructions?

To help you evaluate proof artifacts, the remaining questions are different for traditional (software) artifacts and for proof artifacts. For traditional software artifacts:

• Q13a: Are you able to modify the benchmarks/artifact to run simple additional experiments, similar to, but beyond those discussed in the paper?

For proof artifacts, instead of Q13a, we suggest answering:

• Q13b: Does the proof artifact contain definitions and proofs that can be used in other projects? (Examples of such artifacts include Rocq or Isabelle proofs and plugins.)
• Q14: Does the artifact clearly state all environment dependencies, including supported versions of the proof assistant and required third-party packages?
• Q15: Are all proofs claimed as reusable complete? (no “admit” in Rocq or “sorry” in Lean/Isabelle)

Milestone 4: Evaluating reproducibility

The goal of this milestone is to reproduce all the significant claims made in the paper using the artifact, only excluding claims if there are very good reasons why they cannot be reproduced.

For traditional software artifacts:

• Q16: Does the artifact provide evidence for all the significant claims you noted in Q1-3? Which claims are you able to reproduce and which are you not able to reproduce?
• Q17: What do you expect as a reasonable range of deviation, e.g., for experimental benchmarking results? Are your results within this range?

For proof artifacts, instead of Q16-17 we suggest answering:

• Q16b: Can you locate and map all definitions and theorems to their mechanization? Were you able to understand the mechanized definitions and theorem statements?
• Q17b: Does the artifact contain undocumented assumptions or unfinished proofs?
• Q18b: Do the mechanized definitions and theorems correspond precisely to those in the paper? If not, are the points of departure justified? Does the artifact agree with everything that the paper claims it does?

Milestone 5: Writing reviews and discussion

Write a review by drawing on your answers to Q1–Q17. Make sure to include a specific recommendation of whether to award a badge, and of which badge(s) to award. Once you submit your draft review, we will make all reviews public, so you have a chance to discuss the artifact with other reviewers and the chairs, and reach a consensus. You should feel free to change your mind or revise your review during these discussions.

Q. My artifact requires hundreds of GB of RAM / hundreds of CPU hours / a specialized GPU / etc., that the AEC members may not have access to. How can we submit an artifact?

If the tool can run on an average modern machine, but may run extremely slow in comparison to the hardware used for the paper’s evaluation, document the expected running time and point to examples the AEC may be able to replicate in less time. If your system will simply not work at all without hundreds of GB or RAM, or other hardware requirements that typical machines will not satisfy, please contact the AEC chairs in advance to make arrangements. One option is to get suitable hardware from a cloud provider (for example, Cloudlab), and give reviewers anonymous access. (The AEC chairs will coordinate with reviewers to decide when the cloud reservation needs to be active.) Submissions using cloud instances or similar that are not cleared with the AEC Chairs in advance will be summarily rejected

Q. Can my artifact be accepted if some of the paper’s claims are not supported by the artifact, for example if some benchmarks are omitted or the artifact does not include tools we experimentally compare against in the paper?

It depends. As a general principle, the artifact should follow the Data-Availability Statement presented in the paper. For example, if good explanations are not provided (as explained above) in the Data-Availability Statement, but if such claims are essential to the overall results of the paper, the artifact will be rejected. As an extreme example, an artifact with a working tool but no benchmarks (because they are closed-source) would be rejected. In this case, alternate benchmarks should be provided.

Q. Why do we need a DOI for the Available badge? Why not a Github or institutional URL?

A DOI is a strong assurance that the artifact will remain available indefinitely. By contrast, Github URLs are not permanent: it is possible to rewrite git commit history in a public repository (using rebase and –force, for example), users can delete public repositories, and Github itself might disappear like Google Code did (2015). Institutional URLs may also move and change over time.

Q. Reviewers identified things to fix in documentation or scripts for our artifact, and we’d prefer to publish the fixed version. Can we submit the improved version for the Available badge?

Yes.

Q. Can I get the Available badge for an artifact that was not judged to be Functional? I’m still making the thing available!

Yes.

Q: Does the AEC accept paper proofs?

The AEC process is not equipped to judge paper proofs. Therefore, most paper proofs will not be evaluated. Authors can still submit a paper proof for an Available badge, but should not expect any badges beyond that.