Traditionally, technical research papers are published without including any arteficts (such as tools, data, models, videos, etc.), even though the arteficts may serve as crucial and detailed evidence for the quality of the results that the associated paper offers. They support the repeatability of experiments and precise comparison with alternative approaches, thus enabling higher quality in the research area as a whole. They may also make it easier for other researchers to perform their own experiments, thus helping the original authors disseminating their ideas in detail. Hence, artifacts should be taken seriously and recognized separately.
Last year, PPoPP 2017 successfully ran the artifact evaluation process for the third time. PPoPP 2018 continues this experiment.
Artifacts have been already evaluated at several other conferences and workshops including recent CGO 2017, PACT 2017, and PPoPP 2017. Our eventual goal is to develop common evaluation methodology gradually and collaboratively.
The website for this years artifact evaluation is at http://ctuning.org/ae/ppopp2018.html