Empirical Standards for Software Engineering Research

This is THE MOST USEFUL review checklist I’ve found so far: It clearly defines evidence-based criteria for a good paper, and none of them is the fuzzy concept of “novelty”. Instead, we can choose the type of paper to review, and get the form adapted with fields tailored for that particular type of paper (benchmarking, case study,…).

Some criteria are as simple as “presents results” or “results directly address research questions” – but you would be surprised to see that there are some publications that don’t meet these. Ultimately I agree with the authors that the main criterion should be whether a paper contributes to our collective body of knowledge.

And it really helps me to read papers differently: For example, the question “visualizations/graphs are not misleading” has helped me spot serious flaws in paper figures more than once.

I would be very happy to see this adapted to HPC and Scientific Computing!