Assessing the Quality of the Steps to Reproduce in Bug Reports [ACM SIGSOFT Distinguished Paper Award]

Oscar Chaparro, Carlos Bernal-Cardenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng.
Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 86-96, 2019.

Click here for the PDF version. The talk slides are available here.

Data and Software

The data and replication package can be found on this page.

Abstract

A major problem with user-written bug reports, indicated by developers and documented by researchers, is the (lack of high) quality of the reported steps to reproduce the bugs. Low-quality steps to reproduce lead to excessive manual effort spent on bug triage and resolution. This paper proposes an approach (named Euler) that automatically identifies and assesses the quality of the steps to reproduce in a bug report, providing feedback to the reporters, which they can use to improve the bug report. The feedback provided by Euler was assessed by external evaluators and the results indicate that Euler correctly identified 98% of the existing steps to reproduce and 58% of the missing ones, while 73% of its quality annotations are correct.

BibTeX entry

@InProceedings{Chaparro:19a,
  author = {Oscar Chaparro and Carlos Bernal-Cardenas and Jing Lu and Kevin Moran and Andrian Marcus and Massimiliano Di Penta and Denys Poshyvanyk and Vincent Ng},
  title = {Assessing the Quality of the Steps to Reproduce in Bug Reports},
  booktitle = {Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  pages = {86--96},
  year = 2019}