On Agile Test Automation (May 2017)


While everyone agrees that test automation is essential when using agile methods, not everyone agrees on the “why’s,” “what’s” and “how’s.” Although many license tools and/or develop a dashboard to perform the software testing job, many do not spend the time necessary to identify how they will use the results to improve their testing discipline. This represents what many in the industry call the metrics dilemma. Instead of using metrics to answer the important questions raised during testing, users take whatever information they are given and use it as best they can. This is where the “Goal-Question-Metric paradigm” comes into play. This standard is used for defining the test questions that will be answered by the metrics generated by the tool.  For example, “Have all planned tests been run successfully?” To answer the question, the tool would develop a metric based on the score of the number of tests passed versus failed along with the total number of tests that were and were not run. In order to define more metrics like this with test automation in mind, we are developing a typical list of questions that agile users want answered relative to testing. As part of this list, which we plan to release later this year, we plan to develop and provide a set of metrics that can be used to assess whether the results of software development, regression, and acceptance testing have been satisfactorily performed. Of course, users of the list will need to fine-tune it based on their individual situations. For those interested, inputs are encouraged and solicited. You can send your inputs to us at don@reifer.com.