Automation – metrics and measurements

Often, when the automation is carried out only after the entire product is ready, the number of times the automation execution is carried out is restricted due to the lack of time. On adding the development and maintenance effort to this, the returns on the investment in automation drastically reduce and sometimes move to negative.

To make an effective investment, it is critical to start automation early on for the prioritized features, based on the release schedule.

A regular tracking and monitoring system will accentuate the measurements and course corrections as required. Some of the key metrics to track are automation coverage, automation index, framework and scripts reusability, automation scripting and execution productivity, and automation ROI.

Automation coverage

This is usually measured as the number of automatable test cases out of the total number of test cases in the Regression and Functional test suite. An average of 70%-80% coverage can be achieved by automating the functionality, regression, and service-level automation. Bringing in the end-to-end interface automation can increase the coverage percentage.

Automation index

The number of automated scripts executed out of the total number of test cases that are selected for the given Test Cycle, gives the Automation Index. The higher the index, the higher is the ROI, and the benefits of automation.

Framework and scripts reusability

Having a robust and extensible automation framework makes automation adoption easy and a viable option. It is also key that the framework follows modular programming practices, allowing the reuse of scripts and module libraries. This metric can be measured after initial adoption is completed.

Automation scripting and execution productivity

The number of scripts that can be produced and executed in a given time period, say, a day, shows the team's productivity. This productivity number when compared to the manual productivity, should result in a huge reduction of effort to consider the automation project successful. This productivity data usually shows improvement over time when folded in with the reusability factor as well as increased domain and application knowledge in the team.

Automation ROI

The total effort spent on automation includes time for the following:

• Framework design
• Script creation
• Reviews and testing
• Maintenance effort over releases to incorporate release-specific changes
• The time taken to execute test cases in each test cycle

Comparing this time with manual effort for test execution shows the ROI. If, in spite of having automation scripts, the test cases are still being executed manually, the ROI on automation would be considered negative.

Another aspect to keep in purview while calculating the ROI is the time spent on analyzing failed scripts. Some scripts fail due to the issues or defects in the application and analyzing them does not take much effort. If the scripts fail due to any other reason, such as errors in script logic, network latency, third-party utility connectivity delays, and so on., and if the time taken to analyze such errors is longer, it has a direct and adverse impact on the productivity and ROI.

Automation progress tracking

Since automation work is considered as a separate unit of work, it is important that a project schedule is maintained and tracked closely. Daily, weekly, and monthly status reports should have a section to report the automation metrics as discussed in the preceding section, risks and dependencies, and so on. in order to ensure the obstacles are cleared on time.