Diving deeper with CTS Verifier

We already know that there are lots of APIs and functions that we can automatically test with cts-tradefed, but what about all the other APIs and functions that cannot be tested within an automated environment?

CTS Verifier comes in every time an API or a function cannot be tested on a device without manual input. These are scenarios involving audio quality, touchscreen effectiveness, accelerometer precision and reactivity, camera quality, and features that are meant so specifically for human interaction that they are impossible to test without human interaction.

Setup

All we need to run CTS Verifier is an Android certified device and the appropriate CTS Verifier APK file. As we are testing an Android 4.4 device, we need to pay attention to downloading the proper CTS Verifier version. You can download the APK for your Android version and device architecture here: http://source.android.com/compatibility/downloads.html.

You just need to unzip the downloaded file and you will find a folder hierarchy and two .apk files. You can install CtsVerifier.apk using ADB:

$ adb install –r CtsVerifier.apk

The following screenshot shows the properly installed CTS Verified app and the initial screen:

Manual testing

As we know, CTS Verifier contains tests that need manual input to execute, evaluate, pass, or fail. Every test has its own Info screen that helps the tester to perform the test. As an example, we will run the Accelerometer Test, in the Sensors section.

Launching the test, we are welcomed by the info screen, as shown in the following screenshot:

The Info button explains how to perform the test and what to evaluate. As we move into the testing, we can evaluate if the accelerometer is working as expected. The following screenshot shows three different moments of the test:

• The smartphone lays on the desk
• The smartphone is held in hand, in portrait mode
• The smartphone is held in hand, in landscape mode
  

As specified in the Info button, the arrow is always pointing in the same direction as the gravity: the sensor is working properly. We can consider that we have passed the test and click on the Pass button.

We have passed our first test. CTS Verifier provides dozens of tests and, one by one, we are going to run, verify, and pass them, in the long journey towards our first Android Certified Device.

Retrieving the results

When every test has been executed, we can save the result using the Save icon in the top-right corner of the initial screen, as shown in previous screenshot. The results will be saved on the device and a dialog box will show the precise path, as shown in the following screenshot:

Now, let's open a terminal and copy all the results from the phone to our computer:

$ adb pull /mnt/sdcard/ctsVerifierReports/ .
$ unzip *.zip

At this point, we have a ctsVerifierReport-[…].xml with all the info about our manually executed tests.

Congratulations! You have fully tested an Android device. Step 0 of our journey is complete.