Implementing Complete End to End Testing Automation

Understanding the road ahead

Consider this you just launched a mobile application , people love it and it has become an instant hit but now months have passed and you have not been able to launch version 2.0 of your app and even if you do manage to hastily come up with all the features that can keep your audience intact and perhaps get some more users you are unable to make a quick release . Why ? Because you are stuck in an endless loop of testing individual features of the app and communicating the same back to your developers who then are stuck in a never ending cycle of waiting to hear back from the testers to let them know how the app performs over multiple scenarios and across multiple devices . The end result is an app that performs poorly and may end up in causing lots of unexpected production problems causing a scene similar to this ,

Fixing bugs in production

Clearly the situation is not ideal and can lead to unnecessary stress, often to include multiple devices under testing and to reduce the costs of hiring multiple testers we can end up with testers in situations like this :

Testing on multiple devices , a hassel

Obviously we need a better solution that fits all , Hence testing automation !

I was given the task to implement an entire testing automation process for our android application in my company and after going through multiple solutions online , my team and I decided to create a testing solution that is tailored to my company’s needs . We had few things in mind :

1. To automatically generate an APK on the cloud .
2. Upload the same application to a third party website that can run tests on multiple remote devices .
3. Run multiple tests that can complete end to end regression testing .
4. Generate a report from the tests and notify developers .
5. Go for a release knowing how our application performs across multiple devices and across multiple versions of android OS .

We had to address all these points and perhaps improve on them too .

Requirements :

1. A Version control system eg. Bitbucket
2. Jenkins setup
3. A server running linux OS

So I decided to use as many open source tools as possible. To have control on our tests we decided to write Appium scripts and use AndroidUI automator to identify various elements (Xpath, elementID) and create scripts that tests multiple features of our application through end to end UI testing. To enable building of android application on the cloud we decided to use Jenkins to build our application on one of our servers

Previously we used to build the application using Android Studio on the click of a button but now the task was to do the same and create two versions of the application on Jenkins one, the ‘Debug’ build of the application and the other, ‘Release’ build of the application . To ensure the correct build is made we decided to use Bitbucket which was our version control system

And by defining two branches :

1. App_debug branch
2. Release branch

Any commit made to these two branches would trigger a build on Jenkins and execute stages that is used to build the application , upload the same application on a third party website called pcloudy which could run our application on multiple remote devices and across different versions of android OS and finally run Appium tests on them which would generate reports that will be emailed to the developers in the company who could quickly fix the problems and make another commit

A flow of what the end result would look like ,

Automation Flow

To implement the Jenkins pipeline ,

1. The first stage involved validating the branch to ensure the commit has come from only branches tagged ‘release’ or the branch named ‘app_debug’.
2. The second stage included installing all the node packages.
3. The third stage was to include the SDK platform offered from google.
4. After putting a simple check to identify from which branch the code is coming from we can either create an unsigned APK for ‘an app_debug’ branch or a signed APK if the branch is from ‘release/*’ branch .
5. Now we clone code from another repository where we store all our Appium scripts.
6. Finally we generate the report and include screenshots where the tests fails.
7. The same report is now sent over email to all our developers using Jenkins.

Jenkinsfile :

Validating the branch

Performing Stage 2 and Stage 3

Stages 4 ,5 and 6

To upload the script to pcloudy we use their API documentation and create a simple script named upload_to_pcloudy.py

Finally in the post build stage we send the report which is created using pytest-html-reporter

How a report would look like ,

A failed report is immediately emailed to stakeholders

Finally this report is emailed using the post build stage in Jenkins with the relevant screenshots for failed steps ,

Email sent with screenshots and reports as attachments

Post build stage in Jenkins

Thus this completes our end to end testing automation pipeline !