Why AWS(Amazon Web Services) Device Farm?

We have used Amazon AWS Device Farm for testing the Android Mobile Application under the cloud testing infrastructure. AWSDF help us to ensure the device coverage in mobile application testing including the device interoperability. Test Results will be published with the detailed logs, screenshots, reports and performance data. Mobile Application Automation Testing can be implemented by using the open testing framework of Appium, Calabash and Junit. It will support device instrumentation testing using Robotium and Espresso.

Open Test Frameworks such as Appium, Calabash, Espresso, Robotium will support Acceptance Testing.

Tool Analysis Information

Amazon AWS Device Farm supporting following type of test(Configuration) in their cloud infrastructure;

1. Builtin:Fuzz
2. Appium Java Junit
3. Appium Java TestNG
4. Calabash
5. Instrumentation (Robotium, Espresso, Junit)
6. uiautomator

Work Plan 1

We have tried with the Builtin:Fuzz test type in device farm by uploading a Fontli APK file in Amazon web portal. Random events such as click, scroll, swipe, tap etc,. has been triggered by the monkey API sit in the Amazon server. It will publish the JSON logs and logcat file for the created test run in Amazon AWS device farm.

We have selected the top devices(default) in the device pool facility available in the device farm. Out of five devices, test run got success in the two Samsung & one Motorola mobile phones and two got failed in the tablets such as Samsung Galaxy TAB 4 and Amazon Kindle Fire. We got the logcat for both the success test run and failed test run from the Amazon web portal for debug analysis.

Work Plan 2

We have tried the next test type of Instrumentation with Robotium by creating test package, which is used to test the target package of fontli application. Robotium test package comprised of invoking the android application, login the application and take the screenshot of main page after login.

Robotium is Instrumentation specific Open Test Framework maintained by Google community. Robotium JAR will be integrated with the Eclipse IDE. Test Script will be created by using programming language such as Java and it will be integrated with Android unit. Test package will be created based on the written test script and test package directly interacts with the instrumentation runner available in the Android Dalvik Virtual Machine. The same request will trigger user action in the target package which is nothing but the mobile application to test. We have uploaded the target package and test package in the Amazon AWS device farm web portal but the expected test result was not generated by the Amazon AWS device farm dash board. It will keep on share the pass result in the dash board without executing our test Robotium+Java.

We could see the success result for the same test script running in the local machine Eclipse IDE with Emulator. We have raised our concern to Amazon AWS Device Farm support team and looking forward their solution to proceed for further move. We got a reply from them to share our application Google play store link and ask us try with some other test type available at AWS device farm portal. Test filter in the configuration should be mentioned about the package name of the test APK based on the below solution from Amazon Technical Team.

The Java package in the test APK is com.test.demo but in the app’s package is com.companyname.fontli. By default we look for tests in the same Java package as the app. To override this behavior, set “Test filter” to “com.test.demo” in the “Configure a test” step. After implementing the above procedure, Instrumentation test run is working as expected with the required test results including screenshot capture.

Work Plan 3

We moved to next test type such as Appium Java Junit. We have created a test script using Java as a programming language integrated with Junit to automate a login process of Fontli mobile Application. Open Test Framework Appium JAR has been integrated with the Eclipse IDE and mentioned about the desired capabilities of test device.

Appium is going to work with the Node Server concept in which Test Script event request pushed to the Node server and the same server again push the JSON request to the real device or emulator connected with the machine. Appium Node server will take the control of ADB(Android Device Bridge) in full swing to trigger all the Test Script events. Once run the Test Script under the Junit Run Configuration, test cases will be executed and results has been published in the Eclipse IDE.

Now come to the AWS Device Farm cloud Testing Infrastructure, we have uploaded the Fontli application package(APK) and selected the LG devices in the pool such as Nexus 5, L70 etc,. We have uploaded the Test Script class file after zip in order to ensure the test script going to execute in the APK file. But unfortunately test cases was not executed in the AWS Device Farm and the same issue has been escalated to their Technical Support Team.

We got a solution from Amazon Technical Support team in which we have to generate the zip file using POM based Apache Maven. We have integrated Apache Maven with Eclipse IDE and build the Appium Test Script source in the maven form of POM.XML and successfully run the test script using Maven in the local machine. But again the same test script is getting failed in the Amazon AWS device farm portal with the launcher activity exception. We have escalated the issue to the concern amazon technical support team.

Now the test script got executed successfully after getting the resolution from Amazon technical team in which app activity has been changed as welcome activity instead of holder activity. (Activity name matters here)

Work Plan 4

Calabash is a behavior driven development and it is automated testing framework that can be used in both android and iOS platforms. These specifications are actually written using English language,so to write a script in calabash,it is not required that person should be technically strong in particular language. Writing scripts in calabash is like talking to person in simple English. It is easily understandable by the system. Every *.feature file conventionally consists of a single feature. Lines starting with the keyword Feature. The scripts are written using predefined steps.

A feature folder has a list of following items;

• step_definitions
• support
• Feature file

We have tried few test scenarios using the fontli application (android platform) and it is working as expected in the Amazon Device Farm.