Appium Image Recognition For Mobile Game Testing

Appium is a framework which is able to scale across different use cases with any problem. Various mobile games today are tested on Appium and Image Recognition Platform. All in all, this is a very efficient and effortless way to create test cases for mobile games and graphic-based content.

In this article, we will look at Appium features, how to set it up and how to use it.

Why To Use Image Recognition with Appium

Image Recognition technique is widely used by mobile game developers nowadays as creating automated tests for their mobile games is very easy and handy with Image Recognition Technique. Not only for games, but the technique can also be used for native apps in Android and/or iOS. Hence, developers are not limited when they use Appium.

Creating tests and working with those tests on both the platforms (Android and iOS) is the USP of Appium Image Recognition. Test automation frameworks generally cannot recognize UI components such as OpenGL ES contents, their IDs, attributes etc. Even if a human is clearly able to see the elements, test automation frameworks fail to do so.

This is why Appium Image Recognition is preferred. It can easily recognize all the graphical interface elements and can test them easily.

Required Components of Appium Image Recognition

The following components and/or software is required in order to get started with Appium Image Recognition:

is a web service to run tests on real mobile devices on the cloud.

  • OpenCV:

OpenCV is a machine learning software as well as an open-source computer vision, and it provides a common infrastructure for computer vision applications to accelerate the use of machine perception.

  • Bitbar Akaze:

This is a fork of Accelerated KAZE with added support for JSON.

  • Jsoncpp:

Jsoncpp provides you with C++ support. It is used to represent numbers, ordered sequences of values, strings etc. This is integrated with Bitbar Akaze library.

  • Maven:

This is required to compile the whole code and launch it for use.

These components, along with Bitbar Public Cloud, is what you require to run tests and execute them in a well-timed manner.

Example: Using Java for Appium

Before starting, just take a look at Appium on GitHub. This will provide an example for Java using OpenCV and Akaze to use Appium.

This example project can be used by you to perform local tests or to understand Bitbar cloud. You will also learn the difference between client-side and server-side execution.

Installing and Setting Up OpenCV

The OpenCV java libs are there under ./lib//opencv directory which is present inside the project. If you want to install all the libraries within maven locally, type in the following command:

mvn install:install-file -Dfile=lib//opencv/opencv-2413.jar -DgroupId=opencv -DartifactId=opencv -Dversion=2.4.13 -Dpackaging=jar

Take note that every platform has a different version of libraries, and so the above command is modified accordingly.

Installing and Setting Up Bitbar Akaze Fork

Bitbar Akaze Fork can be used optionally to implement AKAZE C++. The sample project on Github already has the desired ./lib//akaze/ folder with pre-built binaries for different operating systems.

Other things are fetched by Maven automatically.

Additional Information on OpenCV and Bitbar AKAZE Fork

The above example is dependent on the AKAZE algorithm to find out the matching keypoints between two images and then store them into a JSON file. The example also uses OpenCV Java to process the JSON files with key points to find homography.

AkazeImageFinder.java class will run akaze_match, and then it will use the key points from JSON so that it can identify a given image inside a scenery and return the results found.

In this example, we will be using a specific image from ./queryimages/folder. 

After the image is found, the test script can be tapped on, dragged or interacted with the game.

This class also has various functions to work with and manipulate the image captured. Some of the examples of this function are:

 // returns the corners of the image if found, or null if no image was found

public Point[] findImage(String image, String scene)

// takes a screenshot, then tries to find the image on it and return the corners

public Point[] findImageOnScreen(String image)

// takes a screenshot, tries to find the image on the screen, then tap the middle of it

public void tapImageOnScreen(String image)

// takes a screenshot, finds the image on the screen, taps and holds it, then swipes the screen while still holding it

public void swipeScreenWithImage(String image, int repeats)

// takes a screenshot, tries to find the image, taps it and drags it to a certain position given as x and y offsets from middle of the screen

public void dragImage(String image, double x_offset, double y_offset)

Running Tests as Server-Side Run on Bitbar Public Cloud

First of all, you will need to create a project so that Appium can run on Bitbar Public Cloud. You can do this by manually logging in and creating the project, or you can utilize a Bitbar API for creating your project. Using this code with curl will do the task easily:

$ curl

-X POST

-d “name=NewProject”

-H “Authorization: Bearer abcdefgh-1234-ijkl-m5n6-opqrstuvxyxz”

https://cloud.testdroid.com/api/v2/me/projects

One more thing which you need to authorize before running your test is the project type and how you want to execute your test run. You can use the Appium server-side for Android or iOS depending upon what device you want to run the test on.

After the creation of the project, you can use the following script to create a test zip from your project:

./createAndroidZip.sh

// or alternatively …

./createiOSZip.sh

Since you have your test zip and application file available now, you can create test runs for your projects.

Just upload the app and the test zip in the Bitbar public cloud, determine the device group and adjust the timeout of your test.

You need to use this code if you want to change the name of your Android or iOS test class:

# Name of the desired test suite and optionally specific test case, eg: AndroidSample#mainPageTest

TEST=${TEST:=”AndroidSample#mainPageTest”}

# JUnit file wont have the #caseName ending

JUNIT=”AndroidSample”

Optional: Building Up The Latest Libraries For Appium

To do this, first, we need to compile the latest version of jsoncpp.

Jsoncpp

To install jsoncpp, the following command is used:

git clone https://github.com/open-source-parsers/jsoncpp.git

cd jsoncpp

python amalgamate.py

This step will generate the following files and new AKAZE binaries are needed to be compiled.

  • dist/jsoncpp.cpp: source file that is added to your project.
  • dist/json/json.h: corresponding header file for use in your project. 
  • dist/json/json-forwards.h: header that provides forward declaration of all JsonCpp types.

After this, you also need to build up and copy libjsoncpp.a library to the local directory:

mkdir build

cd build

cmake ..

make

cp src/lib_json/libjsoncpp.a /usr/local/lib/

OpenCV

To build a new library in OpenCV, use the following command:

git clone https://github.com/Itseez/opencv.git 

cd opencv

git checkout 2.4 

mkdir build 

cd build/ 

cmake -G “Unix Makefiles” .. 

make -j8

make install

This will create Java jar files which will be used for image recognition samples. Jar files are installed by the command:

mvn install:install-file

 -Dfile=path/to/opencv/build/bin/opencv-2413.jar

 -DgroupId=opencv

 -DartifactId=opencv

 -Dversion=2.4.13

 -Dpackaging=jar

The pom.xml file will need to be updated with every different OpenCV version.  

Bitbar AKAZE

Bitbar Akaze can be built using the following commands:

git clone https://github.com/bitbar/akaze.git

cd akaze

# copy previously built versions of jsoncpp to ‘src/lib/jsoncpp/’, ‘src/lib/json/’ and src/lib/jsoncpp.cpp

mkdir ./src/lib/jsoncpp/

cp -r path/to/jsoncpp/dist/json ./src/lib/jsoncpp/

cp -r path/to/jsoncpp/dist/jsoncpp.cpp ./src/lib/

mkdir build

cd build

cmake .. 

make

cp bin/akaze_* path/to/image-recognition/akaze/bin/ 

This is it. Using this guide, you will be able to run Appium Image Recognition and run your tests smoothly on the environment you like.

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Share on pinterest
Pinterest
Close Bitnami banner
Bitnami