Appium... Image Rec!

It's been a while since my last post--An indication that my life just has been that busy lately! Not really an excuse though.

QA Automation, it is something extremely rare in the mobile video game industry. Why?

1. It takes quite a bit to setup and maintain. Mobile games are small in comparison to large web services or huge enterprise software, so it may not be worth it to have automation for tiny games.

2. QA for video games usually requires platform-specific automation due to things like smoke tests and performance tests. Sure, your game runs fine on a Galaxy S7, but what about on iPhone 7?

3. Automation on single-view applications like Unity can be difficult due to the fact that it's hard or impossible to get references to specific elements in the view. View? Let me explain.

What do I mean by view? That's mostly an iOS term (well it didn't start with iOS, but they did adopt the MVC-style architecture), and the Android equivalent is called an activity. You can think of them as a single "window" where everything rendered in your application will appear. Most importantly, your user interface stuff.

Appium is very popular with automation because of its ability to access elements within a view and simulate clicking or dragging those elements. Unfortunately, tools such as Unity and Xamarin that allow you to build in both iOS and Android platforms don't expose their user interface elements to Appium.

How do we then use Appium to click on a UI element? Coordinates. How do we get the right pixel coordinates when we are dealing with multiple aspect ratios? Image recognition.

When it comes to IR, there's plenty of options, but most people turn to OpenCV. Open Computer Vision has been around for a while now, and it is opensource. With some moderate coding, you can create a wrapper that uses their algorithms to detect images on the screen and feed pixel coordinates into Appium and BAM you have the click coordinates... Automation on cross-platform mobile!