Image Recognition Macro Concepts: How MAS Works

Macro Automation Studio (MAS) automates Android apps the way a person does: it looks at the screen, decides, and taps. This page explains how an image recognition macro works, the words you meet in the app, and where each part runs. Read it once and the rest of the docs will make sense.

How an image recognition macro sees the screen

MAS does not modify APKs and does not read app memory. It works from the outside, through adb, with three building blocks.

Template images

A template is a small crop of the screen: a button, an icon, a badge. You crop it in Asset Lab from the live device, it lands in your Image Library with a numeric ID, and your script refers to it by that ID. At run time find_object takes a fresh screenshot and searches it for the template with OpenCV template matching. A match needs a similarity of 0.8 by default (threshold=0.8) and returns the centre point, so the next line is a tap.

Image files can be jpg, png, gif or webp, up to 10 MB each. Keep templates small and distinctive: a whole screen matches only that exact screen, while a button matches wherever the button appears.

OCR automation on Android

Text is read with Tesseract OCR through read_text. You pass a Region(x1, y1, x2, y2) so the engine looks at one counter or one label rather than the whole screen, and a page segmentation mode (psm=7 for a single line, psm=8 for a single word). ColorConversion.BLACK_WHITE helps with coloured text on noisy backgrounds. Asset Lab has a live OCR test so you can tune a region before it goes into code.

Humanized input

Input is sent as real touch and key events: click, swipe, input_text, key_press, zoom_in and zoom_out. Timing is yours to set. A click waits delay_ms=1000 after the tap, a swipe takes duration_ms=1000, and a key held for 500 ms or more counts as a long press. Slower, spaced-out macros look less mechanical.

Coordinates and screen size

Every coordinate is a pixel offset from the top-left corner of the screen. get_screen_size() returns the width and height; get_device_info() returns the same plus the device name. Templates and regions are tied to the resolution and DPI you captured them on, so keep every device that runs a macro on the same resolution as the one you built it on. The emulator guides recommend a size for each emulator, and cloud devices are created with a fixed display preset.

Vocabulary

Where things run

Three places do the work, and it helps to know which one you are looking at.

Your computer drives emulators

MAS on your PC or Mac finds emulators by their process name and the ports they listen on, then talks to each one with adb connect 127.0.0.1:<port>. Screenshots, taps and OCR for a local device all happen on your machine, so the app must stay open while a macro runs. The same is true of the scheduler.

Cloud devices run in MAS’s cloud

A cloud device lives on MAS’s servers. When you run a macro on it, MAS uploads a snapshot of your project and runs it next to the device. You can watch the screen live in the app or close the app entirely; the run continues either way. Idle devices stop after about an hour of inactivity and keep their data.

Scripts talk to the app over JSON-RPC

A macro is a normal Python process. MAS starts it as python -u -m src.app inside its own Python 3.13 environment and hands it the connection details in environment variables (MAS_RPC_PORT, MAS_RPC_HOST, MAS_DEVICE_ID, MAS_SESSION_TOKEN). Every mas.* call is a JSON-RPC 2.0 request over a local socket; the app owns adb, OpenCV and Tesseract and sends the result back. On a cloud device the same calls travel over a unix socket inside the sandbox. This is why import mas needs no install and why a script started outside MAS reports “Could not discover RPC port”.

AI at authoring time, none at run time

MAS Agent spends AI credits while it explores and compiles. The macro it produces is plain Python and runs like any other, on your emulator or on a cloud device, without the agent in the loop.

The subscription

Every page in the app apart from sign-in and Subscription needs an active entitlement: the free trial or a paid plan. A plan is sized by devices, meaning how many of your own emulators or phones can run at the same time. Cloud devices are an add-on with their own monthly credits, and extra credits can be topped up for agent sessions. Cancelling keeps access until the end of the paid period. The Billing page explains the mechanics; the figures are on the pricing page.

Where to go next

Set up a device on the Devices page, crop your first template in Asset Lab, or read the SDK overview if you would rather start from code.