Python ADB Screenshot, Template Matching and OCR

Vision is how a Macro Automation Studio (MAS) macro sees the screen from Python. The app takes a screenshot of the device, over adb on an emulator, searches it for your template images with OpenCV template matching, and reads text with Tesseract OCR. The SDK only sends the request over JSON-RPC, so there is nothing to install. This page explains the search functions, the wait helpers, OCR, and the types they return. Full signatures are on the API reference. When you describe the task to MAS Agent, it writes this code for you.

Templates for OpenCV template matching on Android

A template is a small crop of the screen: a button, an icon, a badge. You crop it in Asset Lab from the live device, it lands in your Image Library with a numeric ID, and your script refers to it by that ID. Small, distinctive crops match reliably; a whole screen matches only that exact screen.

Declare the IDs once at the top of the script with mas.images. It returns an ImageMap with one ImageRef per name, so the rest of the script reads like prose.

import mas

images = mas.images({
    "claim_btn": 784,
    "energy_icon": 785,
    "close_x": 786,
})

match = mas.find_object(images.claim_btn)

Three rules come from the source. Names must be valid Python identifiers (claim_btn, not claim-btn), IDs must be integers, and the map is read-only. Every search function also accepts a raw integer ID. When a script mixes mas.images() with raw integers, the SDK prints a hint on exit listing the unregistered IDs, because the packer reads image IDs from the mas.images() call when it bundles a project for a cloud run or the Marketplace. An ID that is not in your library raises ImageNotFoundError.

take_screenshot

take_screenshot() captures the device screen and returns a Screenshot with base64 (PNG data), width, height and an ISO 8601 timestamp. Every search function takes a fresh screenshot unless you pass one in, so capture once when you need to look for several things on the same frame.

import base64
import mas

images = mas.images({"claim_btn": 784, "close_x": 786})

shot = mas.take_screenshot()
claim = mas.find_object(images.claim_btn, screenshot=shot)
close = mas.find_object(images.close_x, screenshot=shot)

with open("frame.png", "wb") as f:
    f.write(base64.b64decode(shot.base64))

find_object

find_object(image_id, *, threshold=0.8, screenshot=None, continuous_mode=False, timeout_ms=1000, capture_interval_ms=500, max_matches=1, search_region=None) searches the screen for one template. It returns an ObjectMatch when the best match scores at least threshold, and None otherwise. A miss never raises. Every parameter after image_id is keyword-only.

• threshold: similarity from 0.0 to 1.0, higher is stricter. 0.8 is the default. Go up to 0.9 when a template has near-identical siblings, and down to 0.7 only for compressed or animated art.
• search_region: a Region(x1, y1, x2, y2) that limits the search. A tight region is faster and rules out look-alikes elsewhere on the screen.
• continuous_mode: when True, the app keeps capturing every capture_interval_ms (500 ms) until the template appears or timeout_ms (1000 ms) elapses. timeout_ms has no effect outside continuous mode.
• max_matches: leave it at 1 here and use find_objects when you want several.
• screenshot: a frame from take_screenshot() to reuse.

import mas
from mas import Region

images = mas.images({"claim_btn": 784})

match = mas.find_object(images.claim_btn)
match = mas.find_object(images.claim_btn, threshold=0.9,
                        search_region=Region(x1=0, y1=1400, x2=1080, y2=1920))

find_objects

find_objects(image_id, *, ..., max_matches=10, ...) takes the same parameters, with max_matches defaulting to 10, and returns a list of ObjectMatch sorted best first. The list is empty on a miss. Use it for grids of identical items: reward chests, checkboxes, list rows that share an icon.

find_any_object

find_any_object(image_ids, *, ..., search_strategy="first_match", ...) searches for several templates in one call and returns one ObjectMatch, or None. It is for variants of the same control: two languages, two themes, a pressed and an unpressed state. matched_template_id on the result tells you which template hit.

search_strategy takes one of three values. "first_match" and "priority_order" try the templates in list order and stop at the first hit. "best_match" tries every template and returns the strongest score.

import mas

images = mas.images({"ok_en": 301, "ok_de": 302, "ok_fr": 303})

match = mas.find_any_object([images.ok_en, images.ok_de, images.ok_fr])
if match:
    mas.log(f"matched template {match.matched_template_id}")
    mas.click(match.x, match.y)

Waiting: find_object_retry and find_any_object_retry

find_object_retry(image_id, *, total_tries=3, time_sleep=2.0, **kwargs) is the recommended way to wait for something to appear. It calls find_object up to total_tries times, sleeps time_sleep seconds between failed attempts, never sleeps after the last one, and returns the first match or None. The cost is bounded and easy to read: three tries two seconds apart is at most four seconds of waiting plus three searches.

total_tries and time_sleep are keyword-only. Every other keyword (threshold, search_region, max_matches, screenshot) is forwarded to find_object by name, so order never matters. If you pass a screenshot, the call collapses to a single attempt, because retrying a frozen frame would search the same pixels again.

find_any_object_retry(image_ids, *, total_tries=3, time_sleep=2.0, **kwargs) does the same over a list of templates and forwards to find_any_object.

import mas
from mas import Region

images = mas.images({"claim_btn": 784})

btn = mas.find_object_retry(
    images.claim_btn,
    total_tries=5,
    time_sleep=1.5,
    threshold=0.85,
    search_region=Region(x1=0, y1=1200, x2=1080, y2=1920),
)
if btn is None:
    mas.log("claim button never appeared", level="warning")

wait_for_object

wait_for_object(image_id, timeout_ms=10000, threshold=0.8) waits on the app side until the template appears or the timeout passes, and returns the match or None. It is kept for existing macros. Prefer find_object_retry in new code: discrete tries with a flat pause read plainly, take a predictable amount of time, and behave the same on every host. wait_for_object also has no search_region.

read_text

read_text(region=None, screenshot=None, model="eng_best", psm=7, color_conversion=ColorConversion.NONE, timeout_ms=30000) runs Tesseract OCR and returns a TextRecognitionResult with text, confidence (0.0 to 1.0) and the region it read.

• region: a Region around the text. Without it the whole screen is read, which is slow and noisy. Draw the region in Asset Lab, where you can test it live.
• model: "eng_best" (default), "eng_fast" or "eng".
• psm: the Tesseract page segmentation mode. 7 treats the region as one line (default), 8 as one word, 6 as a uniform block of text, 11 as sparse text in no particular order.
• color_conversion: a ColorConversion member applied before OCR. NONE is the default. BGR_TO_GRAY or RGB_TO_GRAY turns the region grayscale. BLACK_WHITE is a MAS extension that thresholds to pure black and white, the best choice for bright text on a busy background. There is no GRAYSCALE or BINARY member.
• timeout_ms: 30000 by default.

OCR returns text, not numbers. Strip everything that is not a digit before you compare.

ObjectMatch, Region and the other types

ObjectMatch has exactly three fields: x and y, the center of the matched area in screen pixels, and matched_template_id, the image ID that hit. match.center returns (x, y) as a tuple and match.template_id is an alias. There is no confidence, width or height on a match; if you need a stricter answer, raise threshold.

Region is a dataclass with x1, y1 (top-left) and x2, y2 (bottom-right), in pixels. Screenshot, TextRecognitionResult, ColorConversion and SearchStrategy are all importable from mas.

Example: click a button when it appears

import mas

images = mas.images({"claim_btn": 784})

btn = mas.find_object_retry(images.claim_btn, total_tries=4, time_sleep=2.0)
if btn:
    mas.click(btn.x, btn.y, delay_ms=1000)
else:
    mas.log("nothing to claim this round")

Example: read an energy counter

import re
import mas
from mas import Region, ColorConversion

ENERGY = Region(x1=880, y1=40, x2=1060, y2=90)

result = mas.read_text(region=ENERGY, psm=7, color_conversion=ColorConversion.BLACK_WHITE)
digits = re.sub(r"[^0-9]", "", result.text)
energy = int(digits) if digits else 0
mas.log(f"energy={energy} confidence={result.confidence:.2f}")

if energy < 20:
    mas.log("low energy, stopping", level="warning")
    raise SystemExit(0)

Example: find all matches on one Python ADB screenshot

Capture one frame, find every match on it, then tap them in order. Take a new frame afterwards, because the screen has changed.

import mas

images = mas.images({"chest": 412})

shot = mas.take_screenshot()
chests = mas.find_objects(images.chest, screenshot=shot, max_matches=10, threshold=0.85)
mas.log(f"found {len(chests)} chests")

for chest in chests:
    mas.click(chest.x, chest.y, delay_ms=800)

Errors

A miss is None or an empty list, never an exception. The exceptions you will meet are ImageNotFoundError (the ID is not in your Image Library), DeviceNotConnectedError (no device bound to the run) and TimeoutError from wait_for_object. All three extend RPCError; see Errors and exceptions.