Android Accessibility Tree in Python: Elements API

Most Android apps describe their screen to the accessibility service: every button, label and list row has a class, often a resource id, and usually text. Macro Automation Studio (MAS) exposes that Android accessibility tree to your Python macro as the elements API. Where it works it is exact where image matching is approximate: you tap a control by its id, read a counter as the string the app drew, and collect a whole list without cropping a single template. This page explains the functions, the selector shape, when the tree is unavailable, and how to combine it with vision. Full signatures are on the API reference. If you describe the task to MAS Agent, it writes this code for you.

How the Android accessibility tree reaches Python

The device side is one call: dump_hierarchy returns the current screen as uiautomator XML. Everything else (matching, ranking, ambiguity counting, scroll-and-collect) runs inside the SDK on your side of the connection. Both the desktop app and cloud devices serve the dump, so the whole API works on local emulators and on cloud devices.

The host reads the tree through a ladder. When the screen holds still it takes the rich uiautomator dump, which carries text, content description, resource id and class. When the screen keeps repainting (an animation, a running timer), uiautomator refuses, and the host falls back to a structural view dump that has resource ids and classes but no text. That distinction matters for selectors, as explained below.

The selector

Every lookup takes a plain dictionary. All keys are optional, but at least one of rid, desc, text or cls must be present.

CLAIM_SELECTOR = {
    "rid": "com.app:id/claim",   # resource-id; the short form "claim" also matches
    "text": "Claim",             # visible text, case-insensitive
    "desc": "Claim reward",      # content-description, case-insensitive
    "cls": "Button",             # class name; matches the part after the last dot
    "index": 0,                  # which of the equally ranked hits to take
    "clickable": True,           # only consider clickable nodes
}

Ranking: rid outranks desc, which outranks text, which outranks cls. For desc and text an exact match scores higher than a contains match; both ignore case and collapse whitespace. A resource id identifies a control on its own, so when it matches, a class that disagrees does not veto it. When several nodes tie, index picks among them in document order.

Harvest selectors while writing the macro from dump_hierarchy() and keep them as constants at the top of the script. Prefer a resource id: it survives themes, languages, resolutions and the structural fallback; text does not.

Functions: from the uiautomator dump to Python

host_capabilities

host_capabilities() returns the list of RPC method names the connected host dispatches, or an empty list on an older host. Check for "dump_hierarchy" before you rely on the tree, instead of finding out with a MethodNotFoundError mid-run.

dump_hierarchy

dump_hierarchy() returns the raw XML as a string. Use it while writing a macro to see the ids and texts on a screen, or to log the tree when a selector stops matching.

find_element and find_elements

find_element(selector) returns the best-ranked ElementMatch, or None when the tree is readable but nothing matches. find_elements(selector, *, max_matches=10) returns every hit, best first.

An ElementMatch carries x and y (the center, ready for click), text, desc, rid, cls, bounds as (x1, y1, x2, y2), the flags checked, enabled, clickable and scrollable, and match_count. A match_count of 1 means the selector was unambiguous. A larger number means the pick was made by rank and index, so tighten the selector. match.center returns (x, y), mirroring ObjectMatch.

find_element_retry

find_element_retry(selector, *, total_tries=3, time_sleep=2.0) is the wait primitive for elements, with the same shape as find_object_retry: up to total_tries attempts, a flat time_sleep pause between failed ones, none after the last. A tree that cannot be read counts as a failed attempt. Only when the final attempt still cannot read the tree does ElementsUnavailable propagate, so one flaky dump never fails a run.

read_element

read_element(selector) returns the text of the best match, falling back to its content description, or "" when nothing matches. It is the exact string the app drew, where OCR would approximate it.

scroll_to_element

scroll_to_element(selector, *, container=None, max_swipes=8) looks for the selector, swipes, and looks again until it matches or the swipes run out. Each swipe runs from 75 percent to 25 percent of the height of the container (itself a selector), or of the largest scrollable element on screen, or of the screen itself.

read_page

read_page(container=None, *, fields=None, max_swipes=8, row_cap=200) means “give me everything on this list”. It finds the container (by selector, or the largest scrollable element), detects its repeated row structure, scrolls through, and returns the rows deduplicated in reading order. Each row is {"cells": [...]} and each cell has text, desc, rid and center; fields keeps only the keys you name, plus text. max_swipes is limited to 0 to 20 and row_cap to 1 to 500. Scrolling stops early when a full swipe surfaces nothing new.

When the tree is unavailable

The rule the module is built on: an unavailable tree is never the same as an absent element. find_element, find_elements, read_element, scroll_to_element and read_page raise ElementsUnavailable in four cases.

• The host cannot dump at all: an older host without the method, or a device-side dump failure.
• The dump stalls past the read timeout.
• The dump does not describe the screen: a game or video surface (SurfaceView, GLSurfaceView, UnityPlayer, VideoView, TextureView) covers it, almost no node carries text, description or id, or every node is full-screen.
• The screen is animating, only the structural tree could be read, and the selector offers nothing but text or desc. The error message tells you to anchor on a resource id or use a vision asset for that screen. When the selector also has rid, the text keys are dropped and the lookup proceeds on the id.

ElementsUnavailable extends MASError, not RPCError, so catch it by name. Never treat it as “already done”; fall back to find_object or retry.

Elements or vision?

Prefer elements for text-heavy apps: settings screens, forms, chat lists, shops, anything built from native widgets. A resource id does not change with a theme, a language pack or a resolution, and read_element returns exact strings. Prefer vision for games and anything drawn on a canvas, because the tree describes nothing there. Macros that touch both worlds use elements for the app’s menus and templates for the play area, with a fallback in between.

Example: tap a button, fall back to a template

import mas

CLAIM = {"rid": "com.app:id/claim", "text": "Claim"}
images = mas.images({"claim_btn": 784})

def tap_claim() -> bool:
    try:
        el = mas.find_element_retry(CLAIM, total_tries=3, time_sleep=1.5)
    except mas.ElementsUnavailable:
        el = None                                    # tree unavailable, not "absent"
    if el:
        mas.click(el.x, el.y)
        return True
    match = mas.find_object_retry(images.claim_btn, total_tries=3, time_sleep=2.0)
    if match:
        mas.click(match.x, match.y)
        return True
    return False

Example: read a counter and scroll to a row

import mas

GOLD = {"rid": "com.app:id/gold_amount"}
INVITES_ROW = {"text": "Invite friends", "clickable": True}

try:
    gold = mas.read_element(GOLD)
    mas.log(f"gold on screen: {gold or 'unknown'}")

    row = mas.scroll_to_element(INVITES_ROW, max_swipes=6)
    if row and row.enabled:
        mas.click(row.x, row.y)
except mas.ElementsUnavailable as e:
    mas.log(f"no tree here: {e}", level="warning")

Example: collect a list

import mas

try:
    rows = mas.read_page({"rid": "com.app:id/leaderboard"}, fields=["text"], max_swipes=10)
except mas.ElementsUnavailable:
    rows = []

for row in rows:
    names = [cell["text"] for cell in row["cells"] if cell.get("text")]
    mas.log(" | ".join(names))
mas.log(f"{len(rows)} rows collected")

Example: check the host first

import mas

HAS_TREE = "dump_hierarchy" in mas.host_capabilities()

def find_start():
    if HAS_TREE:
        try:
            return mas.find_element({"rid": "com.app:id/start"})
        except mas.ElementsUnavailable:
            pass
    return mas.find_object_retry(901, total_tries=3, time_sleep=2.0)

Both ElementMatch and ObjectMatch expose x, y and center, so the caller can tap whichever comes back.