Get Screen Elements

Summary

• Internal name: GetScreenElements
• Category: Android UI
• Purpose: Dump every element on the current screen into a JSON object keyed by traversal index, with each entry's text, content description, and position.
• Task type: Normal

---

Compatibility

• Minimum AndroMate version: 1.1.0
• Maximum AndroMate version: 1.1.0
• Minimum Android: Android 13 (API 33)
• Maximum Android tested: Android 16 (API 36)

• Supported manufacturers:

  • Samsung (One UI 6.x / 7.x / 8.x)
  • Other manufacturers - not supported yet (Xiaomi HyperOS and Google Pixel are being adapted)
  • Required permissions:
  • ACCESSIBILITY_SERVICE

---

Detailed description

The Get Screen Elements task is a read-only query - unlike the other Android UI tasks in this group, it doesn't act on the screen, it describes it. It walks the entire accessibility tree of the active window in traversal order and builds one JSON object, keyed by a "0", "1", "2", ... index, where each value is { "text": ..., "contentDescription": ..., "position": { "left", "top", "right", "bottom" } }.

Use it to inspect a screen's structure while building a workflow, or feed the result into JsonObjectOperation / For Each Element In Screen-style logic downstream.

---

Input parameters

This task has no configurable input parameters (besides json_object_output, documented below under Output parameters).

---

Output parameters

Field	Type	Trigger condition	Android Compatibility	AndroMate Compatibility	Default
json_object_output	JSON Object (as string)	Always on success - {"0": {text, contentDescription, position}, "1": {...}, ...}	Android 13 (API 33) -> Android 16 (API 36)	1.1.0 -> 1.1.0	<ANDROMATE_NULL_VALUE>

---

Exceptions

Code	Exception Name	Description
SCREEN-AUTOMATOR-ERROR-010	Screen Automator Service Not Connected	The AndroMate accessibility service isn't enabled/connected.

---

Execution flowchart

flowchart TD
    Start([GetScreenElements]) --> CheckConnected{Accessibility service\nconnected?}
    CheckConnected -->|No| E1[SCREEN-AUTOMATOR-ERROR-010\nSERVICE_NOT_CONNECTED]
    CheckConnected -->|Yes| Walk[Walk accessibility tree\nin traversal order]
    Walk --> Build[Build JSON object\nindex -> text/contentDescription/position]
    Build --> Store[Store into json_object_output]
    Store --> Success([JsonObjectTaskResult])

    E1 --> Error([Exception])

    style Start fill:#e3f2fd
    style Success fill:#c8e6c9
    style Error fill:#ffcdd2
    style Walk fill:#f3e5f5
    style Build fill:#fff9c4

How it works:

1. Check connection: fails fast with SERVICE_NOT_CONNECTED if the accessibility service isn't enabled.
2. Walk: recursively visits every node of the active window, in traversal order.
3. Build: for each node, records text, contentDescription, and its screen bounds (left/top/right/bottom), keyed by a 0-based index.
4. Store: writes the resulting JSON object into json_object_output.

---

Code examples

Example 1 - Dump the current screen into a variable

{
  "GetScreenElements": [
    { "id": "1", "title": "Dump screen", "json_object_output": "$screen" }
  ]
}

---

Input parameter details

This task has no configurable input parameters.

---

Output parameter details

1. Output parameter: json_object_output

The name of the workflow variable that receives the JSON object dump. Each entry's position is the node's bounds in screen coordinates (pixels), not a center point.

Example

{
  "0": { "text": "Login", "contentDescription": null, "position": { "left": 40, "top": 120, "right": 300, "bottom": 180 } },
  "1": { "text": null, "contentDescription": "Settings icon", "position": { "left": 900, "top": 40, "right": 960, "bottom": 100 } }
}

---

Complete JSON example

{
  "GetScreenElements": [
    { "id": "1", "title": "Dump current screen", "json_object_output": "$screen_dump" }
  ]
}