Screen OCR

Summary

• Internal name: ocr_task
• Category: Media Projection
• Purpose: Capture the currently-projected screen and extract all visible text from it using on-device OCR (ML Kit Text Recognition).
• Task type: Normal

---

Compatibility

• Minimum AndroMate version: 1.1.0

• Maximum AndroMate version: 1.1.0

• Minimum Android: Android 13 (API 33)

• Maximum Android tested: Android 16 (API 36)

• Supported manufacturers:

  • ✅ All manufacturers

• Required permissions:

  • An active MediaProjection session — obtained via AskProjectionPermission, run earlier in the same workflow

---

Detailed description

The Screen OCR task captures a screenshot of the device via the active MediaProjection session and runs on-device text recognition (Google ML Kit) over it, returning every detected block of text.

This task requires an active projection — always run AskProjectionPermission earlier in the workflow first. Calling ocr_task without an active projection raises an exception immediately.

The output is a JSON object where each key is a zero-based index and each value is one recognized text block (in the order ML Kit detected them — roughly top-to-bottom, left-to-right, but not guaranteed for complex layouts). Empty/whitespace-only blocks are skipped.

---

Input parameters

Parameter	Type	Required	Possible values	Android Compatibility	AndroMate Compatibility	Default
timeout_ms	Integer	No	Milliseconds to wait for OCR recognition to complete	Android 13 (API 33) → Android 16 (API 36)	1.1.0 → 1.1.0	5000

---

Output parameters

Parameter	Type	Condition	Android Compatibility	AndroMate Compatibility	Default
value_output	String (JSON object)	Always, on success	Android 13 (API 33) → Android 16 (API 36)	1.1.0 → 1.1.0	<ANDROMATE_NULL_VALUE>

value_output

A JSON object string mapping a zero-based index to each recognized text block, e.g. {"0": "Settings", "1": "Wi-Fi", "2": "Bluetooth"}.

---

Exceptions

Code	Exception Name	Description
MEDIA-PROJ-002	Media Projection Not Active	No active MediaProjection session — run AskProjectionPermission first.
MEDIA-PROJ-002	Media Projection Capture Failed	The screen bitmap could not be captured from the active projection.
—	Timeout	OCR recognition did not complete within timeout_ms.

---

Execution flowchart

flowchart TD
    Start([▶ ocr_task]) --> CheckActive{Projection\nactive?}
    CheckActive -->|No| E1[❌ MEDIA-PROJ-002\nNOT_ACTIVE]
    CheckActive -->|Yes| Capture[📸 Capture screen bitmap]
    Capture --> CheckBitmap{Bitmap\ncaptured?}
    CheckBitmap -->|No| E2[❌ MEDIA-PROJ-002\nCAPTURE_FAILED]
    CheckBitmap -->|Yes| Recognize[🔍 Run ML Kit text recognition]
    Recognize --> Wait{Completed within\ntimeout_ms?}
    Wait -->|No| E3[❌ Timeout]
    Wait -->|Yes| Build[📋 Build index-to-text JSON]
    Build --> StoreResult[💾 Store value_output]
    StoreResult --> Success([✅ StrTaskResult])

    E1 --> Error([❌ Exception])
    E2 --> Error
    E3 --> Error

    style Start fill:#e3f2fd
    style Success fill:#c8e6c9
    style Error fill:#ffcdd2
    style Recognize fill:#fff9c4
    style StoreResult fill:#c8e6c9

How it works:

1. Check active projection: raises MEDIA-PROJ-002 (NOT_ACTIVE) if no AskProjectionPermission ran successfully first.
2. Capture: takes a bitmap snapshot of the current screen via the projection. Raises MEDIA-PROJ-002 (CAPTURE_FAILED) if the bitmap comes back null.
3. Recognize: runs ML Kit's on-device text recognizer over the bitmap.
4. Wait: blocks until recognition completes or timeout_ms elapses (raises a timeout exception otherwise).
5. Build result: collects every non-empty recognized text block into a {"0": "...", "1": "...", ...} JSON object.
6. Store: writes the JSON string into value_output.

---

Input parameter details

1. Input parameter: timeout_ms

How long to wait for ML Kit to finish recognizing text, in milliseconds.

Example

"timeout_ms": 8000

Details

• Optional — defaults to 5000 (5 seconds) if omitted.
• Not resolved as a $variable — read as a literal integer from the JSON.

---

Complete JSON example

{
  "AskProjectionPermission": [
    { "id": "1", "timeout_ms": 8000 }
  ],
  "ocr_task": [
    {
      "id": "2",
      "title": "Read screen text",
      "timeout_ms": 5000,
      "value_output": "$SCREEN_TEXT"
    }
  ]
}