Investigate agent device as Melvin tool for mobile device testing

[Julesssss]

Problem

We are using playwright within Melvin for simple bug verification and test fixes (limited to web).
Our plans to extend Melvin include improved testing and expanding testing functionality to mobile.

agent-device looks like a great potential tool for this. The demo shows fast UI testing of the Expensify ND app.

Solution

Lets create a POC using the tool. Initially we would like to enable Melvin to:

1. Verify test cases similar to playwright
2. Post screenshots to App PRs (building into Neil's work)

@AndrewGable @justinpersaud

---

• Here's our existing Playwright skill
• Here's documentation on Melvin
• Here's how Melvin uploads images and attatches them to Github comments

Initial Questions:

• Limitations?
• How to integrate with Melvin, which is designed for one-shot requests

Issue Owner

Current Issue Owner: @kacper-mikolajczak

[adhorodyski]

@Julesssss looks like I'm missing access to the main repo with Melvin's setup, can you help with that?

[Julesssss]

@justinpersaud would you mind inviting @adhorodyski to E/E please.

[kacper-mikolajczak]

Hi @Julesssss! After chat with Adam, I will push this effort from CK side forwards.

Let's sync up on the exact success criteria for this exploration. Could you start with more in-depth description of how you'd like to see it being used and integrated? It should help us be on the same page before jumping into implementation phase. Thanks ❤️

[Julesssss]

Hey @kacper-mikolajczak, sure. So this is ideally a quick POC to see if it will be possible without too much compromise.

Plan:

• Give Melvin the ability to test HybridApp builds
• Give Melvin ability to screenshot during these tests
  • We have existing logic for storing the screenshots so we can attach them to Melvin's GitHub comments

We have two Melvin workflows:

• Melvin Anywhere: Has full dev environment
• Melvin App: Has access to App, can make web builds pointing to prod
  • We should aim for this flow only, keep it simple

[justinpersaud]

Added @adhorodyski to Callstack AI development team which should grant access

[kacper-mikolajczak]

Hey folks 👋 Here's a design doc on the PoC after our internal discussion with @Julesssss:

PoC Design: Agentic Mobile Testing for Melvin

Goal

Improve PR QA/review process by enabling agentic mobile testing via agent-device, integrated with the existing MelvinBot App workflow. Additionally, enable agent-device for local developer use.

Background

MelvinBot currently handles ~200 requests per 36 hours on Expensify/App - automated issue analysis, CI fixes, code review, implementation, and more. The workflow (claude-melvinbot-app.yml) includes Playwright browser testing infrastructure, but analysis of ~650 production runs shows it's invoked only ~0.3% of the time. Mobile testing is entirely absent.

agent-device is a CLI tool by Callstack that gives AI agents control over mobile app UIs using accessibility tree snapshots (not screenshots), making interactions token-efficient and deterministic. It supports Android emulators headlessly on Linux - a natural fit for the existing CI setup.

Architecture

                        MelvinBot App Workflow
                     (claude-melvinbot-app.yml)
                               |
            +------------------+------------------+
            |                                     |
     Existing steps                        New steps (PoC)
     (unchanged)                           (added to workflow)
            |                                     |
  +---------+---------+              +------------+------------+
  | Checkout repos    |              | Enable KVM              |
  | Setup Node        |              | Install Android SDK     |
  | Install App deps  |              | Install agent-device    |
  | Configure Git     |              | Install skill           |
  | Playwright setup  |              +------------+------------+
  +---------+---------+                           |
            |                                     |
            +------------------+------------------+
                               |
                    Claude Code Action runs
                               |
                 Agent decides what to do:
                               |
              +----------------+----------------+
              |                |                |
        Code analysis    Web testing      Mobile testing
        (default)        (Playwright)     (agent-device)
                                               |
                                    +----------+----------+
                                    | Fetch APK from Rock |
                                    | Boot emulator       |
                                    | Install + open app  |
                                    | Interact + verify   |
                                    | Upload screenshots  |
                                    | Shutdown emulator   |
                                    +---------------------+

Mobile testing flow (when triggered)

+-------------------+     +-------------------+     +--------------------+
| 1. Fetch APK      |---->| 2. Boot emulator  |---->| 3. Install app     |
| rock remote-cache |     | agent-device boot |     | agent-device       |
| download          |     | --headless        |     | install <app> <apk>|
+-------------------+     +-------------------+     +--------------------+
                                                             |
+-------------------+     +-------------------+     +--------v-----------+
| 6. Shutdown       |<----| 5. Screenshot +   |<----| 4. Interact +      |
| agent-device      |     | upload to R2      |     | verify via         |
| close --shutdown  |     | (existing infra)  |     | accessibility tree |
+-------------------+     +-------------------+     +--------------------+

APK delivery

The workflow leverages Rock's remote build cache to skip native builds entirely:

+-----------------------------+
| remote-build-android.yml    |  Runs on every push,
| (existing App workflow)     |  caches Debug APK to S3
+-------------+---------------+
              |
              | fingerprint + APK artifact
              v
+-----------------------------+
| S3: ad-hoc-expensify-cash   |  Public-read bucket
| rock-android-*-<hash>/      |  x86_64 compatible
+-------------+---------------+
              |
              | npx rock remote-cache download
              | (no AWS creds needed)
              v
+-----------------------------+
| MelvinBot workflow          |  APK ready for emulator
| .rock/cache/android/        |  install
+-----------------------------+

Primary (Option A): Agent runs npx rock remote-cache download -p android -t developmentDebug which resolves the fingerprint and downloads the matching cached APK from S3. This is the most robust approach - fingerprint matching ensures the correct build for the current native state.

Fallback (Option B): If a ROCK_ARTIFACT_URL exists from a prior Adhoc build, agent uses agent-device install-from-source <url> to download and install directly. Simpler but requires a prior build.

Timeline

Phase 1                   Phase 2                   Phase 3
|----- Local dev -------->|----- Melvin CI -------->|----- Expand ---------->

 - agent-device docs       - Testing strategy        - iOS simulator support
 - Example test flows      - PoC validation          - Broader QA automation
 - SKILL.md in App repo    - Workflow changes         - Usage analysis
                           - Melvin skill
                           - Screenshot upload
                           - End-to-end test

Phase 1 and 2 can be taken in parallel.

Phase 1 - Local developer setup

Deliverable	Description
Developer docs	Instructions for running agent-device against Expensify/App locally (Android emulator/iOS simulator)
Example test flows	Scripted flows for common scenarios (onboarding, expense creation, navigation)
Claude Code skill	SKILL.md in App repo so developers can use agent-device through Claude Code locally

Phase 2 - Melvin CI integration

Deliverable	Description
Testing strategy	Define what Melvin should verify on Android, where testing steps come from (PR Mobile QA section), and test account / auth approach
PoC validation	Verify the end-to-end emulator flow works on CI runners (boot, APK install, app interaction, shutdown) and iterate on any issues
Workflow changes	Add Android emulator + agent-device setup steps to claude-melvinbot-app.yml
Melvin skill	Claude Code skill teaching Melvin when/how to use agent-device for mobile testing
Screenshot integration	Reuse existing R2 upload infra for agent-device screenshots
End-to-end test	Trigger Melvin with a mobile-testable request, verify full flow

Phase 3 - Expand

• iOS simulator support (agent-device already supports iOS; requires macOS runner)
• Broader QA triggers and automation
• Analyze usage and value to reviewers

Workflow changes (detailed)

New steps added to claude-melvinbot-app.yml before Claude Code runs:

# 1. Hardware-accelerated Android emulation
- name: Enable KVM
  run: |
    echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
      | sudo tee /etc/udev/rules.d/99-kvm4all.rules
    sudo udevadm control --reload-rules
    sudo udevadm trigger --name-match=kvm

# 2. Android SDK components (runner already has base SDK)
- name: Install Android emulator and system image
  run: |
    yes | sdkmanager --install "emulator"
    yes | sdkmanager --install "system-images;android-34;google_atd;x86_64"

# 3. agent-device requires Node 22; App requires Node 20
- name: Setup Node 22 for agent-device
  uses: actions/setup-node@v4
  with:
    node-version: '22'

- name: Install agent-device
  run: npm install -g agent-device

- name: Restore App Node version
  uses: actions/setup-node@v4
  with:
    node-version-file: App/.nvmrc

# 4. Install agent-device Claude Code skill
- name: Install agent-device skill
  run: cd App && npx skills add callstackincubator/agent-device

What stays the same:

• Runner: blacksmith-4vcpu-ubuntu-2404 (upgrade to 8 vCPU if insufficient)
• Timeout: 30 min (adjust if needed)
• Playwright setup: untouched, remains available
• Screenshot upload: reuses existing R2 infra (melvinbot-assets.exops.io)

Design decisions

Decision	Choice	Rationale
Platform	Android only (PoC)	Avoids macOS runner cost; Linux + KVM is sufficient for validation
APK source	Rock remote-cache (primary)	Fingerprint matching ensures correct build; public-read S3, no creds needed
SDK setup	Raw sdkmanager	Runner already has Android SDK installed; no third-party action needed
Caching	None for PoC	System image cache saves ~60s; not worth the complexity yet
Node version	Dual setup-node (22 + restore 20)	App engines field locks to 20.19.5; agent-device requires 22
Trigger	Skill-based, on-demand	Emulator only boots when agent decides to test; no overhead on every run
Emulator lifecycle	agent-device manages	Handles boot, install, interaction, and shutdown autonomously
Perception model	Accessibility tree snapshots	~3-4x more token-efficient than screenshots; deterministic element refs

Action items

1. Write developer docs and Claude Code skill for local agent-device usage (Phase 1)
2. Define testing strategy - what to verify, where testing steps come from (PR Mobile QA section), auth approach
3. Add Android emulator + agent-device setup steps to claude-melvinbot-app.yml
4. Write Melvin skill for mobile testing with agent-device
5. Integrate screenshot upload with existing R2 infra
6. End-to-end validation - trigger Melvin with a mobile-testable request, verify full flow

Open questions

1. Where should testing steps come from? The PR template already includes # Mobile QA and # Web QA sections. Proposal: Melvin writes Mobile QA steps when creating the PR, then uses those same steps to drive agent-device verification. This mirrors the human QA process and requires no extra input. When mobile testing is triggered manually (e.g. @MelvinBot test this on Android), it can also pull from the existing PR testing steps rather than requiring the user to restate them. Currently the Playwright app-web-tester receives freeform TASK/VERIFY from melvinbot with no defined source - using the PR testing steps would be an improvement for both web and mobile testing.

2. Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.

PoC implementations

Although still under construction, here are the draft PRs if you'd like to peek a bit more than just high level overview:

• Phase 1 (local dev integration): [NoQA] Add agent-device glue-code skill for mobile testing #87662
• Phase 2 (Melvin integration): https://github.com/Expensify/Expensify/pull/622913

---

I am looking forward for your thoughts and improvements, thanks a lot!

CC @adhorodyski @Julesssss @AndrewGable @justinpersaud

[kacper-mikolajczak]

Hi folks! Let me know what you think and how would you like to see it moving forwards. Thanks!

[Julesssss]

The plan sounds good, I am mostly excited for phase 1 currently. Lets see how this goes before moving onto 2/3.

I think it's fine. You can see that we have many possible flows in our current melvin flow. There are plans to extract these out, but for now we need to make sure the flow isn't confused. For example, web screenshot and upload might be tightly coupled with playwright-app-testing.

Caching | None for PoC

Just thinking ahead, this would be cache of the rock build, and only helpful on the second+ test run?

Open questions

Where should testing steps come from?

Testing:

• Local: Directed by user
• PRs authored by Melvin:
  • We recently attempted to make Melvin verify and list it's changes seperately from engineering verifications
  • To begin with, a few standard app smoke tests would be good, following that we could have run tests related to changes

Runner capacity. Running Android emulator + Claude Code on 4 vCPU may be tight. Will test during PoC validation and upgrade to 8 vCPU if needed.

The app testing workflow runs on the larger Github Action runner currently, so I don't think this is a problem.