Mobile AI agents
A model can suggest an action. The system decides whether it is allowed.
Read-only constraints are enforced at separate layers.
Encrypted credential storage in the orchestration system.
Human-in-the-loop recovery for challenges and uncertain states.
The challenge.
Mobile automation is stateful. Screens change, model outputs can be malformed, and external text can contain instructions the agent must ignore.
The engineering challenge was to connect a production mobile client, an API orchestrator, and a device executor without letting model uncertainty become unrestricted device input.
The boundaries.
- Treat screen content as data, never instructions.
- Discover action candidates from the live UI rather than guessing labels.
- Enforce read-only boundaries independently in three places.
- Escalate challenges to a human instead of trying to bypass them.
How it fits
together.
An Expo client connects to a NestJS orchestrator. Paced BullMQ jobs reach a FastAPI executor that drives Android devices over ADB. Structured model outputs pass context-specific allowlists before an action can execute. Credentials use AES-256-GCM encryption.
The decisions
behind the details.
Capabilities before actions
The visible screen determines a temporary set of allowable actions. Model suggestions are checked against those capabilities.
TradeoffMore observation and validation work on each execution step.
Recovery that fails closed
Malformed outputs and uncertain states should stop execution and allow operator review.
TradeoffSome sessions need human intervention instead of automatic completion.
Evaluate against a control
Frozen ground-truth metrics, a measured noise floor, and never-dosed controls separate intervention effects from feed drift.
TradeoffExperiments require additional control runs; results need their limits stated.