Description
Getting an AI to tap around a phone usually means a jailbreak, a Mac running automation, or a developer account and XCTest. esp32-iphone-agent takes another route: a few-dollar ESP32-S3 board plugged into the iPhone serves as both eyes and hands, with no jailbreak and no computer in the loop.
The eyes are AirPlay. The board opens its own Wi-Fi hotspot, shows up in Screen Mirroring as a receiver and forwards the H.264 stream untouched to the app on the same phone. The hands are USB HID. To the iPhone the board is a standard mouse and keyboard, so with AssistiveTouch on, the pointer acts as a finger: a click is a tap and a drag is a swipe. Screenshots go to any OpenAI-compatible vision model, which picks the next move, and the app sends it to the board over Bluetooth.
Finds its own pointer: a mouse only moves relatively, so the app re-homes the pointer in a corner and wiggles it before precise taps, spots it on screen and corrects any miss
Planner and actor: one model breaks the task into steps and another produces actions such as tap, swipe, draw, type, key press, home and finish, and you can set them separately
Runs in the background: keep working in other apps while a task runs, with the Dynamic Island showing what it is doing and a stop button within reach
Resumes after a drop: if the board, hotspot or mirroring link breaks, the run pauses and picks up from the same step once it returns
Step by step replay: every run is stored on the phone and can be replayed with taps and swipes drawn over the screenshots, which makes it easy to see where the model went wrong
Bring your own model: vLLM, SGLang, llama.cpp, Ollama or a cloud API all work, and the prompts are plain text files in the prompts folder you can edit and rebuild
Shortcuts integration: saved tasks can be launched from Siri, Shortcuts, widgets and Control Center
The eyes are AirPlay. The board opens its own Wi-Fi hotspot, shows up in Screen Mirroring as a receiver and forwards the H.264 stream untouched to the app on the same phone. The hands are USB HID. To the iPhone the board is a standard mouse and keyboard, so with AssistiveTouch on, the pointer acts as a finger: a click is a tap and a drag is a swipe. Screenshots go to any OpenAI-compatible vision model, which picks the next move, and the app sends it to the board over Bluetooth.
Features
Finds its own pointer: a mouse only moves relatively, so the app re-homes the pointer in a corner and wiggles it before precise taps, spots it on screen and corrects any miss
Planner and actor: one model breaks the task into steps and another produces actions such as tap, swipe, draw, type, key press, home and finish, and you can set them separately
Runs in the background: keep working in other apps while a task runs, with the Dynamic Island showing what it is doing and a stop button within reach
Resumes after a drop: if the board, hotspot or mirroring link breaks, the run pauses and picks up from the same step once it returns
Step by step replay: every run is stored on the phone and can be replayed with taps and swipes drawn over the screenshots, which makes it easy to see where the model went wrong
Bring your own model: vLLM, SGLang, llama.cpp, Ollama or a cloud API all work, and the prompts are plain text files in the prompts folder you can edit and rebuild
Shortcuts integration: saved tasks can be launched from Siri, Shortcuts, widgets and Control Center
