Autopilot · write-up

Running the Xenon 2 autopilot in-browser: options

xenondoc/AUTOPILOT_BROWSER_PORT_OPTIONS.MD · 8 KB · updated 2026-09-17

Context

The autopilot (xenon_tools/) is currently a ~49,500-line Python codebase that talks to a desktop build of Hatari over a bespoke TCP socket protocol (xenon_client.py <-> src/xenonControl.c). Hatari itself already has a working WASM/browser build (separate effort, documented in xendondoc/XENON2.MD and xendondoc/remove-asyncify.md). The goal explored here is running the autopilot and the game together in a browser, including on low-powered devices (phones). This is a research/decision document — nothing described here has been implemented.

Two hard constraints came out of the research, independent of which option is picked:

1. No TCP sockets in a browser page. xenon_client.py's transport layer must be replaced with in-page message passing (direct calls, postMessage, or a SharedArrayBuffer ring) no matter what. This is forced regardless of the option chosen below.
2. The core loop is lockstep, not real-time-bound. Hatari's STEP_GAME_FRAME blocks waiting for the client's reply, so a slow decision just slows the game down rather than desyncing or corrupting it. This significantly loosens the performance bar for a first working port — it only needs to be "fast enough to feel responsive," not "fast enough to hit a 20ms deadline."

Everything else about the autopilot is good news for portability: the live decision loop (autoplay.py's ReactiveController) has zero third-party dependencies — no PIL, no numpy, no threading/asyncio — it's pure struct-decoded object/memory state in, joystick mask out. PIL only appears in offline visualization tooling that would never ship to the browser anyway. The hard part of any port isn't dependencies, it's the sheer volume of hand-tuned, disassembly-derived heuristics (~15,270 lines / 202 methods in ReactiveController, plus game_model.py's reimplemented 68000 physics and validate_scripted_motion.py's opcode-interpreter fidelity checks) that must not silently drift in behavior during translation.

Options

A. Pyodide (CPython-in-WASM) in a Web Worker

Run the existing Python almost unmodified inside Pyodide, replacing only xenon_client.py's socket calls with a JS bridge (function calls into the Hatari WASM instance via postMessage or a shared buffer). autoplay.py, world_state.py, world_map.py, game_model.py, xenon_symbols.py carry over essentially as-is; the 409-test suite in test_autoplay.py keeps working unchanged as a regression net.

• Pros: By far the smallest amount of new code to write and the lowest risk of behavioral drift — you're not re-deriving 15K lines of tuned heuristics in a new language, you're just changing the transport. Fastest path to "something runs end-to-end in a browser."
• Tk-based autoplay_ui.py/replay_ui.py don't run under Pyodide, but they're explicitly "exploration tools, not part of the dependency-free emulator build" (autoplay_ui.py:3) and aren't needed for the autopilot to function headlessly.
• Cons: Pyodide's runtime is large (~10-20MB) and has multi-second cold-start init — a real cost on a phone, especially on first load / poor connections. CPython-interpreted per-frame decision logic will be markedly slower than native even inside a browser JS engine, which matters more on low-power devices even though there's no hard deadline. This is a bridge, not the stated end-state the docs already point at.

B. Rewrite the controller/model layer to TypeScript, run in a Web Worker

Port autoplay.py, world_state.py, world_map.py, game_model.py, xenon_symbols.py (and the protocol decode logic currently in xenon_client.py) to TypeScript, running alongside the existing TS web renderer already built for the Hatari WASM port.

• Pros: No second heavy runtime to download/boot — just JS, which is plenty fast for this workload (small per-frame object lists and branchy heuristics, not pixel/CV work — exactly what JS engines handle well). Fits naturally next to the already-existing web/src/*.ts code. Best balance of phone-friendliness and development ergonomics.
• Cons: A full manual port of ~20K lines of delicate, disassembly-derived logic (including game_model.py's bit-exact physics and the scripted-motion opcode interpreter) is real, risky work. The 409-test suite would need porting alongside it (or run in parallel against Python during transition) to catch behavioral drift — translation bugs in this kind of code are easy to introduce and hard to notice without that safety net.

C. Rewrite the stabilized controller to C, compiled into the same WASM binary as Hatari

This is the path the project's own docs already commit to ("once observations and tactics stabilize, the final controller can be moved into dependency-free C" — AUTOPILOT.MD:8-9), extended to target the existing Emscripten/WASM build instead of (or in addition to) a native build.

• Pros: Matches the documented long-term plan. Single WASM binary, no cross-module message passing or serialization at all (autopilot and emulator share linear memory) — best possible performance and smallest footprint, which matters most for low-power phones. Reuses the toolchain already set up (_matej_notes.txt: WSL + emsdk + CMake/Ninja, hatari-wasm target).
• Cons: Same translation-risk profile as option B but in a less ergonomic language for this kind of iteration — C is slower to write/debug than TS for heuristic-heavy branchy logic, with no test-runner/tooling ecosystem as convenient as Python's or JS's. Longest-effort option.

D. Hybrid / incremental — don't port everything at once

Keep iterating on the autopilot in Python natively (as now) for levels/behavior that are still unstable (Level 4 is explicitly in-progress per AUTOPILOT.MD), and only port a level's logic to the browser target once it's validated and stops changing — one level at a time, growing the browser-side subset over time. This can be paired with either B or C as the eventual target language, and optionally with A as a stopgap to get something browser-playable immediately without blocking on a rewrite.

• Pros: Directly matches the project's own stated philosophy (AUTOPILOT_AIERRORS.MD's error #1: premature browser-design was already flagged as a past mistake). Avoids porting code that's still churning. Lets desktop Python remain the fast-iteration environment it was built for.
• Cons: Browser build stays partial/behind for longer; needs a per-level "is this stable enough to port" judgment call.

Recommendation

Don't try to get Python itself running in the browser as the end state — the phone constraint argues against Pyodide's runtime weight and cold-start cost for anything beyond a quick demo. The dependency profile (stdlib-only decision loop, no numpy/CV) means either a TS or a C port is realistic, and TS is the better fit specifically because this is browser work: it sits next to the renderer code that already exists, skips a second toolchain, and JS is fast enough for this workload (small object lists + branching, not pixel math). C matches the docs' stated long-term vision and wins on raw performance/footprint, but costs more to develop against for heuristic-heavy code like this.

Practical path: D (incremental) using B (TypeScript) as the target, with A (Pyodide) optionally as a short-lived bridge if you want something demoable in-browser before any porting starts. Port level-by-level as each level's Python behavior stabilizes (Levels 1-3 already read as validated per AUTOPILOT.MD; Level 4 is still moving), porting test_autoplay.py's relevant cases alongside each piece as a translation-correctness check before retiring the Python version of that level's logic. xenon_client.py's protocol layer gets replaced by an equivalent in-page bridge regardless of path chosen, so that piece can be built first and tested against the existing wire format immediately.