Autopilot · write-up

Incremental renderer terrain observations

xenondoc/AUTOPILOT_INCREMENTAL_TERRAIN.MD · 7 KB · updated 2026-09-17

Architecture

Terrain remains a persistent map for every level. Renderer draws remain the source of live changes, including destructible walls; resident offline level maps continue to seed known geometry. This change does not replace navigation or alter its tactics.

The update stages are independently selectable in compare_kernels.py:

• c-map-old: the original per-cell observation and invalidation rules.
• c-map-incremental: compare visible renderer rows with their previous inputs.
• c-map-batched: incremental rows plus one cache invalidation per changed frame.
• c: the preceding stages plus native draw projection and row comparison.

Unchanged rows skip _observe and its geometry processing. Their MapCell bookkeeping copies retain exact first/last frames and observation counts. These copies are deliberate: replay snapshots share historical cell values and must not be corrupted by in-place counter updates. Tile coordinate keys are cached per row rather than reconstructed every frame.

Rows with changed input retain the existing observation rules. A destructible cell that is still solid and missing from an unchanged renderer row still passes through _observe: it needs its second fully visible absence to prove opening. A clipped edge resets that proof as before. Static walls remain monotonic.

Occupancy changes still increment revision per cell, preserving all revision semantics. Navigation grids, routes, and pixel raster caches are invalidated once at the end of the update instead of once per changed cell. Direct _observe calls retain immediate invalidation and invalidate that row's comparison history. Map reset, resident seeding and RAM synchronization discard observation history.

Native API

ABI 18 adds named XapTileDraw, XapTileImage, and XapObservedRow structures, with matching ctypes structures. xap_observe_tile_rows projects all wall draws into the visible tile band and compares its rows in one C call. It handles clipped sources, multi-tile rectangles, negative coordinates, ties-to-even projection, and overlapping draws in renderer order. A separate occupancy mask allows sprite ID zero to remain distinct from an absent tile.

Input, output and previous-row buffers are caller-owned and disjoint. Immutable packed row history belongs to PersistentWorldMap and is copied shallowly with replay snapshots. C allocates nothing, retains no pointers, and calls no Python callbacks. The Python bridge bulk-packs draws into the named ctypes structure array and lazily decodes a whole row once when changed geometry or a pending destructible disappearance requires examination. This avoids allocating ctypes rectangle/draw wrappers for every input and cell wrappers for every output.

This ports projection/comparison, not the entire persistent map: Python retains cell bookkeeping and destruction confirmation. Navigation continues to use the existing shared native wall raster and masks. Unchanged geometry keeps those buffers valid; confirmed occupancy changes invalidate them through the same map owner. The standalone native library and WASM export are maintained; no Hatari driver integration or WASM profiling is included.

Diagnostics and validation

Replay/live diagnostics expose per-update changed/unchanged row counts, native batch count and actual cache invalidations. The test suite passes 922 tests, including new projection differentials, every cell's bookkeeping and revision, clipped disappearance proofs, gate reappearance, namespace resets, historical snapshot isolation, direct observation changes, and invalid native input.

Measurements

The Level 1 recording was replayed over frames 2367–10059 (7,693 observations, 5,425 gameplay frames), with sequential workers pinned to CPU 4 and two repeats in opposite variant orders. Times include the full measured observation pipeline.

Row reuse is the material improvement; batching is too small to distinguish from noise in the total time. The initial ctypes wrapper erased the native projection savings, so input packing and output decoding were changed to work in bulk.

A fresh two-repeat comparison of the final implementation measured:

The final total is 3.14% lower than the original. The native bridge is only 0.34% ahead of Python incremental updates: treat that difference as noise, not a proven native speedup. C remains enabled with the native kernel, and Python projection remains available for the Python kernel and isolated comparisons.

Gameplay averages 13.808 unchanged versus 1.057 changed rows per update: about 93% of visible rows avoid repeated geometry processing. Cache invalidations fall from 0.014 to 0.001 per gameplay observation; this recording has few new occupancy changes after resident-map seeding, explaining the small batching benefit.

All variants in both timing experiments produced identical decision verdicts. Artifacts under xenon_tools/run_logs/: - terrain-observation-timing-0908/comparison.json: isolated initial stages. - terrain-observation-bulk-0908/comparison.json: final repeated timings.

Profile

A separate profiled old/new replay confirms the work removal:

That is 94.8% fewer cell-observation calls and 56.6% less profiled map-update time. These instrumented times explain the work distribution; the uninstrumented 3.14% result above is the end-to-end performance estimate. The native wrapper still costs 0.613 cumulative seconds across 7,692 calls, while stable metadata copies remain in Python to preserve snapshot semantics. Further elimination would require separating historical diagnostics from the live map, rather than another projection micro-optimization.

Profile artifact: terrain-observation-profile-0908/comparison.json and its c-map-old-1.pstats / c-1.pstats files. Its decision verdicts also match.

Cross-level validation

Final old/new replay comparisons have zero changed verdict frames in all checked recordings: Level 1 (7,693 observations), Level 2 (961), Level 3 (225), Level 4 (900), and Level 5 (5,652). The latter four reports are under terrain-observation-level{2,3,4,5}-0908/comparison.json. They are coverage checks, not repeated speed estimates or fresh closed-loop gameplay runs.

The native kernel was rebuilt with hatari_dev.ps1 -Action build-kernel. The final full Python suite passes 922 tests; log: terrain-observation-final-tests-0908.txt. git diff --check also passes.