Autopilot · write-up

Native linear prediction and collision-shape preparation (2026-09-06)

xenondoc/AUTOPILOT_NATIVE_LINEAR_KERNEL.MD · 15 KB · updated 2026-09-17

This document records the linear port and its ABI-4 cleanup. The subsequent combat extension uses ABI 5, adds target indices to body descriptors and retains the Python driver. Measurements below remain the historical linear-port results.

The previous typed body-query implementation was committed first as 16334ce0 (Add typed batched native autopilot body validation). This extension removes repeated constant-velocity fitting and per-frame body-shape packing. It retains the Python backend and the ordered Python destruction fallback.

Two separately measured changes

1. Reuse motion parameters in Python. autopilot_v2/linear_prediction.py computes an eligible hazard's observed collision rectangle and stable world velocity once per observation. Future steps reuse those parameters with the original floating-point arithmetic. This is enabled by prediction_reuse for both kernel backends. The cache is replaced at each observation; replay snapshots cannot accidentally acquire another frame's parameters.
2. Prepare linear sweeps in C. The C backend packs a 40-byte XapLinearBody once per eligible body per observation. C constructs its future swept bounds while checking the existing batches of up to eight player hulls. Those bodies no longer require Python Rect/Group construction and slice packing for each native frame. Existing scripted formation and eye paths keep their path model.

Eligibility follows the original motion and collision-shape dispatch, through two small overridable hooks in autoplay.py. A generic velocity alone is not enough: an animated or specialized collision hull must retain its own model. Scripted motion, linked predictions, active shell motion and extra safety envelopes keep their reference paths. Probing eligibility does not eagerly build scripted trajectories already covered by the formation broadphase.

The body-only exclusion in PredictionService.group_slices deliberately keeps the source's anticipated shots. Moving a shooter body to C must not remove its projectiles. Possible contacts still trigger the complete ordered Python pass, where existing weapon simulation can resolve destruction. Walls, homing, level tactics, cash and pickup decisions remain with their existing implementation.

Representation and ABI

The shared native/WASM ABI is now 4; rebuild the native DLL. Older DLLs are rejected by the wrapper. All public inputs use named C structs mirrored by ctypes.Structure.

XapLinearBody contains an integer XapIntRect, signed 64-bit Q32 X/Y velocity, numeric hazard kind and camera-coupling flag. Q32 means 32 fractional bits: for example, 1.5 pixels/frame is stored as 1.5 * 2**32. Static bodies use zero speed. Bounds must be integral and within the documented coordinate/extent limits; velocity is limited to 64 pixels/frame and the horizon to 1,024 frames.

Fitted velocities are rounded to the nearest Q32 value when packing the descriptor. A value such as 1/3 is handled in C; it does not trigger Python preparation. The maximum rounding drift is less than 0.00000012 pixels per axis over the 1,024-frame limit, so no extra margin or special precision fallback is needed. Geometry generation uses integer/fixed-point arithmetic, then converts to the common floating-point clearance representation. Existing XapPoint, XapRect, generic slices and distance results retain doubles. This extension does not claim that every predictor or distance calculation is integer.

Python owns the packed observation buffers. C borrows them for a call, allocates nothing and retains no pointers. No map transfer is needed: this kernel does not read maps. The caller still controls when an observation is replaced.

Simplified validation and results

xap_validate_body_scene(paths, path_count, points, point_count, linear, linear_count) checks the immutable descriptors and all referenced trajectory points once after packing. BodyScene calls it once per observation. Those buffers must remain unchanged until the scene is discarded; another caller modifying them must revalidate before querying. There is no opaque context, allocation, registration token or retained native pointer.

xap_query_bodies retains pointer/count, path indexing, frame, output-capacity, query-value and dynamic-slice checks per batch. It does not rescan the immutable scene's coordinate values, velocity ranges or type flags. Overflow-sensitive fixed-point ranges are established during scene validation. The caller's immutability obligation is part of the native API contract.

XapBodyResult now contains only clearance and an int32 contact flag (0/1). The unused contact index and hazard kind are removed. Actual contact identity still comes from the ordered Python destruction-aware check when needed. The result remains 16 bytes because of native alignment; this cleanup is not a claimed reduction in returned bytes. Both ctypes and WASM use the same layout.

A prepared-buffer comparison against the saved ABI-3 DLL used 12 paths (972 points), 5 linear bodies, eight queries per batch and four opposite-order passes of 2,000 observations pinned to one CPU. It includes ctypes and the new validation call, but excludes packing and the rest of the autopilot:

The one-time call/full-path scan adds roughly 1.5 microseconds for a scene used only once. Seven batches are effectively unchanged at this scale. This is an interface/validation cleanup, not a demonstrated whole-controller speedup. Both versions returned identical contact flags and clearances for these queries. Raw samples and DLL hashes: body-simplification-micro-0906-01.json.

Native batching requires the formation broadphase and an exactly restored motion suffix of at least eight steps. A supported linear body is now sufficient to create a scene even when there is no supported full path. New search branches remain lazy Python checks; the port does not invent motions to fill batches.

Validation

• 849 tools tests pass with the native library available.
• Tests cover fractional fitted motion, per-observation invalidation, specialized animated hulls, preservation of shots, C/Python struct layouts, rounded Q32 velocities (including 1/3 using C), negative speeds, camera correction, horizon limits and unchanged output after invalid native input.
• Scene validation covers future points beyond the first query batch and rejects invalid fixed-point ranges before querying. Multiple batches validate their shared scene only once; changing slice/query data still receives per-call checks.
• Native and WASM agree with Python on 160 mixed scenes / 1,280 queries, including paths, linear descriptors and generic slices.
• The rebuilt WASM module also matches 1,316 eye cases / 654,496 points.

Vectors: xenon_tools/run_logs/native-linear-vectors-0906-01.json and the existing native-eye-vectors-0906-01.json. The ABI-4 body vectors are native-simple-body-vectors-0906-01.json. WASM checks use check_native_bodies_wasm.mjs and check_native_eye_wasm.mjs.

./xenon_tools/hatari_dev.ps1 -Action build-kernel
python -m unittest discover -s xenon_tools -p "test_*.py"
python xenon_tools/autoplay.py --planner maneuver --navigation-backend spans --kernel-backend c

Measurement method

compare_kernels.py --variants c-baseline,c-reuse,c separates the two changes:

• c-baseline: previous C eye/body behavior, new parameter reuse disabled and no linear descriptors.
• c-reuse: Python parameter reuse enabled, no linear descriptors.
• c: parameter reuse plus native linear sweep preparation.

These are fresh-process benchmark controls, not extra production backends. The two controls use the same current DLL with zero linear descriptors. Thus the timing comparison isolates the new stages instead of crediting them for previous C ports. Use --kernel-backend python or --kernel-backend c in production.

Plain runs use two opposite-order passes pinned to one permitted logical CPU, spans navigation, all architecture features, identical recordings/checkpoints and terrain sidecars. Timings include reconstruction, policy, planning, packing and ctypes overhead; they exclude emulator/rendering work. Source, DLL and input hashes are recorded. cProfile is run separately and is used only to attribute work, not to estimate end-to-end gains.

For example, from xenon_tools, using a new output directory for each run:

python compare_kernels.py run_logs/prediction-reuse-corridor-0906-02.validation/prediction-reuse-corridor-0906-02.x2events --from-frame 801 --to-frame 2555 --variants c-baseline,c-reuse,c --repeats 2 --pin-cpu --output run_logs/linear-comparison

Add --cprofile, use --repeats 1 and a separate output directory for attribution.

Repeated plain timings

These ABI-3 measurements precede the simplification to rounded velocity conversion and the ABI-4 result/validation cleanup. They used an exact-conversion gate, since removed. Historical timings and exact replay matches below are retained as measured; they have not been remeasured for the rounding change.

Means below average the two opposite-order passes. Percentages compare each stage with the preceding stage; combined reduction compares full C with baseline.

Both corridor and boss passes improve at each stage. The Level-5 sample makes zero native body calls: c-reuse and c execute the same relevant work there, yet their timing differs. The reuse pass also varies by 3.7%. Treat those Level-5 differences as noise, not a native speedup or a demonstrated parameter-reuse gain.

The average pass p95 changes from 51.218 to 49.664 ms in the corridor and 22.607 to 22.940 ms at the boss. This is a mean-time improvement, not evidence of consistently improved tail latency. The remaining slower search/repair frames still need Python validation and specialized preparation.

Actions, tactics, authority, verification, contact frame/kind/key, kills, pickups and reported clearance match across 2,899 observations, in both passes. This is replay comparison against the observed trajectories, not a new closed-loop campaign or phone/browser-driver benchmark. Existing Level-5 unverified outcomes remain unchanged; its special tactics are still separate work.

Reports with per-frame timings, counters, differences and fingerprints:

• Corridor
• Boss
• Level 5

Work and boundary size

Per-observation averages include frames that do not use the native scene.

Generic slice bytes fall 45% in the corridor and 96% at the boss. Existing path bytes remain; these reductions concern the generic slice payload, not the entire native scene or total Python memory. More corridor frames can use C because a linear-only scene is now supported. Calls remain batched, never one per body.

With Python reuse alone, there are 1.867 parameter builds and 98.696 reuse hits per corridor observation (boss: 1.827 builds and 98.577 hits). Adding native preparation reduces those hits to 37.151 and 31.916: much of the repeated Python step work itself is now absent. These counters refer to the generic motion fallback, not every use of velocity estimation elsewhere in the policy.

Separate cProfile attribution

Corridor profile: linear-kernel-corridor-profile-0906-01/{c-baseline,c-reuse,c}-1.pstats. All compared verdicts match. The table reports cumulative times, which overlap and must not be added. Use the plain timing table for controller speedups.

Parameter reuse removes repeated fits. Native preparation then removes 108,011 generic Python sweep calls and more packing work, while player-motion work stays identical. The remaining native preparation time includes specialized models and anticipated shots still supplied as Python slices. This port reduces a measured cost; it does not eliminate heterogeneous Python prediction or search costs.

Boss profile: linear-kernel-boss-profile-0906-01/{c-baseline,c-reuse,c}-1.pstats. All 844 compared verdicts match. It independently confirms the same pattern:

Replay diagnostics

The shared autoplay/replay maneuver summary now includes linear_parameter_builds, linear_parameter_hits, native_linear_bodies and native_linear_bytes. Detailed counters also record unsupported native inputs and the existing body calls, frames and slice bytes. Recording metadata identifies ABI 4 and the linear_bodies kernel alongside eye_chain and body_queries. Historical recorded diagnostics remain distinct from current replay analysis. The shared summary also shows native_body_scene_validations alongside query calls, so repeated batches can be distinguished from scene preparation.