Edge OTA: firmware, software, and model delivery to Jetson fleets

Status: Draft for review | Date: 2026-07-23 | Backlog: AURA-759 (Robot Runtime & Integration) | Research: .claude/plans/2026-07-23-edge-ota-model-delivery companion research note

1. The problem

A policy trained and evaluated inside Auraison has to end up running on a robot. Today that last step is manual: someone copies weights onto a Jetson, restarts a container, and hopes the arm still closes its gripper. That does not scale past a bench unit, and it has no answer for the failure that matters most on a physical machine, which is the update that leaves the robot unable to boot or unable to act safely.

The difficulty is that a Jetson is not one thing you update. It is five layers with different cadences, blast radii, and rollback stories: the boot firmware in QSPI flash, the Jetson Linux BSP and kernel, the OS root filesystem, the application containers (the ROS 2 stack and the inference server), and the model artifact itself. A design that treats "update the robot" as a single operation is how robots get bricked. This document sets the architecture for delivering all five over the air, with the model channel — the part no off-the-shelf platform solves well — as the piece Auraison builds.

The fleet is small and NVIDIA-centric: an AR4 arm, a TurtleBot, a USV, and a drone, most on Orin-class modules running stock JetPack. The recommendation is sized for that reality, not for a thousand-robot product, and it names where the choice would change if that scale ever arrives.

2. The five update layers

The boot chain on Orin lives in QSPI-NOR flash and runs BootROM → MB1/MB2 → UEFI. Jetson Linux supports bootloader A/B redundancy: two sets of boot partitions, updates written only to the inactive slot, and automatic fallback to the known-good slot when the updated one fails to boot after a few retries. Secure boot authenticates the chain against RSA keys fused into the SoC and, on Orin, extends that authentication through UEFI Secure Boot to the kernel; NVIDIA documents OTA with secure boot and disk encryption both enabled (Jetson Linux Developer Guide, Update and Redundancy).

The BSP and OS layers have two NVIDIA-supported OTA paths. Debian-package updates from NVIDIA's APT server (apt upgrade, apt dist-upgrade) move userspace and kernel packages but are not atomic; a power cut mid-dist-upgrade can leave an inconsistent system. Image-based OTA is the fail-safe path: a host-side script builds a signed payload, nv_update_engine writes the inactive rootfs or bootloader slot, and an unexpected reboot mid-write falls back to the previous slot. Rootfs A/B redundancy is a distinct feature layered on bootloader A/B and is the mechanism third-party platforms orchestrate remotely.

Application software is the near-universal container story: ROS 2 nodes and the inference server ship as docker-compose sets, updated far more often than the OS. The model artifact is the least standardized layer, and it carries a Jetson-specific wrinkle. A TensorRT engine is compiled for a specific device generation and JetPack version, so a deployable model is often the chain weights → ONNX → engine with a hardware and JetPack compatibility constraint attached. That constraint is the reason a model version has to bind to the software and BSP version that consumes it, a theme that runs through the rest of this design.

3. Platform choice

The recommendation is Mender, self-hosted, for the OS and application layers, with an Auraison-built model channel on top.

Mender is the one mature OTA platform with an explicit, NVIDIA-co-developed integration on Jetson's native bootloader and rootfs A/B redundancy that works on stock L4T-Ubuntu, with no move to Yocto required (Mender × NVIDIA). That matters because VLA work chases CUDA, cuDNN, and TensorRT currency, and L4T-Ubuntu is where that currency is easiest to keep. Mender self-hosts, which fits how Auraison already owns its control plane rather than renting a device cloud, and its update-module system layers the three cadences through one deployment and reporting pipe: image A/B for the BSP and rootfs, a docker-compose module for the containers, and a model channel for the frequent case.

The runner-up is FoundriesFactory, built on the Linux microPlatform with OSTree and an aktualizr-lite client. It has the strongest security architecture in the field, TUF metadata end to end with atomic OSTree deploys and signed docker-compose apps, and documented Jetson AGX Orin support (Foundries LmP Jetson target). Three costs demote it for now: it requires leaving stock JetPack for a Yocto-built platform, it is a paid SaaS, and it is Qualcomm-owned, an awkward dependency for an NVIDIA fleet. If Auraison ever ships robots at fleet scale, Foundries or Toradex Torizon (which runs a full Uptane client and is now available on Jetson) is the point to revisit.

The full field is compared in the companion research note. Two entries are worth stating here because they close off tempting shortcuts. NVIDIA Fleet Command targets NVIDIA-Certified x86 servers, not Jetson modules, so it does not apply. Kubernetes at the edge (k3s, KubeEdge) is orthogonal: it can run on top of the chosen OS layer but never replaces firmware or OS OTA, and reaching for it as "the update system" leaves the layer that actually bricks robots unmanaged.

The deciding trade-off against Foundries is security model. Mender ships signed artifacts and TLS device auth, not TUF/Uptane. Auraison closes that gap in its own control plane by borrowing Uptane's shape (described in section 6) rather than by adopting a Yocto platform to inherit it.

4. The model-update contract

This is the layer Auraison builds, because no platform enforces it. A model release is an immutable, signed bundle: a manifest, the weights as safetensors, an optional precompiled TensorRT engine per target, and the processor and config files. The manifest carries identity and provenance (name, semver, content digests, the W&B run id, the training commit, the lakehouse snapshot reference) and a compatibility contract: the runtime image digest or version range it needs, the observation and action schema version, and the JetPack and compute-capability target for any compiled engine. The consuming runtime advertises its own contract version and refuses to load a bundle it cannot satisfy. A model that is individually valid but paired with an incompatible runtime is treated as a failure, not a deployment.

The control plane publishes the bundle to the artifact store and signs the manifest with an offline-rooted key. Transfer is delta-aware where it pays: Hugging Face's Xet content-defined chunking moves only changed chunks, which helps most when base weights are shared across policy versions (adapters, LoRA, config-only changes) or when the device already holds a related version (Xet deduplication). A full fine-tune shares few chunks across runs, so the honest expectation there is a near-full, resumable, digest-verified download rather than a cheap delta.

Bundles stage into a versioned directory on a data partition the OS A/B slots never touch, so an OS rollback cannot delete a staged model. The on-device promotion is itself A/B: current and candidate symlinks point into versioned directories, and promotion is an atomic rename. The gate between staging and promotion is where a robot's safety is won or lost:

1. Verify the signature and content digests.
2. Check the compatibility contract against the running stack; abort if unmet.
3. Load-test the candidate. Triton's explicit model-control mode gives the right property here: a failed reload leaves the incumbent loaded and serving (Triton model management).
4. Run a shadow or replay evaluation against the incumbent on recent logged episodes.
5. Promote by symlink rename and runtime reload, only while the robot is idle.
6. Apply guarded first-run limits, then commit after a health window; on any failure, repoint current to the previous version.

Every transition reports back to the control plane. A coupled model-and-runtime change ships as one deployment with a single health window and a single rollback decision, so the two never drift out of contract.

5. Where it sits across the planes

Three owners, with a clean split of responsibility.

The control plane owns build, signing, and dispatch: it bundles and signs a model after training and simulation evaluation, runs the Mender server, and holds fleet state. The sim-in-the-loop release gate runs here, before a bundle is ever dispatchable, and it is the cheapest high-value check because the platform already produces simulation. The project's deployments strand owns intent: which model version and application release go to which cohort of robots, the staged-rollout policy, and the promotion and rollback history. This is the director role, composing only what the registry has already blessed. The on-device agent, the Mender client plus an Auraison model agent, owns the mechanics on the robot and must be able to finish or revert any update with zero connectivity.

6. Safety, and what Uptane teaches

Automotive OTA is the rigor benchmark, and its lesson is a threat model: assume some signing key or server will eventually be compromised, and stay safe anyway. Uptane splits the decision into two repositories that must agree (Uptane). An image repository, signed by offline keys, holds metadata for every valid image. A director repository, using online keys, decides which valid images a specific device should install now. A client installs only what both agree on, so a compromised online director cannot ship an image the offline-keyed repository never blessed.

Auraison borrows the shape without adopting the full framework. The artifact registry signs what is valid with offline-rooted keys; the project deployments strand directs what runs where with online credentials; the two stay separate. Metadata carries explicit expiry, so a robot offline for a month re-verifies freshness before it acts on a stale instruction. The compatibility contract is the concrete form of Uptane's mix-and-match defense: a valid model in an invalid combination with its runtime is exactly the attack class the contract refuses.

The robotics-specific rules follow from never bricking and never acting unsafely. Anything that can prevent boot goes only through A/B with boot-attempt counting; package-manager dist-upgrade on a fleet robot is out, because power loss mid-upgrade is unrecoverable where A/B is recoverable by construction. An update is not successful at reboot or reload; it is successful when a health suite passes, meaning the ROS 2 graph is up, controllers respond, inference latency is within budget, and the watchdog is fed. Until that window passes, the previous slot or model stays one atomic step away. A new policy is a safety event rather than a software event, so it earns a sim gate in the control plane, an on-device shadow comparison against the incumbent, and guarded first runs at reduced authority before it drives actuators at full speed. Rollout across the fleet is a canary, one TurtleBot before the AR4 before the USV, not a big-bang push.

7. What this does not cover

The design is scoped to delivery and on-device lifecycle, not to the training that produces a model or the sim evaluation that gates it, which live in the control and user planes. It assumes the fleet stays on stock L4T-Ubuntu; a move to Yocto (and with it Foundries or Torizon and a full Uptane client) is named as the fleet-scale revisit, not built now. It does not specify the wire protocol of the on-device model agent or the manifest schema field by field; those are the first implementation tasks under the issue below. Bring-your-own model registries beyond R2 and HF Hub, and multi-tenant fleet isolation, are out of scope until the management plane exists.

8. Next steps

This is a plane-crossing capability with a new API surface and a new on-device agent, so under the planning policy it needs this merged design doc before issues are cut. Once merged, the work lands as a single issue under AURA-759, "Edge OTA: signed model + software delivery to Jetson fleet (A/B, validation gate, rollback)," with a pilot on one Jetson (the TurtleBot or an AR4 bench unit): a self-hosted Mender server, the rootfs A/B integration, a docker-compose update module for the ROS 2 stack, the model bundle spec and signing, the on-device model agent with its validation gate, and a deployments record in the project wired to the existing jobs API. AURA-758 (Robot Learning) owns only the evaluation-gate content, the sim and replay checks a policy must pass, if that grows into its own issue.