AR4 workload envelopes and Cloudflare Containers feasibility

Status: Evidence record for AURA-808 Phase 0 | Date: 2026-07-25 | Backlog: AURA-809 (Phase 0 showstopper spike), AURA-808 (edge/cloud split)

1. Why this measurement

The AURA-808 Phase 0 table enumerates five candidate showstoppers for mapping a Cloudflare Container to a ROS 2 node and resolves each from documentation. Four concern networking and lifecycle. None asks the simpler question that comes first: do the AR4 workloads fit inside a Cloudflare Container at all. We measured two live workloads to answer that from data rather than from the datasheet, and then checked the answer against Workers AI as well, since Containers are not the only Cloudflare surface that could plausibly host part of this pipeline.

The first is the Gazebo simulation, sim-tabletop, running tabletop.world with ar_model:=mk5 on the workstation. MoveIt, Servo, the recorder, and the policy server were not running, so this is the simulator alone. The second is a SmolVLA fine-tune on gpu-node-3, which had been running twelve minutes when sampling began and shared the machine with a policy inference server, a second simulator, and the Zenoh pair. Neither measurement was taken on an idle host, and the training one in particular was taken under contention.

2. The simulation envelope

Thirty samples at three-second intervals over ninety seconds, while the simulation held a real-time factor of 1.000 at a 1 ms physics step:

The workstation contributes 24 cores and 125.5 GiB, so the simulation occupies roughly a tenth of the machine. The absolute figure matters more than the fraction: 2.37 vCPU and 2.49 GiB is small by workstation standards and large by edge-container standards.

Four processes account for it. The physics server takes 148% CPU and 778 MB, the Gazebo GUI 70% and 900 MB, the ros_gz parameter bridge 15% and 44 MB, and robot_state_publisher under one percent and 21 MB.

3. What headless actually removes

The compose file offers a sim-tabletop-headless service under the headless profile, and gazebo.launch.py now passes -s --headless-rendering --render-engine ogre2. The container we measured does neither. Its live argv is gz sim -r -v 4 --physics-engine gz-physics-bullet-featherstone-plugin --render-engine ogre2, with no -s and no --headless-rendering, and a gz sim gui process runs alongside the server. The launch file is baked into the image at install/, and only worlds, meshes, and the bridge config are live-mounted, so the running image predates that source change. Reasoning about "the headless service" from the compose file alone overstates how headless the current image is.

Dropping the GUI removes 70% CPU and 900 MB, putting headless simulation at roughly 1.7 vCPU and 1.6 GiB. That is the number to carry into a placement decision.

Headless does not remove rendering. The world declares a scene_camera at 640×480 and 15 Hz, and the bridge publishes /wrist_camera/image and /wrist_camera/camera_info as well. Under --headless-rendering, ogre2 still rasterises both sensors, offscreen through EGL instead of onscreen through GLX. The pixels still have to be drawn by something.

4. The training envelope

The job is lerobot-train fine-tuning lerobot/smolvla_base on ar4_pick_place_furinno_v5v6v7v8v9v10_clean: 149 episodes, 62,652 frames, batch size 32, 100,000 steps, eight dataloader workers, vision encoder frozen, VLM reduced to 16 layers. Of 450M total parameters, 100M are learnable. Thirty samples at three-second intervals:

Throughput held at 2.4–2.8 steps per second, about 81 samples per second, with 0.383 s of optimiser update against 0.010 s of data loading per step. The dataloader is not the constraint; the job is GPU-bound, and the board sits at its power limit at 86 °C. At that rate the 100,000-step run needs roughly eleven and a half hours, which is about 51 epochs over the 62,652 frames. Loss was 0.051 with gradient norm 1.26 at step 1,900.

The CPU figure needs a caveat. The host is a 16-core Threadripper PRO 3955WX carrying a load average of 27, because the policy inference server was simultaneously drawing 673% CPU and holding 3.0 GiB of video memory, and a second simulator another 125%. The 101–782% range for the training container reflects that contention rather than the job's intrinsic demand.

Two operational notes fall out of the same inspection. The root filesystem is 91% full with 46 GB free, and a run that writes checkpoints every 50,000 steps will want more than that. The W&B API key is passed as a plain environment variable and is therefore readable by anyone who can run docker inspect on the host; it should move to a file-mounted secret or a Docker secret, and it should be rotated.

5. Against the Cloudflare Containers envelope

Cloudflare offers six predefined instance types, the largest being standard-4 at 4 vCPU, 12 GiB of memory, and 20 GB of disk. Custom instances share those ceilings, with at least 3 GiB of memory per vCPU and at most 2 GB of disk per GiB of memory. An image may be no larger than the instance's disk, and an account may store 50 GB of images in total (Containers limits).

Headless physics at 1.7 vCPU and 1.6 GiB fits standard-4 with room, and fits standard-3 at 2 vCPU and 8 GiB only if nothing shares the instance. The 4.72 GB image needs standard-2 disk or larger and consumes a tenth of the account image budget. On capacity alone the simulation passes.

The training job fails on every axis at once, which is worth stating precisely because it is the clearest case in the analysis. It wants 4.2 vCPU against a 4 vCPU ceiling, 16 GiB of resident memory against a 12 GiB ceiling, a 19.2 GB image against a 20 GB disk that must also hold the dataset and checkpoints, eleven and a half hours of continuous execution against a runtime that sleeps when traffic stops, and 9 GiB of video memory on a platform with no GPU at all. No instance type and no rework changes that. Training belongs on gpu-node-3 or an equivalent, and the question is closed.

6. Workers AI admits models, not code

Containers are not the only Cloudflare surface with GPUs behind it. Workers AI runs models on Cloudflare's GPU fleet without the caller managing infrastructure, so it is the natural place to ask whether the policy could live at the edge even though the simulator and the trainer cannot.

It cannot, and the reason is admission rather than capacity. Workers AI serves a curated catalogue of about fifty open-source models organised by task: text generation, embeddings, image classification, object detection, speech recognition, translation, text-to-image. There is no task type for a vision-language-action policy, and SmolVLA is not in the catalogue. Bringing your own weights is possible only as a LoRA adapter, and the constraints are narrow: under 300 MB, rank 8 with some support up to 32, files named adapter_config.json and adapter_model.safetensors, and a model_type of mistral, gemma, or llama (LoRA adapters). The AR4 fine-tune satisfies none of that. It trains 100M parameters of an action expert on a SmolVLM2-derived backbone and writes a full checkpoint, not a low-rank adapter over a supported base. Private custom models exist as a path, but through a custom-requirements form rather than as a self-serve capability.

Training is excluded a second time here for a simpler reason: Workers AI is inference-only. Nothing in the product runs a training loop.

Throughput was never the obstacle and it is worth recording why. The AURA-808 inventory puts policy inference at roughly 0.6 predict-requests per second, about 36 per minute, against Workers AI rate limits of 300 requests per minute for text generation and higher for most other task types (Workers AI limits). The AR4 policy would fit the rate budget comfortably. It simply cannot be admitted as a model.

Taken with section 5 this closes the general question, and the shape of the answer is worth stating plainly because it recurs. No Cloudflare surface runs arbitrary code against a GPU. Containers give arbitrary code without a GPU; Workers AI gives a GPU without arbitrary code. Every negative result in this document is a restatement of that one gap, which is also why section 9 looks outside Cloudflare to close it.

7. The rendering showstopper

Cloudflare Containers have no GPU, so sensor rendering in a ported simulator has to fall back to software. Mesa is present in the image, including libEGL_mesa.so.0 and the full DRI driver set, so ogre2 would find a software EGL path rather than fail to start. It would run on llvmpipe.

We did not measure llvmpipe rasterisation of these two camera streams, and what follows is inferred rather than observed. The simulation already spends 2.37 vCPU while a discrete GPU absorbs the rendering at a third of its capacity. Moving that work onto the same four vCPU that must also run 1 kHz Featherstone physics leaves very little margin, and Gazebo answers CPU starvation by dropping the real-time factor below one. A simulator running at a fraction of real time is not useful for episode collection and is not cheaper than a GPU host.

This is a sixth showstopper and it is absent from the AURA-808 Phase 0 table: offscreen GPU rendering for sensor simulation has no Cloudflare equivalent. Unlike the networking blockers, no bridge design routes around it. It binds any simulation carrying a camera, which is every simulation the AR4 policy pipeline depends on.

8. Runtime configuration that has no Cloudflare equivalent

The simulation container is coupled to its host in five ways, none of which survives the move. It runs privileged: true, joins the host network and IPC namespaces, binds the X11 socket and an Xauthority file, requests all NVIDIA devices through deploy.resources.reservations.devices, and bind-mounts four host paths for worlds, meshes, and the bridge config. A port would replace the bind mounts with baked-in or R2-sourced assets and drop the rest outright.

Networking is the constraint AURA-808 already identified, and the measurement does not soften it. The simulator is a DDS participant publishing /joint_states, /tf, the two image topics, and the joint_trajectory_controller interface. Cloudflare admits only HTTP, routed through a Worker to a Durable Object and then to the container, and states that end users cannot make non-HTTP TCP or UDP requests to a container instance (Containers architecture). Nothing outside the container can subscribe to any of those topics. Either the whole ROS graph runs inside one container and only results leave over HTTP, or the Zenoh-over-WebSocket bridge in the AURA-809 plan works, which is still unproven.

Lifecycle adds a smaller constraint that still changes the design. Disk is ephemeral, containers sleep when traffic stops, and a restart may land in a different region. An episode-recording run has to stream MCAP or LeRobot output to R2 as it goes rather than write locally and upload at the end.

9. The hybrid Cloudflare and Hugging Face option

The gap identified in section 6 is specific enough to shop for. What the AR4 pipeline needs is a host that runs an arbitrary container against a GPU, on demand, reachable over HTTP. Hugging Face sells exactly that, and the two platforms compose rather than compete: Cloudflare keeps the control plane it is good at, Hugging Face supplies the GPU substrate Cloudflare does not have.

Jobs is the closer fit of the two Hugging Face products. It runs any Docker image from the Hub or Docker Hub on a chosen hardware flavour, billed by the second, with volumes mounted from model repositories, dataset repositories, or Storage Buckets, and with secrets encrypted server-side rather than passed as plain environment variables (Jobs guide). The hardware table includes an rtx-pro-6000 flavour at 23 vCPU, 256 GB of memory, 475 GB of storage, and one RTX PRO 6000 with 96 GB of video memory for $2.75 an hour, which is the same GPU that gpu-node-3 runs. The measured job would therefore transplant without retuning. At the observed 2.4–2.8 steps per second the 100,000-step run takes about eleven and a half hours, or roughly $32 per run on that flavour. Smaller flavours fit the memory profile too, since the job holds only 9 GiB of video memory and 16 GiB of host memory: l40sx1 at $1.80 an hour and a10g-large at $1.50 an hour both have headroom. Whether either is cheaper per run depends on how much wall-clock they add, which we have not benchmarked and should not guess.

The 30-minute default timeout is the one trap. It is configurable up to days, and a training job that does not set it explicitly will die four percent of the way through.

Jobs also dissolves the rendering showstopper from section 7, which is the more interesting result. An arbitrary image on a GPU flavour runs headless Gazebo with EGL exactly as the workstation does, so camera-bearing simulation becomes possible on rented hardware at $0.40 an hour for t4-small or $0.80 for l4x1. The lifecycle caveat from section 8 softens rather than disappears: a job is still batch-shaped and fire-and-forget, but a timeout measured in days is a different proposition from sleeping when HTTP traffic stops. Scheduled jobs and repository webhooks give a scheduling surface for sim regression runs, and the DDS constraint is unchanged, so the whole ROS graph still has to live inside the one container.

Inference Endpoints covers the piece Workers AI rejected. It deploys custom containers, not only catalogue models, with autoscaling and Hub-integrated weight loading (Inference Endpoints), which is the shape ar4_policy_server already has. The existing POST /predict service would move as-is.

The glue is documented rather than improvised. Cloudflare AI Gateway lists Hugging Face among its supported providers, so the caching, rate limiting, retry, and observability layer already in production for the local vLLM server (AURA-705) extends to a Hugging Face-hosted policy without new machinery. One thing to verify before committing: the provider integration is written against Hugging Face's own inference APIs, and we have not confirmed it proxies a custom-container dedicated endpoint as cleanly as it does a catalogue model.

That yields a three-tier split. Cloudflare holds Workers, Durable Objects, D1, Access, R2, and AI Gateway. Hugging Face holds Jobs for training and headless simulation, Inference Endpoints for policy serving, and Hub repositories or Storage Buckets for datasets and checkpoints, which is the same question AURA-770 already opened against R2. The workstation keeps everything real-time, hardware-attached, and safety-critical, unchanged from the AURA-808 conclusion.

Cost is the reason this stays an option rather than a recommendation. gpu-node-3 is owned hardware, so the marginal cost of the measured run is electricity: roughly 5 kWh at the observed 287 W board draw plus host overhead, which is on the order of a dollar against $32 rented. Owned hardware wins by more than an order of magnitude per run while it sits idle enough to absorb the work, and it did not, during this measurement, since the training job was competing with an inference server for the same sixteen cores. The honest framing for Phase 2 is that Hugging Face buys elasticity and parallelism, not cheaper compute: the case for it is running twenty sim-regression jobs at once, or training while the workstation is busy, not replacing a box we already paid for.

10. Verdict

Neither workload runs on Cloudflare Containers as it stands. Training is excluded permanently and on every dimension. The simulation image is excluded as configured, because it wants a GPU, a host network, privileged mode, and four host directories.

A reworked simulation can run there under one condition: self-contained, camera-free, and batch-shaped, running the full ROS graph inside a single container and emitting results over HTTP to R2. Physics-only simulation of that kind fits standard-4 comfortably and is a reasonable target for trajectory validation, controller regression tests, and reachability checks in CI.

Any variant that renders camera sensors needs a GPU host. That rules out Cloudflare Containers for the simulation workloads the policy pipeline actually depends on, independently of how the Phase 1 Zenoh spike turns out.

Cloudflare-only is therefore the wrong question, and the useful answer is the split in section 9. Cloudflare keeps the control plane, Hugging Face Jobs and Inference Endpoints supply the GPU-backed container host that no Cloudflare product offers, and the workstation keeps everything real-time. Nothing about that changes the edge boundary AURA-808 already drew; it only names where the non-real-time GPU work can go when gpu-node-3 is saturated.

11. What this does not prove

The simulation measurement covers one service on one world. The full stack adds MoveIt, Servo, and move_group, and dataset collection adds the recorder and the policy client; none were running, so 2.37 vCPU is a floor for the stack rather than an estimate of it. A scene with more geometry than tabletop.world would raise both physics and rendering cost.

The training measurement was taken under heavy contention on a 16-core host running a load average of 27, so its CPU range describes what the job got rather than what it wants. Ninety seconds beginning twelve minutes into an eleven-hour run also says nothing about checkpoint-write spikes or late-run memory growth. The GPU utilisation, power, and temperature figures are device-wide and include the inference server sharing the board; only the 9.0 GiB video-memory figure is attributed to training alone.

The llvmpipe estimate in section 7 is inferred from the GPU offload we observed, not measured. Confirming it takes one run of the headless service with LIBGL_ALWAYS_SOFTWARE=1 and the NVIDIA device request removed, which is the cheapest remaining experiment and worth doing before the Phase 3 decision. Nothing here measures message rates or bytes per second on the ROS topics, which is what the Phase 2 cost model needs; network_mode: host prevents Docker from attributing that traffic, so those numbers have to come from ros2 topic bw.

The Hugging Face side of section 9 is read from documentation and pricing tables, not from a run. We have not deployed either image there, so the claim that the training job transplants to rtx-pro-6000 without retuning rests on the GPU being the same model, and the claim that headless Gazebo renders on a Jobs flavour rests on it being an ordinary CUDA container rather than on an observed EGL context. The eleven-and-a-half-hour figure carried into the cost estimate is extrapolated from step rate at 1,900 steps of 100,000, measured under contention, so the rented-cost figure inherits that error. A single short job on t4-small running the sim image headless would settle both questions for a few cents, and should precede any commitment. The electricity comparison assumes a whole-host draw we did not meter and excludes capital and depreciation entirely.