Cosmos 3, released by NVIDIA in May 2026, merges the previously separate Cosmos Predict, Cosmos Transfer and Cosmos Reason models into one: a single inference call returns scene reasoning, future frames and robot actions together. A 2026-09-28 post on the AWS Physical AI Blog (evidence level [O], vendor primary source) breaks down its two-tower architecture and lays out an AWS reference architecture and instance sizing from data curation to on-device deployment. For engineering teams, the most useful part is the conditions attached to the instance sizing: the full Nano and Super checkpoints are currently tested in BF16; eight-GPU SFT on H100 is NVIDIA’s tested configuration; multi-node H200 post-training should be benchmarked against that H100 recipe; and on B200 the sharding or replication layout should be validated before making performance claims.
From three models to one
According to the post, earlier Cosmos releases meant orchestrating three separate models, each with its own inference call, model-loading overhead and serving infrastructure. Traditional physical AI pipelines chain specialised models: a vision-language model describes the scene in text, that text conditions a video generation model, and the generated frames go to a policy network that outputs motor commands. The post argues that every handoff loses context and adds latency. Its example is warehouse picking: a VLM describes a package as “a deformed cardboard box on a conveyor belt”, and the surface curvature and the distance to neighbouring objects are lost in that sentence. Cosmos 3 is described as perceiving the geometry, reasoning about grasp stability, generating the trajectory and emitting commands in one forward pass, with no intermediate text bottleneck. NVIDIA calls it the first fully open omni-model for physical AI.
The capabilities the post lists are: vision-language understanding (text, image, video, audio), a world model (future states generated as video), a world action model (executable trajectories such as joint angles and gripper positions), forward dynamics (future observations predicted from actions), inverse dynamics (actions inferred from demonstrations) and a policy model (action sequences predicted from observations plus a task prompt). The three variants:
| Variant | Parameters | Typical use given in the post |
|---|---|---|
| Cosmos 3 Super | 64B (32B Reasoner + 32B Generator) | Highest-fidelity simulation, research, offline generation |
| Cosmos 3 Nano | 16B (8B Reasoner + 8B Generator) | Production inference, balancing cost and quality |
| Cosmos 3 Edge | 4B | On-device robot inference, low-latency control loops |
Two towers: Reasoner and Generator
Cosmos 3 uses a Mixture-of-Transformers (MoT) design that puts autoregressive reasoning and diffusion-based generation into one jointly trained model. The data flow the post describes:
- Inputs split into two paths. Text and ViT-encoded image patches form the autoregressive subsequence that goes to the reasoning tower; video, audio and actions pass through frozen VAE encoders into latents that go to the diffusion tower. 3D multimodal RoPE (MRoPE) aligns all tokens on one physical time axis, with FPS modulation to handle different frame rates.
- The reasoning tower processes visual and language tokens left to right with causal self-attention (the ViT uses DeepStack multi-layer aggregation) and outputs dense representations that encode its scene reasoning.
- The generation tower processes tokens globally with bidirectional attention and iteratively denoises the latents into video, synchronised audio and action trajectories.
- Dual-stream joint attention: diffusion tokens attend over the concatenated keys/values of both towers, so every generated pixel, audio sample and action coordinate is conditioned on the reasoning; autoregressive tokens stay causal and self-contained.
- One forward pass outputs text reasoning, video, audio and action trajectories (joint angles, gripper commands, 6-DoF paths) at the same time.
Seen through the see-think-act loop from this site’s course episode 0.1, the design removes the model handoff between “think” and “act”. Note that the post only describes the architecture; it gives no latency or success-rate comparison with a multi-model pipeline, so “handoffs lose information” is the authors’ argument, not a measurement.
Four stages on AWS and two ways to serve
The reference architecture covers the full lifecycle:
- Data curation: raw video, LiDAR and camera feeds land in Amazon S3; Cosmos Curator on AWS Batch filters, de-duplicates and quality-scores them; Cosmos Dataset Search runs on Amazon EKS with a Milvus vector database and Cosmos-embed NIM for semantic retrieval.
- Post-training: domain SFT on Amazon SageMaker HyperPod; action post-training adapts the action head to a specific robot’s joint configuration, gripper mapping and 6-DoF end-effector paths. The post says the model’s built-in forward dynamics can run closed-loop tests before physical deployment.
- Production inference: Amazon EKS with Cosmos NIM, or Amazon SageMaker JumpStart.
- Edge: Cosmos 3 Edge (4B) deploys to NVIDIA Jetson via AWS IoT Greengrass, with OTA model updates, and edge sensor data flows back to S3 for retraining.
The trade-off between the two serving options is clear. EKS with NIM targets low-latency workloads such as control loops and real-time scene understanding: Karpenter provisions GPU nodes on demand, nodes are spread across Availability Zones, a new NIM image takes traffic only after passing health checks, and GPU utilisation, queue depth and per-request latency flow to CloudWatch and Prometheus to drive horizontal autoscaling. It suits teams that already run Kubernetes and want control over batching and GPU bin-packing. JumpStart is the managed path: per the post, as of August 2026 all three variants (Edge, Nano, Super) are available there; real-time endpoints serve interactive requests, asynchronous endpoints handle long-running world-generation jobs without client-side timeouts, and checkpoints produced on HyperPod and registered in the SageMaker Model Registry deploy to the same kind of endpoint.
On security, the post specifies: SSE-KMS encryption on all S3 buckets; VPC endpoints for S3/ECR so that traffic does not egress over the public internet; HyperPod clusters and EKS node groups in private subnets; IAM roles scoped per stage (curation, training, inference, edge); device certificates for edge devices, rotated on a schedule, with OTA channels restricted to signed model artifacts; PrivateLink for customer or safety-critical data, and CloudTrail logging on every model artifact promotion.
Instance sizing: the caveats the post states itself
The post points out that the full Nano and Super checkpoints are currently tested in BF16; the Cosmos3 Generator NIM has separate FP8 profiles with different memory and parallelism requirements.
| Workload | Recommended instance | Guidance in the post |
|---|---|---|
| Nano single-GPU inference | g7e.2xlarge or p5.4xlarge | One RTX PRO 6000 96 GB or H100 80 GB; BF16 for the full checkpoint; FP8 may be used with supported NIM profiles |
| Nano throughput or post-training | p5.48xlarge | Eight H100s, matching NVIDIA’s tested eight-GPU SFT configuration |
| Super full-model inference | p5e.48xlarge or p5.48xlarge | Use the runtime-recommended HSDP, CFG and Ulysses layout rather than assuming TP8; H200 gives more memory headroom |
| Multi-node post-training | p5en.48xlarge | Eight H200s with EFAv3, suitable for NCCL collectives, but benchmark against NVIDIA’s tested H100 recipe |
| Blackwell inference or training | p6-b200.48xlarge | Eight B200s, 1,432 GiB aggregate GPU memory; validate the runtime’s sharding or replication layout before making performance claims |
The conditions in the table are worded differently. The setup the post explicitly calls NVIDIA-tested is eight-GPU SFT on H100; multi-node H200 post-training should be benchmarked against that H100 recipe; on B200, the runtime’s sharding or replication layout should be validated before making performance claims. FP8 may be used with supported NIM profiles; the post does not say how far FP8 has been tested. Nor does it say who ran the BF16 tests or whether other configurations were tested. On the edge side, this site’s “Frontier Radar deep dive | Edge and real-time” covers NVIDIA’s tutorial on post-training Cosmos 3 Edge for on-device robot control, along with other latency data; experience with VLA post-training on HyperPod is in “Frontier Radar deep dive | Training and self-improvement”.
Limitations
- This is an AWS vendor blog post ([O]), and its statements about model capabilities come from NVIDIA’s description. The post contains no benchmark numbers for quality, latency or success rate, and the material this article relies on includes no independent evaluation. “The first fully open omni-model” is NVIDIA’s own claim.
- One of the post’s architecture subsection headings is wrong: item 1 describes the input split but carries the same heading as item 2, “Autoregressive Tower (Reasoning)”. This article reads item 1 as input processing, based on its content.
- For the Edge variant, the post gives only 4B total parameters and does not split them into Reasoner and Generator as it does for Nano and Super.
- “Validating policies in closed loop with forward dynamics before deployment” means validating inside the model’s own predictions; whether that can replace real-robot evaluation is not discussed in the post.
- The GitHub repository and sample-code links at the end of the post are not in the material this article relies on and are not cited here. The post’s cost claims (GPU price reductions, Spot savings) are also left out; check AWS’s official pricing pages for prices.
Links
- AWS Physical AI Blog: NVIDIA Cosmos 3 on AWS: Omnimodal World Models for Physical AI (2026-09-28)
- Physical AI Lab 0.1: What Is Physical AI? From Chatting to Doing (the see-think-act loop)
- Physical AI Lab course overview (the L3 plan includes an episode on world-model data generation with Cosmos)
- Frontier Radar deep dive | Edge and real-time: where the latency budget of VLAs and world action models goes
- Frontier Radar deep dive | Training and self-improvement: three stages of VLA post-training and task collapse in online RL
