The Simulation Stack in Robotics
On physics engines, world models, and the opportunities for startups in simulation
I’m Tanay Jaipuria, a partner at Wing and this is a weekly newsletter about the business of the technology industry. To receive Tanay’s Newsletter in your inbox, subscribe here for free:
Hi friends,
A few weeks ago I wrote about the data pyramid in robotics, walking through the different sources of data labs are using to train robot foundation models.
I touched on simulation in it, which is what I’ll go deeper on today. Simulation is promising because it can allow for scaling data based on compute, which makes it one of the more promising ways to work through the data bottleneck.
At a high level, simulation gets used for a few things today:
Generating synthetic training data for models to learn from
As an environment for reinforcement learning, where policies practice and can hill-climbed toward higher success rates.
Evaluation and testing policies cheaply before they ever run on hardware.
Given how expensive and difficult it is to get robots into deployment, being able to “simulate” at scale in a much cheaper manner is extremely helpful.
Broadly, there are two camps on how to build the simulator itself. The first is physics-based simulation, using engines that directly model rigid bodies, joints, and contact. The second is via world models which are neural networks typically trained on video that learn to predict what happens next given actions taken in it.
In this piece, I’ll go deeper into both camps, the gap between simulation and reality, where I see startup opportunities, and close with how the labs are using simulation today.
I. Physics-Based Simulation
Physics simulators have been around for a while. MuJoCo (now maintained by Google DeepMind) has long been used in academia and beyond, NVIDIA’s Isaac Sim and Isaac Lab are used extensively commercially, and newer entrants include Genesis (open sourced out of a CMU-led group) and Newton (built jointly by NVIDIA, Google DeepMind, and Disney Research).
Most of these are open source, and all of them now run in parallel on GPUs. A single machine can run 1000s of environments at once, so a robot policy can accumulate years of experience in hours via simulation.
The promise of physics simulation is that data scales with the cost of compute once the environment is built. The simulator also gives you perfect ground truth (poses, contacts, forces) that would be difficult to label in real-world data, and robots can fail as many times as they need to without any cost.
There has been a lot of evidence for simulation working for example in locomotion. Most humanoid and quadrupeds today are trained with reinforcement learning in simulation and then transferred to hardware, including things like stair climbing and recovering from pushes.
It’s also worked for some dexterity. OpenAI’s Rubik’s cube result back in 2019 came from training entirely in simulation in MuJoCo with heavy domain randomization.
The common thread is that these are tasks where the physics that matters are aspects like rigid bodies, hard contact and the robot’s own dynamics which the engines model well.
The limitations of physics engines come for deformable objects like cloth and cables, liquids, suction grippers, transparent objects, and touch or force sensing are generally difficult to model well. Most manipulation tasks need some of these, which is a big part of why sim worked for walking well before it worked for manipulation, and robots can do complex dance manouvers but not relatively simple manipulation.
II. World Models
A somewhat more nascent and emerging approach towards simulation involves “learning the simulator” from data. An action-conditioned world model takes in recent video frames plus an action, and generates what the camera would see next.
As a quick aside, as Fei-Fei Li, CEO of World Labs, touched on in a recent piece world models can be renderers (output pixels), simulators (predict how state evolves under actions), and planners (choose actions). The focus of this piece is on the role as simulator in the context of robotics, but there are other use cases as well, whether in entertainment or other industries and even in robotics as a brain itself1
The appeal is that these models learn from video, so there’s no manual authoring of assets or scenes, and they can in principle represent all the messy stuff like cloth and liquids that physics engines handle badly. Google DeepMind’s Genie 3 can generate an interactive world from a text prompt in real time, and NVIDIA has been pushing its Cosmos family of world models into robotics pipelines.
I don’t think they’re ready to replace the physics engine as a training environment quite yet in robotics but could be in the future, as Waymo and others have shown in AV.
Currently, Genie 3 stays coherent for a few minutes at 720p, while training a policy ideally would need hours of consistent interaction. Also, there are still open questions how well these will actually learn to model the complete physical realities of the world versus mimicking or approximating the physics.
There are a couple of uses of World Models that work quite reliably today. One is evaluation: 1X trained a world model on data from its NEO home robot and uses it to score candidate policies before they go onto their robots, and found that the better policy in their world model maps to reality 90% of the time.
The second is data generation and augmentation. Models like Cosmos can re-render an existing robot demonstration into 1000s of variations with different conditions, which can help scale up real-ish data. This is particularly interesting in the context of using World Models to solve some of the challenges of physics engines around content, since building a single realistic environment with correct articulation, masses, and friction is weeks of manual work, and generative models are starting to remove that constraint by reconstructing real environments into simulation.
World Labs is the most notable example here. They acquired SceniX, a real-to-sim startup and published results from an engine that captures a real robot, environment, and task, reconstructs it as an aligned simulation, and generates 1000s of variations of that world to train and evaluate policies in with some strong results.
If you don’t yet receive Tanay’s newsletter in your email inbox, please join the 10,000+ subscribers who do:
III. The Sim-to-Real Gap
While simulation is promising, the reason it hasn’t simply solved the data bottleneck is that both camps eventually hit a wall that there ends up being a delta between the policy trained in sim vs reality, often known as the sim-2-real gap.
The common approaches to fix this are typically domain randomization (varying things like friction, mass, etc) and system identification.
Real-to-sim work of the kind mentioned above helps as well since reconstructing the actual deployment environment well makes the sim environment match reality better.
World models also have their own sim-to-real gap which one can think of as hallucinations. In particular, the model can end up drifting from performing real physics particularly as time goes on, which can result in policies trained inside it thinking they did things right or wrong which may not have been the case. Here, the biggest lever to improve is training better world models.
IV. Opportunities
For startups, I see a few big opportunity areas in and around simulation:
Building a great world-model simulator. The models are largely there on visual quality but aren’t yet fully there for physics, and whoever can deliver the best physically-grounded world model for robotics can become part of the de-facto simulation stack. There are a number of companies trying to build the best world models for this use case and others such as World Labs, Spaitial, General Intuition, OneWorld and others.
High-fidelity assets and environments for simulation. Physics engines are only as useful as the content inside them, and most generated assets today are visual meshes without the joints, masses, and friction that make them sim-ready. Lightwheel, for example, supplies sim-ready assets and environments to the labs.
Simulation infra for better/easier sim. In my mind this can be something that sits above both sets of stacks providing a better experience to use sim across things like locomotion and manipulation. Some versions of these could be agentic interfaces over the sim engines, a cloud platform that bakes in better techniques like asset generation, domain randomization, system identification and sits above the simulation platforms. Antioch and Drift are two companies building in and around this.
Closing Thoughts
I wanted to close with how the labs actually use simulation today based on the publicly disclosed information.
As you can see from below, most labs use simulation to some extent, but some use it lightly while others use it more heavily. Genesis, Skild and NVIDIA tend to be the most sim-heavy ones, where as Generalist likely uses sim the least. 1X is probably the one that has talked most about their use of world models for eval and simulation, whereas Genesis is one of the few that even built their own physics simulation engine (which they open sourced) as part of their stack.
It is worth nothing that even the most sim-heavy labs pair simulation with real data and that simulation shouldn’t be viewed as a substitute. I think a “recipe” of sorts that seems to be emerging is internet video, egocentric data and real demonstrations for pre-training, simulation RL to harden policies, and sim or world models for evaluation, with the mix varying by lab and by task.
I expect the share of the training mix coming from simulation and synthetic data to keep growing especially as world models improve and the cost to use physics engines from an asset creation perspective continue to go down.
If you’re building in and around simulation for robotics, please feel free to reach out at tanay at wing.vc. I would love to chat!