The Data Pyramid in Robotics

On the data bottleneck in robotics, and the types of data being combined to address it

I’m Tanay Jaipuria, a partner at Wing and this is a weekly newsletter about the business of the technology industry. To receive Tanay’s Newsletter in your inbox, subscribe here for free:

---

Hi friends,

Frontier LLMs were pre-trained on trillions of tokens of text, most of it at least initially taken from the web. The internet was, in effect, a free pre-training corpus that humanity had collectively assembled.

In robotics, there’s no such equivalent. That generally means all attempts to solve and get to physical AGI in some form involve working through this idea of the data bottleneck. In this piece, I’ll go deeper into the bottleneck, and introduce a version of the robot data pyramid highlighting the various kinds of data used in robot learning that can be combined to help address it.

I. The Data Bottleneck in Robotics

Open X-Embodiment, the largest open robot dataset, contains around 1M trajectories spanning 22 robot types. It was assembled by researchers pooling 60 existing datasets across 10s of labs. Collection of trajectories of robot data is very slow and expensive.

Sergey Levine of Physical Intelligence, asked on the Dwarkesh podcast how their robot data compares to what goes into training large models, put it this way:

“Maybe a better comparison is to the datasets that are used for multimodal training. And there, I believe last time we did that count, it was between one and two orders of magnitude.”

And that doesn’t even account for the fact that often robot data is byte-dense but signal-sparse (30 frames a second of a gripper slowly approaching a cup contains a lot less information than 30 tokens of text).

π0 reportedly trained on roughly 10,000 hours of robot data, and closing Levine’s one to two orders of magnitude means something like 1 million hours of robot data. And getting to internet-level scale seems basically out of reach. Consider that on YouTube for example, ~300M hours of video are added annually. There’s no activity that will result in users collectively naturally assembling a corpus of robot data a la the internet to LLMs, so the field recognizes different approaches are needed.

II. The Types of Robot Data

NVIDIA’s GR00T team proposed thinking around the idea of a data pyramid: web data at the base, synthetic data in the middle, real robot data at the top, with less volume but more fidelity and embodiment-specificity as you climb. I’d further break down the data pyramid into 7 types, across the three categories they proposed as below :

Using these different forms allows to address the shortage of robot data, by learning from already available or cheaper to collect data. The lower layers supply priors to models (what objects are, how tasks tend to decompose), and the upper layers ground those priors on actual specific hardware and embodiments. Let’s talk about each one.

1. Internet video

YouTube alone holds billions of hours of humans cooking, cleaning, assembling, and fixing things. These typically aren’t enriched and don’t have specific depth or forces, but can help teach a model about common objects, actions, and environments.

Today it mostly enters robot models indirectly, through the vision-language backbone but various attempts like NVIDIA’s DreamZero and Rhoda’s Direct Video Action model are trying to get the most out of this dataset more directly. DreamZero builds its policy on top of a video generation model pre-trained on internet-scale video while Rhoda AI pre-trains on hundreds of millions of internet videos and then adapting to a given robot with as little as 10 to 20 hours of its data.

2. Egocentric video

Over the last year, many of the robotics labs have increased their purchasing of first-person video from head-mounted cameras. Egoverse and Ego4D are open datasets which have 1000s of hours of footage of humans doing common tasks across household and factory / industrial settings.

The interest in this area has come from both approaches like discussed above that pre-train on video, as well as various research that has found that models pre-trained on carefully filtered egocentric human video beat models pre-trained on the same volume of teleoperated robot data, once both were adapted to the target robot.

3. Enriched egocentric / exocentric

Sometimes, egocentric data can be paired with exocentric data and have actions labeled: hand pose tracking, depth, gaze, or even full motion-capture suit. Ego-Exo4D is one example which has around 1,400 hours of first-person and third-person views of skilled tasks.

Tesla for example has also in the past hired workers to perform tasks in mocap suits.

This has benefits, particularly depending on the level of enrichment and fidelity of the labels. For example, some vendors / labs are even collecting data with full tactile forces which are generally one of the biggest gaps with using human data.

4. Handheld capture

Ultimately, the goal is to collect data in a cheap and easily scalable form that can most closely resemble the robot itself. That’s the idea of approaches like UMI (Universal Manipulation Interface), which involves humans going around doing tasks while wearing / holding the end-effectors or grippers that would be on the real robot.

In effect, it allows you to mimic the form factor of the robot without actually needing the robot making it more scalable.

The most notable example leveraging UMI is Sunday Robotics, which was founded by the researchers behind UMI. They ship skill-capture gloves co-designed with their robot’s hands into contributors’ homes, and say their ACT-1 model was trained on around 10 million household episodes collected this way, with zero robot data. Another one is Toyota Research Institute’s Large Behavior Models which uses UMI capture alongside teleop and simulation.

Sunday Robotics Data Capture Glove

5. Simulated data

Physics simulators like Isaac and MuJoCo have been around for a while, and increasingly an exciting development is the potential for world models that act as learned simulators (NVIDIA’s Cosmos, Google’s Genie 3) which could allow robots to evaluate or train their policies in millions of varied virtual environments.

The promise of simulated data is that you’re only bound by compute. The downside today is the sim-2-real gap where the results from sim don’t translate perfectly in the real world as well as issues with physics simulators working on manipulation / deformables and world models not being quite ready for prime-time yet.

However, World Models like Waymo’s and even 1X’s as well as the use of Simulation in the AV industry have shown the promise of this approach. There’s a lot more to say here which I’ll save for a future piece.

6. Teleoperation

Teleoperation involves someone actively operating the robot, whether virtually through VR setups or through a rig setup where the robot mimics the actions.

This allows for collecting extremely high-fidelity and essentially real data on the exact embodiment that is needed.

The issue with this approach is the economics and the difficult to scale. While costs are going down and have been going down, these still cost over $100 an hour and require relatively skilled operators and you’re limited on the number of robots and the setup / environment as well.

Sergey Levine’s Sporks of AGI has a good writeup on the tension around looking for alternative approaches given how difficult it is to scale teleop, but that those approaches all bring a translation problem that tele-op does not.

The general view currently amongst labs is that Tele-op will continue to be needed and won’t go away, but whether the percent of the training mix can involve a lower percent of tele-op (which could be mostly in post-training) because other sources scale easier.

7. Deployment

Ultimately, the data everyone wants is from real robots doing real work in real settings, ideally work someone is paying for. The robot might run fully autonomously, or with a human teleoperator taking over when it hits trouble, similar to how a safety driver handles disengagements in AVs.

The challenge of course is somewhat chicken-and-egg, since you need a robot good enough to deploy before deployment can make it better.

Over time, this data will form the flywheel for companies in terms of deployments leading to data which then helps improve the deployment. Over time, this could become the highest volume data source as deployments increase, but today the practical number of deployments is small other than structured environments or in AV.

Two notes on this data source. One, if the deployments are for extremely narrow tasks in extremely structured environments, that data would be helpful to improve the performance of robot policies on those tasks in those environments but may not help generalize to other similar tasks or more varied environments. Two, in the short term, it’s likely that most of these deployed robots will be deployed as a hybrid autonomous tele-operated manager where human teleoperators take over when the policy fails. This is both a practical way to get deployments ready in more of these unstructured environments and a really valuable form of data of interventions/failures. The idea goes back to DAgger (dataset aggregation) in imitation learning: run the policy, let a human correct it where it fails, and retrain on the corrections. Approaches like Sirius and HIL-SERL build on this.

Closing Thoughts

Progress is coming on all of these fronts at once: approaches to better leverage human video data like Rhoda’s and DreamZero, collection being scaled up across egocentric, UMI and teleoperated data, tactile and force capture hardware and collection that is making egocentric data richer in the middle, world models chipping at the sim-to-real gap, and the first real deployment flywheels starting to spin somewhat.

No one actually knows how much data is enough. Asked whether robots need 100x or 1,000x more, Sergey Levine’s was honest noting: “that’s the thing, we don’t know that.”

For a robotics team, the more practical question might be, what reliability do you need to get to some level of hybrid deployment given a set of tasks, and what is the cheapest mixture of these sources that clears the reliability bar for that task.

If you’re building in physical AI, whether that’s robot foundation models, data collection, or the picks and shovels around them, feel free to reach out at tanay at wing.vc.