Every robotics team has a sim2real story. A policy that works in simulation, deployed on hardware, and something breaks. The grasp misses by a centimeter. The wheel slips on a floor that looked like every other floor. The camera sees a shadow the simulator never rendered.
We call this the sim2real gap, and we measure it at the point where it shows up: deployment. But that is not where it starts. Most of the gap was already there before training began, baked into the simulated world the policy learned from. The simulator was wrong about the world, and the policy inherited the error.
Sim2real is the symptom. Real2sim is the cause.
Two gaps, not one
It helps to split the gap into its parts, because they have different fixes.
The first is dynamics: contact, friction, compliance, actuator latency, sensor noise. This is a physics problem. No camera measures friction, and nothing in this piece claims otherwise.
The second is perception and geometry: what the world looks like and where things actually are. Scale, depth, layout, surface appearance, lighting. This is a measurement problem. And it is the part of the gap the field has largely chosen to route around rather than solve.
Why randomization became the default
Domain randomization works. Legged locomotion and in-hand manipulation crossed into the real world with crude visuals by randomizing textures, lighting, geometry, and physics until the policy stopped caring about any of them.
But look at why it works. You randomize because the base scene is wrong and you don't know in which direction. Every degree of randomization is training signal spent teaching the policy to ignore something, rather than to understand it. For proprioception-heavy tasks that cost is affordable. For vision-conditioned manipulation in a specific, cluttered, real environment, it scales badly. The policy has to be robust to a thousand worlds because it was never shown the right one.
A more accurate starting world doesn't eliminate randomization. It shrinks the budget. You randomize around the truth instead of around a guess.
Where error enters the chain
Real2sim is a chain: capture the environment, reconstruct it, decompose it into objects and surfaces, assign physical properties, simulate. Each stage adds error. But error does not add evenly.
An error at the last stage is local. A wrong friction coefficient on one surface affects one interaction. An error at the first stage is inherited by everything after it. If depth is off, scale is off; if scale is off, reach, clearance, and collision are all off, and no downstream step can recover what was never measured. The reconstruction is faithful to the wrong input.
This is the argument for measured geometry over estimated geometry. Monocular reconstruction, photogrammetry from phone video, learned depth: all of them produce plausible worlds. Plausible is the problem. A policy trained in a plausible world learns plausible reach and plausible clearance, and the real robot discovers the difference on contact.
The evaluation argument
Even teams that never train in simulation need a faithful replica of where the robot will operate. Not a generic warehouse, the warehouse. Not a representative kitchen, that kitchen.
Without it, evaluation is either done on the physical site, which is slow, expensive, and unrepeatable, or in a simulated stand-in, which tests the policy against a world that is not the deployment world. You cannot measure a gap against a reference that was itself never measured.
This is the least arguable version of the thesis. Whatever your view on training in sim, the test bed has to be right, and a test bed is a real2sim artifact.
The variable treated as free
The field has spent a decade improving simulators, physics engines, and policy architectures. It has spent comparatively little on the quality of the world those simulators are handed. Capture is treated as a solved input: point a sensor at the scene, run reconstruction, move on.
It is not solved. It is the noisiest stage in the chain and the one whose errors compound furthest. Real2sim is not a preprocessing step. It is where the gap is decided.
What we are doing about it
DreamVu works on the first link of the chain. Alia is a 360° camera that captures the physical world at 16K with metric depth: real distances in real units, not relative depth that has to be rescaled and guessed into place. Depth is measured, not predicted. Color, depth, and point cloud come from the same capture, so the geometry a simulator receives is the geometry that was there. Everything downstream inherits that.
