3D pipeline / Simulation
From video
to a robot test.
What has to change before a reconstructed scene becomes something a robot can interact with?
- Status
- Completed pilot / Ongoing exploration
- Tools
- Blender / COLMAP / Brush / MuJoCo / AI agents
Starting with an artist’s question
I wanted to understand what I could contribute as a 3D artist to a real-to-sim workflow. A reconstruction can look convincing, but it still needs editable geometry, usable materials and collision shapes that behave properly.
I worked with an AI agent to take a public video reference through reconstruction, Blender preparation and a small robot-interaction test. I kept pushing the experiment beyond isolated asset repairs: could the scene be inspected, edited and used as a whole?
Video & supplied cameras → Reconstruction comparison → Editable Blender scene → Materials & collision geometry → MuJoCo interaction
Preparing the scene
The environment pilot used the public pixel5-table video sequence and its supplied camera poses. COLMAP and Brush were compared before preparing the scene in Blender. The resulting environment contained 18 pieces, 11,136 triangles and nine material maps, reviewed through seven cameras and three lighting setups. It was later combined with separately sourced Panda and YCB mug assets.
I asked for materials that could be separated from captured lighting, rather than treating the original appearance as a finished asset. Scale, hidden surfaces and physical properties still involved estimates. The trolley was fixed in place for the robot test, and the material work was an authored interpretation, not measured recovery of real materials.
Choosing a collision budget
The visual mesh and collision model had different jobs. The agent's first detailed mug collider used 67 convex pieces and 3,492 proxy triangles to preserve the cup and handle openings. A corrected compact version used 15 pieces and 348 triangles, and was the version carried into MuJoCo.
Both hollow versions passed five rest tests, four pushes and the cavity probe, with 16 geometric handle-clearance rays staying clear. Those rays do not establish finite-gripper access. The compact version's final Blender push positions differed from the detailed reference by up to 3.10 mm, so the broad checks do not establish precision equivalence. A single convex hull was cheaper in this test but closed the functional openings.
For the same four-second +X push, median Blender test time was 0.594 seconds for the detailed collider and 0.389 seconds for the compact one. This compares three interleaved runs per collider and includes scene reload, setup and 481 measured frames; it excludes Blender startup, rendering, result-file writing and artist work. It is a whole-test measurement, not an isolated collision or physics benchmark. More objects, greater contact density and parallel scenes remain untested.
Testing the interaction
A Panda arm from MuJoCo Menagerie picked up the separately prepared YCB mug from the stationary trolley and placed it at a target. The grasp used one finger inside the rim and one outside. A programmed controller used exact object positions; this was a contact-and-control pilot, not a learned policy.
The evaluation used a small, deterministic 5 × 4 grid of 20 nearby starting positions: five X values and four Y values, with yaw fixed per Y row (−8°, 0°, +8°, 0°). The controller received the exact object pose. Twelve predefined checks covered the task, including lift, placement, settled release and unintended contacts; the maximum placement error recorded in this grid was 3.103 mm. Separate controls checked an open gripper, support removal and cavity access. This characterizes the chosen setup, without estimating reliability across new scenes or unknown poses.
The failures were useful
In an earlier Blender asset-validation test, the first compact mug collider had 15 pieces and 332 triangles. It passed rest and push tests and a central floor ray, yet left a test sphere 8.91 mm above the intended contact height. An axial floor point had been included in the wall profile, producing inward slopes. Correcting the profile produced the 348-triangle version, and its tests were rerun. This happened during validation of the mug collider, before the Panda task.
A separate mesh-preparation issue came from a 1 mm bevel on a 2 mm wheel edge. It produced 128 collapsed UV triangles per disk. Reducing the bevel resolved that audit failure.
These cases made the validation requirement concrete: visual inspection, mesh audits and functional contact probes each address different ways an asset can fail.
My part, and the agent’s
I defined the question and the requirements for an editable, reviewable scene, redirected the experiment toward that complete workflow, and reviewed the geometry, materials and multi-view results. The AI agent implemented the scripts and technical checks, and made many of the modeling, material and simulation decisions. My part was directing the scope, evaluation and integration. This was a collaborative experiment using existing tools.
What the pilot established
For this pilot, moving from reconstruction to interaction meant adding collision geometry and physical assumptions, then validating the scene visually and through contact tests. The prepared scene supported scripted pick-and-place under known poses and tightly bounded conditions. A simpler hollow collider retained the interactions required by the tests, while the cavity-profile and UV failures exposed problems that basic checks had missed.
No robot policy was trained, visual perception was not evaluated and nothing was transferred to a physical robot. These tests do not establish physical fidelity, generalization or scalability, or measure artist productivity or the causal benefit of artist intervention.
The next controlled test should keep the controller, physics settings and acceptance criteria fixed while varying collider complexity first, then object count and contact density. Measuring physics-step time separately from loading and logging, alongside contacts, penetration and task outcomes, would show which simplifications remain useful as the scene grows.
Sources & reuse
The trolley, wall and floor were reconstructed and interpreted from pixel5-table in Otto Seiskari and Jerry Ylilammi’s (2024) Gaussian Splatting on the Move — Smartphone Dataset, version 1.0. The source includes video and supplied camera data. Our changes include structural reconstruction, estimated materials and scale, collision geometry, robot-scene assembly and simulation rendering.
The dataset-derived video and poster on this page are distributed under CC BY-SA 4.0. That license applies to these visual adaptations, not to this site’s original text or other content.
The mug is derived from YCB Object and Model Set, Berkeley 025_mug, by Berk C. Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel and Aaron M. Dollar, under CC BY 4.0. Changes include structural reconstruction, a nonuniform nominal-size adjustment and a hollow collision proxy.
The Franka Emika Panda model is from MuJoCo Menagerie, derived from Franka Emika’s robot description, under Apache-2.0. This experiment adds the mount, scene and scripted controller; original vendor geometry is retained. These credits identify sources and do not imply their endorsement.