Chapter 6: There Is No Single Simulator — Roles of Isaac Sim, Isaac Lab, and MuJoCo

Overview

“Which simulator should I install?” is the wrong first question. A tabletop project needs several different services: robot and scene assets, camera and depth rendering, rigid/contact dynamics, thousands of parallel learning environments, ROS 2 integration, and repeatable regression tests. One product can cover several services, but no engine is automatically best at all of them. Simulator choice is therefore an allocation of responsibilities, not a brand contest.

The NVIDIA stack is easiest to understand as layers. OpenUSD represents a composed scene; Omniverse technologies support scene interchange and applications; Isaac Sim is a robotics simulation application with sensors, assets, physics, and ROS bridges; Isaac Lab adds robot-learning tasks and environment structure; PhysX or emerging Newton executes physics responsibilities. MuJoCo is a separate dynamics and modeling stack with compact MJCF models and strong control/contact experimentation. MJX and MuJoCo Playground extend batched learning workflows. These layers overlap, but their names are not synonyms.

This chapter shows when Isaac is stronger for sensor-rich digital twins and GPU-parallel learning, when MuJoCo is valuable for RL and contact-rich or in-hand experiments, and how to connect them without assuming fragile live co-simulation. Status is current to 2026-07-14. The result is a staged installation and validation plan that prepares the first-motion runbook in (Chapter 7).

After reading this chapter... - You can assign asset, application, learning-environment, physics, ROS, and validation responsibilities to named components. - You can choose a simulator workflow for classical planning tests, RL, imitation/VLA data, or agentic robotics. - You can explain why contact-rich work may justify MuJoCo while sensor/digital-twin work may favor Isaac Sim. - You can move a robot across URDF, USD, and MJCF while testing invariants instead of trusting conversion. - You can connect engines through task semantics, datasets, checkpoints, and regression logs without requiring live lockstep.

1. Separate the NVIDIA stack into responsibilities

OpenUSD, Isaac Sim, Isaac Lab, and PhysX or Newton are respectively asset, application, learning-layer, and physics responsibilities rather than interchangeable product names. ^{[1] [5] [3] [4]} OpenUSD composes prims, references, variants, transforms, and material/physics schemas. Isaac Sim supplies a robotics application around scene authoring, sensors, articulation, physics, synthetic data, and ROS 2 integration. Isaac Lab structures tasks, observations, actions, rewards, resets, and parallel environments. The physics backend resolves dynamics and contact.

This map prevents two category errors. First, an OpenUSD asset is not a validated dynamics model merely because it loads. Second, a learning environment is not the simulator kernel; it defines experiment semantics on top of one. Record the version and owner of every layer because a change in importer, application, task code, or physics backend can alter results.

Isaac Sim sensor rendering and ROS 2 bridging do not make it the robot safety controller or prove contact fidelity. ^{[5] [5] [19]} A ROS 2 bridge can publish camera, joint, and clock data and accept simulated commands. It does not certify the real arm, tool, table, or stopping path. A photoreal image does not validate friction, compliance, tendon, or impact behavior.

Figure 6.1 — Responsibility boundaries in the NVIDIA simulation stack and the separate real control and safety boundary. Original synthesis.

2. When Isaac Sim and Isaac Lab are the natural starting point

Choose Isaac Sim when the central risk is scene and sensing: multiple RGB/depth cameras, lighting, occlusion, calibration, synthetic data, camera placement, complex fixtures, or a digital-twin review. It also helps when a team already maintains USD assets or needs one composed scene for perception, planning, operator visualization, and robot-learning experiments. Isaac Lab becomes relevant when the same scene must generate many task instances and parallel policy rollouts ^{[14] [5]}.

For classical motion planning, use Isaac Sim to validate joint directions, collision geometry, TCP, camera frames, planning-scene geometry, and the ROS 2 command/state interface. Do not start by maximizing visual quality. A gray-box scene with correct scale and collision is a better first gate than a beautiful scene with wrong inertials.

For imitation or VLA work, the simulator can test camera coverage, observation schema, action representation, timestamp alignment, and dataset writer behavior. Synthetic images may augment perception, but they do not substitute for real camera calibration and failure data. Agentic robotics can use the scene for high-level plan dry runs and recovery-state tests; deterministic controller and safety authority remain outside the language agent.

Isaac Lab namespace and workflow changes during the 4.5 migration make tutorial version pinning mandatory. ^[3] Copying an older tutorial into a current installation can fail because package names, imports, configuration, or launch workflow moved. Pin Isaac Sim, Isaac Lab commit/release, driver, CUDA/OS, asset revision, and example branch together. A tutorial is evidence only after its exact tuple passes locally.

Newton 1.0 became released software in March 2026, superseding beta-only descriptions at the cutoff. ^{[6] [7] [4]} This dated update matters because a 2025 beta article is not the current support contract. Newton remains a backend choice to qualify per task, not a reason to rewrite a stable PhysX experiment without measured benefit.

Isaac installation and smoke-test ladder

1. Freeze supported OS, GPU/driver, Isaac Sim, Isaac Lab, physics backend, ROS 2, and repository revisions.
2. Launch the vendor's minimal example before importing the custom robot.
3. Load a static robot; verify meters, axes, joint limits, mass, and collision shapes.
4. Command one articulation in simulation without ROS; compare state and expected sign.
5. Add one camera; verify intrinsics, extrinsics, timestamps, depth units, and frame names.
6. Add ROS 2 bridge; replay the Chapter 4 interface and time gates.
7. Add Isaac Lab task only after standalone scene invariants pass.
8. Save a deterministic seed and short regression trajectory before tuning throughput.

3. Why and when MuJoCo is needed

MuJoCo is useful when the research question is dominated by compact dynamics, control, contact, tendons, equality constraints, actuator semantics, or rapid iteration over physically meaningful parameters ^[8]. This often applies to in-hand manipulation, underactuated or tendon hands, force/impedance experiments, and RL tasks where thousands of control steps matter more than camera realism. “Needed” does not mean Isaac cannot represent contact; it means MuJoCo may expose the relevant model and solver knobs more directly and with a smaller experiment surface.

MJCF exposes actuators, tendons, constraints, friction loss, force ranges, and solver parameters for compact dynamics and contact experiments. ^{[9] [8]} URDF can describe kinematic links and joints, but a dexterous hand may need MJCF-specific actuator, coupling, tendon, contact, and sensor semantics. Explicitness is valuable only if parameters are measured and validated; more knobs can also create more ways to tune to a demonstration.

For contact-rich in-hand work, build tests below the task-success level: free-fall energy, joint step response, fingertip force–displacement, static friction threshold, object slip, tendon coupling, grasp hold, and one short manipulation trajectory. Compare the same observables on hardware. A policy success score can conceal an incorrect contact model that the policy exploits.

MuJoCo is also attractive for RL when MJX or Playground provides a maintained batched environment. But GPU speed is not the same as feature completeness or agreement with CPU MuJoCo.

MJX and MuJoCo Playground accelerate batched learning, but feature parity and numerical behavior require per-environment checks. ^{[10] [11]} Run a small CPU MuJoCo reference and compare reset state, one-step transition, termination, reward components, and short rollouts before scaling. Fix seeds and report backend, precision, solver, batch size, and hardware.

The reported 475-fold MuJoCo-Warp manipulation speedup over MJX is hardware- and task-specific, not a universal accuracy ranking. ^[6] Do not transpose that multiplier to a different GPU, hand, solver, batch size, or contact workload. Throughput belongs in a local benchmark with fidelity and stability checks beside it.

Figure 6.2 — Environment and platform scope of MuJoCo Playground. Source: Zakka et al., MuJoCo Playground technical report, Figure 1.

4. URDF, USD, and MJCF share geometry but not semantics

URDF is a common robot-description interchange for links, joints, inertials, visuals, collisions, and transmissions/extensions. USD is a general scene-composition system with robotics/physics schemas and application-specific articulation and drive properties. MJCF is MuJoCo's model language, including simulation and actuator/contact semantics. Conversion must therefore be treated as model reconstruction plus tests, not file-format translation.

URDF, USD, and MJCF overlap but can differ in drive, contact, tendon, constraint, and sensor semantics. ^{[12] [13] [9]} An importer may infer drive gains or merge fixed joints; an exporter may not preserve every USD feature; URDF may lack a native representation for a tendon/coupling used in MJCF. Keep a canonical engineering source and backend-specific overlays rather than round-tripping repeatedly.

Create asset invariant tests:

Use URDF as a seed when a manufacturer provides it, import into USD, and author missing drive/material/sensor semantics explicitly. Build MJCF from the same measured kinematics and inertials, then add actuator/tendon/contact semantics. Store a manifest mapping every source field and every backend override.

5. Connect Isaac and MuJoCo through contracts and data

Isaac and MuJoCo usually exchange validated assets, task semantics, trajectories, checkpoints, and logs rather than run in live lockstep. ^{[12] [9] [10]} The sources do not establish a universal bidirectional converter. Live co-simulation adds clock authority, stepping order, state reconciliation, network latency, and duplicated contact ownership. Unless the experiment genuinely needs two engines in one timestep, offline or episode-level handoff is easier to audit.

A robust shared contract contains:

• named observation fields with units, frames, rates, normalization, and missing-data behavior;
• action fields with reference frame, controller semantics, scale, clipping, delay, and hold behavior;
• reset distribution, randomization parameters, termination, reward components, and success definition;
• robot/tool/object asset versions and calibration/identification parameters;
• common trajectory probes, seeds, episode schema, and regression thresholds;
• checkpoint metadata stating training backend and deployment wrapper.

Train in MuJoCo and validate camera/perception integration in Isaac, or train in Isaac Lab and use MuJoCo as an independent dynamics check. A second engine is not proof of reality; it is a cheap detector of backend-specific exploits. Deployment still passes real held-out tests.

6. Choose by experiment, not by brand

Photorealistic sensing, dynamics fidelity, contact fidelity, and RL throughput are separate axes with no universal winning engine. ^{[5] [8] [14] [15]} Rate each axis against a task metric. Tactile simulation, for example, may require contact geometry and sensor rendering beyond rigid-body success; TACTO demonstrates a dedicated tactile-sensing layer rather than proving all engines equivalent ^[15].

The tactile-skin sim-to-real work summarized in Terry's Korean note and English note illustrates why contact and tactile abstractions must be validated. ExoStart's demonstration interface is covered in Korean and English; it is relevant when simulator actions must match a sensorized human interface.

7. A staged installation and validation workflow

Stage A — define the contract before installation

Write task, observation, action, reset, success, controller, and real deployment interface. Choose the primary risk: geometry/sensing, contact/control, or parallel learning. Select one primary engine and one optional independent checker. Pin hardware, OS, GPU driver, simulator, physics backend, Python, ROS distribution, and asset versions.

Stage B — static and one-step tests

Load a single robot without a task. Check asset invariants. Apply one small joint command and compare sign, limit, state, and timestep. Add gravity and inspect rest pose/free response. Only then add tool, object, contact, sensors, and ROS bridge one at a time.

Stage C — common regression probes

Run named poses, a short free-space trajectory, joint step, object drop, static grasp, slip ramp, camera calibration target, and reset determinism. Save state/action trajectories and backend configuration. Where cross-engine quantities differ, define a task-relevant tolerance rather than demanding bitwise identity.

Stage D — learning scale

Verify one environment, then a small batch, then production batch. Compare reward decomposition, termination, NaN/penetration, determinism, memory, and steps per second. Throughput optimization begins only after semantics pass.

Stage E — policy validation

Evaluate held-out seeds and parameter ranges in the training engine, the second engine if justified, recorded real trajectories, then supervised hardware. Use the same observation/action wrapper and record mismatch. Never let a simulator directly authorize real motion.

Figure 6.3 — A cross-engine asset, data, and evaluation bridge that assumes neither a universal converter nor live lockstep. Original synthesis.

8. Four concrete workflow recipes

For a classical planning cell, make the ROS/MoveIt interface authoritative. Import the arm and installed tool into Isaac Sim or another scene simulator, but judge success by joint names, TF, collision geometry, controller action, and trajectory replay. A policy framework is unnecessary. The deliverable is a regression bag and scene hash that reproduce home, pre-grasp, obstacle avoidance, cancel, and collision rejection.

For GPU RL, begin with one environment whose state transition and reward can be inspected. Isaac Lab is natural when an Isaac scene, sensors, and GPU physics already define the task. MJX/Playground is natural when a maintained MuJoCo environment and compact contact/control model are central. Scale only after reset distribution, action clipping, observation normalization, reward terms, and termination match the written contract. Report useful simulated seconds per wall-clock second, not only aggregate steps per second; excessive substeps or invalid contacts can make the headline misleading.

For imitation or VLA data, treat the episode schema as the shared product. Isaac may render wrist and external cameras and test occlusion; MuJoCo may validate dynamics under recorded actions. Both must consume the same action reference, timestamps, camera calibration version, and task phase. Compare replay divergence at named checkpoints. Synthetic episodes without an explicit relationship to real camera noise, operator actions, and failure labels do not close the deployment gap.

For a contact-rich hand, start with MJCF or a qualified Isaac/Newton hand model only after measuring joint zeros, coupling, fingertip geometry, actuator response, and representative friction. Create separate probes for free joint motion, tendon coupling, static grasp, slip, rolling, and impact. Train after those probes pass; then replay the learned policy in a second backend only if its action/observation semantics are demonstrably equivalent. Finally, use reduced-energy hardware tests. A cross-engine success that fails the hardware force trace remains a model failure.

9. Bounded Codex prompts

Prompt A — simulator-role decision audit

Read my task, robot files, sensor list, ROS interfaces, learning requirements, and the
official version documents I provide. Do not install packages or modify files. Classify
needs into scene/assets, sensor rendering, rigid/contact dynamics, batched learning,
ROS integration, and validation. Recommend one primary simulator and at most one checker.

For each choice state exact version tuple, supported evidence, missing feature, test,
and stop condition. Do not infer fidelity from rendering or throughput from marketing.
Return a decision table and minimal staged installation plan.

Prompt B — cross-format asset audit

Compare this URDF, USD metadata/export report, and MJCF read-only. Build a mapping for
links, joints, axes, limits, inertials, collision, drives/actuators, tendons/coupling,
contacts, sensors, and frames. Mark lost, inferred, backend-only, and unknown semantics.

Propose invariant tests and expected numeric tolerances, but do not silently rewrite
assets. Done when a human can approve a canonical source, backend overlays, and blockers.

Prompt C — small simulator regression suite

Using only simulator/fake interfaces, propose tests for named poses, joint step, short
trajectory, object drop, static grasp, slip ramp, reset, observation/action schema,
reward terms, and termination. Pin seeds, timestep, solver, precision, and versions.

Do not connect to hardware or generate a live co-simulation bridge. Save comparable
CSV/episode summaries and flag mismatches by task metric. Never weaken collision,
force, or safety limits to make tests pass.

10. Evidence tiers, disagreements, and limitations

OpenUSD, Isaac, Newton, MuJoCo, importer, and Playground claims use primary official documentation or repositories. Foundational dynamics, GPU simulation, and tactile-simulation claims use primary papers. Vendor throughput figures describe one reported configuration; they are not independent accuracy rankings.

The Isaac-versus-MuJoCo disagreement is conditional. Isaac is often stronger when scene composition, sensor realism, ROS integration, and one GPU workflow matter. MuJoCo is often stronger when compact actuator/tendon/contact semantics and control iteration dominate. Either can be misused. The correct comparison holds task/action/observation and validation metrics fixed.

Throughput versus fidelity is not one axis. Faster stepping can enable broader randomization but may expose missing features or different numerics. A slower calibrated contact model can be more useful for a particular hand, while a high-throughput approximate model can be appropriate for robust policy search. Report both speed and held-out error.

Limitations: versions change after 2026-07-14; official documentation does not independently validate every contact; converters are incomplete; tactile and deformable behavior remain model-specific; a second engine is not reality; and simulator success does not replace cell risk assessment or vendor-controller safety ^[19].

Manufacturing Cell Checkpoint

For a camera-guided pick cell, assign USD/Isaac Sim to fixture and sensor layout, MoveIt/ROS to interface regression, and real logs to calibration truth. If the gripper only closes in free space, MuJoCo may add little. If the study becomes slip-sensitive in-hand manipulation, create an MJCF hand/object model and contact probes while preserving the same action and episode schema.

Log asset_version, backend/version, timestep/solver, randomization, calibration_id, observation_schema, action_schema, controller_model, seed, probe_metrics, policy_checkpoint, real_episode_id. Owners are separate: asset owner, physics owner, learning owner, ROS owner, calibration owner, and safety owner.

What to Learn Next

You now have a role map and validated handoff between simulators. (Chapter 7) uses it to perform the first movement: static description, fake controller, physics simulation, drives-disabled connection, one small joint move, Cartesian motion, and grasp. The simulator does not authorize the next gate; it supplies evidence for a human-owned runbook.

Before continuing, choose one primary engine, write what it will not validate, and define three common probes that can run in simulation and reality. That small error budget is more valuable than installing every simulator.

Annotated research trail

These sources deepen simulator scope, sensor models, and model fidelity. They are grouped by the assumption or experiment to inspect, rather than used as a list of borrowed success rates. Read each result within its platform and protocol.

References

1. OpenUSD (2026). OpenUSD Documentation. Official documentation.
2. NVIDIA et al. (2026). NVIDIA Isaac Sim: A Reference Architecture for Robotics Simulation. arXiv:2606.03551.
3. Isaac Lab (2026). Isaac Lab Migration Guide. Official documentation.
4. Newton (2026a). Newton Physics Repository. Official repository.
5. NVIDIA (2026a). Isaac Sim ROS 2 Bridge. Official documentation.
6. NVIDIA (2026b). Newton 1.0 Contact-Rich Manipulation and Locomotion. Official release.
7. NVIDIA (2025). Isaac Lab and Newton Beta. Official technical article.
8. Todorov, E., et al. (2012). MuJoCo: A Physics Engine for Model-Based Control. IEEE/RSJ IROS. DOI: 10.1109/IROS.2012.6386109.
9. MuJoCo (2026a). MJCF XML Reference. Official documentation.
10. Zakka, K., et al. (2025). MuJoCo Playground Technical Report. Technical report.
11. Google DeepMind (2026). MuJoCo Playground Repository. Official repository.
12. NVIDIA (2026c). Isaac Sim URDF Importer. Official documentation.
13. NVIDIA (2026d). Isaac Sim URDF Exporter. Official documentation.
14. Makoviychuk, V., et al. (2021). Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning. arXiv:2108.10470.
15. Wang, S., et al. (2022). TACTO: A Fast, Flexible, and Open-Source Simulator for High-Resolution Vision-Based Tactile Sensors. IEEE Robotics and Automation Letters. arXiv:2012.08456.
16. Jakobi, N., et al. (1995). Noise and the Reality Gap. ECAL. DOI: 10.1007/3-540-59496-5_337.
17. Tobin, J., et al. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv:1703.06907.
18. Kasolowsky, U., & Bäuml, B. (2024). Fine Manipulation Using a Tactile Skin. IROS 2024.
19. ISO (2025). ISO 10218-2:2025 Robotics — Safety Requirements — Robot Applications and Cells. International Standard.