Chapter 8: Teleoperation and a First Learned Policy — Build the Leader–Follower Loop

Overview

Teleoperation is not “stream the controller pose to the robot.” A usable loop transforms a device sample with a clock and tracking status into a bounded robot command, then returns robot and task feedback to the operator. Frames, scale, clutch, workspace, filtering, IK, retargeting, rate, collision checks, command ownership, and a stale-input watchdog must all be explicit. A direct leader arm reduces some mapping ambiguity; VR or optical tracking increases flexibility but adds calibration and loss modes.

The same loop is also a data-production system. A demonstration is useful only if camera observations, operator input, sent actions, robot state, calibration, interventions, resets, and outcomes share a reconstructable timeline. The smallest learning exercise in this chapter records a simple teleoperated pick/place, trains a behavior-cloning baseline, and evaluates it repeatedly in simulation before supervised hardware. It is not a full RL, VLA, or world-model treatment; those belong to #S12/#S13.

After reading this chapter... - You can draw and validate the complete leader–follower packet and feedback loop. - You can compare VR, Vive trackers, optical hands, gloves, joysticks, and mechanical leaders by observable failure modes. - You can retarget a gripper or dexterous hand without blindly copying human joints. - You can collect synchronized episodes with intervention and failure labels. - You can run a small behavior-cloning exercise with separated data and repeated gated evaluation.

1. Write the leader–follower contract before connecting devices

A leader-follower loop must define frames, scale, clutch, workspace constraints, filtering, IK or retargeting, rate, watchdog, and stale-input behavior. ^{[1] [2] [3]} Add command ownership, units, timestamps, tracking confidence, stop behavior, and feedback. If any field is unknown, keep the follower simulated.

The packet path is: device sample→device clock/status→calibrated operator frame→clutch and zero offset→translation/rotation scaling→workspace projection→arm IK/differential IK→joint/rate/collision limits→follower command→robot state and visual/haptic feedback. Every arrow needs an owner and a failure output. A clutch freezes follower reference while the operator recenters; releasing it must not create a pose jump.

Figure 8.1 — A leader–follower closed loop with timestamps, frames, clutch, and watchdog. IK/Servo and collision scaling are not safety-rated stops. Original synthesis.

2. Choose an input device from the task and observability

A VR controller or OpenXR device provides a tracked 6-DoF pose, buttons, and a natural clutch. It is useful for arm teleoperation, but headset/runtime frame conventions, recentering, controller loss, and network/render latency must be tested ^[2]. A Vive tracker provides a compact tracked rigid body and can be mounted on a handheld prop, but base-station visibility and tracker-to-tool calibration remain.

Optical hand tracking such as Leap Motion reduces worn hardware and produces hand landmarks or joints. It is vulnerable to occlusion, rotated/calibration failure, self-contact, and uncertain metric scale ^[10]. A data glove provides richer finger state and can work under some occlusion, but fit, magnetic/environmental effects, drift, calibration, SDK licensing, and no intrinsic force feedback matter. A joystick or SpaceMouse has low dimensionality and excellent clutch/rate control; it is often the safest first Cartesian leader.

A mechanical leader robot offers close joint correspondence, proprioception, and sometimes natural bilateral coupling. SO-101/OpenArm-style leaders make demonstration collection transparent when follower morphology matches ^{[14] [15]}. They are embodiment-specific, occupy space, need calibration, and can transmit operator bumps. An exoskeleton may give fingertip correspondence and richer demonstrations but increases fit, calibration, service, and safety burden.

Optical tracking trades low hardware burden for occlusion and calibration risk, while mechanical leaders trade embodiment specificity for correspondence and proprioception. ^{[9] [10] [11] [12]} There is no universal best device. Choose the smallest input space that preserves task intent and produces testable validity signals.

3. Calibrate time, frames, scale, and latency

Calibrate the transform from device tracking space to robot base or teleoperation reference. Use several known orientations and positions, not one point. Verify right/left-handed coordinates, quaternion convention, meters, and tool approach axis. Store device, mount, operator, robot, and calibration revisions. Recenter events must create a new mapping epoch.

Timestamp at acquisition, after transport, after mapping, at command send, and at follower state. Measure median and tail latency plus jitter; average latency alone hides a one-in-a-hundred jump. Filter noise without making the operator chase delayed motion. Log raw and filtered signals so smoothing can be revised offline.

Clutch state is data. When disengaged, operator movement must not become follower motion. On reengagement, set a new relative origin to avoid a jump. Scale translation and rotation separately. Workspace projection should preserve intent near boundaries while clearly signaling saturation; silently clipping for long periods produces poor demonstrations.

Tracking loss must revoke motion authority through a tested watchdog instead of replaying the last pose. ^{[2] [1]} Define maximum sample age, confidence, missed count, and recovery. Test device occlusion, unplug, process crash, frozen timestamp, repeated packet, and network interruption in simulation. Recovery requires fresh valid samples, clutch reset, and operator acknowledgment.

A calibration rehearsal that produces evidence

Treat calibration as a short experiment, not a GUI step. Fix the robot, display the tracked leader pose, and record at least a neutral pose plus separated translations and rotations about each device axis. Ask a second person to name the expected robot-base direction before revealing the mapped marker. This catches swapped axes, sign errors, degrees-versus-radians mistakes, and tool frames that look plausible in one pose. Repeat the set after recentering and after restarting every device process. The resulting transform, residual error, device serial, mount revision, operator profile, timestamp, and software versions form one calibration record.

Next, measure the loop dynamically. Use a visible or logged step-like leader motion and compare acquisition, mapped-target, sent-command, and follower-state timestamps. Report median, 95th and 99th percentile age, jitter, packet gaps, and effective command frequency for a stationary segment and a representative motion segment. Repeat with cameras and visualization enabled because rendering or encoding can contend for compute. If clocks live on different machines, verify synchronization and estimate its uncertainty; a timestamp with an unknown clock is not timing evidence.

Finish with an authority drill. While moving only in simulation, release the clutch, cover or disconnect the tracker, freeze a timestamp, restart the leader process, and recover. For every injection, write the expected state transition: ACTIVE → DEGRADED/HOLD → REVOKED, the allowed follower output, the operator indication, and the explicit action required to return. Promotion requires observed transitions to match this table without a pose jump. A convenient teleoperation demo that cannot survive this rehearsal is not ready for hardware.

4. Map arm motion with IK and differential IK

Pose teleoperation may solve IK for an incremental target, while velocity teleoperation uses differential IK. Both need current state, TCP, joint limits, collision geometry, singularity handling, and controller compatibility. Global motion to a distant pre-grasp should be planned; local teleoperation is suitable for bounded adjustments.

Differential IK is safe for responsive teleoperation only when singularity, joint-limit, collision, and discrete-time amplification are bounded. ^{[4] [5] [1]} Damping and scaling limit amplification but do not authorize motion. Stop or retreat when margins are exhausted. Validate discrete rates and delay because a stable continuous formulation can behave poorly with stale sampled feedback.

Use three envelopes: a hard forbidden region; a soft region where commands scale down; and an operating region with normal scale. Apply them to Cartesian workspace and joint margins. Command timeout must be shorter than a dangerous accumulation but compatible with measured device jitter. Only one controller or Servo process may own follower interfaces.

5. Retarget a gripper and dexterous hand differently

A parallel gripper often maps one operator variable—trigger, pinch distance, or leader width—to one normalized opening. Define open/closed calibration, deadband, maximum force setting, close speed, object-detection semantics, and behavior when arm clutch is open. Finger closure should not continue merely because the last trigger packet was high.

A dexterous hand has different morphology and coupling from a human. Human joint-angle copying can violate robot limits or destroy fingertip relationships. Decide which task features matter: fingertip positions, pairwise distances, palm orientation, grasp aperture, contact intent, or synergy coordinates.

Hand retargeting should optimize task-space or fingertip feasibility rather than blindly copy human joint angles across morphologies. ^{[6] [7] [8]} Include joint/velocity limits, collision, coupling, reachability, temporal smoothness, and task weights. Log both human input and realized robot target so retargeting error remains observable.

DexUMI reports 86 percent average success across two dexterous hand platforms but relies on embodiment-specific exoskeleton and visual adaptation. ^[12] The result supports a carefully designed interface, not universal transfer. Terry's DexUMI note is available in Korean and English.

ExoStart reports real-student success of 56, 50, 54, 94, 62, 56, and 2 percent across seven 50-episode tasks, so only five exceed 50 percent; it also exposes control-rate and contact-model mismatches. ^[13] Nut Unscrew equals 50 percent, while Bulb Install reaches 2 percent and is linked in the paper to a MuJoCo soft-contact penetration exploit. These mismatches are precisely what a commissioning log must retain. Terry's ExoStart note is available in Korean and English.

Figure 8.2 — DexUMI overview from human-hand demonstration to dexterous robot-hand execution. Source: Xu et al., 2025, Figure 1.

6. Simulation-first acceptance ladder

1. Replay recorded device data with no follower and inspect stamps/validity.
2. Map to a visual marker; verify axes, scale, clutch, recenter, and loss.
3. Drive a kinematic robot in simulation; sweep workspace and singularities.
4. Add collision and physics; inject delay, jitter, drop, occlusion, and process crash.
5. Test gripper/hand separately, including stale input and saturation.
6. Connect real robot read-only; compare simulated and real state/interface.
7. Run one reduced joint/Cartesian axis with observer and deadman.
8. Run free-space pose teleoperation, then gripper, then a simple object.

7. Turn demonstrations into auditable episodes

A synchronized episode should retain observations, actions, robot state, timing, calibration version, interventions, resets, and failure reason. ^{[16] [17] [19]} Store raw device input, mapped target, command actually sent, follower state, camera/tactile/force, clutch/watchdog, operator intervention, task phase, outcome, hardware/software manifest, and dropped-sample counters.

Do not delete failures. Label failure onset and cause: tracking loss, retargeting infeasible, collision scaling, missed grasp, slip, perception error, operator abort, protective stop, or ambiguous. Keep aborted episodes in an analysis split even if excluded from initial training. Intervention records reveal where policy deployment will need supervision.

Split by episode and object/scene condition, never random adjacent frames. Prevent near-duplicate demonstration segments from leaking across train and evaluation. Inspect action and observation distributions, episode duration, success, intervention, saturation, missing data, and calibration epochs before training.

Review an episode before scaling collection

Review the first five episodes manually before collecting fifty. Reconstruct a contact event on one timeline: the image in which the fingers approach, the operator command, the mapped target, the command sent, the measured joint or gripper response, and any force or tactile change. Check that clutch periods are represented as intentional states rather than mislabeled zero actions. Plot sample age and action saturation beside the video; a visually successful episode can still teach delayed or clipped control.

Create a one-page collection report for each session: attempted and retained episodes, successes, operator interventions, watchdog events, protective stops, calibration epoch, object and lighting conditions, missing streams, maximum timestamp gap, and exclusion reasons. Freeze the schema before the main collection run. When a field changes, start a new schema version and provide a converter or keep the datasets separate. This discipline makes a later policy regression traceable to data, mapping, or model rather than folklore.

8. The smallest safe learned-policy exercise

Choose one light object, fixed start region, one camera arrangement, one gripper action, and a short pick/place. Collect a modest set of successful demonstrations plus labeled failures under one stable calibration. First train a simple behavior-cloning policy in a LeRobot/robomimic-style offline workflow ^{[19] [18]}. The objective is to validate the data→training→checkpoint→evaluation contract, not win a benchmark.

Use three episode groups: training, validation for model selection, and untouched evaluation with different seeds/object placements. Save preprocessing, normalization, architecture, seed, steps, checkpoint, code/data hashes, and evaluation script. Inspect predicted actions offline and in simulation. Add action clipping, workspace/collision filters, stale-observation rejection, rate limit, and a human-held enable path before supervised hardware.

Diffusion Policy reports a 46.9 percent relative average benchmark improvement, not a guarantee for a new robot or dataset. ^[20] A stronger sequence model cannot repair wrong frames, mixed calibration, stale images, or unsafe action wrappers. Terry's Diffusion Policy note is available in Korean and English.

A first learned policy should use separated episodes, seeds, checkpoints, and repeated evaluation rather than one successful rollout. ^{[18] [19] [20]} Report attempts, success confidence, intervention, collision scaling, aborts, object damage, latency, and failure categories. A video is not an evaluation.

Evaluation ladder

This is the handoff to #S12/#S13. RL, VLA, world models, larger datasets, policy fine-tuning, and autonomy require separate evidence. Do not expand task scope because the first cloning rollout succeeded.

A reproducible first-exercise record

Before training, write a compact experiment card. It names the task success predicate, allowed start region, object set, camera and robot calibration IDs, observation and action fields, control frequency, episode split rule, model input horizon, and prohibited deployment conditions. The evaluation plan predeclares the number of simulation attempts and distinct reset seeds. Validation chooses a checkpoint; evaluation is opened only after that choice.

For diagnosis, compare three baselines: replay a recorded action in the same simulated reset, run the behavior-cloning prediction open loop without authority, and run it closed loop through the same limits used by teleoperation. Replay failure suggests environment or reset inconsistency. Open-loop divergence suggests observation/action alignment or model error. A policy that looks reasonable until the safety wrapper clips it reveals distribution or scaling problems. Record these outcomes even when the final success count is zero; the purpose of this exercise is a trustworthy pipeline.

Figure 8.3 — Evidence gates from episodes through behavior cloning and repeated evaluation. LeRobot episode counts are current guidance, not a universal threshold. Original synthesis.

9. Failure diagnosis and recovery

After an unexpected real event, revoke motion, preserve logs, start a new attempt ID, and return to the last deterministic gate from (Chapter 7). A tracking or policy failure must never auto-restart into motion.

10. Bounded Codex prompts

Read the device API, robot description, MoveIt Servo/controller config, and official docs.
Do not connect to hardware or edit files. Produce a leader-follower contract listing
frames, units, timestamps, validity, clutch, scale, workspace, IK/retargeting, limits,
command owner, rate, watchdog, feedback, stop, and recovery. Mark unknowns and fake tests.

Generate simulator-only tests for known axes, recenter/clutch, singularity, joint/workspace
boundary, collision scaling, delayed/reordered/frozen packets, tracking loss, process crash,
and command timeout. No real IP, hardware plugin, enable, limit bypass, or auto restart.

Audit this episode dataset read-only. Check timestamps, schema, calibration epochs, raw vs
mapped vs sent actions, missing samples, intervention/failure labels, episode-level split,
near-duplicate leakage, and train/eval distributions. Return blockers and a minimal
behavior-cloning/evaluation plan; do not claim deployment readiness.

11. Evidence tiers, disagreements, and limitations

Device interfaces and LeRobot/MoveIt behavior use official documentation; IK, teleoperation, retargeting, imitation, and policies use primary papers. Product claims describe interfaces, not independent latency or ergonomics rankings. Research success rates remain tied to platform, task, data, and evaluation.

Optical versus mechanical leadership is conditional. Optical devices are flexible and quick to deploy but fail under occlusion/calibration. Mechanical leaders offer correspondence and proprioception but are embodiment-specific. Gripper teleoperation often needs only one scalar, while dexterous manipulation justifies more complex interfaces.

Teleoperation data is not automatically policy-ready. Operator correction, latency, clutch use, and task selection bias the distribution. Behavior cloning is simple but compounds errors outside demonstrations ^[21]. Intervention or on-policy data may later help, but it broadens safety and research scope.

Limitations: versions change after 2026-07-14; no generic latency/watchdog threshold fits every cell; haptic feedback is not guaranteed; simulation does not validate human fatigue or real contact; this small exercise does not cover full RL/VLA/world-model design; local safety review remains authoritative.

Manufacturing Cell Checkpoint

For a VR-guided pick cell, authorize only free-space arm teleoperation first. Fix tracker-to-base calibration, clutch, scale, workspace, Servo limits, watchdog, observer, and stop. Add gripper after separate testing, then one light object. Log episode_id, device/calibration, raw/mapped/sent action, stamps, validity, clutch, IK/scale reason, follower state, camera, gripper, intervention, safety/fault, task phase/outcome.

What to Learn Next

The loop and first policy are now auditable. (Chapter 9) addresses a custom robot whose URDF is not trustworthy by default: kinematic calibration, TCP/camera alignment, dynamics and actuator identification, contact validation, and a measured sim-to-real error budget. Teleoperation data is only as good as that model and calibration.

Before continuing, demonstrate a tested tracking-loss stop, a clutch with no jump, a reproducible episode, and repeated simulation evaluation. One attractive rollout is not a completed loop.

Annotated research trail

These sources deepen leader–follower interfaces and policy-quality data. They are grouped by the assumption or experiment to inspect, rather than used as a list of borrowed success rates. Read each result within its platform and protocol.

References

1. MoveIt (2025). MoveIt Servo Tutorial. Official documentation.
2. Khronos (2024). OpenXR 1.1 Specification. Official specification.
3. Foote, T. (2013). tf: The Transform Library. DOI: 10.1109/TePRA.2013.6556373.
4. Whitney, D. E. (1969). Resolved Motion Rate Control. DOI: 10.1109/TMMS.1969.299896.
5. Nakamura, Y., & Hanafusa, H. (1986). Singularity-Robust Inverse Kinematics. DOI: 10.1115/1.3143764. [Nakamura & Hanafusa, 1986]
6. Qin, Y., et al. (2023). AnyTeleop. RSS. DOI: 10.15607/RSS.2023.XIX.015.
7. NVIDIA (2025). SPIDER. arXiv:2511.09484.
8. Lee, et al. (2026). DexTwist. arXiv:2605.12182.
9. Ding, et al. (2024). Bunny-VisionPro. arXiv:2407.03162.
10. Ultraleap (2024). Leap Motion Controller 2 Calibration Notice. Official support notice.
11. Fang, et al. (2025). DEXOP. arXiv:2509.04441.
12. Xu, M., et al. (2025). DexUMI. PMLR 305.
13. Si, et al. (2025). ExoStart. arXiv:2506.11775.
14. Hugging Face (2025). SO-101. Official documentation.
15. OpenArm (2026). OpenArm in LeRobot. Official documentation.
16. Hugging Face (2026a). Teleoperation Processors. Official documentation.
17. Hugging Face (2026b). Human-in-the-Loop Data Collection. Official documentation.
18. Hugging Face (2026c). Bring Your Own Policies. Official documentation.
19. Mandlekar, A., et al. (2021). robomimic. arXiv:2108.03298.
20. Chi, C., et al. (2023). Diffusion Policy. arXiv:2303.04137.
21. Argall, B., et al. (2009). A Survey of Robot Learning from Demonstration. Robotics and Autonomous Systems. DOI: 10.1016/j.robot.2008.10.024.