Chapter 3: Add Eyes, Controls, and Compute — Cameras, Teleoperation, GPUs, and Fixtures

Overview

A robot cell does not become ready for visual learning when a camera is plugged into USB, nor does it become teleoperable when a VR pose appears on screen. The useful unit is a synchronized sensing-and-authority system: every image has known geometry and time, every operator command has a validity state, compute is sized for the peak concurrent workload, and the physical cell can stop independently of the workstation.

This chapter turns that system into purchasing and acceptance decisions. The goal is not to reproduce a particular high-end workstation quotation. That imported brief is useful as a list of questions, but it is not evidence. Current official specifications and primary studies instead define capacity tiers, while measurements in the reader's own cell decide whether a tier is sufficient.

The emphasis is also deliberately pre-algorithmic. A future VLA may consume images, language, proprioception, and actions, but it cannot repair an undocumented optical frame, an overwritten timestamp, or a camera mount that moves when the table is bumped. RL, VLA, and world-model design belong to #S12 and #S13; here we build the trustworthy data plane they will inherit.

After reading this chapter, you will be able to... - choose fixed, wrist, RGB-D, stereo, or triggered industrial cameras from task geometry rather than headline resolution; - distinguish Quest-class VR, handheld controllers, SpaceMouse-style joysticks, trackers, optical hand tracking, gloves, and mechanical leaders by the motion and feedback they actually provide; - size GPU memory, host RAM, network, and storage from concurrent workloads and measured episode bandwidth; - specify tables, mounts, cable relief, UPS behavior, hardwired stopping, guarding, and operator access as one cell design; - ask Codex for bounded adapters and audits with simulator-only tests, explicit failure states, and no authority over safety functions.

3.1 The cell is a time-indexed authority graph

Before selecting devices, draw two graphs. The data graph runs from photons and operator motion through drivers, timestamps, transforms, buffers, recording, and models. The authority graph runs from a human intent through clutch/deadman logic, workspace limits, IK or retargeting, the robot controller, and independent stop hardware. A component can be healthy in one graph and dangerous in the other. A tracker may continue publishing numerically plausible poses after its confidence is invalid; a camera may show a crisp image whose timestamp belongs to a delayed queue.

For each stream, write six fields before integration: physical quantity, coordinate frame, clock, nominal rate, validity signal, and owner of the stop decision. “Hand pose at 120 Hz” is incomplete. A usable contract says which hand landmarks, expressed in which frame, stamped by which clock, accompanied by which confidence or tracking flags, and converted into what bounded robot command.

The transform graph is time-indexed, not merely a collection of frame names ^[8]. That fact changes procurement. A rigid camera bracket is part of the measurement system; a PTP-capable camera is useful only when the rest of the clock path is designed; a wireless tracker needs a stale-pose policy; and a leader device needs a clutch even if its nominal mapping is one-to-one.

Figure 3.1 — Synchronized sensing-and-authority architecture. Original synthesis.

Use a one-page interface sheet for every device. Record connector and power, supported operating system and SDK version, frame convention, timestamp origin, calibration artifact, expected loss mode, and a simulator substitute. This sheet becomes the input to Chapter 4, where ROS 2 topics, TF, QoS, and lifecycle states will carry these contracts rather than invent them.

3.2 Camera geometry: see the task, not the brochure

Start from the smallest task feature and the worst occlusion. A fixed overview camera should see the full reachable workspace and a margin for the operator's hand. A wrist camera should resolve the contact region at the closest safe working distance without colliding with the object, gripper, or table. A side camera should reveal depth or insertion error hidden from the overview view. Three redundant views of the same occlusion are not a multi-camera design.

For VLA-ready collection, a practical starting layout is one fixed oblique overview RGB or RGB-D camera, one wrist or near-wrist view, and an optional lateral view for bimanual or insertion tasks. Keep the viewpoints stable across episodes. Put a calibration target at representative heights and corners, not only at the table center. Record intrinsics, distortion model, exposure, white balance, depth scale, and every camera-to-base or camera-to-tool transform as versioned artifacts.

RealSense D455 lists depth up to 1280×720 at 90 fps, while OAK-D Pro adds on-device processing; neither headline proves robot-state timestamp alignment. ^{[14] [18]}

The D455's stated 0.6 m minimum operating range can be awkward for close eye-in-hand geometry, while the OAK-D Pro's published ideal depth range begins at 0.7 m ^{[14] [18]}. Those numbers do not ban either product; they tell the reader to reproduce the actual wrist-to-object distance before purchase. Reflective tools, transparent containers, dark objects, repetitive texture, and finger occlusion deserve physical test samples. A comparative study of four stereo/depth cameras also shows why performance must be read against scene and object conditions, not treated as one universal ranking ^[4].

Resolution is a budget, not a trophy. Higher resolution increases pixel throughput, VRAM use, encoder load, storage, and annotation cost. Increase it only if a task feature remains unresolved after lens, working distance, lighting, and viewpoint are fixed. For many manipulation datasets, stable geometry and timing produce more reusable evidence than one oversized image stream.

Calibration is an experiment, not a wizard

Intrinsic calibration estimates how a camera maps rays to pixels; hand-eye calibration estimates a rigid transform between camera and robot frames. Classical formulations such as AX=XB remain foundational ^{[25] [23]}. A modern workflow should add pose diversity, held-out poses, uncertainty, and periodic revalidation. Low reprojection error alone can hide poor coverage or target bias ^[31]. Uncertainty-aware work reinforces the need to model robot-pose error instead of treating every calibration input as exact ^[28].

An acceptance test should therefore reserve poses that were not used for fitting. Place a known target or object in the working volume, predict its location from the calibrated chain, and report translation and orientation residuals by region. Save the raw images, detected points, robot states, solver version, result, and mounting photograph. If a mount is moved, a lens is refocused, or a wrist tool is changed, invalidate the relevant calibration explicitly.

3.3 Camera time: synchronize what happened, not only clocks

Clock synchronization, exposure synchronization, and data alignment are different. PTP can align device clocks; a hardware trigger can align exposures; neither guarantees that a frame reaches the recorder without queueing. USB transport, decoding, GPU copies, middleware queues, and batching add age after exposure. Measure both the timestamp relationship and end-to-end arrival delay.

Global-shutter cameras with hardware trigger and PTP can create a more auditable synchronization path than commodity RGB-D cameras, at higher integration cost. PTP precision depends on the complete clock and network design. ^{[1] [2] [4]}

A representative ace 2 model specifies a 5 MP global-shutter sensor, 2600×2160 output, a 65 fps default rate, and hardware-trigger support ^[3]. That is useful evidence for a triggered path, not a command to buy that model. It still needs a lens, lighting, trigger wiring, network interface, storage test, and separate metric-depth strategy. Conversely, a commodity RGB-D camera may be entirely adequate for slow pick-and-place if measured alignment error stays inside the task tolerance.

Run a timestamp audit with a visible or electrical event observable by multiple streams. Examples include an LED driven from a logged signal, a moving calibration wand observed by cameras and robot state, or a gripper closure visible in images and motor data. Estimate offset, jitter, drift over an episode, dropped frames, and the 95th/99th-percentile frame age. Repeat with all planned cameras and recording enabled; an unloaded test hides contention.

Bounded Codex prompt — timestamp audit “In this repository, design a simulator-first timestamp audit for three cameras, robot joint state, and one teleoperation stream. First inspect the existing message schemas and clock sources. Produce an interface table, a measurement plan for offset/jitter/drift/frame age, and tests using synthetic delayed, reordered, and missing samples. Do not invent device APIs, do not connect to real hardware, and do not issue robot commands. Put all thresholds in configuration, fail the audit when clock provenance is unknown, and finish by running the relevant tests and reviewing the diff.”

This prompt asks for an auditable tool, not a magic synchronizer. The human team must choose tolerances from task speed and geometry. A 30 ms offset may be irrelevant while the robot is static and destructive during fast contact; the correct gate is spatial error at the task, not a fashionable latency number.

3.4 Choose operator input by information and feedback

Teleoperation devices differ along four independent axes: absolute versus incremental motion, hand/finger observability, feedback to the operator, and embodiment match. “VR support” does not answer any of them. A Quest-class headset and controller can provide mobile 6-DoF poses through an OpenXR runtime, but the robot adapter must define reference spaces, recentering, tracking validity, clutch behavior, and what happens when the headset sleeps or leaves the tracking volume ^[16].

A handheld 6-DoF controller is a good default for Cartesian end-effector demonstrations: buttons naturally implement clutch, gripper toggle, mode selection, and deadman behavior. A SpaceMouse-style device is even simpler at a desk and maps well to bounded incremental motion. A gamepad or 3-axis joystick is inexpensive and robust for coarse modes, but mode switching can make simultaneous translation and rotation cumbersome.

Optical hand tracking such as Leap Motion removes the controller and exposes finger landmarks. That is attractive for retargeting a dexterous hand, but self-occlusion, object occlusion, ambiguous contact, and missing haptics become central. A glove provides more persistent joint sensing and may tolerate visual occlusion, yet fit, calibration, wireless transport, battery state, and morphology mapping still need validation. A tracker attached to a prop or handheld gripper can capture absolute rigid-body motion while leaving grasp state to another sensor.

A mechanical leader is the highest-commitment option. It can make joint correspondence, proprioception, and sometimes force feedback more direct, but it consumes workspace, requires per-hand/arm co-design, and introduces its own calibration, backlash, singularities, and pinch hazards. Recent DEXOP and DOGlove systems illustrate why “more hardware” can improve demonstration rate or contact awareness without becoming transparent bilateral teleoperation ^{[6] [7]}.

Leap Motion Controller 2 specifies up to 120 fps, but an official 2024 notice documents units that could show rotated views or no tracking. ^{[26] [27]}

The notice was batch-specific, so it should not be generalized to every unit. It is nevertheless valuable negative evidence: commissioning needs a known-pose sanity test and a tracking-loss test, not just a nominal frame-rate check. The same principle applies to every consumer tracker.

MANUS Quantum Metagloves specify 120 Hz sampling and at most 7.5 ms signal latency, excluding transport, retargeting, IK, control, and feedback latency. ^[19]

The current product page also lists Windows support, a finite wireless range, and battery duration. Treat those as integration constraints. If the main robot workstation runs Ubuntu, decide whether a separate Windows capture host is acceptable, how clocks will align, how packets cross the boundary, and what stale input means. A glove data rate is not a robot loop rate.

Independent robotic-ground-truth testing shows that VIVE Ultimate Tracker precision changes with operating conditions rather than matching fixed-camera capture universally. ^{[12] [10]}

Inside-out tracking depends on lighting, scene texture, occlusion, radio conditions, and runtime state. Test the tracker on the actual path and speed, then compare it with a fixed-camera or robot-ground-truth reference. A device can be excellent for coarse teaching and still be unsuitable for millimeter-scale insertion.

Figure 3.2 — Leap Motion Controller 2 desktop setup. Source: Ultraleap hand-tracking documentation; educational fair use.

3.5 From pose to command: validity comes before IK

The operator device should never publish an unqualified “desired robot pose.” Publish or log raw observation, device frame, reference space, timestamp, validity flags, buttons, and calibration version. A separate mapping layer applies clutch offsets, scale, workspace bounds, rate limits, and robot-frame transforms. IK or retargeting comes after that boundary. Chapter 8 will implement the full leader–follower loop; here the requirement is to preserve enough information to test it.

OpenXR exposes reference-space and tracking-state semantics, so an adapter must propagate invalid tracking instead of replaying the last pose indefinitely. ^[16]

On invalid tracking, the safe software behavior is normally to stop producing new motion targets, command a bounded hold through an already validated controller path, and require deliberate reacquisition. Exact stop behavior belongs to the robot's validated application design. The adapter is not a safety controller, and “hold last pose” is not equivalent to replaying the last velocity or continuing an integrated target.

A SpaceMouse provides six-DoF incremental motion without finger pose or haptic contact and is therefore not equivalent to a glove or mechanical leader. ^{[9] [6] [7]}

That limitation is often a strength. Incremental input plus a spring-centered device, clutch, and conservative scale is easier to inspect during first integration. Use it to validate the command path before adding absolute VR frames or hand retargeting. Optical systems such as DexPilot and AnyTeleop demonstrate broader arm-hand retargeting, while also exposing occlusion, morphology, and no-force-feedback limits ^{[11] [24]}.

Bounded Codex prompt — simulator-only OpenXR adapter “Implement a simulator-only OpenXR input adapter after inspecting the repository's current interfaces. Preserve raw pose, reference-space identifier, timestamp, position/orientation validity, tracking flags, and controller buttons. Add configurable clutching, scale, workspace bounds, velocity limits, and a watchdog. When tracking is invalid or stale, emit no new target and expose an explicit fault state. Use a fake runtime to test recentering, discontinuity, packet loss, and recovery. Do not add a real-robot transport, do not bypass existing controller limits, and do not label this component safety-rated. Run tests and provide a short acceptance report.”

3.6 Compute by workload, not by prestige

Separate four workloads: robot I/O and recording, visual simulation/rendering, policy inference, and model training. Their peaks may occur together or on different machines. Robot I/O benefits from predictable scheduling and network isolation; simulation wants GPU memory and rendering throughput; inference wants bounded latency; training wants memory capacity, utilization, and recoverable jobs. One enormous workstation can still be a poor controller if training saturates its GPU, storage, or network during an experiment.

Isaac Sim 5.0 lists 32 GB RAM, an RTX 4080-class GPU, and 16 GB VRAM as its minimum desktop tier, not as a sufficient specification for every VLA workload. Large models, multi-camera rendering, and concurrent processes can require more memory. ^[21]

At the cutoff, the same official requirements page describes 64 GB RAM with a 16 GB RTX 5080-class GPU as a “good” tier and a 48 GB RTX PRO 6000 Blackwell-class GPU as “ideal” ^[21]. Those are Isaac Sim platform tiers, not universal purchase prescriptions. A VLA training job can exceed them; a ROS 2 control and recording host may need far less. The 2026 Isaac Sim paper is a platform description, but its benchmark conditions should not be converted into a blanket workstation ranking ^[22].

Treat the middle rows as engineering guidance, not vendor minima. Measure peak allocated and reserved GPU memory, host RAM, encoder use, GPU utilization, thermal throttling, and frame latency with the real scene and models. Reserve headroom for driver/runtime changes. If training is rare, buying remote capacity can be more rational than placing a maximum-tier GPU beside the robot.

Do not let the model-training environment own the emergency stop, protective stop, or low-level safety path. Keep experiment orchestration restartable. A failed model or out-of-memory process should end an episode, not leave an operator guessing whether the robot command stream is still active.

3.7 Network and storage: make an episode reconstructable

Design the network as at least two logical concerns: latency-sensitive robot/control traffic and high-volume camera/storage traffic. They may share managed hardware in a small lab, but their VLAN, addressing, firewall, multicast, PTP, and bandwidth assumptions should be documented. Industrial GigE cameras can support long cables and PTP, while consumer USB cameras move contention to the host controller ^{[1] [2]}. Draw actual cable paths and USB root hubs instead of counting only ports.

Compute an uncompressed upper bound before choosing disks. For an RGB stream,

bytes per second, where C is bytes per pixel and f is frames per second. Three 1920×1080, three-byte, 30 fps RGB streams are about 560 MB/s, or roughly 2.0 TB per hour before compression and container overhead. A 1280×720, 16-bit, 30 fps depth stream adds about 55 MB/s. Compression can reduce the total dramatically, but the ratio depends on motion, noise, codec, and quality. Measure actual p95 write bandwidth and episode size.

Camera, robot, teleoperator, force, and tactile streams need a common episode clock and calibration version because sample rates alone cannot reconstruct alignment. ^{[2] [5] [8]}

An episode manifest should include start/end time, task and reset identifiers, robot and end-effector configuration, device serials, firmware/driver versions, calibration hashes, clock sources, topic/stream schema versions, dropped-sample counters, operator interventions, stop/fault events, and checksums. Keep raw device observations when feasible; derived actions can then be regenerated after a mapping fix.

UMI reports less than 1 cm and 4 degrees tracking error in its tested setup and makes relative trajectories and latency matching explicit. The measurement does not certify another camera, robot, or environment. ^[5]

UMI is useful here because it treats latency and action representation as part of the data interface, not post-processing trivia. FastUMI extends the collection direction to more than 10,000 demonstrations across 22 tasks, yet its tracker dependence remains a hardware dependence ^[29]. DexUMI reports transfer across two dexterous-hand platforms, while requiring a hand-specific exoskeleton and controlled camera relationship ^[30]. These systems argue for recording the interface assumptions that produced a dataset.

Figure 3.3 — Multiple times and their provenance inside one episode. Original synthesis.

Use a storage lifecycle: fast local NVMe for acquisition, immutable episode packaging after validation, replicated project storage for accepted episodes, and a documented deletion policy for rejected raw data. Test recovery from a full disk, unplugged destination, permission error, and partial episode. A recorder that silently drops the final minute is a data-integrity fault.

3.8 Tables, mounts, power, and independent stopping

The physical support changes both safety and data quality. A flexible table moves the robot base relative to world cameras. A camera tripod can be nudged without changing a configuration file. A dangling wrist-camera cable changes payload and may enter a pinch zone. A glossy unshaded light source changes exposure across the day. Treat tables, posts, lighting, cable chains, and fixtures as metrology hardware.

Anchor the robot using the manufacturer's mounting pattern and verify table stiffness under expected accelerations. Put fixed cameras on a rigid structure referenced to the robot base where possible. Add witness marks or tamper indicators at adjustable joints. Route power and data separately when practical, provide strain relief at every moving transition, and keep service loops outside reachable pinch and sweep volumes.

The current application-level safety standard assigns safety reasoning to the integrated robot application and cell, including tooling, workpieces, layout, and operation ^[15]. A collaborative-capable arm does not make a camera boom, sharp gripper, heavy fixture, or operator-access pattern safe. A qualified local risk assessment determines guarding, enabling, speed/force limits, and validation.

A UPS maintains selected electronics long enough to preserve logs or shut down; it is not a safety function unless specifically designed and validated as one. Decide whether the robot controller, workstation, network switch, cameras, and storage ride through together. Mixed behavior can be worse than a clean shutdown: cameras may disappear while the controller remains active, or the recorder may survive without its clock master.

Place the physical E-stop and enabling controls where the operator can reach them without entering the robot path. Keep software cancellation, protective stop, and emergency stop conceptually and electrically distinct. Verify reset behavior: recovery should require an intentional action and should not replay a queued teleoperation target.

3.9 Four purpose-sized assemblies

The following assemblies extend the four arm-and-hand purposes from Chapter 2. They are not fixed bills of materials; they are coherent starting points whose interfaces can be accepted.

Assembly A — low-cost learning and pick/place

Use a low-cost leader–follower arm or compact educational arm, one fixed RGB or RGB-D camera, and the mechanical leader or a simple gamepad. A modest computer can record 640×480 or similar streams; official SO-101 examples use 640×480 at 30 fps and preserve calibration by robot ID ^[13]. Put the cell on a rigid small table, add a physical power cutoff appropriate to the device, and keep the workspace enclosed or out of casual reach.

The acceptance target is not industrial repeatability. It is a complete loop: calibrated leader and follower, one camera with verified time, 20 repeatable reset-and-pick episodes, no unexplained dropped frames, and a recoverable dataset. This setup cannot validate high-speed contact, heavy payloads, or industrial safety functions.

Assembly B — reliable ROS 2 and MoveIt research

Use a supported research arm with a parallel gripper, one overview RGB-D camera, one compact wrist RGB camera, and a SpaceMouse or handheld controller. Choose a capture/control host that can run drivers, visualization, and recording without GPU contention. Add a managed switch, rigid camera post, calibrated board, robot-rated base, E-stop integration, and a controlled access boundary.

Walk one command from operator input to simulator before enabling hardware. The SpaceMouse validates scaling, clutch, frame direction, rate limiting, and watchdog behavior with few hidden states ^[9]. Chapter 4 will turn this physical interface contract into ROS 2 nodes, topics, actions, TF, and lifecycle transitions.

Assembly C — VLA-ready vision and teleoperation collection

Use a supported arm-hand pair, fixed overview and side cameras plus a wrist view, a Quest-class OpenXR input or tracked controller, and a workstation in the vision/teleop tier. Hardware-triggered global-shutter cameras are justified when fast motion or cross-camera timing dominates; otherwise validated RGB-D devices may reduce integration cost. Provide fast local acquisition storage, replicated project storage, calibration versioning, and a scene-independent clock audit.

The acceptance target is a synchronized episode, not a policy score. Require a manifest, p95/p99 frame age, clock drift, dropped-frame counts, tracking-loss events, operator intervention, and a replay that overlays robot state and images correctly. Keep future language labels and model outputs separate from raw sensor provenance.

Assembly D — contact-rich dexterous manipulation

Use a torque-capable research arm and a dexterous hand whose payload, cabling, tactile access, spares, and simulator assets were already accepted in Chapter 2. Add multiple high-quality views, force/tactile logging, and either a calibrated glove, haptic glove, or mechanical exoskeleton. A larger GPU is useful for multi-camera replay and learning, but deterministic capture and safe control should remain independent.

Mechanical and haptic interfaces can improve contact awareness, yet their feedback is not automatically transparent or stable. Classical teleoperation work shows a fundamental stability–transparency tension under delay ^{[20] [17]}. Record both human and robot-side states, force limits, saturation, and intervention. This assembly is the most expensive because it buys observability and embodiment, not merely compute.

3.10 Acceptance day: one synchronized episode walkthrough

Take Assembly C as the concrete walkthrough. First, lock the robot, tool, camera posts, lenses, lighting, switch, and clocks. Photograph the cell. Assign device IDs and record firmware and driver versions. Calibrate intrinsics, fixed-camera extrinsics, wrist-camera hand-eye transform, and controller-to-robot mapping. Validate every chain on held-out poses.

Figure 3.4 — Official OAK-D Pro product image. Source: Luxonis hardware documentation; educational fair use.

Second, run with the robot disabled or in simulation. Move the controller through the intended workspace, recenter twice, block a tracker, remove a controller from view, pause the runtime, and reconnect it. The adapter must expose each invalid state, the mapping must cease creating new targets, and recovery must require clutch reacquisition. Record the sequence.

Third, load every camera and recorder at once. Run the representative motion and lighting pattern for longer than the planned episode. Measure timestamp offset, drift, frame age, lost samples, CPU/GPU memory, temperatures, write rate, and storage growth. Fill the disk boundary in a separate safe test and confirm that acquisition fails explicitly.

Fourth, exercise power and stopping with qualified supervision and no autonomous policy. Test software cancellation, protective behavior, physical E-stop, controlled UPS shutdown, and restart as separate cases. Confirm that no buffered target is executed after reset. The workstation must never decide that the cell is safe merely because all processes restarted.

Finally, record at least 20 short teleoperated trials with scripted resets. Replay them side-by-side with the same frame and calibration version. Reject an episode when its provenance is missing, even if the motion looked successful. Only after this gate should a learning pipeline consume the data.

Acceptance checklist

• [ ] Every camera has saved intrinsics, distortion, exposure settings, serial number, and a held-out residual report.
• [ ] Every camera-to-base/tool and operator-to-robot transform has a version and invalidation rule.
• [ ] Offset, drift, p95/p99 frame age, and dropped samples were measured under full concurrent load.
• [ ] Tracking invalidity, recentering, stale input, disconnection, and deliberate reacquisition were tested.
• [ ] GPU/host memory, encoder, network, and storage headroom were measured with the real workload.
• [ ] Episode manifests include clocks, calibrations, schema versions, faults, interventions, and checksums.
• [ ] Robot base, cameras, lighting, fixtures, and cables remain mechanically stable through the task.
• [ ] UPS behavior, E-stop, protective behavior, guarding, reset, and queued-command clearing were independently validated.
• [ ] Simulator-only and read-only tests pass before any teleoperation target reaches hardware.
• [ ] A named person owns calibration, data integrity, network, safety validation, and release of the cell for use.

3.11 Evidence boundaries, disagreements, and prior surveys

Official datasheets are the primary source for connector, range, rate, and supported-platform claims. They are not independent rankings. Peer-reviewed or primary experimental papers can compare a defined setup, but their error and success numbers do not certify a different room, camera, robot, or controller. Company demos are useful for discovering interfaces; without task definitions, denominators, failure accounting, and reproducible settings, they do not set acceptance thresholds.

Two disagreements should remain conditional. First, consumer RGB-D/inside-out tracking is cheaper and faster to integrate, while industrial trigger/PTP capture is more auditable and more expensive. Slow pick/place may favor the former; fast, multi-view, contact-sensitive measurement may justify the latter. Second, optical input offers freedom and portability, while a mechanical leader offers correspondence and feedback at the cost of embodiment-specific hardware. The correct choice follows the task's tolerance and data objective.

This chapter refreshes the commercial-hand and simulator-era context from #S1 and #S9 without copying their product lists. The new conclusion is operational: a hand, camera, or simulator is admitted only through a versioned interface and a measured acceptance gate. Broad VLA and agentic-robotics synthesis is deferred, but the data provenance and authority boundary established here are prerequisites for those volumes.

For deeper project-specific readings, see Terry's bilingual notes for UMI (KO, EN), DexUMI (KO, EN), and DEXOP (KO, EN). These links are reading aids; the claims above cite the original sources.

Open questions and recurring failure modes

End-to-end pose truth remains hard when wireless runtimes, separate operating systems, and proprietary filtering intervene. PTP cannot repair unknown sensor pipelines. A high-rate glove cannot reveal contact force it does not sense. A large GPU cannot compensate for inconsistent camera geometry. More cameras can reduce occlusion while multiplying clocks, calibration edges, storage, and failure combinations.

Watch for symptoms rather than brands: good static overlay but bad moving overlay suggests time error; consistent error in one workspace corner suggests geometric coverage; sudden jumps after recentering suggest reference-space handling; successful live view but corrupt replay suggests manifest or queue failure; stable robot state with missing operator validity suggests unsafe adapter semantics; calibration drift after maintenance suggests an untracked physical change.

Manufacturing Cell Checkpoint

Before leaving Part I, freeze a cell-interface release. It should state the task envelope, camera fields of view and calibration residuals, input device and validity semantics, peak compute/load measurements, episode schema and retention, network/clock topology, physical drawing, stop/guarding design, and named owners. Give the release a version and make experiments cite it.

The minimum go/no-go review has five questions:

1. Task: Can every critical feature and occlusion be observed at the speed and lighting of the real task?
2. Authority: Does every input loss, stale state, recentering, saturation, and reset have a tested bounded outcome?
3. Data: Can an accepted episode be reconstructed from raw streams, clocks, transforms, calibrations, and schema versions?
4. Capacity: Does the full concurrent workload meet latency, memory, bandwidth, thermal, and retention gates with headroom?
5. Safety and ownership: Can the cell stop independently of software, and has a qualified person approved the integrated layout and recovery procedure?

If any answer is unknown, the cell is not “almost ready.” It has a named experiment to run before hardware authority expands.

What to Learn Next

Chapter 4 turns these hardware contracts into software motion. The camera and operator schemas become ROS 2 topics; long-running motion becomes an action; transforms become time-indexed TF edges; drivers acquire lifecycle states; QoS and discovery meet the workstation–robot network. The key question will be not “How do I publish a pose?” but “Which process is allowed to transform a valid, timed observation into a bounded controller request, and how is failure visible?”

Annotated research trail

These sources deepen teleoperation, calibration, and data capture. They are grouped by the assumption or experiment to inspect, rather than used as a list of borrowed success rates. Read each result within its platform and protocol.

References

1. Basler AG (2026a). Basler ace 2 Camera Family. Official product documentation.
2. Basler AG (2026b). Precision Time Protocol for Basler GigE Cameras. Official technical documentation.
3. Basler AG (2026c). Basler ace 2 a2A2600-64ucBAS Specification. Official product specification.
4. Borghi et al. (2025). Empirical Comparison of Four Stereoscopic Depth Sensing Cameras for Robotics Applications. arXiv:2501.07421.
5. Chi et al. (2024). Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS 2024, arXiv:2402.10329.
6. Fang et al. (2025a). DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulation. arXiv:2509.04441.
7. Fang et al. (2025b). DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove. RSS 2025, arXiv:2502.07730.
8. Foote, T. (2013). tf: The Transform Library. IEEE TePRA. DOI: 10.1109/TePRA.2013.6556373.
9. Franka Robotics (2026). Franka SpaceMouse Teleoperation Example. Official repository.
10. Grunwald et al. (2024). Evaluating the Precision of the HTC VIVE Ultimate Tracker under Varied Conditions. arXiv:2409.01947.
11. Handa et al. (2020). DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System. ICRA 2020, arXiv:1910.03135.
12. HTC VIVE (2026). VIVE Ultimate Tracker Developer Guidelines and Product Support. Official developer documentation.
13. Hugging Face LeRobot / TheRobotStudio (2025). SO-101 Hardware and Calibration Guide. Official documentation.
14. Intel RealSense (2025). Intel RealSense D455 Product Specification. Official product specification.
15. ISO (2025). ISO 10218-2:2025 Robotics — Safety Requirements — Part 2. International Standard, edition 3.
16. Khronos Group (2024). OpenXR 1.1 Specification. Official specification.
17. Lawrence, D. A. (1993). Stability and Transparency in Bilateral Teleoperation. IEEE Transactions on Robotics and Automation. DOI: 10.1109/70.258054.
18. Luxonis (2025). OAK-D Pro Technical Specification. Official product specification.
19. MANUS (2026). MANUS Quantum Metagloves Technical Specifications. Official product documentation.
20. Niemeyer, G., and Slotine, J.-J. E. (1991). Stable Adaptive Teleoperation. IEEE Journal of Oceanic Engineering. DOI: 10.1109/48.64895. [Niemeyer & Slotine, 1991]
21. NVIDIA (2025). Isaac Sim 5.0 System Requirements. Official platform requirements.
22. NVIDIA Isaac Sim Team (2026). NVIDIA Isaac Sim: Enabling Scalable, GPU-Accelerated Simulation for Robotics. arXiv:2606.03551.
23. Park, F. C., and Martin, B. J. (1994). Robot Sensor Calibration: Solving AX=XB on the Euclidean Group. IEEE Transactions on Robotics and Automation. DOI: 10.1109/70.326576. [Park & Martin, 1994]
24. Qin et al. (2023). AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System. RSS 2023. DOI: 10.15607/RSS.2023.XIX.015.
25. Tsai, R. Y., and Lenz, R. K. (1989). Position and Orientation Calibration of a Robot Camera as a Pan-Tilt Unit. IEEE Transactions on Robotics and Automation. DOI: 10.1109/70.34770. [Tsai & Lenz, 1989]
26. Ultraleap (2024a). Leap Motion Controller 2 Datasheet. Issue 15.
27. Ultraleap (2024b). Leap Motion Controller 2 December 2024 Calibration Notice. Official support notice.
28. Ulrich, M., and Hillemann, M. (2024). Uncertainty-Aware Hand–Eye Calibration. IEEE Transactions on Robotics 40. DOI: 10.1109/TRO.2023.3330609. [Ulrich & Hillemann, 2024]
29. Wu et al. (2024). FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset. arXiv:2409.19499.
30. Xu et al. (2025). DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation. CoRL 2025, PMLR 305.
31. Zhang, Z. (2000). A Flexible New Technique for Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: 10.1109/34.888718.