Eval (sim)

Part of the OpenRAL public-symbol inventory. Hand-curated; (LNN) markers are refreshed by tools/refresh_methods_linenos.py.

`python/sim/src/openral_sim/policy.py`

Policy adapter protocol — the contract every VLA backend must satisfy.

class PolicyAdapter(Protocol) — Uniform VLA / policy interface. (L25)
attr spec: VLASpec, device: str
reset() -> None — Reset action queue / RNG at episode start. (L36)
step(observation, instruction) -> NDArray[np.float32] — Next action. (L39)
close() -> None — Release GPU / file handles. (L57)

`python/sim/src/openral_sim/rollout.py`

Sim rollout protocol — the typed contract every scene adapter must satisfy.

class StepResult — One environment transition. (L67) fields: observation, reward, terminated, truncated, info
class SimRollout(Protocol) — Minimal gym-style env contract. (L86)
attr scene: SceneSpec, task: TaskSpec
reset(seed=None) -> Observation (L155)
step(action) -> StepResult (L158)
render() -> NDArray[np.uint8] | None — HWC uint8 RGB or None. (L161)
close() -> None (L164)
duck-typed extension: mujoco_handles() -> tuple[mujoco.MjModel, mujoco.MjData] | None — Optional, NOT part of the Protocol; MuJoCo-backed adapters implement it so openral sim run --view can open a passive viewer. Callers MUST getattr(env, "mujoco_handles", None) and tolerate None.
duck-typed extension: sim_time_ns() -> int | None — Optional, NOT part of the Protocol; the backend's authoritative elapsed sim time in ns, the seam a sim /clock publisher reads. MuJoCo-backed adapters return round(MjData.time * 1e9) via sim_time_ns_from_mujoco_handles; sidecars that carry sim_time_ns expose that wire value. Monotonic non-decreasing within an episode; backends that rewind MjData.time on reset (robocasa) restart it, so a cross-reset-monotonic consumer maintains its own offset (SimAttachedHAL.sim_time_ns). None = no sim clock (PushT or a sidecar lacking time). Callers MUST getattr(env, "sim_time_ns", None) and treat both missing + None as "no clock" (fall back to wall time).
duck-typed extension: enable_intrinsic_viewer() -> None — Optional, NOT part of the Protocol; adapters whose engine draws its own window (e.g. gym_pusht) implement it so SimRunner can switch them into live-view mode at activate() time. When present, SimRunner skips the MuJoCo viewer path entirely.
sim_time_ns_from_mujoco_handles(handles: tuple[Any, Any] | None) -> int | None — Shared helper: round(MjData.time * 1e9) from a mujoco_handles() (model, data) tuple, None when handles is None. The single place the MuJoCo-backed adapters' sim_time_ns() implementations route through. (L21)
class EpisodeResult — Outcome of one episode. (L169) fields: success, steps, total_reward, mean_step_latency_ms, max_step_latency_ms, latency_budget_ms, budget_violations, frames, metadata
summary() -> str — Human-readable single line. (L221)

`python/sim/src/openral_sim/registry.py`

Registries that map ID strings to backend factories.

class _Registry(Generic[T]) — Tiny ID → factory map. (L43)
__init__(kind) (L54)
kind -> str [@property] (L61)
register(name, *, fixed_robot=None, provision=None) -> Callable[[Callable[..., T]], Callable[..., T]] — Decorator. The optional fixed_robot kwarg (only meaningful on SCENES) declares which robot_id the scene's physics backend hard-wires; the CLI rejects mismatched --robot values with ROSConfigError instead of silently swapping the robot. The optional provision kwarg (also SCENES-only) declares the scene's out-of-tree provisioning step — the multi-GB download, fork clone, or sidecar-venv build the factory would otherwise trigger on first call — so callers that can afford slow work up front (openral deploy sim before ros2 launch) keep it out of the HAL's 300 s-bounded on_configure. Must be idempotent; the factory calls the same helpers again and they short-circuit. Leave None for pip-only backends. (L64)
get(name) -> Callable[..., T] — Look up by ID. (L136)
fixed_robot(name) -> str | None — Scene's hard-fixed robot id (None for free-axis scenes or unregistered names). (L117)
provision(name) -> Callable[[], None] | None — Scene's pre-launch provisioner (None for backends with nothing to fetch and for unregistered names — a preflight is advisory, so unlike get this does not raise on an unknown id). (L125)
names() -> list[str] — Sorted IDs. (L151)
__contains__(name) -> bool (L155)
module-level globals: SCENES, POLICIES, ROBOTS — three _Registry[T] singletons.

`python/sim/src/openral_sim/factory.py`

make_env(env_cfg) -> SimRollout — Build the simulated environment. (L25)
make_policy(env_cfg) -> PolicyAdapter — Build the policy. (L43)
make_robot(env_cfg) -> RobotDescription | None — Resolve robot description if registered. (L61)

`python/sim/src/openral_sim/sim_runner.py`

Per-step InferenceRunner for the simulation runtime.

class SimRunner(InferenceRunnerBase) — One-tick = one-env-step inference runner that drives a SimEnvironment for n_episodes episodes. Subclasses InferenceRunnerBase so sim and hardware (DeployRunner) share the InferenceRunner Protocol. (L249)
SimRunner.__init__(env_cfg, *, view=False, strict_view=False, instruction_override=None, deadline_overrun_policy=WARN, recorder=None) — Defer env / policy construction to activate(); rate_hz is fixed at 1000 Hz (sim is not real-time, deadline policy defaults to WARN). instruction_override is the explicit --instruction CLI value (or None) that wins over a scene's per-episode obs["task"] language via the private _resolve_step_instruction helper. recorder is an optional openral_dataset.RolloutRecorder fanned out alongside _EpisodeBuffer — additive, never a substitute. (L293)
SimRunner._record_to_recorder(action, reward, terminated, truncated) -> None — Internal helper that extracts per-step state / rendered frame / action and forwards to the attached recorder; broadcasts the single rendered viewpoint to every camera key declared on the robot (sim envs typically expose one render but multiple vla_feature_keys). Errors are logged, not raised. (L380-ish)
SimRunner.activate() -> None — Validate manifest via _check_rskill_compatibility, build env + policy concurrently via _build_env_and_policy (GH-134: make_env + make_policy run on a 2-worker ThreadPoolExecutor by default), arm the first reset-tick, open the outer sim.run OTel span. (L368)
SimRunner.deactivate() -> None — Flush a trailing episode if any, close the viewer / policy / env, close the outer span. Idempotent. (L452)
SimRunner._should_terminate() -> bool — Returns True once n_episodes EpisodeResults have been emitted. (L489)
SimRunner._tick_impl(tick_idx) -> TickResult — Dispatch reset-tick vs step-tick. (L537)
SimRunner._reset_tick(tick_idx) -> TickResult — env.reset + policy.reset; action_applied=False, inference_ms=0.0, step_idx=None. (L547)
SimRunner._step_tick(tick_idx) -> TickResult — policy.step + env.step; populates step_idx, reward, terminated, truncated, action_applied=True. (L597)
SimRunner._finalize_episode() -> None — Build an EpisodeResult from the per-step _EpisodeBuffer, append to episode_results, reset the buffer. (L804)
class _EpisodeBuffer — Private dataclass accumulating per-step latencies / frames / rewards inside one episode; reset on each boundary. (L217)
_check_rskill_compatibility(env_cfg) -> RSkillManifest | None — Load the rSkill manifest, run rSkill.check_compatibility against the registered RobotDescription, return the manifest or None for built-in mock policies. Strict-by-construction. (L915)
_SEQUENTIAL_INIT_ENV: str = "OPENRAL_SIM_SEQUENTIAL_INIT" — Module-level constant: the env var that forces _build_env_and_policy onto the legacy sequential path (set to "1"). (L1014)
_build_env_and_policy(env_cfg) -> (SimRollout, PolicyAdapter) — GH-134: build env + policy concurrently on a 2-worker ThreadPoolExecutor by default; sequential when OPENRAL_SIM_SEQUENTIAL_INIT=1. Logs structured sim_init_parallel / sim_init_sequential records with env_ms / policy_ms / total_ms / saved_ms. Exceptions from either side propagate verbatim — the helper does not catch ROSError (or anything else). (L1095)
_seed_global_rngs(seed) -> None — Seed Python / NumPy / Torch RNGs so stochastic policies reproduce per (seed + episode_idx). (L1206)
_open_viewer_and_pacing(env, env_cfg, *, strict_view) -> (Any, float | None) — Open a passive mujoco.viewer against the adapter's mujoco_handles() with show_left_ui=False, show_right_ui=False (only the sim renders), set the camera + geom visibility via _aim_viewer_camera, and compute the per-step sleep budget so the viewer renders at the env's natural sim-time. (L1255)
_aim_viewer_camera(viewer, env, mj_model, mj_data) -> None — Set the viewer's opening camera + geom visibility (lazily imports openral_hal.depth_cloud.{apply_robosuite_visual_geomgroups, initial_viewer_camera} — paying openral_hal's torch/lerobot import cost only at interactive viewer-open, mirroring openarm_robosuite/_assets.py): hides robosuite collision shells so textures render, then sets the free-camera opening pose via initial_viewer_camera (eye at a 3rd-person scene camera, orbit pivot on the base; base-aligned default for camera-less models). Camera stays mjCAMERA_FREE so the user can orbit (drag) / zoom (scroll); only the initial view is set. Best effort — any failure logs viewer_camera_aim_failed and leaves MuJoCo's default camera. (L1317)

`python/sim/src/openral_sim/benchmark.py`

Benchmark runner — loops a bare list[BenchmarkScene] (loaded via load_benchmark_suite + raise_on_invalid_suite) and emits a RSkillEvalResult.

check_benchmark_task_compatibility(manifest, *, task_id, scene_id) -> None — Task-data gate: raises ROSCapabilityMismatch when manifest.evaluated_tasks is non-empty and none of its entries cover the scene's task_id/scene_id (prevents e.g. a LiftCube policy running on PickCube). Permissive (logs rskill_task_compat_undeclared) when evaluated_tasks is empty. Called from run_benchmark_scene before the rollout, skipped for mock policies + hf:// URIs. (L80)
_task_matches(task_id, scene_id, declared) -> bool — Whether a declared entry covers the scene: exact task.id, a "<scene>/<…>" family prefix ("libero_spatial" covers libero_spatial/0..9), or the bare scene.id. (L66)
filter_scenes_for_skill(scenes, manifest) -> tuple[list[BenchmarkScene], list[BenchmarkScene]] — Suite analogue of check_benchmark_task_compatibility: partitions a suite into (kept, skipped) by matching each scene against manifest.evaluated_tasks (via _task_matches). None/empty evaluated_tasks is permissive (keep all, mirrors the single-scene gate's legacy branch). Lets one benchmark run execute every task an rSkill supports and skip the rest, and closes the gap where the suite path never gated tasks (mismatched-but-same-embodiment ran to a silent 0). (L143)
_manifest_for_filter(vla) -> RSkillManifest | None — Loads the rSkill manifest for filter_scenes_for_skill, or None for unfilterable skills (built-in mock policies / raw hf:// URIs — same guard as the single-scene gate). (L126)
run_benchmark(scenes, *, suite_id, vla, device=None, save_dir=None, video_dir=None) -> tuple[RSkillEvalResult, list[EpisodeResult]] — Auto-filters scenes to the rSkill's evaluated_tasks (filter_scenes_for_skill, logging a benchmark_suite_task_filter summary of skips and raising ROSCapabilityMismatch if nothing matches), then iterates the kept scenes × range(seed, seed + n_episodes), drives each (BenchmarkScene, seed) tuple with a fresh SimRunner, and aggregates into a validated RSkillEvalResult. video_dir (default None) enables per-step frame capture and writes one MP4 per episode via _website_video.write_world_videos (<task>[_seed<n>]_<rskill>_<success|fail>.mp4 + videos.json), freeing each episode's frames after the write so large suites stay memory-flat (the runner additionally caps in-memory frames at 8192 with stride-doubling subsampling for very long episodes, and write_world_videos removes the opposite-outcome stale sibling MP4 + manifest record when a re-run's outcome flips). Per-scene robot_id / task / max_steps pulled from each BenchmarkScene; suite-level invariants pre-checked by raise_on_invalid_suite (the runner does not re-validate). All args after scenes are keyword-only — callers must name suite_id and vla. (L182)
_aggregate_results(scenes, *, suite_id, vla, per_task, episodes) -> RSkillEvalResult — Roll per-task booleans into per-task / avg success rates. Suite-level benchmark.name / benchmark.simulator come from scenes[0].metadata.display_name / .simulator when present, else fall back to suite_id / scenes[0].scene.id. benchmark.arxiv auto-derived from scenes[0].metadata.paper when the URL contains arxiv.org/. max_steps in the protocol summary is max(scene.task.max_steps for scene in scenes) so the bound is the suite worst-case, not just scenes[0]. Pulled out for unit-test reuse. (L349)
run_benchmark_scene(scene, vla, *, device=None, save_dir=None, config_path=None, view=None, record_video=False) -> tuple[RSkillEvalResult, list[EpisodeResult]] — Single-scene sibling of run_benchmark; backs openral benchmark scene. record_video (default False) captures per-step world frames into each EpisodeResult.frames so benchmark scene --save-video can write clean website MP4s. Iterates range(scene.seed, scene.seed + scene.n_episodes) against the one (scene, task) pair carried by a BenchmarkScene and emits the same RSkillEvalResult shape so openral benchmark report does not need to distinguish entrypoints. Raises ROSConfigError when scene.robot_id is None. view (tri-state, default None) is the opt-in viewer flag for parity with sim run: None keeps the historical headless behaviour (eval/CI unaffected), an explicit True/False is resolved through cli._resolve_view and passed to SimRunner. (L463)
_aggregate_scene_results(scene, vla, successes, episodes, config_path) -> RSkillEvalResult — Single-scene counterpart of _aggregate_results; shares the output schema. PushT special-case mirrors the suite path. Embeds config_path into reproduction_cli for byte-identical reruns from disk. (L608)
default_output_path(weights_uri, benchmark_id) -> str — Canonical mapping rskills/<dir> (or bare name) → rskills/<dir>/eval/<id>.json. (L699)
update_rskill_benchmarks(skill_dir, benchmark_id, score) -> Path — Surgical rewrite of the benchmarks: block in <skill_dir>/rskill.yaml that preserves every other comment + line; re-validates the merged manifest through RSkillManifest before writing. Closes the openral benchmark run → rskill.yaml loop so manifest headlines stay in sync with the eval JSONs. Raises FileNotFoundError if no manifest, ROSConfigError on unknown benchmark_id / out-of-range score. (L742)
update_rskill_benchmarks_from_uri(weights_uri, benchmark_id, score) -> Path — Resolve the skill reference to a local dir and delegate to the manifest updater; mirrors default_output_path so the CLI passes the same ref it already holds. (L842)

`python/sim/src/openral_sim/cli.py`

sim_app: typer.Typer — Public openral sim Typer group. Mounted into the top-level openral Typer tree by openral_cli.main. Hosts the run leaf (--config / --robot / --scene / --task / --rskill / …) and the list leaf (registry printer).
sim_run_app: typer.Typer — The leaf Typer (invoke_without_command=True) exposing every rollout CLI flag; users invoke it as openral sim run.
_sim_run_callback(...) — Typer callback carrying every rollout CLI flag; builds a SimpleNamespace and dispatches to _run. The optional --dashboard / --dashboard-port flags wrap _run in attached_dashboard(...). Same flag is mirrored on openral deploy run and openral benchmark run.
_discover_sim_configs() -> list[Path] — Recursive read-only walk of scenes/**/*.yaml (benchmark / sim / deploy) under the repo root, sorted by relative path. Safe to call without any sim dependencies. (L486)
sim_list() -> None — @sim_app.command("list") callback that prints every sim config under scenes/**/*.yaml, each a paste-able --config path for openral sim run. No rollout, no OTel span, no GPU. (L507)
_resolve_save_video(raw) -> Path | None — Map the --save-video Typer string to the legacy Path | None semantics (empty string ⇒ example_videos/). (L293)
_load_or_build_env(args) -> SimEnvironment — --config XOR explicit flags; --config combined with any of --task / --robot / --scene / --rskill / --instruction raises ROSConfigError. Also enforces the scene-fixed-robot guard: when SCENES.fixed_robot(scene.id) is set and disagrees with env.robot_id, raises ROSConfigError naming the scene's required robot. Rejects a non-VLA rSkill (manifest.model_family is None — detector / reward / playbook kinds) with the same message openral benchmark run gives, rather than letting VLASpec(id=None) surface a raw pydantic string_type error. openral sim run --dry-run runs _check_rskill_compatibility before returning, so the embodiment / sensor gate a real rollout hits at SimRunner.activate also gates the dry run. (L309)
_resolve_view(flag) -> tuple[bool, bool] — tri-state resolver returning (view, strict_view) from the --view/--no-view/auto flag plus MUJOCO_GL / DISPLAY env. (L525)
main(argv=None) -> int — Thin wrapper that invokes sim_run_app with standalone_mode=False so tests can get the return code without sys.exit. The legacy standalone console script was removed in 2026-05; main stays for internal callers. (L454)
_run(args) -> int — Body of the callback after argv parsing + OTel setup. Configures observability with service name ral-sim. (L714)
_write_videos(args, results, env_cfg) -> None — Dispatch --save-video to the debug or world writer per --video-style; raises ROSConfigError for any other style. (L819)
_write_debug_videos(args, results, env_cfg) -> None — Render the 3-panel debug MP4(s) via openral_sim._video.save_episode_mp4. (L834)
_write_website_videos(args, results, env_cfg) -> None — Thin adapter that pulls scene/rskill/section from the run and delegates to openral_sim._website_video.write_world_videos. --video-style world. (L878)

`python/sim/src/openral_sim/_video.py`

Shared 3-panel rollout-debug MP4 helper (was examples/_video.py).

save_episode_mp4(result: EpisodeResult, path: Path, *, title: str = "") -> Path — Render (policy input grid | observation-state plot) for one episode, falling back to the rollout/world stream only when the adapter recorded no input frames. Re-exported from openral_sim. (L62)
_stack_padded_states(states) -> NDArray[np.float32] — Pad ragged observation-state arrays for plotting. (L167)
_resize_sequence(frames, target) -> list[NDArray[np.uint8]] (L182)
_resize_frame(frame, target) -> NDArray[np.uint8] (L220)
class _JointPlotRenderer — Reusable matplotlib canvas rasteriser; labels generic state channels s0... rather than claiming every backend's observation.state is a joint vector. (L240)
__init__, render_at_step, _snapshot, __del__

`python/sim/src/openral_sim/_website_video.py`

Clean single-view world MP4 helper for website hero clips (overlays rendered by the page, not burned into pixels).

save_world_mp4(result: EpisodeResult, path: Path, *, fps: int = 20, size: int = 1024, min_duration_s: float = 2.0) -> Path — Write only result.frames (the world/viewer render), center-cropped to a square and resized to size × size, libx264/yuv420p, holding the final frame when needed so short successes remain watchable. Raises ValueError on empty frames, non-.mp4 suffix, non-positive size, or negative min_duration_s. (L43)
write_world_videos(episodes, out_dir, *, scene, rskill, section, size=1024, fps=20) -> list[dict] — Write one clean world MP4 per episode named <scene>_<rskill>[_ep<i>]_<success|fail>.mp4 + merge a videos.json manifest. Shared by openral sim run --video-style world and openral benchmark scene --save-video. (L117)
append_video_manifest(manifest, records) -> None — Merge video records into videos.json, replacing same-file entries; no-op on empty. (L189)
_square(frame, size) -> NDArray[np.uint8] — Center-crop one HWC frame to a square and resize to (size, size) RGB. (L225)

Eval adapters

`python/sim/src/openral_sim/backends/robocasa.py`

RoboCasa kitchen + GR1 tabletop adapter. - provision_robocasa(backend_id) -> None — The slow half of a first run, split out of _build_robocasa_sim so openral deploy sim can run it in front of ros2 launch instead of inside the HAL's 300 s-bounded on_configure: ensure_backend_deps(backend_id) (clone + editable-install the fork) → import robocasa probe (raises the actionable libero↔robocasa robosuite-conflict hint instead of a bare cannot import name 'PandaOmron') → ensure_robocasa_assets() (~11 GB). Order matters — the kitchen downloader is driven via runpy.run_module and needs an importable robocasa. Registered per scene as provision=partial(provision_robocasa, …) and called again by the build path, so preflight and build cannot drift. Idempotent (install probe + readiness sentinel). - _robocasa_backend_id(scene_id) -> str — "robocasa_gr1" for the robocasa/gr1/<Task> family, else "robocasa_kitchen" (both the procedural robocasa id and every robocasa/<Task> prebuilt). The two packages both import as robocasa and ship different asset trees, so provisioning the wrong one leaves the scene unrunnable. - _fit_panda_mobile_action(action, *, env_dim, state_layout) -> NDArray[np.float32] — Reconciles the known 12↔11 RoboCasa dataset/BASIC skew; only state_layout="xr1_8d" may zero-fill a 7-D arm+gripper action into PandaMobile base+torso slots. Unknown widths raise ROSConfigError. - _xr1_robocasa_state(raw) -> NDArray[np.float32] — Builds XR-1 RoboCasa's 8-D [arm_joint_pos(7), gripper(1)] state from the real robosuite observation. - read_panda_mobile_base_velocity(model, data) -> NDArray[np.float32] — Returns body-frame (vx, vy, wz) 3-vec for the robosuite OmronMobileBase; reads data.qvel at the three planar joint addresses and de-rotates the world-frame (vx, vy) using the live yaw. Returns zeros(3) when the base joints aren't in this model (silently no-ops for non-PandaMobile envs). (L1175) - synthesize_laser_scan_2d(*, model, data, base_body_id=None, n_beams=360, max_range_m=12.0, laser_height_m=0.30) -> NDArray[np.float32] — Single-origin batched mj_multiRay 2D laser fan from the panda_mobile base. Returns (n_beams,) float32 ranges in metres, clamped to max_range_m for "no hit" beams (NEVER NaN/inf, so Nav2 costmap consumers don't poison the grid). Self-exclusion via bodyexclude=mj_name2id("base") so the chassis doesn't pollute the scan. (L1248) - head_cam_enabled() -> bool — True iff OPENRAL_ROBOCASA_HEAD_CAM is set (non-empty, non-"0"). Gates the synthetic forward head navigation camera below, so a manipulation run pays for no second offscreen render. The operator no longer sets it by hand: openral deploy sim/run derive it from the palette before launch (openral_cli.deploy_sim._apply_palette_head_cam — on when a capability-matched rSkill declares observation.images.head), and an explicit env value still wins either way. - render_head_view(renderer, model, data, base_joint_names, *, height_m=1.30, forward_offset_m=0.42) -> NDArray[np.uint8] | None — Renders the forward egocentric head frame: a mjCAMERA_FREE camera forward_offset_m ahead of the base body at height_m looking 3 m forward + 0.5 m down. Geometry comes from the robot.yaml head sensor's metadata.height_m / metadata.forward_offset_m (via _RoboCasaSim._head_cam_geometry, cached); the keyword defaults are only the Omron-tuned fallback. Top-down HxWx3 uint8 (no flip; mujoco.Renderer emits rows top-first). None for non-mobile-base models. _RoboCasaSim._wrap_obs calls it (rebuilding its cached renderer when the model identity changes across resets) to add observation.images.head; consumed by the InternVLA-N1 VLN rSkill. - _emit_panda_mobile_extras(obs) (method on _RoboCasaSim) — Attaches obs["robot0_base_vel"] + obs["robot0_scan"] when _has_mobile_base_robot() (name ends in "Mobile" or contains "Omron" — RoboCasa canonicalises PandaMobile → the PandaOmron composition, so the "Omron" arm is what matches at runtime); no-op for other compositions. (L569 in _wrap_obs) - sim_time_ns() -> int | None (method on _RoboCasaSim) — round(MjData.time * 1e9) off mujoco_handles(); covers the robocasa kitchen / GR1 / so100_robosuite scenes. RoboCasa rewinds the clock on reset, so it is monotonic only within an episode — SimAttachedHAL.sim_time_ns adds the cross-reset offset. (L574) - refresh_obs() -> Observation | None (method on _RoboCasaSim) — Re-reads observations + camera frames via robosuite's non-stepping env._get_observations(force_update=True), so SimAttachedHAL._apply_body_twist_to_qpos can refresh the dashboard/WorldState after a direct base-qpos write without advancing physics. It must never step: it previously drove a zero-action env.step on the reasoning that a zero action means no controller effort, but robosuite's mobile-base controller reads that as "hold your setpoint" and regulated the qpos write away — 40 commands at 0.5 m/s on robocasa_baguette wrote 1.0000 m and kept 0.0396 m (4.0%), and the extra step also double-advanced MjData.time, so Nav2 aborted every navigate_to_pose goal with "Failed to make progress". None for backends exposing no such refresh (those keep the cached frame). Guarded by tests/sim/test_panda_mobile_hal_robocasa_body_twist.py. - Constants _OMRON_BASE_JOINT_NAMES, _OMRON_BASE_JOINT_NAMES_FALLBACK, _LASER_DEFAULT_N_BEAMS=360, _LASER_DEFAULT_MAX_RANGE_M=12.0. (L1016)

`python/sim/src/openral_sim/backends/depth_camera.py`

Simulated depth camera via MuJoCo CPU ray-casting (the 3-D analogue of synthesize_laser_scan_2d); robot-agnostic, no GL/EGL context. Feeds the deploy-sim HAL → octomap_server → the kernel world-collision voxel check. - synthesize_depth_pointcloud(*, model, data, camera_name, width, height, fx, fy, cx, cy, max_range_m, min_range_m=0.0, stride=1, exclude_body_id=None, exclude_body_ids=None) -> NDArray[np.float32] — One mj_multiRay ray per (strided) pixel through a pinhole model anchored on the named MJCF camera's live world pose. Returns (N, 3) float32 hit points in the camera optical frame (REP-103: +x right, +y down, +z forward), filtered to [min_range_m, max_range_m]; empty (0, 3) when nothing is in range. exclude_body_id is mj_multiRay's single bodyexclude; exclude_body_ids (a frozenset[int]) drops hits on the robot's own bodies after casting — the self-filter that keeps a base-mounted camera from voxelising the arm into its own world map (else the kernel flags the arm against itself). Raises ROSConfigError if camera_name is absent. (L148) - synthesize_depth_image(*, model, data, camera_name, width, height, fx, fy, cx, cy, max_range_m, min_range_m=0.0, stride=1, exclude_body_id=None, exclude_body_ids=None) -> NDArray[np.float32] — Image counterpart of synthesize_depth_pointcloud, sharing the same pinhole ray-cast (_cast_depth_rays) but keeping every pixel as a dense (ceil(height/stride), ceil(width/stride)) raster of perpendicular optical-Z depth in metres (range · ẑ, the ROS depth-image convention), 0.0 where the ray missed / fell out of range / hit a self-filtered body. This is the input nvblox's projective depth integrator consumes (it rejects the sparse hit-only cloud, whose unorganised layout matches no intrinsic model). The companion CameraInfo must scale intrinsics by 1/stride (see openral_hal.depth_cloud.camera_info_from_intrinsics). Raises ROSConfigError if camera_name is absent. (L235)

`python/sim/src/openral_sim/backends/libero.py`

class _LiberoSim — SimRollout wrapping LiberoEnv. (L114) — reset/step/render/close/action_dim/mujoco_handles/sim_time_ns/_wrap_obs/enable_continuous/_apply_ignore_done/_robosuite_env. enable_continuous() (called by SimAttachedHAL.__init__ on deploy-sim, no-op for openral sim run) makes the episode run continuously — it sets the wrapped robosuite env's ignore_done (via _apply_ignore_done()/_robosuite_env()) and has step() swallow lerobot's inline LiberoEnv.step → if terminated: self.reset(), so a task success / horizon no longer re-randomises the scene mid-mission (which also re-creates the MjData and orphans the passive viewer). Because LiberoEnv builds its robosuite env lazily (first reset) and enable_continuous runs before connect()'s reset, reset() re-applies ignore_done after every reset — otherwise the horizon done hard-raises mid-goal and the HAL's raised-terminal recovery resets the scene under the running policy. mujoco_handles() reaches through robosuite's env.sim.{model,data}._{model,data} for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9). action_dim first materializes lerobot 0.6.0's lazily-built OffScreenRenderEnv (LiberoEnv._ensure_env, which is None until the first reset — the probe runs at HAL connect before that), then walks the LiberoEnv→OffScreenRenderEnv wrapper chain to sum robosuite robots[*].action_dim (LIBERO OSC_POSE = 7) so SimAttachedHAL can size cartesian actions on the openral deploy sim suite-scene path.
_parse_task_id(task_id, scene_id) -> int — Validate <suite>/<int> format. (L83)
_quat_to_axisangle(quat) -> NDArray[np.float32] — [x,y,z,w] → axis-angle. (L359)
_build_libero_scene(env_cfg) -> _LiberoSim (L455)

`python/sim/src/openral_sim/backends/vlabench.py`

VLABench (ICCV 2025) Franka adapter. The dependency plan pins the upstream-tested MuJoCo 3.2.2 + dm_control 1.0.22 pair; loose newer resolution crashes during dm_control model indexing. Policy cameras bypass LeRobot 0.6.0's stale [0,1,2] mapping and select Xiaomi/VLABench's real raw order [front=2, base=0, wrist=3] without flipping rows. - class _VLABenchSim — SimRollout over lerobot's VLABenchEnv; emits three RGB cameras, 7-D EE/gripper state, and info["is_success"]. - _parse_task_id(task_id) -> str — Validate vlabench/<task-name>. - _select_policy_cameras(raw_rgb) -> dict[str, NDArray[np.uint8]] — Select raw camera indices (2,0,3) into camera1/2/3; normalize HWC channels without vertical/horizontal flips. - provision_vlabench() -> None — Pre-launch provisioner: ensure_backend_deps("vlabench") + _resolve_vlabench_root() + _check_vlabench_assets(). The ~12 GB CC-BY bundle is a Google-Drive pull we deliberately never automate, so this verifies it and raises the recipe when absent. Registered as provision= on the vlabench scene. - _build_vlabench_scene(env_cfg) -> _VLABenchSim — Verify assets, build the single-env vector wrapper, and register scene.id="vlabench" with fixed robot franka_panda (and provision=provision_vlabench).

`python/sim/src/openral_sim/policy_deps.py`

Import-probe + install-hint helpers for policy runtimes. Family-keyed helpers (can_import_policy_family / model_family_install_groups / model_family_install_hint) key on RSkillManifest.model_family; XR-1 probes only the shared ZMQ/msgpack wire because its torch stack lives in a sidecar. The manifest-keyed trio below wraps families with manifest-selected runtime exceptions.

Probe tiers. The default probe resolves only the top-level package of each required import via importlib.util.find_spec (~0 ms). It deliberately does not touch the deep module: lerobot/policies/__init__.py eagerly imports every family's config class, so resolving lerobot.policies.<x>.modeling_<x> costs the whole tree — 6.6 s measured, and identically so via find_spec, which must import the parent to find the child. That price was paid in three processes per deploy (the CLI's _preflight_palette_deps, the reasoner's _maybe_seed_palette_from_search_paths, and runtime_node) when only runtime_node needs the modules resolved; probing all 12 families went 6.74 s → 0.49 s with an identical verdict. The fast tier catches the failure this module exists to catch (a group that was never installed) but not an installed-but-broken group — that still surfaces at dispatch, where rskill_runner_node already translates the factory ImportError into a ROSRuntimeError carrying model_family_install_hint. Set OPENRAL_STRICT_POLICY_PROBE=1 to restore the deep-import probe (_deep_import_probe, which is also the only tier that calls purge_partial_imports).

can_import_policy_manifest(manifest) -> tuple[bool, str | None] — Probe the manifest-selected runtime: the BEHAVIOR GR00T sidecar rSkill probes zmq + msgpack (its openral-side wire), everything else falls through to can_import_policy_family. (L318)
manifest_install_groups(manifest) -> tuple[str, ...] — Dependency groups for the manifest-selected runtime (("behavior-groot",) for the BEHAVIOR sidecar rSkill, else model_family_install_groups). (L325)
manifest_install_hint(manifest) -> str — Paste-able install hint for the manifest-selected runtime. (L332)

`python/sim/src/openral_sim/backends/metaworld.py`

MetaWorld MT-50 scene adapter. Opt-in via the metaworld dependency group + a metaworld==3.0.0 --no-deps pip install (its transitive deps conflict with the workspace lock); the scene factory calls openral_sim._deps.ensure_backend_deps("metaworld") first so the user gets an interactive auto-install banner on first use. Scene id metaworld. Task id metaworld/<task-name> (e.g. metaworld/reach-v3). - class _MetaworldSim — SimRollout wrapping MetaworldEnv. (L46) — reset/step/render/close/mujoco_handles/sim_time_ns/_wrap_obs. mujoco_handles() reaches through unwrapped.{model,data} for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9). - _parse_task_id(task_id) -> str (L36) - _build_metaworld_scene(env_cfg) -> _MetaworldSim (L133)

`python/sim/src/openral_sim/backends/maniskill3.py`

ManiSkill3 (SAPIEN-backed) free-axis scene adapter. Opt-in via the maniskill3 dependency group; the scene factory calls openral_sim._deps.ensure_backend_deps("maniskill3") first so the user gets an interactive auto-install banner on first use (bypass with OPENRAL_AUTO_INSTALL_DEPS=1). Scene id maniskill3. Task id maniskill3/<env_id> (e.g. maniskill3/PickCube-v1). - _MANISKILL3_SCENE_ID = "maniskill3" — module constant; scene-registry key. (L40) - _sapien_sim_time_ns(env) -> int | None — Derive elapsed SAPIEN/ManiSkill sim time from a live env's elapsed step counter (elapsed_steps / _elapsed_steps) and control period (control_timestep / control_dt / control_freq). Returns None when the env does not expose a usable clock seam. - class _ManiSkill3Sim — SimRollout wrapping a MS3 gym env with num_envs=1; unwraps the leading batch dim on every obs / step. (L215) — reset/step/action_dim/sim_time_ns/render/close/_wrap_obs. action_dim returns the single-env width from the live gym action space for SimAttachedHAL; sim_time_ns() returns SAPIEN elapsed control time via _sapien_sim_time_ns. - _parse_task_id(task_id) -> str — Validates maniskill3/<env_id> and returns <env_id>. (L50) - _task_id_for_env(env_cfg) -> str — Resolves the concrete ManiSkill env id. Normal sim tasks parse maniskill3/<env_id>; deploy-sim's synthetic _hal_deploy_noop task maps to scene.backend_options.deploy_task_id or PickCube-v1 so taskless DeployScenes still build a real backend env. - _reconcile_robot_uids(env_id, robot_uids) -> None — Validate a scene's requested robot_uids against the task's SUPPORTED_ROBOTS. Accepts a registered camera-variant subclass of a supported base (walks the agent's MRO uids — e.g. panda_wristcam→panda); raises ROSCapabilityMismatch for genuinely-unsupported robots instead of MS3's vague warning + downstream crash. (L66) - class _DropUnsupportedRobotWarning(logging.Filter) / _suppress_unsupported_robot_warning() — Context manager that drops MS3's false "not in the task's list of supported robots" log record (only that message) around gym.make, after _reconcile_robot_uids has validated the variant. (L113 / L123) - _unbatch(value), _unbatch_info(info), _unbatch_obs(obs) — recursive numpy / torch unbatch helpers shared with the SimplerEnv adapter. (L106 / L114 / L130) - _extract_rgb(flat) — Returns the first MS3 sensor_data.<camera>.rgb stream as NDArray[uint8]. (L366) - _extract_state(flat) — Concatenates agent.qpos + agent.qvel into a 1-D float32 vector (returns 0-D when the obs mode doesn't expose the nested agent block). (L409) - _build_maniskill3_scene(env_cfg) -> _ManiSkill3Sim — gym.make with obs_mode / control_mode overridable via scene.backend_options; default state_dict+rgb + pd_ee_delta_pose. (L425) - Module side effect: SCENES.register("maniskill3")(_build_maniskill3_scene) at import (L425).

`python/sim/src/openral_sim/backends/simpler_env.py`

SimplerEnv real-to-sim correlator adapter. Opt-in via the simpler-env dependency group (the package has no PyPI release; install hint in the typed ROSConfigError). Reuses the obs-extraction helpers from backends/maniskill3 because SimplerEnv now sits on top of MS3 v3.0.x. Scene id simpler_env. Task id simpler_env/<friendly_name> (e.g. simpler_env/widowx_carrot_on_plate); friendly names are translated via simpler_env.ENVIRONMENT_MAP to the underlying MS3 env id + kwargs. Today only the four WidowX bridge tasks are wired end-to-end against MS3 v3.0.x; google_robot_* friendly names resolve to env ids that are not yet registered upstream. - _SIMPLER_ENV_SCENE_ID = "simpler_env" — module constant; scene-registry key. (L71) - _DEFAULT_OBS_MODE = "rgb+segmentation" — Only obs mode the MS3 v3.0.x Bridge envs advertise; overridable via scene.backend_options.obs_mode. (L79) - class _SimplerEnvSim — SimRollout wrapping a SimplerEnv-via-MS3 gym env. (L270) — reset/step/action_dim/sim_time_ns/render/close/_wrap_obs. Reshapes the single-env action to (1, action_dim) to satisfy MS3's batched API. action_dim returns the single-env width from the live gym action space for SimAttachedHAL; sim_time_ns() returns SAPIEN elapsed control time via the shared ManiSkill helper. - _parse_task_id(task_id) -> str — Validates simpler_env/<friendly_name> and returns <friendly_name>. (L95) - _task_name_for_env(env_cfg) -> str — Resolves the concrete SimplerEnv friendly/raw task name. Normal sim tasks parse simpler_env/<friendly_name>; deploy-sim's synthetic _hal_deploy_noop task maps to scene.backend_options.deploy_task_id or widowx_carrot_on_plate. - _bump_version_if_deprecated(env_id) -> str — Rounds an upstream -v0 env id up to the highest registered -v* suffix; upstream simpler_env.ENVIRONMENT_MAP still ships -v0 but MS3 v3.0.x registers -v1. (L111) - _resolve_friendly_name(task_name) -> tuple[str, dict[str, Any]] — Translates a SimplerEnv friendly task name into (ms3_env_id, kwargs). Falls back to passing the input through unchanged so users can author configs against raw MS3 env ids. (L134) - _build_simpler_env_scene(env_cfg) -> _SimplerEnvSim — Calls gym.make directly (bypassing the broken upstream simpler_env.make() which still passes prepackaged_config=True / obs_mode='rgbd' that MS3 v3.0.x rejects). (L375) - Module side effect: SCENES.register("simpler_env")(_build_simpler_env_scene) at import (L375).

`python/sim/src/openral_sim/sidecar.py`

Canonical openral-side out-of-process sidecar transport — ZMQ REQ/REP + a numpy-aware msgpack codec. Shared by new sidecar integrations (the Isaac Sim backend); the RLDX-1 adapter predates it and keeps its own wire-locked copy (its codec must match the upstream __ndarray_class__ sentinel and its real path is un-runnable in CI). - encode_ndarray(obj) -> Any / decode_ndarray(obj) -> Any — msgpack default / object_hook codec (np.save into a {"__ndarray__": True, "npy": bytes} sentinel; decode returns a sentinel missing npy unchanged rather than raising KeyError). - require_key(reply, key, *, name) -> Any — typed-ROSRuntimeError guard for a reply missing a required key. - class SidecarClient — owns the ZMQ REQ socket + the optional child Popen. connect (ping existing → else spawn + boot-poll, ROSConfigError on failure), call(endpoint, data) (ROSRuntimeError on a sidecar-side fault / non-dict reply), require, close; boot helpers _try_ping/_spawn/_wait_for_boot/_terminate_child/_is_port_busy/_boot_failure_error; recreates the REQ socket on timeout to clear the EFSM lock. _boot_failure_error classifies a failed boot three ways instead of blaming a slow bootstrap: child exited non-zero → it crashed; port bound but mute → it reached its serve loop, so raising the timeout cannot help; still running and never bound → slow or stalled, and the message gives both readings (Isaac's Kit reached app ready at 12 s then burned the remaining 1188 s on a failed extension load — issue #89). name parametrizes log/error text. _spawn strips PYTHONPATH/VIRTUAL_ENV from the child env so the parent's (different-interpreter) site-packages don't shadow the sidecar venv's numpy (openral deploy sim injects the py3.12 site onto PYTHONPATH; the py3.11 Isaac sidecar must use its own).

`python/sim/src/openral_sim/backends/isaac_sim.py`

NVIDIA Isaac Sim (Omniverse + PhysX + RTX) free-axis scene adapter. Drives an Isaac Sim env that runs in a separate py3.11 sidecar venv (Isaac Sim ships per-interpreter wheels; the openral workspace is py3.12), over the shared openral_sim.sidecar.SidecarClient. Opt-in via the isaacsim dependency group (pyzmq + msgpack on the openral side only); the heavy isaacsim/isaaclab install is an externally-provisioned sidecar venv (Omniverse Kit is proprietary, never vendored — CLAUDE.md §1.9). The factory calls ensure_backend_deps("isaac_client"), resolves the sidecar interpreter (OPENRAL_ISAAC_SIDECAR_PYTHON) + script (tools/isaac_sidecar.py), and auto-spawns the sidecar on first use. Scene id isaac_sim. Task id isaac_sim/<name>. - _ISAAC_SCENE_ID = "isaac_sim" — module constant; scene-registry key. - class _IsaacSimSidecar — SimRollout proxying reset/step/render/close to a SidecarClient; unwraps the eval-shaped Observation (images/state/task) via client.require(...) and caches the last RGB frame. The action_dim property (read from the sidecar ping, cached) lets openral deploy sim wrap it in SimAttachedHAL (_probe_env_action_dim); deploy scene scenes/deploy/isaac_franka.yaml (taskless DeployScene, lift_cube). Minimal bring-up — /joint_states is zeros for a non-MuJoCo backend. The task-level isaac_franka_lift SimScene was removed because no in-tree or downloadable rSkill emits the required 8-D Franka joint-delta action. - _opt_num(opts, key, default, cast) -> int|float — coerce a scene.backend_options value (typed object) via cast, ignoring bool and swallowing ValueError/TypeError (returns default, never raises). - _sidecar_python() -> Path / _provision_isaac_venv() -> Path / _locate_sidecar_script() -> Path — resolve the py3.11 interpreter (env override → opt-in auto-provision → cache default → typed ROSConfigError with provisioning hint) and tools/isaac_sidecar.py (env override → walk-up). Auto-provision precedes the existing-venv shortcut so a venv built from superseded pins is repaired: _provision_isaac_venv passes spec=(*_ISAAC_DEPS, *_ISAAC_CUDA_DEPS) to ensure_pip_venv, which reuses the venv when its sentinel matches and re-installs when it does not. The old order returned any existing venv untouched, so a raised pin never took effect (issue #89). - _ISAAC_CUDA_FLOORS = {"nvidia-nvjitlink-cu12": "12.8", "nvidia-cusparse-cu12": "12.5"} → _ISAAC_CUDA_DEPS — the CUDA runtime floors forced on top of the Isaac install (--upgrade --no-deps), and the single source of both the pip spec and the sidecar's --require-min boot probe. Isaac's omni.isaac.ml_archive prebundles a CUDA-12.8 libcusparse but no libnvJitLink, so nvJitLink resolves against the venv's copy; torch 2.7's own nvidia-nvjitlink-cu12==12.6.85 is too old for it (undefined symbol: __nvJitLinkCreate_12_8) and the mismatch hangs Kit past app ready instead of raising. - _sensor_dict(sensor) -> dict / _build_robot_spec(desc, robot_id) -> dict / _write_robot_spec(env_cfg) -> str — robot-agnostic --layout manifest marshalling. The py3.11 sidecar cannot import openral_core, so _build_robot_spec serialises the RobotDescription to plain JSON — the urdf_path wire field resolved from assets.urdf.ref to a file (openral_core.assets.resolve_asset), ALL non-fixed joints in manifest order with normalised role (base for base_joints, gripper, else arm), the action contract (arm_n + gripper + 3·base-twist; a normalised [0,1] gripper limit falls back to the Panda 0.04 m so it never tears the Isaac finger DOF), and the sensors — and _write_robot_spec writes it to a temp file passed via --robot-spec. - _build_isaac_sim_scene(env_cfg) -> _IsaacSimSidecar — factory: builds the launch argv (incl. --layout from backend_options.layout); for layout == "manifest" writes the robot spec and appends --robot-spec, unlinking it after connect() (the sidecar consumes it at boot). Connects a SidecarClient(name="isaac", …). - provision_isaac_sim() -> None — Pre-launch provisioner: ensure_backend_deps("isaac_client") + _sidecar_python(), which auto-builds the multi-GB RTX-only NVIDIA-index venv under OPENRAL_ISAAC_AUTO_PROVISION=1 and otherwise raises the manual recipe. Registered as provision= so openral deploy sim runs it before ros2 launch rather than inside the HAL's 300 s-bounded on_configure; idempotent (venv sentinel). Covers provisioning only — the Omniverse Kit boot still runs inside on_configure via connect(), and _DEFAULT_BOOT_TIMEOUT_S = 900.0 (raised to 1200 by all four scenes/deploy/isaac_*.yaml) exceeds the launcher's hardcoded 300 s. See the boot-timeout note under tools/lifecycle_autostart.py. - Module side effect: SCENES.register("isaac_sim", fixed_robot=None, provision=provision_isaac_sim)(_build_isaac_sim_scene) at import.

`tools/isaac_sidecar.py` + `tools/_isaac_scene_base.py` + `tools/isaac_scene.py` + `tools/isaac_bowl_plate_scene.py` + `tools/isaac_manifest_scene.py`

Isaac-side sidecar (runs under the py3.11 Isaac Sim venv only). isaac_sidecar.py verifies its dependency floors (_check_required_versions, fed by repeatable --require-min DIST>=VERSION args carrying the openral side's _ISAAC_CUDA_FLOORS) before launching the headless Omniverse Kit SimulationApp (sets OMNI_KIT_ACCEPT_EULA=YES), then serves a ZMQ REP loop (ping/reset/step/render/close) speaking the same msgpack+ndarray framing as the openral side. _isaac_scene_base.IsaacSceneBase owns the shared lifecycle (reset warmup, step physics-substep loop, _observe assembly, _grab RGBA→HWC); subclasses override build/_apply_action/_images/_state/_reward_terminated (+ _on_reset/_extra_info/_joint_positions). _isaac_scene_base.franka_joint_positions(franka) maps the Isaac Franka's 9 DOF to the manifest's 8 joints (7 arm + mean-finger gripper); both scenes return it from _joint_positions() so obs["joint_positions"] feeds openral deploy sim's SimAttachedHAL.read_state real /joint_states (non-MuJoCo backends). The --layout arg picks the scene class: - lift_cube (isaac_scene.IsaacLiftScene) — World + Franka + DynamicCuboid + Camera; 8-D joint-delta action, cube-height reward. Kept as a deploy/wire PoC only; no SimScene YAML is shipped because the repo has no task-capable 8-D Franka joint-delta rSkill. - bowl_plate (isaac_bowl_plate_scene.IsaacBowlPlateScene) — table + YCB 024_bowl USD + thin-cylinder plate + Franka + agent-view & eye-in-hand cameras, mirroring the LIBERO contract (camera1/camera2 + 8-D [eef_pos‖axisangle‖gripper_qpos] state, 7-D OSC-pose-delta action). End-effector control uses the core isaacsim.robot_motion.motion_generation Lula kinematics solver (LulaKinematicsSolver + ArticulationKinematicsSolver on the right_gripper frame) for position-delta IK — no Isaac Lab and no Isaac Lab OSC term required. Drives act-libero / smolvla-libero through openral sim run (verified e2e; success is OOD, the check is pipeline + arm motion). Scene scenes/sim/isaac_franka_bowl_plate.yaml. - manifest (isaac_manifest_scene.IsaacManifestScene) — robot-agnostic, URDF-driven scene. Instead of a hardcoded Isaac Franka asset it imports the manifest robot's URDF via omni.kit.commands URDFCreateImportConfig → URDFParseAndImportFile (Isaac's isaacsim.asset.importer.urdf), wraps it as isaacsim.core.api.robots.Robot, and drives a JOINT_POSITION-delta articulation controller. Action layout [arm deltas, gripper, base twist]. map_dof_to_manifest(values, *, dof_index, manifest_joints, finger_dof_idx, base_values=None, base_joints=None) (module-level) maps the articulation DOF vector to the full manifest joint order: base joints ← the kinematic base pose, arm joints ← URDF DOF by name, the two-finger→one-gripper collapse ← mean of the finger DOFs, else 0.0 (generic replacement for franka_joint_positions). Kinematic holonomic base (M3): a robot with base_joints is imported fix_base=True and _integrate_base(vx, vy, wyaw) teleports the whole articulation root each step from a base-frame-twist-integrated (x, y, yaw) — real base motion + a base_pose for /odom, no PhysX base joints (the base exists nowhere as an Isaac asset; robosuite composes it in MuJoCo only). Built from the --robot-spec JSON. Verified live: scenes/deploy/isaac_franka_urdf.yaml (tests/sim/test_franka_urdf_isaac.py — franka imports from URDF, /joint_states carries the imported pose, JOINT_POSITION drives the arm) and scenes/deploy/isaac_panda_mobile_urdf.yaml (tests/sim/test_panda_mobile_isaac.py — 11-D action, 11-joint /joint_states = 3 base + 7 arm + 1 gripper, forward base-twist moves the base). Manifest-driven sensors: _plan_cameras makes one base-relative Isaac Camera per RGB/depth SensorSpec (keyed by the RGB vla_feature_key suffix camera1… / the depth sensor name); _update_camera_poses rides them on the kinematic base; _images returns every RGB frame and _depth_clouds returns {sensor: (N,3) base_link} via Isaac's Camera.get_pointcloud(world_frame=True) (Isaac owns the camera convention) transformed world→base_link by the base pose → obs["depth_points"] (SimSensorBridge publishes them as PointCloud2). A modality the manifest does not declare is never created. Verified live: the deploy graph publishes /openral/cameras/front_depth/points (62 k pts, base_link) → octomap → /openral/world_voxels. 2-D lidar: _add_obstacles seeds a few static boxes and _scan_ranges casts a PhysX raycast_closest fan (each ray starting range_min_m past the base to clear the robot's own chassis; robot /panda hits ignored) → obs["scan"] → SimAttachedHAL.read_scan → /scan. The full slam-map + obstacle-aware Nav2 loop additionally needs the deploy-sim /clock publisher (merged).

All three layouts use Isaac Sim core (not Isaac Lab's env machinery, which the PyPI isaaclab wheel does not ship). NOT imported by the openral venv — invoked as a subprocess; the openral-side backends/isaac_sim.py forwards scene.backend_options.layout (and, for manifest, the --robot-spec JSON).

`python/sim/src/openral_sim/backends/robotwin.py`

RoboTwin 2.0 dual-arm SAPIEN scene adapter. Single-robot (fixed) scene bound to the aloha_agilex embodiment (14-DoF), run in a separate py3.10 sidecar venv (RoboTwin pins SAPIEN/CuRobo/mplib/pytorch3d against py3.10/CUDA-12.1, incompatible with the openral py3.12 venv), over the shared openral_sim.sidecar.SidecarClient. Opt-in via the robotwin dependency group (pyzmq + msgpack on the openral side only); the heavy SAPIEN+RoboTwin stack is an externally-provisioned sidecar venv (large + CUDA-pinned, never vendored — CLAUDE.md §1.9; RoboTwin is MIT). The factory calls ensure_backend_deps("robotwin_client"), resolves the sidecar interpreter (OPENRAL_ROBOTWIN_SIDECAR_PYTHON) + script (tools/robotwin_sidecar.py), and auto-spawns on first use. Scene id robotwin. Task id robotwin/<snake_case_task>. - _ROBOTWIN_SCENE_ID = "robotwin" / _ROBOTWIN_ROBOT_ID = "aloha_agilex" — module constants; scene-registry key + fixed robot. - class _RoboTwinSimSidecar — SimRollout proxying reset/step/render/close to a SidecarClient; unwraps the eval-shaped Observation (images keyed camera1/camera2/camera3 / state / task) via client.require(...), caches the head frame for render, and exposes action_dim (14, read from the sidecar ping, cached) + sim_time_ns from the sidecar reply. - _scene_default_port(task_id, robot_id) -> int — deterministic per-scene ZMQ port in [_SIDECAR_PORT_MIN, _SIDECAR_PORT_MAX) (SHA-256 digest, not the salted builtin hash), so distinct tasks never share a sidecar endpoint; an explicit backend_options.port still wins. - _robotwin_task_name(task_id) -> str — strips the robotwin/ namespace to the bare upstream task name the LeRobot env wants. - _opt_num(opts, key, default, cast) — coerce a scene.backend_options value (ignores bool, swallows ValueError/TypeError). - _provision_robotwin_venv() -> Path / _sidecar_python() -> Path / _locate_sidecar_script() -> Path — opt-in (OPENRAL_ROBOTWIN_AUTO_PROVISION=1) provisioning of the py3.10 venv (lerobot from git main + SAPIEN + wire — the RoboTwin task package + multi-GB assets remain a manual step), interpreter resolution (env override → cache default → opt-in provision → typed ROSConfigError carrying the full conda recipe), and tools/robotwin_sidecar.py location (env override → walk-up). - _build_robotwin_scene(env_cfg) -> _RoboTwinSimSidecar — factory: builds the launch argv (--task, --cameras, --episode-length, obs h/w, host/port), connects a SidecarClient(name="robotwin", expected_identity={"env": "robotwin", "task": <name>}). - provision_robotwin() -> None — Pre-launch provisioner: ensure_backend_deps("robotwin_client") + _sidecar_python() (opt-in multi-GB LeRobot + SAPIEN venv under OPENRAL_ROBOTWIN_AUTO_PROVISION=1, else the manual recipe). Registered as provision=; keeps the venv build out of the HAL's 300 s on_configure. Note this covers provisioning only — the sidecar boot still happens inside on_configure via _build_robotwin_scene's connect(), and this backend's _DEFAULT_BOOT_TIMEOUT_S = 600.0 exceeds that bound (latent: no in-tree deploy scene selects robotwin). - Module side effect: SCENES.register("robotwin", fixed_robot="aloha_agilex", provision=provision_robotwin)(_build_robotwin_scene) at import.

`tools/robotwin_sidecar.py`

RoboTwin-side sidecar (runs under the py3.10 lerobot-main + RoboTwin + SAPIEN venv only). Constructs LeRobot's native robotwin gym env (RoboTwinEnvConfig + make_env, single non-vectorised env) for the requested task and serves a ZMQ REP loop (ping/reset/step/render/close) speaking the same msgpack+ndarray framing as the openral side. _RoboTwinEnv adapts the env's {pixels, agent_pos} obs to the eval-layer {images, state, task} shape, re-keying the env's native cameras (head_camera/left_camera/right_camera, _ENV_CAMERA_NAMES) to the openral scene's camera1/camera2/camera3 in order, and includes sim_time_ns in reset/step replies by deriving elapsed SAPIEN time from the wrapped env. NOT imported by the openral venv — invoked as a subprocess.

`python/sim/src/openral_sim/backends/rlbench.py`

RLBench (CoppeliaSim/PyRep) single-robot (fixed franka_panda) scene adapter. Drives an RLBench task that runs in a separate externally-provisioned py3.10 sidecar venv (CoppeliaSim is proprietary, free-EDU, never vendored — CLAUDE.md §1.9; the released 3D policies pin the MohitShridhar/RLBench@peract fork), over the shared openral_sim.sidecar.SidecarClient. Opt-in via the rlbench dependency group (pyzmq + msgpack on the openral side only). The factory calls ensure_backend_deps("rlbench_client"), resolves the sidecar interpreter (OPENRAL_RLBENCH_SIDECAR_PYTHON) + COPPELIASIM_ROOT + script (tools/rlbench_sidecar.py), wraps the launch with env VAR=… to inject CoppeliaSim's runtime vars, and auto-spawns on first use. Scene id rlbench; step takes an 8-D keyframe [x y z qx qy qz qw gripper_open]. - _RLBENCH_SCENE_ID = "rlbench"; _scene_default_port(rlbench_task, variation) — deterministic per-task ZMQ port (SHA-256, range 21000–21999). - _RLBenchSidecar(scene, task, _client) — SimRollout proxy; _wrap_obs carries images/point_clouds (dict per camera) + gripper_pose/gripper_open for the 3D keyframe policy. Caches sim_time_ns from reset/step replies and exposes sim_time_ns() for clock projection. - _build_rlbench_scene(env_cfg) -> _RLBenchSidecar — factory; reads backend_options.{rlbench_task,variation,port,max_tries}. Connects a SidecarClient(name="rlbench", …). - provision_rlbench() -> None — Pre-launch provisioner: ensure_backend_deps("rlbench_client") + _sidecar_python(). CoppeliaSim is proprietary with no auto-install plan, so this only locates the externally-provisioned venv or raises the manual recipe — surfacing that refusal during preflight instead of as an opaque 300 s on_configure timeout is the point. - Module side effect: SCENES.register("rlbench", fixed_robot="franka_panda", provision=provision_rlbench)(_build_rlbench_scene) at import.

`tools/rlbench_sidecar.py`

RLBench-side scene sidecar (runs under the externally-provisioned py3.10 venv only; no openral import). Launches CoppeliaSim/PyRep headless via RLBench's Environment (peract fork, EndEffectorPoseViaPlanning + Discrete gripper), serves a ZMQ REP loop (ping/reset/step/render/close) speaking the same msgpack+ndarray framing as the openral side. step appends the peract-fork 9-D ignore_collisions channel and executes the keyframe via a plan-and-retry mover (re-tries until the EE reaches the target pose < 5 mm). Cameras: left_shoulder/right_shoulder/wrist/front (RGB + point cloud). Reset/step replies include sim_time_ns, read from PyRep's CoppeliaSim simulation clock (get_simulation_time()).

`python/sim/src/openral_sim/backends/aloha.py`

gym-aloha bimanual MuJoCo scene adapter. Opt-in via the sim dependency group (gym-aloha lives there alongside mujoco / gymnasium / lerobot); the scene factory calls openral_sim._deps.ensure_backend_deps("aloha") first so the user gets an interactive auto-install banner on first use. - class _AlohaSim — SimRollout wrapping a gym_aloha env. — reset/step/render/close/mujoco_handles/sim_time_ns/_wrap_obs. mujoco_handles() reaches through env.unwrapped._env.physics.{model,data}.ptr (dm_control wrapper) for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9).

`python/sim/src/openral_sim/backends/pusht.py`

class _PushTSim — SimRollout wrapping gym_pusht/PushT-v0 (pymunk 2-D rigid body). — reset/step/render/close/enable_intrinsic_viewer/_wrap_obs/_paint_view. enable_intrinsic_viewer() opens a pygame window from inside the adapter and _paint_view() blits the last pixel frame on each reset / step, leaving the env in render_mode="rgb_array" so the Diffusion Policy still gets observation.image.

`python/sim/src/openral_sim/backends/so100_robosuite/`

robosuite integration for the Hugging Face SO-100 follower. NOT a SCENES.register(...) adapter (the SO-100 has no benchmarked VLA-driven suite yet); a standalone subpackage that registers the SO-100 with robosuite's robot / gripper factories and provides a runnable scripted-pick demo. Used by tests/sim/test_so100_robosuite_lift.py and examples/so100_robosuite_lift.py. - __init__.py — re-exports SO100, SO100Gripper, make_so100_lift_env, so100_osc_controller_config; importing the package side-effect-registers the robot + gripper. - _assets.py — ensure_so100_assets() -> SO100Assets lazily rewrites the DeepMind mujoco_menagerie trs_so_arm100 MJCF into two robosuite-compatible XMLs (arm with 5 motor actuators + base/right_hand body, gripper with the Jaw joint + eef body + finger pads), caches under $OPENRAL_CACHE_DIR/so100_robosuite/<menagerie-fingerprint>/. class SO100Assets(frozen dataclass) carries robot_xml, gripper_xml, menagerie_dir. The rewrite handles three robosuite quirks: nested <default> flattening (so _replace_defaults_inline resolves classes), absolute mesh paths (robosuite's resolve_asset_dependency ignores meshdir), and childclass stripping (robosuite drops the defaults block before MuJoCo compiles). - model.py: - class SO100(ManipulatorModel) — 5-DOF arm (Rotation / Pitch / Elbow / Wrist_Pitch / Wrist_Roll); default_base = "NullMount", default_gripper = {"right": "SO100Gripper"}, init_qpos matches the menagerie home keyframe. Registered in REGISTERED_ROBOTS and ROBOT_CLASS_MAPPING (as a FixedBaseRobot) at import. - class SO100Gripper(GripperModel) — 1-DOF Jaw with _important_geoms for left_fingerpad / right_fingerpad so robosuite's _check_grasp resolves cleanly. Registered in GRIPPER_MAPPING at import. - env.py: - class _So100Lift(Lift) — Lift with the SO-100 bolted onto the standard TableArena top, a small upright redwood block (1.2 cm half-edge × 4 cm tall) sized for the SO-100 jaw aperture, and a _check_success that scales with the block height (success when the bottom face clears the table by lift_height_m). - so100_osc_controller_config() -> dict[str, Any] — Loads robosuite's shipped parts/osc_position.json, narrows output_max from ±5 cm/step to ±1 cm/step (SO-100's small mass matrix would otherwise overshoot), bumps kp 150 → 1500 to match the 30-50× smaller mass-matrix entries, and pins input_ref_frame = "world" so the policy's world-frame Cartesian targets aren't re-rotated by the SO-100's 90°-z base orientation. - make_so100_lift_env(*, has_renderer, has_offscreen_renderer, use_camera_obs, camera_names, camera_heights, camera_widths, horizon, control_freq, table_full_size, cube_half_extent_m, cube_block_height_m, x_range, y_range, seed, lift_height_m, reward_shaping) -> _So100Lift — composes the registered robot + gripper + OSC_POSITION config; cube placement reference matches the Panda-default table_offset = (0, 0, 0.8) so the stock agentview / frontview cameras frame the scene correctly. - policy.py: - class PolicyTelemetry (dataclass) — Per-step diagnostics: phase / eef_to_cube_distance_m / gripper_command / cartesian_delta / cube_height_m. - class ScriptedPickPolicy (dataclass) — Four-phase Cartesian state machine (approach → descend → close → lift); step(env, obs) -> (action, PolicyTelemetry) returns a 4-vec [dx, dy, dz, gripper] normalised to [-1, 1] for OSC_POSITION + GRIP. No grid-search IK, no Jacobian glue — OSC owns the IK; the policy just emits clip((target - eef) / cartesian_step_m, -1, 1) against the latched initial cube pose. SO-100 jaw direction convention: positive opens, negative closes (named explicitly via open_cmd / closed_cmd locals to guard against sign flips).

`python/sim/src/openral_sim/backends/openarm_robosuite/`

Custom MJCF composer + SCENES.register("openarm_tabletop_pnp") adapter for the bimanual OpenArm v2 pick-and-place scene. The composer rewrites the vendor enactic/openarm_mujoco v2 bimanual MJCF in-place to add scene bodies (table, target object, world skybox), substitute its <position> actuators with motor actuators of compatible torque limits, and inject a camera. State / action dimensions and the actuator inventory are now derived from the RobotDescription.sim block (this branch) rather than hard-coded module constants. Opt-in via the robocasa dependency group (only place robosuite>=1.5 is declared in the workspace; this backend uses robosuite purely as an MJCF wrapper and does not need the robocasa kitchen / GR1 forks); the scene factory calls openral_sim._deps.ensure_backend_deps("openarm_robosuite") first so the user gets an interactive auto-install banner on first use instead of a bare ModuleNotFoundError: robosuite. - _assets.py: - load_openarm_description() -> RobotDescription — Resolves the canonical robots/openarm/robot.yaml manifest as the single source of truth for actuator metadata. (L83) - actuator_specs_from_description(desc) -> list[ActuatorSpec] — Build the ordered list of MJCF actuator specs (name, joint, ctrlrange, gear, side) from desc.joints + desc.sim.grippers. Mirrors the OpenArm v2 actuator block but stays robot-data-driven so adding a new joint in robot.yaml is enough — no edit here. (L122) - motor_actuator_names_from_description(desc) -> list[str] — Convenience wrapper returning just the actuator names in MJCF order; used by the env's action wiring. Replaces the removed MOTOR_ACTUATOR_NAMES module-level tuple. (L169) - _render_actuator_block(specs) -> str — Render an <actuator> XML block from the spec list (motor actuators with per-actuator ctrlrange + gear). (L302) - compose_openarm_tabletop_mjcf(env_cfg) -> str — Compose the scene MJCF: pulls the v2 bimanual MJCF, substitutes position actuators with the motor block from _render_actuator_block, injects scene bodies (table / target / base sites) + the top camera, and lifts the robot bases. (L478) - Module constants _FALLBACK_TOP_CAMERA_POS / _FALLBACK_TOP_CAMERA_TARGET / _FALLBACK_TOP_CAMERA_FOVY (L251–L253) — Fallbacks consumed only when RobotDescription.scene_defaults.top_camera is unset AND scene.backend_options.top_camera_* is unset. Renamed from _DEFAULT_TOP_CAMERA_* (this branch) to reflect that the canonical defaults now live on the robot manifest (SceneDefaults / TopCameraDefaults on openral_core). - Private helpers: _look_at_quat, _inject_base_center_sites, _lift_robot_bases, _rename_upstream_wrist_cameras, _inject_white_skybox, _strip_position_actuators. - env.py: - _resolve_state_dim(env_cfg, rskill_manifest=None) -> int — Derive the observation-state dim from the rSkill manifest's state_contract.dim when present, else from OPENARM_DESCRIPTION.observation_spec.state_shape. Replaces the removed _OBS_STATE_DIM module-level constant so the scene is self-consistent when the rSkill ships a different state contract. (L152) - _resolve_initial_pose_from_rskill(rskill_manifest) — Read the per-rSkill initial joint pose if the manifest pins one. (L219) - _resolve_base_translation(env_cfg) -> tuple[float, float] — Parse the scene's base_translation override; defaults to the OpenArm tabletop layout. (L104) - class _ArmHandles / _build_arm_handles(model, side) -> _ArmHandles — Per-side MuJoCo qpos/qvel/actuator handles. (L275, L294) - class _OpenArmTabletopRollout — SimRollout for the openarm_tabletop_pnp scene: composes the MJCF via _build_openarm_tabletop_scene, drives both arms through the manifest-derived actuator block, exposes mujoco_handles() + sim_time_ns() (round(MjData.time * 1e9)) for the viewer / sim-clock, and action_dim (== bimanual state_dim) so SimAttachedHAL._probe_env_action_dim resolves the deploy-sim action width (probe-gap fix). (L397) - _build_openarm_tabletop_scene(env_cfg) -> _OpenArmTabletopRollout — Scene factory registered as SCENES.register("openarm_tabletop_pnp")(_build_openarm_tabletop_scene). (L684)

`python/sim/src/openral_sim/backends/so101_box/`

Parameterised raw-MuJoCo scene: SO-101 in a configurable box arena, registered as @SCENES.register("so101_box", fixed_robot="so101_follower"). Defaults match the user-supplied sketch (100 × 61.5 × 75 cm box, SO-101 back-centre on floor, OAK-D Pro overhead RGB-D, terminal gripper-mounted wrist camera, 44.5 × 44.5 × 20 mm slotted block with Ø 23 mm hole + 5 mm slot, Ø 21.9 × 90 mm tube (0.55 mm radial clearance)). The wrist camera is parented to the SO-101 gripper body, mounted on the static finger face and rolled 180° (inverted phone) to match the real lerobot SO-101 wrist rig (Cornito/so101_test2) — the open jaws hang into the bottom ~20% of the frame with the workspace + grasped object filling the rest (pos=[-0.0084, 0.0834, -0.0545], target=[-0.0074, -0.091, -0.1886], up=[0.0, 0.0, -1.0], fovy=90). Every dimension and threshold is driven by scene.backend_options via a typed BoxSceneOptions dataclass — no scene geometry is hard-coded in Python. Each reset() randomises both the block and the tube on the floor at independent (x, y, yaw) draws within configurable ranges; success fires when the tube is inserted vertically into the block hole within configurable tolerances. - _assets.py: - class BoxSceneOptions — Typed dataclass holding every scene-geometry knob (arena, robot mount, two cameras, block + tube dimensions, spawn ranges, insertion thresholds, control_hz, LeRobot joint-units affine). Fed from scene.backend_options by _options_from_backend_options. (L41) - compose_so101_box_mjcf(options=None, robot_description=None) -> tuple[str, Path] — Read the robot's MJCF (robot_description.assets.mjcf, defaulting to so_arm101_mj_description), re-anchor its <body name="base"> to options.robot_base_xyz + yaw — rewriting pos/quat when present, else injecting them (SO-100 Base schema) — splice a wrist camera into the <body name="gripper"> body (options.wrist_camera_{pos,target,up}_local + wrist_camera_fovy), and append the arena floor + 4 walls + ceiling light + OAK-D Pro overhead camera + slotted block + tube to the worldbody. The base/gripper splice anchors are fixed MJCF body names (issue #88 follow-up), so deploy sim composes the twin off the robot's own MJCF; every dimension and camera pose is driven by the typed BoxSceneOptions. Output written next to the upstream MJCF so meshdir="assets" resolves at compile time without copying STLs. (L531) - Private helpers: _resolve_so101_mjcf, _look_at_quat, _reanchor_robot_base, _splice_wrist_camera, _render_arena_geoms, _render_overhead_camera, _render_slot_block (5-box decomposition with a square hole + slot), _render_tube (cylinder + two end-tip sites). - env.py: - _options_from_backend_options(raw) -> BoxSceneOptions — Validate + parse scene.backend_options into a BoxSceneOptions; rejects unknown keys loudly so YAML typos surface immediately. (L66) - class _So101BoxRollout — SimRollout driving the SO-101's 6 position actuators, two MuJoCo renderers (RGB + depth-mode on the same overhead camera), random spawn at every reset, and the geometric insertion success check. Each step() advances a full control period of physics (steps_per_control_period, from options.control_hz), and joint_units: degrees mode uses the shared _so_arm_units LeRobot conversions (arm affine + gripper channel normalised [0, 100] over the jaw range) in both directions — a single 2 ms mj_step per action and a gripper-as-degrees mapping previously left the arm unable to reach any 30 FPS checkpoint target and pinned the jaw at the clipped limit. Exposes mujoco_handles() + sim_time_ns() (round(MjData.time * 1e9)) for openral sim run --view / the sim clock, and action_dim (== 6) so SimAttachedHAL._probe_env_action_dim resolves the deploy-sim action width (probe-gap fix). (L154) - build_so101_box_scene(env_cfg) -> _So101BoxRollout — Scene factory registered as SCENES.register("so101_box", fixed_robot="so101_follower")(build_so101_box_scene). Composes the MJCF, resolves the 6 arm actuators by their upstream numeric names ("1"…"6"), and caches the block / tube / hole / tip site indices for the success check. (L533)

`python/sim/src/openral_sim/backends/_so_arm_units.py`

Shared unit + cadence conversions for the raw-MuJoCo SO-ARM bench scenes (so101_eraser, so101_box) — one home for the two conventions LeRobot-trained SO-101 checkpoints impose, so a fix in one scene can never miss the other. - steps_per_control_period(timestep_s, control_hz, *, scene) -> int — Physics steps that make up one control period: round(1 / (control_hz * timestep_s)), at least 1; raises ROSConfigError on non-positive control_hz. One policy action must cover a control PERIOD of physics — a 30 FPS-trained checkpoint's absolute joint targets each assume ~33 ms of travel, and stepping one 2 ms tick per action gives the position actuators 1/17th of that (proprio never progresses, the policy re-issues near-home commands forever). (L29) - lerobot_action_to_radians(action, *, joint_signs, joint_offsets_deg, gripper_range) -> NDArray[float64] — LeRobot degrees-mode action → MuJoCo radian ctrl targets: arm channels invert the calibration affine (lerobot_deg = signs * mujoco_deg + offsets) then convert to radians; the LAST (gripper) channel is NOT degrees — LeRobot SO-ARM datasets store it normalised [0, 100] over the jaw travel, so it maps that fraction onto the jaw's radian range (the same [0, 1]-style surface the deploy HAL exposes). (L48) - radians_to_lerobot_state(qpos, *, joint_signs, joint_offsets_deg, gripper_range) -> NDArray[float64] — Inverse map for proprio: arm qpos → LeRobot servo degrees via the affine, gripper qpos → normalised [0, 100] (a plain degrees(qpos) would report the closed jaw as ≈ -8, below the checkpoint normalizer's observed minimum). (L67)

`python/sim/src/openral_sim/backends/tabletop_push/`

Greenfield robot-agnostic native scene: a push-cube-to-goal task on a configurable tabletop, registered FREE-AXIS as @SCENES.register("tabletop_push") (no fixed_robot). The robot is a flag — env_cfg.robot_id resolves a RobotDescription whose sim.mjcf_uri provides the base arm MJCF; the table/cube/goal/cameras are appended to that robot's MjSpec worldbody and the robot root body is re-anchored, so no robot-specific scene code is needed (verified for SO-100, SO-101, Franka, UR5e). When the scene requests a wrist camera and does not hardcode an MJCF body, the adapter infers the mount from robots/<id>/robot.yaml (sensors[].sim_placement.parent_body). Success is geometric (cube centre within goal_radius of the goal disc and still resting on the table), so it makes no gripper/end-effector assumption. Action/state dim = the compiled model's actuator count nu, with the appended task world preserving the robot's low actuator/qpos indices (the same contract MujocoArmHAL._sim_kwargs_for relies on). - _assets.py: - class TabletopOptions — Typed dataclass holding every scene-geometry knob (table slab, robot-mount fallback, cube, goal disc + radius, two world cameras, opt-in wrist camera, settle steps, optional reset joint pose, scaled joint-unit affine, lighting). Fed from scene.backend_options by _options_from_backend_options. (L51) - compose_tabletop_mjcf(description, options=None, *, base_pose=None) -> mujoco.MjModel — Resolve the robot MJCF from the manifest (assets.mjcf via _resolve_robot_mjcf → openral_core.assets.resolve_asset), load it into an MjSpec, re-anchor the robot root body (worldbody.bodies[0]) to base_pose (full 6-DOF) or the yaw-only robot_base_xyz fallback, append the table + cube (freejoint) + goal site + overhead/front cameras + light, and compile. Robot-agnostic — no body-name regex. (L189) - Private helpers: _resolve_robot_mjcf, _base_pos_quat, _append_table, _append_cube, _append_goal_marker, _append_world_cameras, _append_overhead_light, _append_wrist_camera, _look_at_quat. - env.py: - _options_from_backend_options(raw) -> TabletopOptions — Validate + parse scene.backend_options into a TabletopOptions; rejects unknown keys loudly. (L62) - class _TabletopPushRollout — SimRollout driving the robot's nu actuators by index (clipping each to its transmission joint's range), rendering the world cameras, randomising the cube + goal each reset (goal via model.site_pos), and the robot-agnostic on-goal success check. Exposes mujoco_handles() + sim_time_ns() (round(MjData.time * 1e9)) for openral sim run --view / the sim clock, and action_dim (== robot actuator count nu) so SimAttachedHAL._probe_env_action_dim resolves the deploy-sim action width (probe-gap fix). (L135) - build_tabletop_push_scene(env_cfg) -> _TabletopPushRollout — Scene factory registered as SCENES.register("tabletop_push")(build_tabletop_push_scene) (free-axis). Composes the model, resolves the robot's actuator→joint transmissions for state read + action clipping, and caches the cube body/freejoint + goal site indices. Raises ROSConfigError when robot_id has no registered manifest. (L344)

`python/sim/src/openral_sim/policies/mock.py`

class _MockSim — Tiny gym-like env for tests. (L31)
class _ZeroPolicy — Always emits zero-vector actions. (L117)
class _RandomPolicy — Fixed-seed Gaussian samples. (L136)
_coerce_int(value, default) -> int (L88)
_build_mock_scene(env_cfg) -> _MockSim (L100)
_resolve_action_dim(env_cfg) -> int (L159)
_build_zero_policy(env_cfg) -> _ZeroPolicy (L202)
_build_random_policy(env_cfg) -> _RandomPolicy (L211)

`python/sim/src/openral_sim/policies/smolvla.py`

Chunk-executor wiring (all chunked adapters) — policy factories assign build_chunk_executor(...) to their adapter. SmolVLA/xVLA use policy.predict_action_chunk; pi05, GR00T, MolmoAct2, and OpenVLA pass family-specific producers. Declared chunk sizes are enforced for custom producers so telemetry equals actions consumed. Diffusion Policy remains excluded because it consumes observation history every tick; ZMQ sidecars remain synchronous because REQ sockets cannot be shared with a prefetch thread. Real-Time Chunking rides the same seam: an enabled policy_extras.rtc block (gated to smolvla + pi05, and requiring chunk_prefetch) makes build_chunk_executor install the policy's lerobot RTCProcessor and hand the executor an ActionQueue that blends each prefetched chunk into the executing one. The producer must accept the extra inference_delay / prev_chunk_left_over kwargs — _PI05Adapter._chunk_forward(batch, **kwargs) forwards them straight to predict_action_chunk; SmolVLA's default predict_action_chunk producer already takes them.
class _SmolVLAAdapter — Lerobot-style policy adapter. (L181) — reset/step/close/_prepared_batch/_build_batch/_update_input_preview. Steps through the executor buffer (see wiring note above; per-step fallback via run_inference when n_action_steps <= 1). Benchmark/eval leaves prefetch off, preserving synchronous fresh-observation replans and published success semantics; real-time deploy sets chunk_prefetch and overlaps predict_action_chunk with replay (the live eraser rollout improved 14.6 Hz → 20.9 Hz and reduced average boundary stalls 0.73 s → 0.52 s); batch build is lazy (only on inference-launching ticks); reset()/close() delegate to the executor; _update_input_preview keeps the debug-video frame fresh from raw numpy only. Zero-copy NVMM vision leg (Phase 3): when the observation carries image_handles (NVMM descriptors from the co-located sensor leg), _maybe_encode_image_handles lazily builds an NvmmVisionEncoder sharing the TRT runtime's cached vision engine, orders the handles by the checkpoint's image_features (_handles_in_engine_order — a swapped order would silently cross the camera embeddings), encodes straight on the device pointers, and stashes the result via set_precomputed_img_embs; _build_batch(gpu_frames=True) then emits device-resident placeholder pixels that only keep lerobot's prepare_images/tokenizer plumbing satisfied. This shared sampler-side embedding state is serialized: step() tears down the chunk executor for any observation carrying image_handles and runs in the foreground, so a later tick cannot overwrite embeddings while a background inference reads them. Handles without an attached TRT runtime, or partial camera coverage, raise ROSRuntimeError (no silent blind fallback — handle frames carry no CPU pixels). Mixed-precision input cast: lerobot's from_pretrained loads SmolVLA natively mixed — a bf16 VLM backbone but a float32 action expert (the flow-matching sampler allocates float32 noise/time internally and needs the expert in float32). step() therefore casts only the image inputs to the backbone dtype (_image_dtype); observation.state stays float32 to match the expert. Unifying the whole policy is wrong: fp32 ~doubles memory (OOMs the reward sidecar on 8 GB), bf16 breaks the sampler.
_build_smolvla(env_cfg) -> _SmolVLAAdapter — Resolves device + rSkill via _vla_core.resolve_device / resolve_rskill_repo_id(adapter_name="SmolVLA"); resolves the manifest through the shared openral_sim.policies._policy_loading.load_manifest_for_spec; defers torch + lerobot imports through the shared lazy_import_lerobot("SmolVLA") (both helpers extracted in the 2026-05 cleanup to drop the parallel local copies). Calls _vla_core.apply_chunk_replay and _vla_core.maybe_compile_chunk_forward to enable vla.extra.n_action_steps / compile — torch.compile is skipped (logging smolvla.compile_skipped_for_rtc) when _vla_core.rtc_enabled_in_extra is true, because RTC rewrites the same flow-matching forward torch.compile would capture; the skip is keyed on the parsed enabled flag, so rtc: {enabled: false} keeps its compile. Loads the lerobot PolicyProcessorPipeline via _vla_core.materialize_processor_dir(manifest) — per-file hf_hub_download driven by manifest.processors (rSkill self-containment audit Gap 1+3). No snapshot_download. Stats-fallback path: when the per-file download 404s (community finetunes routinely ship only config.json + model.safetensors) the adapter logs smolvla_processor_files_missing_falling_back_to_dataset_stats and rebuilds the processors from manifest.dataset_uri's normalization stats via _load_lerobot_dataset_stats(...) + make_pre_post_processors(..., dataset_stats=...). If dataset_uri is also unset a typed ROSConfigError is raised. Every load phase (imports, from_pretrained, to_device, processor_dir, make_processors) is wrapped in _smolvla_phase(...). After to_device it computes _image_dtype (the model's majority param dtype — the bf16 backbone) for the adapter's image-input cast; None on a fully-float32 load. (L557)
_smolvla_phase(name, **fields) -> ContextManager[None] — Adapter-local shortcut for phase_timer(name, prefix="smolvla", log=_log). (L162)
_is_processor_missing(exc: BaseException) -> bool — Walks __cause__ / __context__ looking for a HF Hub RemoteEntryNotFoundError / EntryNotFoundError. Detects 404s that materialize_processor_dir re-raised as ROSConfigError; stays decoupled from HF Hub's exception module path. (L65)
_load_lerobot_dataset_stats(dataset_uri: str) -> dict[str, dict[str, Any]] — Aggregates per-feature stats from a LeRobotDataset on HF Hub. Tries v3 (single meta/stats.json) first; on 404 falls back to v2.1 (meta/episodes_stats.jsonl) aggregated via lerobot.datasets.compute_stats.aggregate_stats. Returns a {feature_key: {mean|std|min|max|count: np.ndarray}} dict suitable for make_pre_post_processors(..., dataset_stats=...). (L80)

`python/sim/src/openral_sim/policies/openvla.py`

class _OpenVLAAdapter — Transformers custom-code OpenVLA/OpenVLA-OFT policy adapter. Loads one RGB frame + OpenVLA prompt into the checkpoint processor, calls either predict_action or RLinf's generate_action_verl, normalizes returned single actions / OFT chunks to (chunk, action_dim), applies optional manifest policy_extras action postprocess (action_scale, binary gripper threshold), and replays the chunk from an internal queue. reset() clears the queue and reapplies openvla_torch_seed after SimRunner's per-episode RNG seeding so stochastic generation is reproducible.
_build_openvla(env_cfg) -> _OpenVLAAdapter (@POLICIES.register("openvla")) — Resolves the rSkill manifest from spec.weights_uri, requires OPENRAL_ALLOW_REMOTE_CODE=1, loads the HF repo via AutoModelForVision2Seq.from_pretrained(..., trust_remote_code=True), uses NF4/device-map placement on CUDA ({"": cuda_index}), patches _unnormalize_actions to move accelerate CUDA tensors to CPU before NumPy, resolves cameras from the manifest/scene, validates openvla_generation_method, and threads OpenVLA policy_extras into the adapter.
Pure helper surface: _decode_prompt(instruction) -> str, _unnormalize_action(norm, stats) -> np.ndarray, _as_action_chunk(arr, action_dim) -> np.ndarray, _postprocess_action_chunk(arr, *, action_scale, binarize_gripper, gripper_threshold) -> np.ndarray. Unit-tested without GPU; the opt-in sim test covers the real RLinf checkpoint on SimplerEnv WidowX.

`python/sim/src/openral_sim/policies/xr1.py`

Xiaomi Robotics XR-1 / MiBoT adapter. The openral process owns history, state-layout conversion, action replay, and the shared sidecar wire; manifest-selected bitsandbytes NF4 runs inside the torch 2.9.1 / transformers 4.57.1 / FlashAttention custom-code sidecar. - class _XR1Adapter — PolicyAdapter for robocasa_mg, robocasa365, and vlabench_choice; requires OPENRAL_ALLOW_REMOTE_CODE=1, validates three cameras and per-profile state/action dimensions, keeps the RC365 interval-two history, and replays the upstream cadence. - _rc365_state(state) -> NDArray[np.float32] — 16-D OpenRAL quaternion layout → 14-D XR-1 axis-angle layout. - _history_sample(values) -> NDArray[np.float32] — left-pad a seven-frame history and select offsets [-6,-4,-2,0]. - _vlabench_targets(deltas, state) -> NDArray[np.float32] — integrate XR-1's VLABench deltas into absolute env targets with wrapped Euler angles. - _quantization_mode(manifest) -> str — map quantization.dtype=int4 to the sidecar's nf4 loader and bf16 to the unquantized path; reject unsupported dtypes. - The module docstring carries the local persistent-NF4 export command. A manifest with policy_extras.prequantized_nf4: true selects prequantized_nf4, which reloads Transformers-native packed shards directly.

`python/sim/src/openral_sim/policies/act.py`

class _ACTAdapter — ACT policy adapter. Manifest-first image_preprocessing resolution (_cam_alias + _image_input_template + _flip_images_180) lets LIBERO camera1 / camera2 feed an ACT checkpoint whose input features are observation.images.image / observation.images.image2; legacy _state_mean / _action_std path stays for act-aloha-style checkpoints with norm stats in model.safetensors.
_build_act(env_cfg) -> _ACTAdapter — Snapshots the policy weights for ACTPolicy.from_pretrained (config.json sanitized via _sanitize_act_config_json before load). Dispatches the processor branch on manifest.processors is not None: modern (e.g. rskills/act-libero) calls _vla_core.materialize_processor_dir(manifest) and composes the lerobot factory pipelines with preprocessor_overrides={"device_processor": {"device": <resolved>}} so checkpoints with a baked-in device: mps don't crash on CUDA hosts; legacy (rskills/act-aloha) keeps the _try_load_act_norm_stats path that reads norm stats from model.safetensors. Calls _vla_core.apply_chunk_replay (manifest-aware default) and _vla_core.maybe_compile_chunk_forward. Resolves camera keys / state dim / image aliases via _vla_core.resolve_* helpers (mirrors smolvla); default cam tuple derives from ip.aliases.keys() when set so a LIBERO scene that emits camera1 / camera2 "just works". rSkill self-containment audit Gap 1+3.
_load_manifest_for_spec(spec) -> RSkillManifest | None — Mirror of smolvla's helper; loads the rSkill manifest from a skill reference in spec.weights_uri; returns None when the URI is not resolvable to a manifest.
_sanitize_act_config_json(snapshot_dir) -> None — Drops ACTConfig fields the installed lerobot version doesn't accept (e.g. n_state_dim on training-fork checkpoints). Mutates config.json in-place, no-op when there's nothing to strip.
_apply_temporal_ensemble(policy, spec_extra) -> float | None — Sets policy.config.temporal_ensemble_coeff and builds the missing ACTTemporalEnsembler (lerobot only attaches one when the coeff is non-None at __init__ time).
_try_load_act_norm_stats(repo_id, device, torch, cam_keys) -> dict — Pulls normalize_* / unnormalize_* tensors from model.safetensors for the legacy act-aloha-shaped checkpoints.

`python/sim/src/openral_sim/_quantization.py`

Shared bitsandbytes NF4 quantization helpers + prequantized-state-dict fast path + manifest-driven dtype resolution. Family-agnostic — the same primitives serve pi05 today and any future bnb-quantized backbone (pi0.6, smolvla-large). All helpers defer torch / bitsandbytes imports so installing openral-sim does not pull them transitively.

DEFAULT_MIN_PARAMS_TO_QUANTIZE: int = 4_000_000 — Per-Linear weight-element threshold for the nf4 rewrite. PaliGemma / SmolVLA paper default. (L60)
quantize_nf4_in_place(policy, *, torch, compute_dtype, min_params=DEFAULT_MIN_PARAMS_TO_QUANTIZE, new_modules_on_meta=False) -> None — Walks the policy, replaces every torch.nn.Linear whose weight has ≥ min_params elements with a bnb.nn.Linear4bit. The actual nf4 pack runs on the next .to(<cuda>); bias terms stay in compute_dtype for numerical safety. new_modules_on_meta=True wraps the replacement walk in accelerate.init_empty_weights() so the bnb constructor's bf16 placeholder allocation lands on the meta device — saves ~5–10 s on a 3.4 B-param load when the caller is going to to_empty(device=...) the tree afterwards. (L72)
quantize_int8_in_place(policy, *, torch, compute_dtype, min_params=DEFAULT_MIN_PARAMS_TO_QUANTIZE, threshold=6.0, new_modules_on_meta=False) -> None — Sibling of quantize_nf4_in_place that swaps the same large Linears for bnb.nn.Linear8bitLt (LLM.int8 mixed decomposition, ~50% the bf16 footprint, lossless on most attention workloads). bitsandbytes only offers 4-bit and 8-bit Linears — there is no nf8; int8 here means LLM.int8, not torchao dynamic int8. No prequant fast-path: SCB sub-state ownership inside Int8Params makes a separate Hub artefact brittle. (L174)
install_prequantized_linears(policy, state, *, device, torch) -> tuple[int, set[str]] — Replaces every Linear4bit.weight with Params4bit.from_prequantized(...) data read from state. Returns (n_modules_rebuilt, consumed_state_keys) so the caller can subtract the consumed keys before calling policy.load_state_dict for the residual. (L294)
detect_prequantized_nf4(spec) -> str | None — Probes the rSkill's HF repo for a quantization_metadata.json sentinel; returns the repo id when the pack is present, None otherwise. Routed through _hf_download_cached_first so a cache hit avoids the HEAD request. (L376)
load_prequantized_state_for_rskill(policy, spec, *, torch, log_event_prefix="rskill") -> None — Combined entry point: validates the metadata sentinel, downloads model.safetensors, calls install_prequantized_linears, then applies the residual via policy.load_state_dict(leftover, strict=False). Silent no-op when the rSkill ships bf16 weights — adapters can call it unconditionally after their own quantize_nf4_in_place. (L442)
peek_safetensors_keys(repo_id, *, filename="model.safetensors") -> set[str] | None — Reads only the safetensors header (~10 ms warm) and returns its key set. Works for both prequantized packs (nf4 fast path) and bare source checkpoints (int8 fast path that loads bf16 weights via load_state_dict instead of going through lerobot's from_pretrained). Used by targeted_reset_parameters to skip the kaiming / normal init walk for modules whose params will be overwritten by the upcoming state load. (L571)
targeted_reset_parameters(policy, *, covered_keys) -> None — Walks the policy and calls module.reset_parameters() only on modules whose direct parameter keys are NOT a subset of covered_keys (the safetensors key set about to be loaded). Skips containers (modules with no direct params). Pass covered_keys=None for the historical unconditional reset. Model-agnostic — promoted out of pi05.py so π0.5 / MolmoAct2 / future meta-init families share it. (L713)
tie_transformers_weights(policy) -> None — Walks the policy in pre-order and calls module.tie_weights() on each outermost transformers backbone, skipping descendants of already-tied modules; a raising tie_weights (e.g. a meta-init expert backbone) is non-fatal. Promoted out of pi05.py alongside targeted_reset_parameters. (L773)
normalise_manifest_dtype(manifest) -> str | None — Pulls manifest.quantization.dtype.value as a string; returns None for manifests without a quantization block. Lifted out of pi05.py into the shared module so smolvla / xvla / future quantized adapters share one implementation. (L635)
manifest_dtype(spec, manifest=None) -> str | None — Resolves the adapter's load dtype: spec.extra["dtype"] (per-run override) wins, falling back to manifest.quantization.dtype via normalise_manifest_dtype, then None (default_dtype_for_device picks a CUDA-aware default). Lifted out of pi05.py. (L653)
torch_dtype_for(torch, dtype_str, device) -> Any — Map a manifest dtype string (bf16/bfloat16, fp16/float16/half, fp32/float32) to a torch dtype, with a CUDA-aware default (bf16 on CUDA, fp32 elsewhere). Pass-through dtypes (nf4, int8) fall through to the default so adapters can pick a sensible compute dtype for the leaves that won't be quantized. Lifted out of pi05.py. (L678)
default_dtype_for_device(device) -> str — Picks a default load dtype when the manifest doesn't specify one: nf4 on CUDA (so 3.4 B-param backbones fit in ~4 GiB), fp32 elsewhere. Lifted out of pi05.py. (L701)

`python/sim/src/openral_sim/policies/_policy_loading.py`

Shared loader helpers for openral_sim policy adapters (extracted in this branch to remove the parallel _load_manifest_for_spec copies from smolvla.py / rldx.py / pi05.py). Module docstring explains why the manifest-resolution branch is generic but the quantization branch deliberately stayed family-specific.

load_manifest_for_spec(spec) -> RSkillManifest | None — Returns the parsed openral_core.RSkillManifest when spec.weights_uri is a resolvable skill reference; returns None for bare hf:// URIs and local paths so the caller can decide whether the missing manifest is fatal (SmolVLA raises; pi05 / RLDX fall back to the URI directly). Tolerant of spec=None / spec.weights_uri=None. (L53)
lazy_import_lerobot(adapter_name, *, install_hint="just sync --all-packages --group libero") -> tuple[Any, Any] — Imports torch + lerobot's make_pre_post_processors factory behind a typed ROSConfigError with the install hint. Centralises the same import-time ceremony SmolVLA / π0.5 used to duplicate inline. Returns (torch, make_pre_post_processors); the caller imports the adapter-specific Policy class separately. (L84)

`python/sim/src/openral_sim/policies/pi05.py`

_build_pi05(env_cfg) -> _PI05Adapter — Calls _vla_core.apply_chunk_replay. compile is intentionally NOT plumbed: the adapter sets pi05_cfg.compile_model = False to keep the quantization path stable. Supports quantization.dtype ∈ {nf4/int4, int8, bf16, fp16, fp32}: nf4/int4 runs quantize_nf4_in_place + optional prequant fast-path; int8 runs quantize_int8_in_place against bnb.nn.Linear8bitLt (LLM.int8, CUDA-only); everything else casts and moves to device. The manifest's quantization.dtype is consulted via openral_sim._quantization.manifest_dtype when no spec.extra["dtype"] override is set (the four _manifest_dtype / _normalise_manifest_dtype / _torch_dtype_for / _default_dtype helpers used to live here — moved to _quantization.py so smolvla / xvla can reuse them). Manifest resolution routes through openral_sim.policies._policy_loading.load_manifest_for_spec. Processor sidecars resolved via _resolve_pretrained_path(spec, repo_id) → delegates to _processors.resolve_processor_dir (manifest-first, per the rSkill self-containment audit Gap 1+3; snapshot fallback for non-rSkill refs). Every load phase is wrapped in _pi05_phase(...) so the operator sees a per-phase wall-time + GPU footprint in the logs and in openral dashboard. Both nf4 and int8 take a fast meta-init path on CUDA that skips lerobot's slow PI05Policy.from_pretrained (~152 s for the 3.4 B-param backbone): nf4 loads from the prequant safetensors via load_prequantized_state_for_rskill; int8 loads the source bf16 safetensors via _load_bf16_state_for_int8 + _rebuild_int8_params_for_linear8bitlt. Combined effect on warm RTX 4070 cache: 95 s → 10 s (9×) for nf4 / 165 s → 11 s (14.6×) for int8. (L475)
_PI05Adapter._chunk_forward(batch, **kwargs) -> Any — Chunk producer for the executor; predicts under the adapter's autocast. **kwargs carries the executor's RTC arguments (inference_delay / prev_chunk_left_over) straight through to predict_action_chunk, and is empty on the non-RTC path.
_pi05_phase(name, **fields) -> ContextManager[None] — Adapter-local shortcut for phase_timer(name, prefix="pi05", gpu_mb=True, log=_log). (L427)
_targeted_reset_parameters, _tie_transformers_weights — module-level aliases re-importing _quantization.targeted_reset_parameters / tie_transformers_weights (promoted to the shared module so MolmoAct2's fast meta-init reuses them; the int8 fast path here still calls them via the alias).
_expand_covered_keys_via_tied_storage(policy, covered_keys) -> set[str] — Detects tied parameters via Tensor.untyped_storage().data_ptr() and extends covered_keys to include every key in a tied group whenever any member is already covered. Uses named_parameters(remove_duplicate=False) because the default dedups tied params away. (L263)
_load_bf16_state_for_int8(policy, repo_id, *, torch) -> None — Downloads <repo>/model.safetensors via _hf_download_cached_first and applies it via policy.load_state_dict(strict=False). The int8 fast meta-init path's substitute for lerobot's ~152 s PI05Policy.from_pretrained. (L361)
_rebuild_int8_params_for_linear8bitlt(policy) -> int — Re-wraps each Linear8bitLt.weight as a fresh bnb.nn.Int8Params(has_fp16_weights=False). to_empty(device=...) strips Parameter subclasses; without this rewrap the downstream policy.to(<cuda>) would never trigger bnb's int8 pack. (L308)
_resolve_pretrained_path(spec, repo_id) -> str — Returns a local directory containing the lerobot processor sidecars. Local path → verbatim; otherwise routes through _processors.resolve_processor_dir. (L441)

`python/sim/src/openral_sim/policies/molmoact2.py`

MolmoAct2 loads lerobot's in-tree MolmoAct2ForConditionalGeneration directly (via lerobot's MolmoAct2Config + AutoProcessor) — no trust_remote_code, no AutoModelForImageTextToText, no OPENRAL_ALLOW_REMOTE_CODE guard — and is not a lerobot policy. The adapter drives its predict_action(...) continuous-action API and replays the returned chunk one step at a time (own queue, not lerobot's select_action). Model graph + processor + norm_stats.json load from the manifest's source_repo (hf://allenai/MolmoAct2-LIBERO); the NF4 weights overlay from the manifest's weights_uri prequant pack. Verified end-to-end on LIBERO-Spatial (NF4, 8 GiB RTX 4070, success on task 0).

_build_molmoact2(env_cfg) -> _MolmoAct2Adapter (L406, @POLICIES.register("molmoact2")) — Loads Ai2's MolmoAct2 (model_family: "molmoact2", ~5.49 B params; Molmo2-ER VLM + flow-matching action expert, arXiv:2605.02881). Resolves the manifest + dtype, delegates the load to _load_molmoact2_model, then wires replay cadence (clamped to config.max_action_horizon, LIBERO = 10), norm tag, image flips, state/action dims, and autocast. (L685)
_load_molmoact2_model(*, torch, model_cls, config_cls, processor_cls, source_repo, spec, device, dtype_str, max_crops) -> tuple[model, processor, use_nf4, torch_dtype] (L382) — Always loads the processor (AutoProcessor.from_pretrained(...), optional image_processor.max_crops override). nf4-on-CUDA with a prequant pack takes a fast meta-init path (mirrors π0.5): detect_prequantized_nf4 → MolmoAct2Config.from_pretrained → build on the meta device via accelerate.init_empty_weights() + MolmoAct2ForConditionalGeneration._from_config(...) → quantize_nf4_in_place(new_modules_on_meta=True) → to_empty("cpu") → tie_transformers_weights → targeted_reset_parameters(covered_keys=peek_safetensors_keys(pack)) → load_prequantized_state_for_rskill → .to(device). Skips the ~200 s bf16 from_pretrained materialisation; measured 202 s → 14 s on a warm RTX 4070 cache. No manual buffer reconstruction needed — MolmoAct2RotaryEmbedding self-heals a meta/garbage inv_freq (persistent=True → restored by the pack). Falls back to the slow path (from_pretrained on CPU → precast_bf16 → quantize_nf4_in_place → prequant overlay → .to(device)) for bf16 / non-CUDA / no-pack. Supports quantization.dtype ∈ {nf4/int4, bf16, fp16, fp32}; nf4 is CUDA-only and the default on CUDA (bf16 ≈ 11 GiB → OOMs an 8 GiB GPU, nf4 ≈ 4 GiB). Every load phase wrapped in _molmoact2_phase(...). (L526)
_resolve_max_crops(spec, manifest) -> int | None (L305) — Resolve the image-processor max_crops override: vla.extra["image_max_crops"] → OPENRAL_MOLMOACT2_MAX_CROPS env → manifest.image_preprocessing.image_max_crops → None (checkpoint default 8). A secondary vision-activation lever: measured on an 8 GiB RTX 4070 (transformers 5.x) it does not by itself decide the 8 GiB fit — the inference peak is set by the LM token-embedding, and the fast MolmoAct2ImageProcessor largely ignores max_crops. The actual 8 GiB enabler is _enable_expandable_segments. (L499)
_enable_expandable_segments() -> None (L131) — os.environ.setdefault(_CUDA_ALLOC_ENV, "expandable_segments:True") before the first CUDA allocation (called at the top of _build_molmoact2 when device is CUDA). _CUDA_ALLOC_ENV is resolved once at import via openral_sim._sidecar_common.installed_alloc_conf_var (PYTORCH_ALLOC_CONF on torch ≥2.9, else PYTORCH_CUDA_ALLOC_CONF — the name was renamed in 2.9 and the old spelling now logs a deprecation warning). MolmoAct2 NF4 is ~6 GiB resident and peaks ~7.63 GiB; on an 8 GiB card (~7.6 GiB usable) the first forward's ~1.5 GiB embedding cat OOMs without expandable segments and fits with them (verified). No-op if the operator already set the var. (L134)
_molmoact2_phase(name, **fields) -> ContextManager[None] (L105) — Adapter-local shortcut for phase_timer(name, prefix="molmoact2", gpu_mb=True, log=_log). (L160)
_import_molmoact2() -> tuple[Any, Any, Any] (L118) — Imports lerobot's in-tree MolmoAct2ForConditionalGeneration + MolmoAct2Config + transformers AutoProcessor behind a typed ROSConfigError install hint. Returns Any (lerobot / transformers are optional, unstubbed deps). (L173)
_strip_hf_uri(uri, *, field_name) -> str (L145) — Strip the hf:// prefix off a manifest URI, validating it is present. (L289)

`python/sim/src/openral_sim/policies/_processors.py`

Shared resolve_processor_dir(spec, repo_id) -> str helper used by the diffusion / xvla / pi05 adapters to fetch policy_preprocessor.json / policy_postprocessor.json. Mirrors the smolvla / modern-ACT pattern and closes the three sister TODOs on the rSkill self-containment audit (2026-05-18).

resolve_processor_dir(spec, repo_id) -> str — Manifest-first: when spec.weights_uri resolves to a manifest that declares a processors block, delegates to materialize_processor_dir(manifest) (per-file hf_hub_download). Otherwise falls back to snapshot_download(repo_id, ignore_patterns=["*.md"]) — the path legacy hf://lerobot/diffusion_pusht URIs still rely on. (L32)

`python/sim/src/openral_sim/policies/rldx.py`

Auto-managed sidecar adapter for RLWRLD/RLDX-1 (Qwen3-VL-8B + Multi-Stream Action Transformer, ~6.9 B params). Runs the upstream policy in an out-of-process Python 3.10 venv and speaks the server's native ZMQ + msgpack wire protocol — necessary because the rldx package pins requires-python = "==3.10.*" (incompatible with our 3.12 workspace) and ships a custom architectures=["RLDX"] class not in HF Transformers (the HF checkpoint does NOT include modeling_rldx.py, so trust_remote_code is not an escape). The adapter auto-spawns the sidecar on first observation (OPENRAL_RLDX_AUTO_SPAWN=1, default) so users run openral sim run once and never invoke the boot helper. Sidecar boot helper: tools/rldx_sidecar.py. Used as policy_id: "rldx" in rskills/rldx1-*/rskill.yaml. - class _RLDXSidecarAdapter — ZMQ-backed RLDX policy adapter. On __post_init__ it pings the server; if no answer and auto_spawn=True, it forks tools/rldx_sidecar.py (in its own start_new_session) with the manifest-resolved model id + port + quantization + embodiment tag, then polls ping until success or boot_timeout_s elapses (default 900 s — covers the first-run git clone + uv sync). Replays the upstream MSAT 16-action chunk. Replan precedence: vla.extra.replan_steps > manifest.n_action_steps > legacy _RLDX_CHUNK_LEN // 2 fallback — the rldx1-ft-{libero,gr1,rc365} manifests all ship n_action_steps: 16 (replay the full chunk; halves inference round-trips vs the old half-chunk RTC default at the cost of 16 open-loop env steps between observations). Manifest-driven state_layout dispatch ("libero" → LIBERO-flat keys; "gr1" → Fourier-native general_embodiment; "rc365" → PandaMobile general_embodiment; "simpler_widowx" → SimplerEnv WidowX bridge_orig with OXE_BRIDGE_ORIG embodiment_tag; "simpler_google" → SimplerEnv Google fractal20220817_data with OXE_FRACTAL embodiment_tag (the FT-SIMPLER-* checkpoints' processor_config.json only ships bridge_orig / fractal20220817_data modality buckets — the OXE_WIDOWX / OXE_GOOGLE enum names exist but crash PolicyLoader.load with KeyError because their .value strings are not registered modality buckets)). Public contract: reset/step/close/last_input_frame. close() tears down the spawned child via SIGTERM → SIGKILL fallback; no-op when we connected to a pre-existing server. Before adopting a pre-existing sidecar (mode="existing") it calls _verify_existing_identity, which cross-checks the on-disk identity record (family/model/embodiment_tag/quantization, written by run_sidecar) and raises ROSConfigError on a mismatch — closing the "two checkpoints share the default port → second run silently serves the first one's model" hole; a missing record is treated as unverifiable (warn + proceed) so operator-managed boots keep working. - _encode_ndarray(obj) -> Any — msgpack default hook; serialises ndarrays via np.save → BytesIO wrapped in {"__ndarray_class__": True, "as_npy": <bytes>} (mirrors MsgSerializer.encode_ndarray in rldx/policy/server_client.py). - _decode_ndarray(obj) -> Any — msgpack object_hook; reverse of _encode_ndarray. - Manifest resolution: the adapter resolves skill references in weights_uri through the shared openral_sim.policies._policy_loading.load_manifest_for_spec helper so it can read state_contract.layout for LIBERO / GR1 / RC365 dispatch, image_preprocessing.flip_180, and the canonical hf:// model id. (The private _load_manifest_for_spec copy that used to live here was removed in the 2026-05 cleanup.) - _RLDXSidecarAdapter._init_socket / _try_ping / _verify_existing_identity / _wait_for_boot / _spawn_sidecar / _terminate_child / _is_port_busy / _locate_sidecar_script / _resolve_model_id — auto-spawn lifecycle helpers. _verify_existing_identity reads the sidecar identity record via openral_sim._sidecar_common.read_sidecar_identity and fails closed on a checkpoint mismatch. _init_socket (re)creates the ZMQ REQ socket with LINGER=0 + RCV/SND timeouts and connects to tcp://host:port; called from __post_init__ AND from _try_ping on failure because a REQ socket whose recv() timed out is stuck in EFSM (strict REQ state machine: every send() must be followed by a matching recv()) and every subsequent _call would raise Operation cannot be accomplished in current state until the socket is reopened. _try_ping does one timeout-bounded ZMQ round-trip and resets the socket on failure so _wait_for_boot actually makes forward progress instead of looping against a dead socket for the full boot_timeout_s; _spawn_sidecar Popens the boot script (skipped if _is_port_busy reports a listener already); _wait_for_boot polls every 2 s until success or boot_timeout_s or child death; _terminate_child does best-effort SIGTERM → SIGKILL teardown; _locate_sidecar_script walks upwards from __file__ to find tools/rldx_sidecar.py (or honours OPENRAL_RLDX_SIDECAR_SCRIPT); _resolve_model_id picks vla.extra.model_id → manifest weights_uri → spec weights_uri. - _build_libero_obs / _build_gr1_obs / _build_rc365_obs / _build_simpler_widowx_obs / _build_simpler_google_obs / _pick_single_camera / _pick_images / _pick_state / _normalize_action_column / _assemble_libero_chunk / _assemble_gr1_chunk / _assemble_rc365_chunk / _assemble_simpler_chunk — wire-format builders / parsers split by embodiment. The SimplerEnv builders mirror the upstream rldx/eval/sim/SimplerEnv/simpler_env.py reference: WidowX feeds video.image_0 + 8 state scalars (bridge-rotated Euler state.roll/pitch/yaw + state.pad=0 sentinel + raw state.gripper); Google feeds video.image + position/xyzw-quat split state + state.gripper = 1 - raw_open. _assemble_simpler_chunk binarizes the WidowX gripper column (2*(g>0.5)-1) so MS3's bridge digital twin sees [-1, +1] per the upstream WidowXBridgeEnv._postprocess_gripper; Google's sticky-gripper state machine is intentionally NOT applied here (per-rollout state belongs in the env wrapper, not the chunk assembler). The LIBERO chunk assembler rescales the gripper column from the RLDS dataset convention ([0, 1], 0=close/1=open) to LIBERO/robosuite ([-1, +1], -1=open/+1=close) via _rldx_gripper_to_libero before returning — without this the Franka gripper never actuates (GH-133). The GR1 path concatenates Fourier-native general_embodiment action groups (right_arm + left_arm + waist + right_hand + left_hand) into the Fourier GR-1 BASIC 29-D composite. The RC365 path concatenates the 5 PandaMobile groups (eef_pos + eef_rot + gripper + base + control_mode) into 12-D and lets openral_sim.backends.robocasa trim to the 11-D BASIC env action. The matching unflatten path on the RoboCasa side is openral_sim.backends.robocasa.GrootRoboCasaEnv._split_gr1_action (shared validator + slicer) → _to_gr1_action_dict_gym (gymnasium action.{waist,right_arm,left_arm,right_hand,left_hand} keys) / _to_gr1_action_dict (raw robosuite robot0_{torso,right,left,right_gripper,left_gripper} keys); both helpers reuse the same _GR1_BASIC_DIM=29 constant. - _rldx_gripper_to_libero(gripper) -> NDArray[float32] — Maps an RLDS-convention gripper column ([0, 1], 0=close/1=open) to the LIBERO/robosuite convention ([-1, +1], -1=open/+1=close), via out = -sign(2*g - 1). Mirrors the two-step transform (normalize_gripper_action + invert_gripper_action) that the upstream rldx/eval/sim/LIBERO/libero_env.py::LiberoEnv.step applies before stepping the env. Called from _assemble_libero_chunk; consumed by LiberoEnv.step via the 7-D LIBERO action vector at index 6. Fixes GH-133 (Franka gripper stuck open). - _env_bool(name, default) -> bool — permissive boolean env-var parser (1 / true / yes / on). - _resolve_state_layout(manifest) -> str — module-level helper shared by the rldx and gr00t factories; maps manifest.state_contract.layout to one of gr1/rc365/simpler_widowx/simpler_google, else "libero". Single source of truth for obs/action dispatch so neither factory hardcodes an embodiment. - _derive_sidecar_port(*, family, model, embodiment_tag, quantization, layout) -> int / _resolve_sidecar_port(*, port_env, extra_port, …) -> int — per-identity default port (SHA-1-bucketed into 20000–39999, non-crypto) so two different checkpoints never collide on the old hard 5555; _resolve_sidecar_port applies precedence env-pin > vla.extra.port > derived default. - _build_rldx(env_cfg) -> _RLDXSidecarAdapter — @POLICIES.register("rldx") factory. Honours OPENRAL_RLDX_HOST / OPENRAL_RLDX_PORT / OPENRAL_RLDX_AUTO_SPAWN / OPENRAL_RLDX_BOOT_TIMEOUT_S / OPENRAL_RLDX_QUANTIZATION / OPENRAL_RLDX_EMBODIMENT_TAG / OPENRAL_RLDX_MODEL_ID / OPENRAL_RLDX_SIDECAR_SCRIPT env-var overrides; reads replan_steps / image_size / timeout_ms / camera_keys / auto_spawn / boot_timeout_s / quantization / embodiment_tag / model_id from vla.extra; dispatches the obs/action contract via _resolve_state_layout and the port via _resolve_sidecar_port (per-identity default when unpinned).

`python/sim/src/openral_sim/policies/gr00t.py`

NVIDIA Isaac GR00T N1.7 policy adapter — standard checkpoints run in-process under the workspace's Python 3.12 via lerobot 0.6.0's native GrootPolicy (lerobot.policies.groot), mirroring the smolvla adapter. NF4 (backbone-only) keeps the ~3 B model on an 8 GiB card (~5.2 GiB peak); live-validated LIBERO-spatial 5/5 at ~19 ms/step. A manifest with policy_extras.implementation=behavior_b1k_sidecar branches before HF/native checkpoint resolution into behavior_groot.py, because the official organizer checkpoint is pinned to the separate wensi-ai/Isaac-GR00T Python 3.10 runtime. RLDX-1 (a GR00T-N1.5 finetune) still runs on its own ZMQ sidecar via the rldx adapter.

`python/sim/src/openral_sim/policies/behavior_groot.py`

Official BEHAVIOR-1K GR00T policy adapter. The organizer checkpoint depends on the pinned wensi-ai/Isaac-GR00T behavior branch rather than lerobot's native N1.7 loader, so it runs in an externally-provisioned Python 3.10 sidecar. The OpenRAL process carries only pyzmq/msgpack (behavior-groot dependency group).

build_behavior_groot_policy(env_cfg, manifest, extra) -> _BehaviorGrootAdapter — Resolve the local organizer checkpoint, sidecar endpoint, task/instruction, control mode, and quantization (policy_extras.quantization, default nf4, plus nf4_min_params); auto-spawn tools/behavior_groot_sidecar.py or connect to an operator/remote sidecar.
_BehaviorGrootAdapter — PolicyAdapter implementation that preserves the official flattened evaluator observation when present, otherwise maps canonical images.{head,left_wrist,right_wrist} + 61-D state back to the R1Pro wire keys; validates a finite 23-D action.
_behavior_wire_observation(observation, *, instruction) -> dict[str, object] — Pure official-wire assembler used by the adapter and unit tests.

`python/sim/src/openral_sim/backends/behavior.py`

BEHAVIOR-1K / OmniGibson scene adapter. Runs the official evaluator environment in its own sidecar and registers scene.id=behavior, fixed to r1pro.

class _BehaviorSidecar — ZMQ-backed SimRollout; surfaces 61-D state/policy_state, manifest-order 22-joint positions/velocities, three RGB views, and sim time.
step_action_group(actions) -> StepResult — Validate six equal-tick safety-approved slots and commit them as one official 23-D action.
_compose_action_group(actions) -> NDArray[float32] — Compose base twist + torso/arm joint targets + dual grippers into the official action order.
provision_behavior() -> None — Pre-launch provisioner: ensure_backend_deps("behavior_groot_client") + _sidecar_python(). OmniGibson + the BEHAVIOR dataset install out-of-band via upstream's ./setup.sh, so this only locates that environment or raises the setup command — printed before the launch instead of read as a 300 s on_configure timeout. Registered as provision= on the behavior scene. Covers provisioning only — the OmniGibson boot still runs inside on_configure, and _DEFAULT_BOOT_TIMEOUT_S = 1_200.0 (also set explicitly by scenes/deploy/behavior_r1pro.yaml) exceeds the launcher's hardcoded 300 s. See the boot-timeout note under tools/lifecycle_autostart.py.
_build_behavior_scene(env_cfg) -> _BehaviorSidecar — Resolve the official BEHAVIOR Python and auto-spawn tools/behavior_scene_sidecar.py.
_GrootAdapter(spec, device, _policy, _preprocessor, _postprocessor, _torch, ...) — In-process GrootPolicy adapter. libero_sim uses native relative actions whose GrootN17ActionDecodeStep refuses per-step decoding, so it predicts the full chunk via predict_action_chunk, decodes it while the pack-step state is fresh, queues the first _replan_steps (GR00T's 8-of-16 libero_sim horizon) and pops one per step (chunk-replay, like the rldx adapter). _build_batch feeds float CHW [0,1] images to the embodiment's modality keys (_image_input_keys — image/wrist_image for LIBERO, front/wrist for SO-101) + a _state_dim-wide proprio state; honours rSkill image_preprocessing.flip_180 / flip_vertical. Public contract: reset/step/close/last_input_frame.
_build_groot_config(*, local_path, embodiment_tag, quantize, image_keys, state_dim, action_dim) -> GrootConfig — Constructs a GrootConfig with embodiment_tag set at construction (the libero_sim gripper-flip / action-decode transform resolves in __post_init__; mutating it afterwards is too late) and explicit input_features / output_features pinning the head to the real 7-D LIBERO action instead of the 132-D padded default. model_params_fp32=not quantize (quantized params cannot be fp32-cast).
_quantize_groot_nf4(policy, torch, *, scope) -> None — NF4-rewrites the GR00T model via the accelerate-free openral_sim._quantization.quantize_nf4_in_place. scope="backbone" (default) packs only _groot_model.backbone (the ~2 B Qwen3-VL), leaving the diffusion head bf16 — enough for the 16-layer LIBERO head to fit 8 GB. scope="model" packs the whole _groot_model (backbone and the DiT head's large Linears), needed for heavier heads (the SO-101 fruit checkpoint's 32-layer DiT overshoots 8 GB otherwise). In both scopes the >=4M-param threshold spares the small TimestepEncoder MLP, so its params stay bf16 and the GR00T DiT uint8/silu bug cannot recur.
_patch_groot_dtype_property(torch) -> None — Rebinds GR00TN17.dtype to report the first floating-point param dtype. After NF4 the first backbone param is a Params4bit (.dtype == uint8), which would make prepare_input cast float image/state inputs to uint8 and produce garbage. A strict no-op for a non-quantized load, applied unconditionally.
_import_real_groot_policy() -> Any — Returns the real lerobot GrootPolicy, evicting the empty lerobot.policies.groot.modeling_groot compat stub (openral_rskill._lerobot_compat installs it only when the real module fails to import on older lerobot) so it cannot shadow the importable N1.7 class. Belt-and-suspenders on lerobot >= 0.6.0.
_to_groot_state(state, expected_dim) -> NDArray[float32] — Returns the GR00T proprio vector as flat float32, width-checked against expected_dim (the rSkill's state_contract.dim: 8 for the LIBERO eef-pose vector eef_pos(3) ‖ axisangle(3) ‖ gripper_qpos(2), 6 for the SO-101 single_arm(5) ‖ gripper(1) joint vector). GR00T pads to max_state_dim internally, so only a width check is needed.
_groot_phase(name, **fields) / _env_bool(name, default) — phase_timer shortcut (prefix gr00t) and permissive bool env-var parser.

`python/sim/src/openral_sim/policies/rlbench_3dda.py`

3D Diffuser Actor RLBench keyframe policy adapter. MIT. Proxies tools/rlbench_3dda_sidecar.py over the shared SidecarClient; the heavy DiffuserActor model + the 3-step obs history live in the externally-provisioned py3.10 venv (shared with the rlbench scene sidecar). step(observation, instruction) marshals the scene's multi-cam images/point_clouds + gripper_pose/gripper_open to the sidecar's get_action and returns an 8-D keyframe [x y z qx qy qz qw gripper_open] for backends/rlbench.py to plan + execute. Resolves repo/checkpoint/instructions/bounds via OPENRAL_3DDA_* env overrides (defaults to the provisioned cache locations). - _Diffuser3DActorAdapter(spec, device, _client) — PolicyAdapter; last_input_frame() for episode-video capture. - _build_diffuser_actor(env_cfg) -> _Diffuser3DActorAdapter — @POLICIES.register("diffuser_actor") factory; reads backend_options.{rlbench_task,variation,policy_port} (the policy needs the task to pick the matching CLIP instruction embedding). Connects a SidecarClient(name="rlbench-3dda", …).

`tools/rlbench_3dda_sidecar.py`

3D Diffuser Actor policy sidecar (runs under the py3.10 venv only; no openral import). Loads the published PerAct checkpoint into trajectory_optimization.DiffuserActor (embedding_dim 120, 256², 6D rot, wxyz, nhist 3, 100 DDIM steps), keeps per-episode obs history, serves ping/reset/get_action/close. Runs inference under no_grad (~0.43 GB VRAM). dgl.geometry.farthest_point_sampler is replaced by a pure-torch FPS shim; no flash-attn / pytorch3d needed. - _build_gr00t(env_cfg) -> _GrootAdapter — @POLICIES.register("gr00t") factory. Resolves the rSkill manifest, snapshot_downloads the raw N1.7 checkpoint locally (the GR00T processor factory only recognises a local raw-checkpoint dir), builds the config, loads GrootPolicy.from_pretrained on CPU under a float32 default dtype, NF4-rewrites the backbone, then a pure policy.to(device) packs the Linear4bit shells onto the GPU. Reads OPENRAL_GR00T_EMBODIMENT_TAG / OPENRAL_GR00T_QUANTIZATION env + vla.extra (embodiment_tag default libero_sim, quantization default nf4, camera_keys). Execution horizon (_replan_steps) resolves to GR00T's libero_sim 8-of-16.

`python/sim/src/openral_sim/policies/lingbot_vla2.py`

Robbyant LingBot-VLA 2.0 policy adapter (Apache-2.0 code + weights). Proxies the auto-provisioning boot helper tools/lingbot_vla2_sidecar.py over the shared SidecarClient; the 6.38 B model (Qwen3-VL-4B backbone + sparse-MoE flow-matching action expert) runs in its own Python 3.12 + torch-2.9.1 venv — the upstream lingbotvla package pins torch==2.8.0 / transformers==4.57.3 + custom Triton MoE kernels, incompatible with the workspace transformers>=5 (CLAUDE.md §3); the torch half of that pin set is overridden up to 2.9.1 / triton 3.5.1 because 2.8.0 has no linux-aarch64 cu128 wheel (docs/reference/aarch64-support.md), and the HF release ships no config.json architecture (no trust_remote_code escape). Used as model_family: "lingbot_vla2" in rskills/lingbot-vla2-robotwin/rskill.yaml; default embodiment robotwin (dual-arm AgileX Cobot Magic). - _LingBotVla2Adapter(spec, device, _client, _camera_keys, _replan_steps, ...) — PolicyAdapter proxying the sidecar. Predicts a (50, 14) chunk on the sidecar and replays one 14-D step per step, refilling (a new inference) when the queue drains (_DEFAULT_REPLAN_STEPS=25, mirroring the upstream --use_length 25 deploy). _refill marshals the 3 scene cameras (positionally re-keyed onto cam_high/cam_left_wrist/cam_right_wrist) + the 14-D proprio state + instruction to get_action; last_input_frame() for episode-video capture; close() sends close then tears down the client. (L216) - _build_lingbot(env_cfg, *, variant) -> _LingBotVla2Adapter — shared factory behind an auto-managed SidecarClient. variant selects the model family (v2 6B Qwen3-VL MoE / v1 4B Qwen2.5-VL dense expert). Launches tools/lingbot_vla2_sidecar.py --variant <v> with sys.executable. Reads vla.extra (model_id / robo_name / camera_keys / quantization / device / attn / port / replan_steps / auto_spawn); env overrides use the variant prefix (OPENRAL_LINGBOT_VLA2_* for v2, OPENRAL_LINGBOT_VLA_* for v1). expected_identity={model,robo_name} guards adopting a stale sidecar on the per-(variant,model,embodiment) default port. (L294) - _build_lingbot_vla2(env_cfg) / _build_lingbot_vla(env_cfg) — @POLICIES.register("lingbot_vla2") (6B) and @POLICIES.register("lingbot_vla") (4B RoboTwin post-train, model_family: "lingbot_vla" in rskills/lingbot-vla-4b-robotwin/rskill.yaml) thin wrappers over _build_lingbot. (L391) - _policy_default_port(model_id, robo_name, variant="v2") -> int / _resolve_camera_keys(env_cfg, extra) -> tuple[str, ...] / _resolve_model_id(spec, extra, default_model_id) -> str / _locate_sidecar_script() -> Path / _opt_int(value, default) -> int — per-(variant, model, embodiment) SHA-256-bucketed default port (20000–39999); scene-camera resolution (order load-bearing: top/left-wrist/right-wrist); model-id resolution (vla.extra.model_id > manifest weights_uri > spec.weights_uri > per-variant default); boot-helper locator; vla.extra int coercion. (L124)

`python/sim/src/openral_sim/policies/lingbot_va_a1.py`

Thin LingBot-VA real-deployment adapter for the Galaxea A1 embodiment. It connects to the Runtime-owned versioned policy gateway and returns its absolute six-joint plus normalized-gripper proposals through OpenRAL's normal candidate-action and safety path.

_LingBotVaA1Adapter(spec, *, robot_description) — Validates the A1 embodiment and rSkill model identity, negotiates the active OpenRAL joint envelope and policy-owned step bound, sends joints plus paired RGB observations, and validates the returned 7-D action proposal. Runtime owns its exact deployment config, EEF transforms, chunk/cache replay, and IK.
_joint_contract(spec, robot_description) — Extracts ordered finite command limits and rejects a policy substep above either active HAL phase limit.
_model_identity(spec) -> tuple[str, str] — Resolves the rSkill's immutable model repo/revision for the gateway identity handshake.
_build_lingbot_va_a1(env_cfg) -> _LingBotVaA1Adapter — @POLICIES.register("lingbot_va_a1") factory. The adapter explicitly accepts only the galaxea_a1 embodiment.

`tools/lingbot_vla2_sidecar.py` + `tools/_lingbot_vla2_server.py`

Boot helper + server for the LingBot-VLA 2.0 sidecar, companion to openral_sim.policies.lingbot_vla2. The boot helper runs under the openral interpreter: it clones github.com/robbyant/lingbot-vla-v2 at the pinned SHA 69729b4 into <home>/source (shallow-fetch of the exact commit; $OPENRAL_LINGBOT_VLA2_REPO reuses a checkout), builds a Python 3.12 venv from the upstream fully-pinned requirements.txt (transformers==4.57.3 / numpy==1.26.4 / …) under the _V2_OVERRIDES torch stack (torch==2.9.1 / torchvision==0.24.1 / torchaudio==2.9.1 / triton==3.5.1 / torchcodec==0.9.1 marker-scoped off aarch64, passed as uv pip install --overrides, replacing upstream's torch 2.8.0 / triton 3.4.0 / torchcodec 0.6.0 which publish no linux-aarch64 wheels — docs/reference/aarch64-support.md) + pyzmq + bitsandbytes via the shared ensure_pip_venv ($OPENRAL_LINGBOT_VLA2_SIDECAR_PYTHON reuses an interpreter), stamps the sidecar identity record, then os.execvpes into the server with make_isolated_env. The server (sidecar venv, no openral import) loads LingbotVLAv2Server (NF4 Qwen3-VL backbone via _nf4_backbone_in_place, bf16 MoE expert), reconstructs the missing lingbotvla_cli.yaml from configs/vla/robotwin/robotwin.yaml, and answers ping/reset/get_action/close over the same msgpack ndarray wire as openral_sim.sidecar. flash-attn is NOT installed; _install_attn_fallback / _coerce_attn_config patch the upstream flash_attention_2 hardcode to sdpa (or eager) before the model is built. _install_lerobot_stub installs a meta-path finder that stubs the training-only lerobot.* imports the model class pulls in (uninstallable here — it wants transformers 5.x), and _write_cli_yaml stringifies the joints/norm_type entries the upstream loader ast.literal_evals; both are required for the inference-only load to succeed (verified live). _install_moe_logger binds the logger name upstream's MoE fallback handler references but never defines (qwen2_action_expert.py: 2 references, 0 bindings), so a kernel fault surfaces as the real exception instead of NameError: name 'logger' is not defined — purely diagnostic, and it masked a genuine PTXASError during the aarch64 bring-up. A --variant {v2,v1} switch selects the model family end-to-end. v1 (LingBot-VLA 1.0, 4B) clones the separate V1 repo github.com/robbyant/lingbot-vla at pinned SHA 4eb34b7 and provisions its own transformers==4.51.3 / lerobot==0.4.2 (flat layout) venv via _install_v1 — it uses real lerobot (no stub) and its checkpoint ships real config.json + lingbotvla_cli.yaml (no _write_cli_yaml reconstruction). The V1 model (_LingBotV1Policy, upstream LingbotVLAServer) is a Qwen2.5-VL-3B backbone + a dense Qwen2 flow-matching expert; three server-side patches make its flash-free path correct: inject the missing rotate_half into the eager vision attention, force attn=eager (its custom attention has no sdpa kernel), and skip o_proj in NF4 (the interleaved attention reads o_proj.weight.dtype). - _ensure_source(home, *, url, sha, repo_env) -> Path (boot) — pinned-SHA shallow clone into <home>/source; per-variant URL/SHA/override env. - _ensure_venv(home, source, *, install, venv_env) -> Path / _install_v1(uv, py) (boot) — Python 3.12 venv; v2 installs the upstream requirements.txt + pyzmq/bitsandbytes, v1 installs the explicit V1 stack (torch cu128, lerobot==0.4.2, transformers==4.51.3, numpy==1.26.4, + wire/quant extras); per-variant *_SIDECAR_PYTHON override. - main() -> int (boot) — argparse (--model/--robo-name/--quantization/--device/--attn/--variant/--host/--port/--home); per-variant provisioning, writes the identity record (family="lingbot_vla_<variant>"), then os.execvpes tools/_lingbot_vla2_server.py --variant <v> with the per-variant repo + QWEN path env set. - _install_attn_fallback(*, target) / _install_attn_fallback_v1(*, target) / _patch_eager_vision_rotary_v1() / _coerce_attn_config(config, target) (server) — coerce _attn_implementation off flash before construction: v2 patches the Qwen3-VL / Qwen2 _from_config; v1 patches the shared PreTrainedModel._from_config (many sites) and injects rotate_half. - _LingBotPolicy / _LingBotV1Policy / _serve / _nf4_backbone_in_place(*, skip_names=…) / _write_cli_yaml (server) — load the NF4-backbone/bf16-expert model (v1 skips o_proj), flatten the upstream action.arm.position(12)+action.effector.position(2) to a (chunk, 14) array, and serve the ZMQ REP loop.

`python/sim/src/openral_sim/_sidecar_common.py`

Shared boot scaffolding for the out-of-process rldx VLA sidecar (a GR00T-N1.5 finetune, hence the _Gr00tFamilySidecarAdapter class name — GR00T N1.7 itself now runs in-process): clone, venv, install, env isolation, exec — plus the sidecar identity registry. - run_sidecar(*, label, family, repo_url, args, install_deps, make_wrapper) -> int — orchestrates a boot (uv → clone → install → wrapper → exec). Writes the identity record (write_sidecar_identity) just before exec_server so every sidecar this repo starts — auto-spawned or operator-launched — is identifiable. - sidecar_identity_path(port) -> Path / write_sidecar_identity(*, port, family, model, embodiment_tag, quantization) -> None / read_sidecar_identity(port) -> dict[str,str] | None — the per-port identity record under ~/.cache/openral/sidecars/port-<port>.json. The adapter reads it back in _verify_existing_identity to refuse reusing a sidecar serving a different checkpoint. None = no record (unverifiable, not a mismatch). - ensure_pip_venv(*, label, home, python, install, override=None, override_env=None, sentinel_name=".deps-installed", spec=None) -> Path — create / reuse / repair the <home>/.venv of a pip-installable sidecar (LocateAnything, Qwen VLM, DA3, XR-1, Cosmos 3, LingBot-VLA 2, Isaac, RoboTwin). spec is the dependency spec install applies (pinned requirement strings, or a lockfile's text); it is hashed into the completion sentinel via spec_marker, so correcting a pin invalidates the sentinel and re-runs install instead of being ignored forever. Before that the sentinel was an opaque ok and a venv was frozen at whatever it first resolved — how an Isaac venv kept an nvidia-nvjitlink-cu12 too old for its own prebundled cusparse (issue #89). spec=None keeps the opaque marker for callers with no stable spec. - spec_marker(spec) -> str — the sentinel content for a dependency spec: a NUL-separated sha256 digest, or "ok\n" for None. - ensure_uv / ensure_source / make_isolated_env / exec_server / build_parser / run_cmd — uv resolver lookup, shallow clone, 3.10-venv env scrubbing, os.execvpe into the server, shared --model/--port/--quantization/--embodiment-tag/--home CLI, echoed subprocess runner. make_isolated_env also setdefaults TRITON_PTXAS_PATH to a venv-local CUDA 12.9 ptxas when one is installed — triton 3.5.1 bundles a CUDA 12.8 ptxas that cannot target sm_121, so on GB10 / Jetson Thor every Triton kernel fails to compile until it is redirected (docs/reference/aarch64-support.md). - venv_ptxas(venv) -> Path | None — the nvidia-cuda-nvcc-cu12 ptxas inside venv, or None when that wheel isn't installed (the normal case on x86_64). Split out of make_isolated_env so a sidecar execing by another route, or a test, can ask the same question.

`python/sim/src/openral_sim/policies/internvla_n1.py`

InternVLA-N1 / DualVLN vision-language navigation policy adapter (InternRobotics, arXiv:2512.08186; weights CC-BY-NC-SA-4.0, code MIT). Proxies tools/internvla_n1_sidecar.py over the shared SidecarClient; the 8.3B dual-system model (Qwen2.5-VL-7B System-2 + NextDiT System-1) runs in an auto-provisioned py3.11 venv (upstream pins transformers 4.51). step(observation, instruction) takes observation["images"][cam] (RGB uint8), obtains metric depth from the DA3 sidecar (_da3, monocular — no robot depth sensor), sends both to the sidecar's step endpoint, and maps the returned twist=[v_forward, w_yaw] into a 6-D BODY_TWIST row [vx,0,0,0,0,wz]. Latches on the model's STOP (returns zero twist, stops calling the sidecar until reset()). Depth note: the DualVLN checkpoint's nextdit_async System-1 is RGB+latent conditioned and does not consume depth (verified against the model source), so OPENRAL_INTERNVLA_N1_DEPTH=none sends a unit-depth placeholder for it; a depth-consuming navdp checkpoint keeps the default da3. - _InternVLAN1Adapter(spec, device, _client, _camera_key, _da3, _stopped, _last_input) — PolicyAdapter; last_input_frame() for episode-video capture. - _build_internvla_n1(env_cfg) -> _InternVLAN1Adapter — @POLICIES.register("internvla_n1") factory. Resolves the rSkill manifest + NF4/int8 quantization + first scene/vla.extra camera key, derives a per-identity port, connects a SidecarClient(name="internvla-n1", …, expected_identity={family,model,quantization}) (auto-spawns tools/internvla_n1_sidecar.py), then builds a Da3DepthClient unless OPENRAL_INTERNVLA_N1_DEPTH=none.

`python/sim/src/openral_sim/da3_depth.py`

Thin ZMQ client for the DA3 monocular metric-depth sidecar (tools/_da3_depth_server.py, depth-anything/DA3-SMALL) — the SAME model the SLAM/nvblox depth provider uses. RGB in → float32-metres depth out (resized to the RGB frame). Monocular, so it feeds a policy identically in sim and on real hardware; speaks the perception bus's {"op": ...} protocol directly (not the SidecarClient framing). - Da3DepthClient(host, port=5771, process_res, auto_spawn) — connect() pings the default port and reuses an existing sidecar (e.g. SLAM's) when present, else auto-spawns tools/da3_depth_sidecar.py; infer(rgb) -> depth(H,W) float32; close() reaps an auto-spawned child. - DEFAULT_DA3_PORT = 5771 — the shared bind port (matches depth_provider_node).

`tools/internvla_n1_sidecar.py` + `tools/_internvla_n1_server.py`

Boot helper + server for the InternVLA-N1 nav sidecar, companion to openral_sim.policies.internvla_n1. The launcher provisions a py3.11 venv under ~/.cache/openral/internvla-n1-sidecar (clones InternRobotics/InternNav, installs the inference-only pin set — torch 2.9.1 on cu128 + transformers 4.51 + diffusers 0.32.2 + diffusion_policy --no-deps, no flash-attn — plus bitsandbytes for NF4; every uv pass carries --torch-backend=cu128 and the shared aarch64 nvrtc override), writes an argv shim, and run_sidecar(..., family="internvla_n1", …) execs the server. The server loads the checkpoint with a quantization-aware from_pretrained (NF4 on the Qwen backbone, attn_implementation="sdpa", the NavDP head + embeddings left bf16) and answers a ZMQ REQ/REP + msgpack protocol (ping/reset/step/close); the look-down re-step is handled server-side. Refuses an all-zero depth frame. - discrete_action_to_twist(actions, *, forward_mps, turn_radps) -> (v, w, stop) (server) — pure VLN-CE discrete-action → base-twist mapping; unit-tested in tests/unit/test_internvla_n1_action_mapping.py. - main() -> int (both) — sidecar: run_sidecar(..., family="internvla_n1"); server: argparse (--model/--host/--port/--quantization/--resize/--num-history/--plan-step-gap/--forward-mps/--turn-radps/--work-dir) + the ZMQ serve loop.

`python/sim/src/openral_sim/policies/init.py`

_register_policies() -> None — Side-effect imports of the policy-adapter modules so each registers its factory in openral_sim.POLICIES at import time. (L15)

`python/sim/src/openral_sim/backends/init.py`

_register_backends() -> None — Side-effect imports of the scene-backend modules so each registers its factory in openral_sim.SCENES at import time. (L61)

Eval (sim)

python/sim/src/openral_sim/policy.py

python/sim/src/openral_sim/rollout.py

python/sim/src/openral_sim/registry.py

python/sim/src/openral_sim/factory.py

python/sim/src/openral_sim/sim_runner.py

python/sim/src/openral_sim/benchmark.py

python/sim/src/openral_sim/cli.py

python/sim/src/openral_sim/_video.py

python/sim/src/openral_sim/_website_video.py

Eval adapters

python/sim/src/openral_sim/backends/robocasa.py

python/sim/src/openral_sim/backends/depth_camera.py

python/sim/src/openral_sim/backends/libero.py

python/sim/src/openral_sim/backends/vlabench.py

python/sim/src/openral_sim/policy_deps.py

python/sim/src/openral_sim/backends/metaworld.py

python/sim/src/openral_sim/backends/maniskill3.py

python/sim/src/openral_sim/backends/simpler_env.py

python/sim/src/openral_sim/sidecar.py

python/sim/src/openral_sim/backends/isaac_sim.py

tools/isaac_sidecar.py + tools/_isaac_scene_base.py + tools/isaac_scene.py + tools/isaac_bowl_plate_scene.py + tools/isaac_manifest_scene.py

python/sim/src/openral_sim/backends/robotwin.py

tools/robotwin_sidecar.py

python/sim/src/openral_sim/backends/rlbench.py

tools/rlbench_sidecar.py

python/sim/src/openral_sim/backends/aloha.py

python/sim/src/openral_sim/backends/pusht.py

python/sim/src/openral_sim/backends/so100_robosuite/

python/sim/src/openral_sim/backends/openarm_robosuite/

python/sim/src/openral_sim/backends/so101_box/

python/sim/src/openral_sim/backends/_so_arm_units.py

python/sim/src/openral_sim/backends/tabletop_push/

python/sim/src/openral_sim/policies/mock.py

python/sim/src/openral_sim/policies/smolvla.py

python/sim/src/openral_sim/policies/openvla.py

python/sim/src/openral_sim/policies/xr1.py

python/sim/src/openral_sim/policies/act.py

python/sim/src/openral_sim/_quantization.py

python/sim/src/openral_sim/policies/_policy_loading.py

python/sim/src/openral_sim/policies/pi05.py

python/sim/src/openral_sim/policies/molmoact2.py

python/sim/src/openral_sim/policies/_processors.py

python/sim/src/openral_sim/policies/rldx.py

python/sim/src/openral_sim/policies/gr00t.py

python/sim/src/openral_sim/policies/behavior_groot.py

python/sim/src/openral_sim/backends/behavior.py

python/sim/src/openral_sim/policies/rlbench_3dda.py

tools/rlbench_3dda_sidecar.py

python/sim/src/openral_sim/policies/lingbot_vla2.py

python/sim/src/openral_sim/policies/lingbot_va_a1.py

tools/lingbot_vla2_sidecar.py + tools/_lingbot_vla2_server.py

python/sim/src/openral_sim/_sidecar_common.py

python/sim/src/openral_sim/policies/internvla_n1.py

python/sim/src/openral_sim/da3_depth.py

tools/internvla_n1_sidecar.py + tools/_internvla_n1_server.py

python/sim/src/openral_sim/policies/__init__.py

python/sim/src/openral_sim/backends/__init__.py