Skip to content

Eval (sim)

Part of the OpenRAL public-symbol inventory. Hand-curated; (LNN) markers are refreshed by tools/refresh_methods_linenos.py.

python/sim/src/openral_sim/policy.py

Policy adapter protocol — the contract every VLA backend must satisfy.

  • class PolicyAdapter(Protocol) — Uniform VLA / policy interface. (L25)
  • attr spec: VLASpec, device: str
  • reset() -> None — Reset action queue / RNG at episode start. (L36)
  • step(observation, instruction) -> NDArray[np.float32] — Next action. (L39)
  • close() -> None — Release GPU / file handles. (L57)

python/sim/src/openral_sim/rollout.py

Sim rollout protocol — the typed contract every scene adapter must satisfy.

  • class StepResult — One environment transition. (L67) fields: observation, reward, terminated, truncated, info
  • class SimRollout(Protocol) — Minimal gym-style env contract. (L86)
  • attr scene: SceneSpec, task: TaskSpec
  • reset(seed=None) -> Observation (L153)
  • step(action) -> StepResult (L156)
  • render() -> NDArray[np.uint8] | None — HWC uint8 RGB or None. (L159)
  • close() -> None (L162)
  • duck-typed extension: mujoco_handles() -> tuple[mujoco.MjModel, mujoco.MjData] | None — Optional, NOT part of the Protocol; MuJoCo-backed adapters implement it so openral sim run --view can open a passive viewer. Callers MUST getattr(env, "mujoco_handles", None) and tolerate None.
  • duck-typed extension: sim_time_ns() -> int | None — Optional, NOT part of the Protocol (ADR-0048 Phase 1); the backend's authoritative elapsed sim time in ns, the seam a sim /clock publisher reads. MuJoCo-backed adapters return round(MjData.time * 1e9) via sim_time_ns_from_mujoco_handles. Monotonic non-decreasing within an episode; backends that rewind MjData.time on reset (robocasa) restart it, so a cross-reset-monotonic consumer maintains its own offset (SimAttachedHAL.sim_time_ns). None = no sim clock (PushT, Isaac Sim sidecar). Callers MUST getattr(env, "sim_time_ns", None) and treat both missing + None as "no clock" (fall back to wall time).
  • duck-typed extension: enable_intrinsic_viewer() -> None — Optional, NOT part of the Protocol; adapters whose engine draws its own window (e.g. gym_pusht) implement it so SimRunner can switch them into live-view mode at activate() time. When present, SimRunner skips the MuJoCo viewer path entirely.
  • sim_time_ns_from_mujoco_handles(handles: tuple[Any, Any] | None) -> int | None — Shared helper (ADR-0048 Phase 1): round(MjData.time * 1e9) from a mujoco_handles() (model, data) tuple, None when handles is None. The single place the MuJoCo-backed adapters' sim_time_ns() implementations route through. (L21)
  • class EpisodeResult — Outcome of one episode. (L167) fields: success, steps, total_reward, mean_step_latency_ms, max_step_latency_ms, latency_budget_ms, budget_violations, frames, metadata
  • summary() -> str — Human-readable single line. (L220)

python/sim/src/openral_sim/registry.py

Registries that map ID strings to backend factories.

  • class _Registry(Generic[T]) — Tiny ID → factory map. (L43)
  • __init__(kind) (L54)
  • kind -> str [@property] (L60)
  • register(name, *, fixed_robot=None) -> Callable[[Callable[..., T]], Callable[..., T]] — Decorator. The optional fixed_robot kwarg (only meaningful on SCENES) declares which robot_id the scene's physics backend hard-wires; the CLI rejects mismatched --robot values with ROSConfigError instead of silently swapping the robot. (L63)
  • get(name) -> Callable[..., T] — Look up by ID. (L110)
  • fixed_robot(name) -> str | None — Scene's hard-fixed robot id (None for free-axis scenes or unregistered names). (L102)
  • names() -> list[str] — Sorted IDs. (L125)
  • __contains__(name) -> bool (L129)
  • module-level globals: SCENES, POLICIES, ROBOTS — three _Registry[T] singletons.

python/sim/src/openral_sim/factory.py

  • make_env(env_cfg) -> SimRollout — Build the simulated environment. (L25)
  • make_policy(env_cfg) -> PolicyAdapter — Build the policy. (L43)
  • make_robot(env_cfg) -> RobotDescription | None — Resolve robot description if registered. (L61)

python/sim/src/openral_sim/sim_runner.py

Per-step InferenceRunner for the simulation runtime (ADR-0010 amendment 1).

  • class SimRunner(InferenceRunnerBase) — One-tick = one-env-step inference runner that drives a SimEnvironment for n_episodes episodes. Subclasses InferenceRunnerBase so sim and hardware (HardwareRunner) share the InferenceRunner Protocol. (L188)
  • SimRunner.__init__(env_cfg, *, view=False, strict_view=False, instruction_override=None, deadline_overrun_policy=WARN, recorder=None) — Defer env / policy construction to activate(); rate_hz is fixed at 1000 Hz (sim is not real-time, deadline policy defaults to WARN). instruction_override is the explicit --instruction CLI value (or None) that wins over a scene's per-episode obs["task"] language via the private _resolve_step_instruction helper. recorder (ADR-0019) is an optional openral_dataset.RolloutRecorder fanned out alongside _EpisodeBuffer — additive, never a substitute. (L232)
  • SimRunner._record_to_recorder(action, reward, terminated, truncated) -> None — Internal helper that extracts per-step state / rendered frame / action and forwards to the attached recorder; broadcasts the single rendered viewpoint to every camera key declared on the robot (sim envs typically expose one render but multiple vla_feature_keys). Errors are logged, not raised. (L380-ish, ADR-0019)
  • SimRunner.activate() -> None — Validate manifest via _check_rskill_compatibility, build env + policy concurrently via _build_env_and_policy (GH-134: make_env + make_policy run on a 2-worker ThreadPoolExecutor by default), arm the first reset-tick, open the outer sim.run OTel span. (L307)
  • SimRunner.deactivate() -> None — Flush a trailing episode if any, close the viewer / policy / env, close the outer span. Idempotent. (L392)
  • SimRunner._should_terminate() -> bool — Returns True once n_episodes EpisodeResults have been emitted. (L429)
  • SimRunner._tick_impl(tick_idx) -> TickResult — Dispatch reset-tick vs step-tick. (L477)
  • SimRunner._reset_tick(tick_idx) -> TickResult — env.reset + policy.reset; action_applied=False, inference_ms=0.0, step_idx=None. (L487)
  • SimRunner._step_tick(tick_idx) -> TickResult — policy.step + env.step; populates step_idx, reward, terminated, truncated, action_applied=True. (L537)
  • SimRunner._finalize_episode() -> None — Build an EpisodeResult from the per-step _EpisodeBuffer, append to episode_results, reset the buffer. (L732)
  • class _EpisodeBuffer — Private dataclass accumulating per-step latencies / frames / rewards inside one episode; reset on each boundary. (L163)
  • _check_rskill_compatibility(env_cfg) -> RSkillManifest | None — Load the rSkill manifest, run rSkill.check_compatibility against the registered RobotDescription, return the manifest or None for built-in mock policies. Strict-by-construction. (L848)
  • _SEQUENTIAL_INIT_ENV: str = "OPENRAL_SIM_SEQUENTIAL_INIT" — Module-level constant: the env var that forces _build_env_and_policy onto the legacy sequential path (set to "1"). (L947)
  • _build_env_and_policy(env_cfg) -> (SimRollout, PolicyAdapter) — GH-134: build env + policy concurrently on a 2-worker ThreadPoolExecutor by default; sequential when OPENRAL_SIM_SEQUENTIAL_INIT=1. Logs structured sim_init_parallel / sim_init_sequential records with env_ms / policy_ms / total_ms / saved_ms. Exceptions from either side propagate verbatim — the helper does not catch ROSError (or anything else). (L1028)
  • _seed_global_rngs(seed) -> None — Seed Python / NumPy / Torch RNGs so stochastic policies reproduce per (seed + episode_idx). (L1139)
  • _open_viewer_and_pacing(env, env_cfg, *, strict_view) -> (Any, float | None) — Open a passive mujoco.viewer against the adapter's mujoco_handles() with show_left_ui=False, show_right_ui=False (only the sim renders), set the camera + geom visibility via _aim_viewer_camera, and compute the per-step sleep budget so the viewer renders at the env's natural sim-time. (L1188)
  • _aim_viewer_camera(viewer, env, mj_model, mj_data) -> None — Set the viewer's opening camera + geom visibility (lazily imports openral_hal.depth_cloud.{apply_robosuite_visual_geomgroups, initial_viewer_camera} — paying openral_hal's torch/lerobot import cost only at interactive viewer-open, mirroring openarm_robosuite/_assets.py): hides robosuite collision shells so textures render, then sets the free-camera opening pose via initial_viewer_camera (eye at a 3rd-person scene camera, orbit pivot on the base; base-aligned default for camera-less models). Camera stays mjCAMERA_FREE so the user can orbit (drag) / zoom (scroll); only the initial view is set. Best effort — any failure logs viewer_camera_aim_failed and leaves MuJoCo's default camera. (L1250)

python/sim/src/openral_sim/benchmark.py

Benchmark runner — loops a bare list[BenchmarkScene] (loaded via load_benchmark_suite + raise_on_invalid_suite) and emits a RSkillEvalResult (ADR-0009 PR D + ADR-0042).

  • run_benchmark(scenes, *, suite_id, vla, device=None, save_dir=None) -> tuple[RSkillEvalResult, list[EpisodeResult]] — Iterate scenes × range(seed, seed + n_episodes), drive each (BenchmarkScene, seed) tuple with a fresh SimRunner (ADR-0010 amendment 1 / ADR-0041 Task 10 / ADR-0042), aggregate into a validated RSkillEvalResult. Per-scene robot_id / task / max_steps pulled from each BenchmarkScene; suite-level invariants pre-checked by raise_on_invalid_suite (the runner does not re-validate). All args after scenes are keyword-only — callers must name suite_id and vla. (L65)
  • _aggregate_results(scenes, *, suite_id, vla, per_task, episodes) -> RSkillEvalResult — Roll per-task booleans into per-task / avg success rates. Suite-level benchmark.name / benchmark.simulator come from scenes[0].metadata.display_name / .simulator when present (ADR-0042), else fall back to suite_id / scenes[0].scene.id. benchmark.arxiv auto-derived from scenes[0].metadata.paper when the URL contains arxiv.org/. max_steps in the protocol summary is max(scene.task.max_steps for scene in scenes) so the bound is the suite worst-case, not just scenes[0]. Pulled out for unit-test reuse. (L180)
  • run_benchmark_scene(scene, vla, *, device=None, save_dir=None, config_path=None, view=None) -> tuple[RSkillEvalResult, list[EpisodeResult]] — Single-scene sibling of run_benchmark; backs openral benchmark scene. Iterates range(scene.seed, scene.seed + scene.n_episodes) against the one (scene, task) pair carried by a BenchmarkScene and emits the same RSkillEvalResult shape so openral benchmark report does not need to distinguish entrypoints. Raises ROSConfigError when scene.robot_id is None. view (tri-state, default None) is the opt-in viewer flag for parity with sim run: None keeps the historical headless behaviour (eval/CI unaffected), an explicit True/False is resolved through cli._resolve_view and passed to SimRunner. (L294)
  • _aggregate_scene_results(scene, vla, successes, episodes, config_path) -> RSkillEvalResult — Single-scene counterpart of _aggregate_results; shares the output schema. PushT special-case mirrors the suite path. Embeds config_path into reproduction_cli for byte-identical reruns from disk. (L415)
  • default_output_path(weights_uri, benchmark_id) -> str — Canonical mapping rskills/<dir> (or bare name) → rskills/<dir>/eval/<id>.json. (L506)
  • update_rskill_benchmarks(skill_dir, benchmark_id, score) -> Path — Surgical rewrite of the benchmarks: block in <skill_dir>/rskill.yaml that preserves every other comment + line; re-validates the merged manifest through RSkillManifest before writing. Closes the openral benchmark run → rskill.yaml loop so manifest headlines stay in sync with the eval JSONs. Raises FileNotFoundError if no manifest, ROSConfigError on unknown benchmark_id / out-of-range score. (L549)
  • update_rskill_benchmarks_from_uri(weights_uri, benchmark_id, score) -> Path — Resolve the skill reference to a local dir and delegate to the manifest updater; mirrors default_output_path so the CLI passes the same ref it already holds. (L649)

python/sim/src/openral_sim/cli.py

  • sim_app: typer.Typer — Public openral sim Typer group (ADR-0009 PR C). Mounted into the top-level openral Typer tree by openral_cli.main. Hosts the run leaf (--config / --robot / --scene / --task / --rskill / …) and the list leaf (registry printer).
  • sim_run_app: typer.Typer — The leaf Typer (invoke_without_command=True) exposing every rollout CLI flag; users invoke it as openral sim run.
  • _sim_run_callback(...) — Typer callback carrying every rollout CLI flag; builds a SimpleNamespace and dispatches to _run. ADR-0009 PR C. The optional --dashboard / --dashboard-port flags wrap _run in attached_dashboard(...). Same flag is mirrored on openral deploy run and openral benchmark run.
  • _discover_sim_configs() -> list[Path] — Recursive read-only walk of scenes/**/*.yaml (benchmark / sim / deploy) under the repo root, sorted by relative path. Safe to call without any sim dependencies. (L441)
  • sim_list() -> None — @sim_app.command("list") callback that prints every sim config under scenes/**/*.yaml, each a paste-able --config path for openral sim run. No rollout, no OTel span, no GPU. (L462)
  • _resolve_save_video(raw) -> Path | None — Map the --save-video Typer string to the legacy Path | None semantics (empty string ⇒ example_videos/). (L258)
  • _load_or_build_env(args) -> SimEnvironment — --config XOR explicit flags; --config combined with any of --task / --robot / --scene / --rskill / --instruction raises ROSConfigError. Also enforces the scene-fixed-robot guard: when SCENES.fixed_robot(scene.id) is set and disagrees with env.robot_id, raises ROSConfigError naming the scene's required robot. (L274)
  • _resolve_view(flag) -> tuple[bool, bool] — tri-state resolver returning (view, strict_view) from the --view/--no-view/auto flag plus MUJOCO_GL / DISPLAY env. (L480)
  • main(argv=None) -> int — Thin wrapper that invokes sim_run_app with standalone_mode=False so tests can get the return code without sys.exit. The legacy standalone console script was removed in 2026-05; main stays for internal callers. (L409)
  • _run(args) -> int — Body of the callback after argv parsing + OTel setup. Configures observability with service name ral-sim (ADR-0009 PR C). (L669)
  • _write_videos(args, results, env_cfg) -> None — Render the 3-panel debug MP4(s) via openral_sim._video.save_episode_mp4. (L756)

python/sim/src/openral_sim/_video.py

Shared 3-panel rollout-debug MP4 helper (was examples/_video.py).

  • save_episode_mp4(result: EpisodeResult, path: Path, *, title: str = "") -> Path — Render (vla input | env render | joint plot) panels for one episode. Re-exported from openral_sim. (L63)
  • _stack_padded_states(states) -> NDArray[np.float32] — Pad ragged joint-position arrays for plotting. (L165)
  • _resize_sequence(frames, target) -> list[NDArray[np.uint8]] (L180)
  • _resize_frame(frame, target) -> NDArray[np.uint8] (L204)
  • class _JointPlotRenderer — Reusable matplotlib canvas rasteriser. (L224)
  • __init__, render_at_step, _snapshot, __del__

Eval adapters

python/sim/src/openral_sim/backends/robocasa.py

RoboCasa kitchen + GR1 tabletop adapter. ADR-0011 / ADR-0015. - read_panda_mobile_base_velocity(model, data) -> NDArray[np.float32] — Returns body-frame (vx, vy, wz) 3-vec for the robosuite OmronMobileBase; reads data.qvel at the three planar joint addresses and de-rotates the world-frame (vx, vy) using the live yaw. Returns zeros(3) when the base joints aren't in this model (silently no-ops for non-PandaMobile envs). (L949) - synthesize_laser_scan_2d(*, model, data, base_body_id=None, n_beams=360, max_range_m=12.0, laser_height_m=0.30) -> NDArray[np.float32] — Single-origin batched mj_multiRay 2D laser fan from the panda_mobile base. Returns (n_beams,) float32 ranges in metres, clamped to max_range_m for "no hit" beams (NEVER NaN/inf, so Nav2 costmap consumers don't poison the grid). Self-exclusion via bodyexclude=mj_name2id("base") so the chassis doesn't pollute the scan. (L1022) - _emit_panda_mobile_extras(obs) (method on _RoboCasaSim) — Attaches obs["robot0_base_vel"] + obs["robot0_scan"] when "PandaMobile" in self._robots; no-op for other compositions. (L569 in _wrap_obs) - sim_time_ns() -> int | None (method on _RoboCasaSim) — round(MjData.time * 1e9) off mujoco_handles() (ADR-0048 Phase 1); covers the robocasa kitchen / GR1 / so100_robosuite scenes. RoboCasa rewinds the clock on reset, so it is monotonic only within an episode — SimAttachedHAL.sim_time_ns adds the cross-reset offset. (L504) - Constants _OMRON_BASE_JOINT_NAMES, _OMRON_BASE_JOINT_NAMES_FALLBACK, _LASER_DEFAULT_N_BEAMS=360, _LASER_DEFAULT_MAX_RANGE_M=12.0. (L880)

python/sim/src/openral_sim/backends/depth_camera.py

ADR-0030 — simulated depth camera via MuJoCo CPU ray-casting (the 3-D analogue of synthesize_laser_scan_2d); robot-agnostic, no GL/EGL context. Feeds the deploy-sim HAL → octomap_server → the kernel world-collision voxel check. - synthesize_depth_pointcloud(*, model, data, camera_name, width, height, fx, fy, cx, cy, max_range_m, min_range_m=0.0, stride=1, exclude_body_id=None, exclude_body_ids=None) -> NDArray[np.float32] — One mj_multiRay ray per (strided) pixel through a pinhole model anchored on the named MJCF camera's live world pose. Returns (N, 3) float32 hit points in the camera optical frame (REP-103: +x right, +y down, +z forward), filtered to [min_range_m, max_range_m]; empty (0, 3) when nothing is in range. exclude_body_id is mj_multiRay's single bodyexclude; exclude_body_ids (a frozenset[int]) drops hits on the robot's own bodies after casting — the self-filter that keeps a base-mounted camera from voxelising the arm into its own world map (else the kernel flags the arm against itself). Raises ROSConfigError if camera_name is absent. (L35)

python/sim/src/openral_sim/backends/libero.py

  • class _LiberoSim — SimRollout wrapping LiberoEnv. (L74) — reset/step/render/close/action_dim/mujoco_handles/sim_time_ns/_wrap_obs. mujoco_handles() reaches through robosuite's env.sim.{model,data}._{model,data} for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9) (ADR-0048 Phase 1). action_dim walks the LiberoEnv→OffScreenRenderEnv wrapper chain to sum robosuite robots[*].action_dim (LIBERO OSC_POSE = 7) so SimAttachedHAL can size cartesian actions on the openral deploy sim suite-scene path.
  • _parse_task_id(task_id, scene_id) -> int — Validate <suite>/<int> format. (L43)
  • _quat_to_axisangle(quat) -> NDArray[np.float32] — [x,y,z,w] → axis-angle. (L213)
  • _build_libero_scene(env_cfg) -> _LiberoSim (L226)

python/sim/src/openral_sim/backends/libero_custom_bddl.py

Robosuite-backed custom-scene adapter — drives pi0.5-LIBERO against any user-authored BDDL file. Delegates control (OSC_POSE) and rendering to libero.libero.envs.env_wrapper.OffScreenRenderEnv so the policy stays inside its training distribution; the adapter only bridges the OpenRAL SimRollout Protocol. Scene id franka_libero_custom_bddl. - _CUSTOM_SCENE_ID = "franka_libero_custom_bddl" — module constant; scene-registry key. (L72) - _PI05_STATE_DIM = 8 — lerobot-style state vector length: [eef_pos(3), eef_axisangle(3), gripper_qpos(2)]. (L73) - _quat_to_axisangle_xyzw(quat_xyzw) -> NDArray[np.float32] — robosuite returns robot0_eef_quat in xyzw order; mirrors lerobot.processor.env_processor.LiberoEnvProcessorStep._quat2axisangle. (L120) - class _LiberoCustomBDDLSim — SimRollout wrapping OffScreenRenderEnv. (L137) — reset/step/render/close/mujoco_handles/sim_time_ns/_wrap_obs. mujoco_handles() reaches through OffScreenRenderEnv.env.sim.{model,data}._{model,data} for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9) (ADR-0048 Phase 1). - reset(seed) -> Observation — Optionally applies a pickled .pruned_init state at row init_state_index. (L153) - step(action) -> StepResult — Forwards 7-D action to robosuite's OSC_POSE controller. (L167) - _wrap_obs(raw) -> Observation — Builds the 8-D state from robot0_eef_pos/quat + robot0_gripper_qpos. (L226) - _build_libero_custom_bddl_scene(env_cfg) -> _LiberoCustomBDDLSim — Reads scene.backend_options.bddl_file (required) and init_state_file / init_state_index (optional); calls openral_sim._deps.ensure_backend_deps("libero") so the user gets the same interactive auto-install banner as the LIBERO benchmark backend (instead of a raw "run uv sync --group libero" hint); then instantiates OffScreenRenderEnv. Raises ROSConfigError when paths are missing, the user declines the install prompt, or the post-install import still fails. (L266) - Module side effect: SCENES.register("franka_libero_custom_bddl")(_build_libero_custom_bddl_scene) at import (L266).

python/sim/src/openral_sim/backends/metaworld.py

MetaWorld MT-50 scene adapter. Opt-in via the metaworld dependency group + a metaworld==3.0.0 --no-deps pip install (its transitive deps conflict with the workspace lock); the scene factory calls openral_sim._deps.ensure_backend_deps("metaworld") first so the user gets an interactive auto-install banner on first use. Scene id metaworld. Task id metaworld/<task-name> (e.g. metaworld/reach-v3). - class _MetaworldSim — SimRollout wrapping MetaworldEnv. (L46) — reset/step/render/close/mujoco_handles/sim_time_ns/_wrap_obs. mujoco_handles() reaches through unwrapped.{model,data} for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9) (ADR-0048 Phase 1). - _parse_task_id(task_id) -> str (L36) - _build_metaworld_scene(env_cfg) -> _MetaworldSim (L132)

python/sim/src/openral_sim/backends/maniskill3.py

ManiSkill3 (SAPIEN-backed) free-axis scene adapter. ADR-0014. Opt-in via the maniskill3 dependency group; the scene factory calls openral_sim._deps.ensure_backend_deps("maniskill3") first so the user gets an interactive auto-install banner on first use (bypass with OPENRAL_AUTO_INSTALL_DEPS=1). Scene id maniskill3. Task id maniskill3/<env_id> (e.g. maniskill3/PickCube-v1). - _MANISKILL3_SCENE_ID = "maniskill3" — module constant; scene-registry key. (L37) - class _ManiSkill3Sim — SimRollout wrapping a MS3 gym env with num_envs=1; unwraps the leading batch dim on every obs / step. (L56) — reset/step/render/close/_wrap_obs. - _parse_task_id(task_id) -> str — Validates maniskill3/<env_id> and returns <env_id>. (L46) - _unbatch(value), _unbatch_info(info), _unbatch_obs(obs) — recursive numpy / torch unbatch helpers shared with the SimplerEnv adapter. (L106 / L114 / L130) - _extract_rgb(flat) — Returns the first MS3 sensor_data.<camera>.rgb stream as NDArray[uint8]. (L188) - _extract_state(flat) — Concatenates agent.qpos + agent.qvel into a 1-D float32 vector (returns 0-D when the obs mode doesn't expose the nested agent block). (L223) - _build_maniskill3_scene(env_cfg) -> _ManiSkill3Sim — gym.make with obs_mode / control_mode overridable via scene.backend_options; default state_dict+rgb + pd_ee_delta_pose. (L239) - Module side effect: SCENES.register("maniskill3")(_build_maniskill3_scene) at import (L239).

python/sim/src/openral_sim/backends/simpler_env.py

SimplerEnv real-to-sim correlator adapter. ADR-0014. Opt-in via the simpler-env dependency group (the package has no PyPI release; install hint in the typed ROSConfigError). Reuses the obs-extraction helpers from backends/maniskill3 because SimplerEnv now sits on top of MS3 v3.0.x. Scene id simpler_env. Task id simpler_env/<friendly_name> (e.g. simpler_env/widowx_carrot_on_plate); friendly names are translated via simpler_env.ENVIRONMENT_MAP to the underlying MS3 env id + kwargs. Today only the four WidowX bridge tasks are wired end-to-end against MS3 v3.0.x; google_robot_* friendly names resolve to env ids that are not yet registered upstream. - _SIMPLER_ENV_SCENE_ID = "simpler_env" — module constant; scene-registry key. (L68) - _DEFAULT_OBS_MODE = "rgb+segmentation" — Only obs mode the MS3 v3.0.x Bridge envs advertise; overridable via scene.backend_options.obs_mode. (L75) - class _SimplerEnvSim — SimRollout wrapping a SimplerEnv-via-MS3 gym env. (L246) — reset/step/render/close/_wrap_obs. Reshapes the single-env action to (1, action_dim) to satisfy MS3's batched API. - _parse_task_id(task_id) -> str — Validates simpler_env/<friendly_name> and returns <friendly_name>. (L78) - _bump_version_if_deprecated(env_id) -> str — Rounds an upstream -v0 env id up to the highest registered -v* suffix; upstream simpler_env.ENVIRONMENT_MAP still ships -v0 but MS3 v3.0.x registers -v1. (L87) - _resolve_friendly_name(task_name) -> tuple[str, dict[str, Any]] — Translates a SimplerEnv friendly task name into (ms3_env_id, kwargs). Falls back to passing the input through unchanged so users can author configs against raw MS3 env ids. (L110) - _build_simpler_env_scene(env_cfg) -> _SimplerEnvSim — Calls gym.make directly (bypassing the broken upstream simpler_env.make() which still passes prepackaged_config=True / obs_mode='rgbd' that MS3 v3.0.x rejects). (L339) - Module side effect: SCENES.register("simpler_env")(_build_simpler_env_scene) at import (L339).

python/sim/src/openral_sim/sidecar.py

Canonical openral-side out-of-process sidecar transport (ADR-0045) — ZMQ REQ/REP + a numpy-aware msgpack codec. Shared by new sidecar integrations (the Isaac Sim backend); the RLDX-1 adapter predates it and keeps its own wire-locked copy (its codec must match the upstream __ndarray_class__ sentinel and its real path is un-runnable in CI). - encode_ndarray(obj) -> Any / decode_ndarray(obj) -> Any — msgpack default / object_hook codec (np.save into a {"__ndarray__": True, "npy": bytes} sentinel; decode returns a sentinel missing npy unchanged rather than raising KeyError). - require_key(reply, key, *, name) -> Any — typed-ROSRuntimeError guard for a reply missing a required key. - class SidecarClient — owns the ZMQ REQ socket + the optional child Popen. connect (ping existing → else spawn + boot-poll, ROSConfigError on failure), call(endpoint, data) (ROSRuntimeError on a sidecar-side fault / non-dict reply), require, close; boot helpers _try_ping/_spawn/_wait_for_boot/_terminate_child/_is_port_busy; recreates the REQ socket on timeout to clear the EFSM lock. name parametrizes log/error text. _spawn strips PYTHONPATH/VIRTUAL_ENV from the child env so the parent's (different-interpreter) site-packages don't shadow the sidecar venv's numpy (openral deploy sim injects the py3.12 site onto PYTHONPATH; the py3.11 Isaac sidecar must use its own).

python/sim/src/openral_sim/backends/isaac_sim.py

NVIDIA Isaac Sim (Omniverse + PhysX + RTX) free-axis scene adapter. ADR-0045. Drives an Isaac Sim env that runs in a separate py3.11 sidecar venv (Isaac Sim ships per-interpreter wheels; the openral workspace is py3.12), over the shared openral_sim.sidecar.SidecarClient. Opt-in via the isaacsim dependency group (pyzmq + msgpack on the openral side only); the heavy isaacsim/isaaclab install is an externally-provisioned sidecar venv (Omniverse Kit is proprietary, never vendored — CLAUDE.md §1.9). The factory calls ensure_backend_deps("isaac_client"), resolves the sidecar interpreter (OPENRAL_ISAAC_SIDECAR_PYTHON) + script (tools/isaac_sidecar.py), and auto-spawns the sidecar on first use. Scene id isaac_sim. Task id isaac_sim/<name>. - _ISAAC_SCENE_ID = "isaac_sim" — module constant; scene-registry key. - class _IsaacSimSidecar — SimRollout proxying reset/step/render/close to a SidecarClient; unwraps the eval-shaped Observation (images/state/task) via client.require(...) and caches the last RGB frame. The action_dim property (read from the sidecar ping, cached) lets openral deploy sim wrap it in SimAttachedHAL (_probe_env_action_dim); deploy scene scenes/deploy/isaac_franka.yaml (taskless DeployScene, lift_cube). Minimal bring-up — /joint_states is zeros for a non-MuJoCo backend (ADR-0045 follow-up note). - _opt_num(opts, key, default, cast) -> int|float — coerce a scene.backend_options value (typed object) via cast, ignoring bool and swallowing ValueError/TypeError (returns default, never raises). - _sidecar_python() -> Path / _locate_sidecar_script() -> Path — resolve the py3.11 interpreter (env override → cache default → typed ROSConfigError with provisioning hint) and tools/isaac_sidecar.py (env override → walk-up). - _sensor_dict(sensor) -> dict / _build_robot_spec(desc, robot_id) -> dict / _write_robot_spec(env_cfg) -> str — robot-agnostic --layout manifest marshalling (ADR-0045 amendment). The py3.11 sidecar cannot import openral_core, so _build_robot_spec serialises the RobotDescription to plain JSON — the urdf_path wire field resolved from assets.urdf.ref to a file (openral_core.assets.resolve_asset), ALL non-fixed joints in manifest order with normalised role (base for base_joints, gripper, else arm), the action contract (arm_n + gripper + 3·base-twist; a normalised [0,1] gripper limit falls back to the Panda 0.04 m so it never tears the Isaac finger DOF), and the sensors — and _write_robot_spec writes it to a temp file passed via --robot-spec. - _build_isaac_sim_scene(env_cfg) -> _IsaacSimSidecar — factory: builds the launch argv (incl. --layout from backend_options.layout); for layout == "manifest" writes the robot spec and appends --robot-spec, unlinking it after connect() (the sidecar consumes it at boot). Connects a SidecarClient(name="isaac", …). - Module side effect: SCENES.register("isaac_sim", fixed_robot=None)(_build_isaac_sim_scene) at import.

tools/isaac_sidecar.py + tools/_isaac_scene_base.py + tools/isaac_scene.py + tools/isaac_bowl_plate_scene.py + tools/isaac_manifest_scene.py

Isaac-side sidecar (runs under the py3.11 Isaac Sim venv only; ADR-0045). isaac_sidecar.py launches the headless Omniverse Kit SimulationApp (sets OMNI_KIT_ACCEPT_EULA=YES), then serves a ZMQ REP loop (ping/reset/step/render/close) speaking the same msgpack+ndarray framing as the openral side. _isaac_scene_base.IsaacSceneBase owns the shared lifecycle (reset warmup, step physics-substep loop, _observe assembly, _grab RGBA→HWC); subclasses override build/_apply_action/_images/_state/_reward_terminated (+ _on_reset/_extra_info/_joint_positions). _isaac_scene_base.franka_joint_positions(franka) maps the Isaac Franka's 9 DOF to the manifest's 8 joints (7 arm + mean-finger gripper); both scenes return it from _joint_positions() so obs["joint_positions"] feeds openral deploy sim's SimAttachedHAL.read_state real /joint_states (ADR-0034 amendment — non-MuJoCo backends). The --layout arg picks the scene class: - lift_cube (isaac_scene.IsaacLiftScene) — World + Franka + DynamicCuboid + Camera; 8-D joint-delta action, cube-height reward. The cube-lift PoC. - bowl_plate (isaac_bowl_plate_scene.IsaacBowlPlateScene) — table + YCB 024_bowl USD + thin-cylinder plate + Franka + agent-view & eye-in-hand cameras, mirroring the LIBERO contract (camera1/camera2 + 8-D [eef_pos‖axisangle‖gripper_qpos] state, 7-D OSC-pose-delta action). End-effector control uses the core isaacsim.robot_motion.motion_generation Lula kinematics solver (LulaKinematicsSolver + ArticulationKinematicsSolver on the right_gripper frame) for position-delta IK — no Isaac Lab and no Isaac Lab OSC term required. Drives act-libero / smolvla-libero through openral sim run (verified e2e; success is OOD, the check is pipeline + arm motion). Scene scenes/sim/isaac_franka_bowl_plate.yaml. - manifest (isaac_manifest_scene.IsaacManifestScene) — robot-agnostic, URDF-driven scene (ADR-0045 amendment). Instead of a hardcoded Isaac Franka asset it imports the manifest robot's URDF via omni.kit.commands URDFCreateImportConfig → URDFParseAndImportFile (Isaac's isaacsim.asset.importer.urdf), wraps it as isaacsim.core.api.robots.Robot, and drives a JOINT_POSITION-delta articulation controller. Action layout [arm deltas, gripper, base twist]. map_dof_to_manifest(values, *, dof_index, manifest_joints, finger_dof_idx, base_values=None, base_joints=None) (module-level) maps the articulation DOF vector to the full manifest joint order: base joints ← the kinematic base pose, arm joints ← URDF DOF by name, the two-finger→one-gripper collapse ← mean of the finger DOFs, else 0.0 (generic replacement for franka_joint_positions). Kinematic holonomic base (M3): a robot with base_joints is imported fix_base=True and _integrate_base(vx, vy, wyaw) teleports the whole articulation root each step from a base-frame-twist-integrated (x, y, yaw) — real base motion + a base_pose for /odom, no PhysX base joints (the base exists nowhere as an Isaac asset; robosuite composes it in MuJoCo only). Built from the --robot-spec JSON. Verified live: scenes/deploy/isaac_franka_urdf.yaml (tests/sim/test_franka_urdf_isaac.py — franka imports from URDF, /joint_states carries the imported pose, JOINT_POSITION drives the arm) and scenes/deploy/isaac_panda_mobile_urdf.yaml (tests/sim/test_panda_mobile_isaac.py — 11-D action, 11-joint /joint_states = 3 base + 7 arm + 1 gripper, forward base-twist moves the base). Manifest-driven sensors: _plan_cameras makes one base-relative Isaac Camera per RGB/depth SensorSpec (keyed by the RGB vla_feature_key suffix camera1… / the depth sensor name); _update_camera_poses rides them on the kinematic base; _images returns every RGB frame and _depth_clouds returns {sensor: (N,3) base_link} via Isaac's Camera.get_pointcloud(world_frame=True) (Isaac owns the camera convention) transformed world→base_link by the base pose → obs["depth_points"] (SimSensorBridge publishes them as PointCloud2). A modality the manifest does not declare is never created. Verified live: the deploy graph publishes /openral/cameras/front_depth/points (62 k pts, base_link) → octomap → /openral/world_voxels. 2-D lidar: _add_obstacles seeds a few static boxes and _scan_ranges casts a PhysX raycast_closest fan (each ray starting range_min_m past the base to clear the robot's own chassis; robot /panda hits ignored) → obs["scan"] → SimAttachedHAL.read_scan → /scan. The full slam-map + obstacle-aware Nav2 loop additionally needs the deploy-sim /clock publisher (ADR-0048, merged).

All three layouts use Isaac Sim core (not Isaac Lab's env machinery, which the PyPI isaaclab wheel does not ship). NOT imported by the openral venv — invoked as a subprocess; the openral-side backends/isaac_sim.py forwards scene.backend_options.layout (and, for manifest, the --robot-spec JSON).

python/sim/src/openral_sim/backends/aloha.py

gym-aloha bimanual MuJoCo scene adapter. Opt-in via the sim dependency group (gym-aloha lives there alongside mujoco / gymnasium / lerobot); the scene factory calls openral_sim._deps.ensure_backend_deps("aloha") first so the user gets an interactive auto-install banner on first use. - class _AlohaSim — SimRollout wrapping a gym_aloha env. — reset/step/render/close/mujoco_handles/sim_time_ns/_wrap_obs. mujoco_handles() reaches through env.unwrapped._env.physics.{model,data}.ptr (dm_control wrapper) for openral sim run --view. sim_time_ns() returns round(MjData.time * 1e9) (ADR-0048 Phase 1).

python/sim/src/openral_sim/backends/pusht.py

  • class _PushTSim — SimRollout wrapping gym_pusht/PushT-v0 (pymunk 2-D rigid body). — reset/step/render/close/enable_intrinsic_viewer/_wrap_obs/_paint_view. enable_intrinsic_viewer() opens a pygame window from inside the adapter and _paint_view() blits the last pixel frame on each reset / step, leaving the env in render_mode="rgb_array" so the Diffusion Policy still gets observation.image.

python/sim/src/openral_sim/backends/so100_robosuite/

robosuite integration for the Hugging Face SO-100 follower. NOT a SCENES.register(...) adapter (the SO-100 has no benchmarked VLA-driven suite yet); a standalone subpackage that registers the SO-100 with robosuite's robot / gripper factories and provides a runnable scripted-pick demo. Used by tests/sim/test_so100_robosuite_lift.py and examples/so100_robosuite_lift.py. - __init__.py — re-exports SO100, SO100Gripper, make_so100_lift_env, so100_osc_controller_config; importing the package side-effect-registers the robot + gripper. - _assets.py — ensure_so100_assets() -> SO100Assets lazily rewrites the DeepMind mujoco_menagerie trs_so_arm100 MJCF into two robosuite-compatible XMLs (arm with 5 motor actuators + base/right_hand body, gripper with the Jaw joint + eef body + finger pads), caches under $OPENRAL_CACHE_DIR/so100_robosuite/<menagerie-fingerprint>/. class SO100Assets(frozen dataclass) carries robot_xml, gripper_xml, menagerie_dir. The rewrite handles three robosuite quirks: nested <default> flattening (so _replace_defaults_inline resolves classes), absolute mesh paths (robosuite's resolve_asset_dependency ignores meshdir), and childclass stripping (robosuite drops the defaults block before MuJoCo compiles). - model.py: - class SO100(ManipulatorModel) — 5-DOF arm (Rotation / Pitch / Elbow / Wrist_Pitch / Wrist_Roll); default_base = "NullMount", default_gripper = {"right": "SO100Gripper"}, init_qpos matches the menagerie home keyframe. Registered in REGISTERED_ROBOTS and ROBOT_CLASS_MAPPING (as a FixedBaseRobot) at import. - class SO100Gripper(GripperModel) — 1-DOF Jaw with _important_geoms for left_fingerpad / right_fingerpad so robosuite's _check_grasp resolves cleanly. Registered in GRIPPER_MAPPING at import. - env.py: - class _So100Lift(Lift) — Lift with the SO-100 bolted onto the standard TableArena top, a small upright redwood block (1.2 cm half-edge × 4 cm tall) sized for the SO-100 jaw aperture, and a _check_success that scales with the block height (success when the bottom face clears the table by lift_height_m). - so100_osc_controller_config() -> dict[str, Any] — Loads robosuite's shipped parts/osc_position.json, narrows output_max from ±5 cm/step to ±1 cm/step (SO-100's small mass matrix would otherwise overshoot), bumps kp 150 → 1500 to match the 30-50× smaller mass-matrix entries, and pins input_ref_frame = "world" so the policy's world-frame Cartesian targets aren't re-rotated by the SO-100's 90°-z base orientation. - make_so100_lift_env(*, has_renderer, has_offscreen_renderer, use_camera_obs, camera_names, camera_heights, camera_widths, horizon, control_freq, table_full_size, cube_half_extent_m, cube_block_height_m, x_range, y_range, seed, lift_height_m, reward_shaping) -> _So100Lift — composes the registered robot + gripper + OSC_POSITION config; cube placement reference matches the Panda-default table_offset = (0, 0, 0.8) so the stock agentview / frontview cameras frame the scene correctly. - policy.py: - class PolicyTelemetry (dataclass) — Per-step diagnostics: phase / eef_to_cube_distance_m / gripper_command / cartesian_delta / cube_height_m. - class ScriptedPickPolicy (dataclass) — Four-phase Cartesian state machine (approach → descend → close → lift); step(env, obs) -> (action, PolicyTelemetry) returns a 4-vec [dx, dy, dz, gripper] normalised to [-1, 1] for OSC_POSITION + GRIP. No grid-search IK, no Jacobian glue — OSC owns the IK; the policy just emits clip((target - eef) / cartesian_step_m, -1, 1) against the latched initial cube pose. SO-100 jaw direction convention: positive opens, negative closes (named explicitly via open_cmd / closed_cmd locals to guard against sign flips).

python/sim/src/openral_sim/backends/openarm_robosuite/

Custom MJCF composer + SCENES.register("openarm_tabletop_pnp") adapter for the bimanual OpenArm v2 pick-and-place scene. The composer rewrites the vendor enactic/openarm_mujoco v2 bimanual MJCF in-place to add scene bodies (table, target object, world skybox), substitute its <position> actuators with motor actuators of compatible torque limits, and inject a camera. State / action dimensions and the actuator inventory are now derived from the RobotDescription.sim block (this branch) rather than hard-coded module constants. Opt-in via the robocasa dependency group (only place robosuite>=1.5 is declared in the workspace; this backend uses robosuite purely as an MJCF wrapper and does not need the robocasa kitchen / GR1 forks); the scene factory calls openral_sim._deps.ensure_backend_deps("openarm_robosuite") first so the user gets an interactive auto-install banner on first use instead of a bare ModuleNotFoundError: robosuite. - _assets.py: - load_openarm_description() -> RobotDescription — Resolves the canonical robots/openarm/robot.yaml manifest as the single source of truth for actuator metadata. (L83) - actuator_specs_from_description(desc) -> list[ActuatorSpec] — Build the ordered list of MJCF actuator specs (name, joint, ctrlrange, gear, side) from desc.joints + desc.sim.grippers. Mirrors the OpenArm v2 actuator block but stays robot-data-driven so adding a new joint in robot.yaml is enough — no edit here. (L122) - motor_actuator_names_from_description(desc) -> list[str] — Convenience wrapper returning just the actuator names in MJCF order; used by the env's action wiring. Replaces the removed MOTOR_ACTUATOR_NAMES module-level tuple. (L169) - _render_actuator_block(specs) -> str — Render an <actuator> XML block from the spec list (motor actuators with per-actuator ctrlrange + gear). (L302) - compose_openarm_tabletop_mjcf(env_cfg) -> str — Compose the scene MJCF: pulls the v2 bimanual MJCF, substitutes position actuators with the motor block from _render_actuator_block, injects scene bodies (table / target / base sites) + the top camera, and lifts the robot bases. (L478) - Module constants _FALLBACK_TOP_CAMERA_POS / _FALLBACK_TOP_CAMERA_TARGET / _FALLBACK_TOP_CAMERA_FOVY (L251–L253) — Fallbacks consumed only when RobotDescription.scene_defaults.top_camera is unset AND scene.backend_options.top_camera_* is unset. Renamed from _DEFAULT_TOP_CAMERA_* (this branch) to reflect that the canonical defaults now live on the robot manifest (SceneDefaults / TopCameraDefaults on openral_core). - Private helpers: _look_at_quat, _inject_base_center_sites, _lift_robot_bases, _rename_upstream_wrist_cameras, _inject_white_skybox, _strip_position_actuators. - env.py: - _resolve_state_dim(env_cfg, rskill_manifest=None) -> int — Derive the observation-state dim from the rSkill manifest's state_contract.dim when present, else from OPENARM_DESCRIPTION.observation_spec.state_shape. Replaces the removed _OBS_STATE_DIM module-level constant so the scene is self-consistent when the rSkill ships a different state contract. (L152) - _resolve_initial_pose_from_rskill(rskill_manifest) — Read the per-rSkill initial joint pose if the manifest pins one. (L219) - _resolve_base_translation(env_cfg) -> tuple[float, float] — Parse the scene's base_translation override; defaults to the OpenArm tabletop layout. (L104) - class _ArmHandles / _build_arm_handles(model, side) -> _ArmHandles — Per-side MuJoCo qpos/qvel/actuator handles. (L275, L294) - class _OpenArmTabletopRollout — SimRollout for the openarm_tabletop_pnp scene: composes the MJCF via _build_openarm_tabletop_scene, drives both arms through the manifest-derived actuator block, exposes mujoco_handles() + sim_time_ns() (round(MjData.time * 1e9), ADR-0048 Phase 1) for the viewer / sim-clock, and action_dim (== bimanual state_dim) so SimAttachedHAL._probe_env_action_dim resolves the deploy-sim action width (ADR-0034 probe-gap fix). (L397) - _build_openarm_tabletop_scene(env_cfg) -> _OpenArmTabletopRollout — Scene factory registered as SCENES.register("openarm_tabletop_pnp")(_build_openarm_tabletop_scene). (L684)

python/sim/src/openral_sim/backends/so101_box/

Parameterised raw-MuJoCo scene: SO-101 in a configurable box arena, registered as @SCENES.register("so101_box", fixed_robot="so101_follower"). Defaults match the user-supplied sketch (100 × 61.5 × 75 cm box, SO-101 back-centre on floor, OAK-D Pro overhead RGB-D, gripper-mounted wrist camera, 44.5 × 44.5 × 20 mm slotted block with Ø 23 mm hole + 5 mm slot, Ø 21.9 × 90 mm tube (0.55 mm radial clearance)). Every dimension and threshold is driven by scene.backend_options via a typed BoxSceneOptions dataclass — no scene geometry is hard-coded in Python. Each reset() randomises both the block and the tube on the floor at independent (x, y, yaw) draws within configurable ranges; success fires when the tube is inserted vertically into the block hole within configurable tolerances. - _assets.py: - class BoxSceneOptions — Typed dataclass holding every scene-geometry knob (arena, robot mount, two cameras, block + tube dimensions, spawn ranges, insertion thresholds). Fed from scene.backend_options by _options_from_backend_options. (L41) - compose_so101_box_mjcf(options=None) -> tuple[str, Path] — Read the upstream robot_descriptions:so_arm101_mj_description MJCF, re-anchor its <body name="base"> to options.robot_base_xyz + yaw, splice a wrist camera into <body name="gripper">, and append the arena floor + 4 walls + ceiling light + OAK-D Pro overhead camera + slotted block + tube to the worldbody. Output written next to the upstream MJCF so meshdir="assets" resolves at compile time without copying STLs. (L295) - Private helpers: _resolve_so101_mjcf, _yaw_quat_z, _look_at_quat, _reanchor_robot_base, _splice_wrist_camera, _render_arena_geoms, _render_overhead_camera, _render_slot_block (5-box decomposition with a square hole + slot), _render_tube (cylinder + two end-tip sites). - env.py: - _options_from_backend_options(raw) -> BoxSceneOptions — Validate + parse scene.backend_options into a BoxSceneOptions; rejects unknown keys loudly so YAML typos surface immediately. (L66) - class _So101BoxRollout — SimRollout driving the SO-101's 6 position actuators, two MuJoCo renderers (RGB + depth-mode on the same overhead camera), random spawn at every reset, and the geometric insertion success check. Exposes mujoco_handles() + sim_time_ns() (round(MjData.time * 1e9), ADR-0048 Phase 1) for openral sim run --view / the sim clock, and action_dim (== 6) so SimAttachedHAL._probe_env_action_dim resolves the deploy-sim action width (ADR-0034 probe-gap fix). (L154) - build_so101_box_scene(env_cfg) -> _So101BoxRollout — Scene factory registered as SCENES.register("so101_box", fixed_robot="so101_follower")(build_so101_box_scene). Composes the MJCF, resolves the 6 arm actuators by their upstream numeric names ("1"…"6"), and caches the block / tube / hole / tip site indices for the success check. (L533)

python/sim/src/openral_sim/backends/tabletop_push/

Greenfield robot-agnostic native scene (ADR-0033): a push-cube-to-goal task on a configurable tabletop, registered FREE-AXIS as @SCENES.register("tabletop_push") (no fixed_robot). The robot is a flag — env_cfg.robot_id resolves a RobotDescription whose sim.mjcf_uri provides the base arm MJCF; the table/cube/goal/cameras are appended to that robot's MjSpec worldbody and the robot root body is re-anchored, so no robot-specific scene code is needed (verified for SO-101, Franka, UR5e). Success is geometric (cube centre within goal_radius of the goal disc and still resting on the table), so it makes no gripper/end-effector assumption. Action/state dim = the compiled model's actuator count nu, with the appended task world preserving the robot's low actuator/qpos indices (the same contract MujocoArmHAL._sim_kwargs_for relies on). - _assets.py: - class TabletopOptions — Typed dataclass holding every scene-geometry knob (table slab, robot-mount fallback, cube, goal disc + radius, two world cameras, opt-in wrist camera, settle steps, lighting). Fed from scene.backend_options by _options_from_backend_options. (L51) - compose_tabletop_mjcf(description, options=None, *, base_pose=None) -> mujoco.MjModel — Resolve the robot MJCF from the manifest (assets.mjcf via _resolve_robot_mjcf → openral_core.assets.resolve_asset), load it into an MjSpec, re-anchor the robot root body (worldbody.bodies[0]) to base_pose (full 6-DOF) or the yaw-only robot_base_xyz fallback, append the table + cube (freejoint) + goal site + overhead/front cameras + light, and compile. Robot-agnostic — no body-name regex. (L189) - Private helpers: _resolve_robot_mjcf, _base_pos_quat, _append_table, _append_cube, _append_goal_marker, _append_world_cameras, _append_overhead_light, _append_wrist_camera, _look_at_quat. - env.py: - _options_from_backend_options(raw) -> TabletopOptions — Validate + parse scene.backend_options into a TabletopOptions; rejects unknown keys loudly. (L62) - class _TabletopPushRollout — SimRollout driving the robot's nu actuators by index (clipping each to its transmission joint's range), rendering the world cameras, randomising the cube + goal each reset (goal via model.site_pos), and the robot-agnostic on-goal success check. Exposes mujoco_handles() + sim_time_ns() (round(MjData.time * 1e9), ADR-0048 Phase 1) for openral sim run --view / the sim clock, and action_dim (== robot actuator count nu) so SimAttachedHAL._probe_env_action_dim resolves the deploy-sim action width (ADR-0034 probe-gap fix). (L135) - build_tabletop_push_scene(env_cfg) -> _TabletopPushRollout — Scene factory registered as SCENES.register("tabletop_push")(build_tabletop_push_scene) (free-axis). Composes the model, resolves the robot's actuator→joint transmissions for state read + action clipping, and caches the cube body/freejoint + goal site indices. Raises ROSConfigError when robot_id has no registered manifest. (L344)

python/sim/src/openral_sim/policies/mock.py

  • class _MockSim — Tiny gym-like env for tests. (L31)
  • class _ZeroPolicy — Always emits zero-vector actions. (L117)
  • class _RandomPolicy — Fixed-seed Gaussian samples. (L136)
  • _coerce_int(value, default) -> int (L88)
  • _build_mock_scene(env_cfg) -> _MockSim (L100)
  • _resolve_action_dim(env_cfg) -> int (L159)
  • _build_zero_policy(env_cfg) -> _ZeroPolicy (L202)
  • _build_random_policy(env_cfg) -> _RandomPolicy (L211)

python/sim/src/openral_sim/policies/smolvla.py

  • class _SmolVLAAdapter — Lerobot-style policy adapter. (L177) — reset/step/close/_build_batch. Calls openral_rskill._vla_core.run_inference for instrumented select_action.
  • _build_smolvla(env_cfg) -> _SmolVLAAdapter — Resolves device + rSkill via _vla_core.resolve_device / resolve_rskill_repo_id(adapter_name="SmolVLA"); resolves the manifest through the shared openral_sim.policies._policy_loading.load_manifest_for_spec; defers torch + lerobot imports through the shared lazy_import_lerobot("SmolVLA") (both helpers extracted in the 2026-05 cleanup to drop the parallel local copies). Calls _vla_core.apply_chunk_replay and _vla_core.maybe_compile_chunk_forward to enable vla.extra.n_action_steps / compile. Loads the lerobot PolicyProcessorPipeline via _vla_core.materialize_processor_dir(manifest) — per-file hf_hub_download driven by manifest.processors (rSkill self-containment audit Gap 1+3). No snapshot_download. Stats-fallback path: when the per-file download 404s (community finetunes routinely ship only config.json + model.safetensors) the adapter logs smolvla_processor_files_missing_falling_back_to_dataset_stats and rebuilds the processors from manifest.dataset_uri's normalization stats via _load_lerobot_dataset_stats(...) + make_pre_post_processors(..., dataset_stats=...). If dataset_uri is also unset a typed ROSConfigError is raised. Every load phase (imports, from_pretrained, to_device, processor_dir, make_processors) is wrapped in _smolvla_phase(...). (L286)
  • _smolvla_phase(name, **fields) -> ContextManager[None] — Adapter-local shortcut for phase_timer(name, prefix="smolvla", log=_log). (L158)
  • _is_processor_missing(exc: BaseException) -> bool — Walks __cause__ / __context__ looking for a HF Hub RemoteEntryNotFoundError / EntryNotFoundError. Detects 404s that materialize_processor_dir re-raised as ROSConfigError; stays decoupled from HF Hub's exception module path. (L61)
  • _load_lerobot_dataset_stats(dataset_uri: str) -> dict[str, dict[str, Any]] — Aggregates per-feature stats from a LeRobotDataset on HF Hub. Tries v3 (single meta/stats.json) first; on 404 falls back to v2.1 (meta/episodes_stats.jsonl) aggregated via lerobot.datasets.compute_stats.aggregate_stats. Returns a {feature_key: {mean|std|min|max|count: np.ndarray}} dict suitable for make_pre_post_processors(..., dataset_stats=...). (L76)

python/sim/src/openral_sim/policies/act.py

  • class _ACTAdapter — ACT policy adapter. Manifest-first image_preprocessing resolution (_cam_alias + _image_input_template + _flip_images_180) lets LIBERO camera1 / camera2 feed an ACT checkpoint whose input features are observation.images.image / observation.images.image2; legacy _state_mean / _action_std path stays for act-aloha-style checkpoints with norm stats in model.safetensors.
  • _build_act(env_cfg) -> _ACTAdapter — Snapshots the policy weights for ACTPolicy.from_pretrained (config.json sanitized via _sanitize_act_config_json before load). Dispatches the processor branch on manifest.processors is not None: modern (e.g. rskills/act-libero) calls _vla_core.materialize_processor_dir(manifest) and composes the lerobot factory pipelines with preprocessor_overrides={"device_processor": {"device": <resolved>}} so checkpoints with a baked-in device: mps don't crash on CUDA hosts; legacy (rskills/act-aloha) keeps the _try_load_act_norm_stats path that reads norm stats from model.safetensors. Calls _vla_core.apply_chunk_replay (manifest-aware default) and _vla_core.maybe_compile_chunk_forward. Resolves camera keys / state dim / image aliases via _vla_core.resolve_* helpers (mirrors smolvla); default cam tuple derives from ip.aliases.keys() when set so a LIBERO scene that emits camera1 / camera2 "just works". rSkill self-containment audit Gap 1+3.
  • _load_manifest_for_spec(spec) -> RSkillManifest | None — Mirror of smolvla's helper; loads the rSkill manifest from a skill reference in spec.weights_uri; returns None when the URI is not resolvable to a manifest.
  • _sanitize_act_config_json(snapshot_dir) -> None — Drops ACTConfig fields the installed lerobot version doesn't accept (e.g. n_state_dim on training-fork checkpoints). Mutates config.json in-place, no-op when there's nothing to strip.
  • _apply_temporal_ensemble(policy, spec_extra) -> float | None — Sets policy.config.temporal_ensemble_coeff and builds the missing ACTTemporalEnsembler (lerobot only attaches one when the coeff is non-None at __init__ time).
  • _try_load_act_norm_stats(repo_id, device, torch, cam_keys) -> dict — Pulls normalize_* / unnormalize_* tensors from model.safetensors for the legacy act-aloha-shaped checkpoints.

python/sim/src/openral_sim/_quantization.py

Shared bitsandbytes NF4 quantization helpers + prequantized-state-dict fast path + manifest-driven dtype resolution. Family-agnostic — the same primitives serve pi05 today and any future bnb-quantized backbone (pi0.6, smolvla-large). All helpers defer torch / bitsandbytes imports so installing openral-sim does not pull them transitively.

  • DEFAULT_MIN_PARAMS_TO_QUANTIZE: int = 4_000_000 — Per-Linear weight-element threshold for the nf4 rewrite. PaliGemma / SmolVLA paper default. (L60)
  • quantize_nf4_in_place(policy, *, torch, compute_dtype, min_params=DEFAULT_MIN_PARAMS_TO_QUANTIZE, new_modules_on_meta=False) -> None — Walks the policy, replaces every torch.nn.Linear whose weight has ≥ min_params elements with a bnb.nn.Linear4bit. The actual nf4 pack runs on the next .to(<cuda>); bias terms stay in compute_dtype for numerical safety. new_modules_on_meta=True wraps the replacement walk in accelerate.init_empty_weights() so the bnb constructor's bf16 placeholder allocation lands on the meta device — saves ~5–10 s on a 3.4 B-param load when the caller is going to to_empty(device=...) the tree afterwards. (L72)
  • quantize_int8_in_place(policy, *, torch, compute_dtype, min_params=DEFAULT_MIN_PARAMS_TO_QUANTIZE, threshold=6.0, new_modules_on_meta=False) -> None — Sibling of quantize_nf4_in_place that swaps the same large Linears for bnb.nn.Linear8bitLt (LLM.int8 mixed decomposition, ~50% the bf16 footprint, lossless on most attention workloads). bitsandbytes only offers 4-bit and 8-bit Linears — there is no nf8; int8 here means LLM.int8, not torchao dynamic int8. No prequant fast-path: SCB sub-state ownership inside Int8Params makes a separate Hub artefact brittle. (L173)
  • install_prequantized_linears(policy, state, *, device, torch) -> tuple[int, set[str]] — Replaces every Linear4bit.weight with Params4bit.from_prequantized(...) data read from state. Returns (n_modules_rebuilt, consumed_state_keys) so the caller can subtract the consumed keys before calling policy.load_state_dict for the residual. (L293)
  • detect_prequantized_nf4(spec) -> str | None — Probes the rSkill's HF repo for a quantization_metadata.json sentinel; returns the repo id when the pack is present, None otherwise. Routed through _hf_download_cached_first so a cache hit avoids the HEAD request. (L375)
  • load_prequantized_state_for_rskill(policy, spec, *, torch, log_event_prefix="rskill") -> None — Combined entry point: validates the metadata sentinel, downloads model.safetensors, calls install_prequantized_linears, then applies the residual via policy.load_state_dict(leftover, strict=False). Silent no-op when the rSkill ships bf16 weights — adapters can call it unconditionally after their own quantize_nf4_in_place. (L441)
  • peek_safetensors_keys(repo_id, *, filename="model.safetensors") -> set[str] | None — Reads only the safetensors header (~10 ms warm) and returns its key set. Works for both prequantized packs (nf4 fast path) and bare source checkpoints (int8 fast path that loads bf16 weights via load_state_dict instead of going through lerobot's from_pretrained). Used by targeted_reset_parameters to skip the kaiming / normal init walk for modules whose params will be overwritten by the upcoming state load. (L570)
  • targeted_reset_parameters(policy, *, covered_keys) -> None — Walks the policy and calls module.reset_parameters() only on modules whose direct parameter keys are NOT a subset of covered_keys (the safetensors key set about to be loaded). Skips containers (modules with no direct params). Pass covered_keys=None for the historical unconditional reset. Model-agnostic — promoted out of pi05.py so π0.5 / MolmoAct2 / future meta-init families share it. (L712)
  • tie_transformers_weights(policy) -> None — Walks the policy in pre-order and calls module.tie_weights() on each outermost transformers backbone, skipping descendants of already-tied modules; a raising tie_weights (e.g. a meta-init expert backbone) is non-fatal. Promoted out of pi05.py alongside targeted_reset_parameters. (L772)
  • normalise_manifest_dtype(manifest) -> str | None — Pulls manifest.quantization.dtype.value as a string; returns None for manifests without a quantization block. Lifted out of pi05.py into the shared module so smolvla / xvla / future quantized adapters share one implementation. (L634)
  • manifest_dtype(spec, manifest=None) -> str | None — Resolves the adapter's load dtype: spec.extra["dtype"] (per-run override) wins, falling back to manifest.quantization.dtype via normalise_manifest_dtype, then None (default_dtype_for_device picks a CUDA-aware default). Lifted out of pi05.py. (L652)
  • torch_dtype_for(torch, dtype_str, device) -> Any — Map a manifest dtype string (bf16/bfloat16, fp16/float16/half, fp32/float32) to a torch dtype, with a CUDA-aware default (bf16 on CUDA, fp32 elsewhere). Pass-through dtypes (nf4, int8) fall through to the default so adapters can pick a sensible compute dtype for the leaves that won't be quantized. Lifted out of pi05.py. (L677)
  • default_dtype_for_device(device) -> str — Picks a default load dtype when the manifest doesn't specify one: nf4 on CUDA (so 3.4 B-param backbones fit in ~4 GiB), fp32 elsewhere. Lifted out of pi05.py. (L700)

python/sim/src/openral_sim/policies/_policy_loading.py

Shared loader helpers for openral_sim policy adapters (extracted in this branch to remove the parallel _load_manifest_for_spec copies from smolvla.py / rldx.py / pi05.py). Module docstring explains why the manifest-resolution branch is generic but the quantization branch deliberately stayed family-specific.

  • load_manifest_for_spec(spec) -> RSkillManifest | None — Returns the parsed openral_core.RSkillManifest when spec.weights_uri is a resolvable skill reference; returns None for bare hf:// URIs and local paths so the caller can decide whether the missing manifest is fatal (SmolVLA raises; pi05 / RLDX fall back to the URI directly). Tolerant of spec=None / spec.weights_uri=None. (L53)
  • lazy_import_lerobot(adapter_name, *, install_hint="just sync --all-packages --group libero") -> tuple[Any, Any] — Imports torch + lerobot's make_pre_post_processors factory behind a typed ROSConfigError with the install hint. Centralises the same import-time ceremony SmolVLA / π0.5 used to duplicate inline. Returns (torch, make_pre_post_processors); the caller imports the adapter-specific Policy class separately. (L84)

python/sim/src/openral_sim/policies/pi05.py

  • _build_pi05(env_cfg) -> _PI05Adapter — Calls _vla_core.apply_chunk_replay. compile is intentionally NOT plumbed: the adapter sets pi05_cfg.compile_model = False to keep the quantization path stable. Supports quantization.dtype ∈ {nf4/int4, int8, bf16, fp16, fp32}: nf4/int4 runs quantize_nf4_in_place + optional prequant fast-path; int8 runs quantize_int8_in_place against bnb.nn.Linear8bitLt (LLM.int8, CUDA-only); everything else casts and moves to device. The manifest's quantization.dtype is consulted via openral_sim._quantization.manifest_dtype when no spec.extra["dtype"] override is set (the four _manifest_dtype / _normalise_manifest_dtype / _torch_dtype_for / _default_dtype helpers used to live here — moved to _quantization.py so smolvla / xvla can reuse them). Manifest resolution routes through openral_sim.policies._policy_loading.load_manifest_for_spec. Processor sidecars resolved via _resolve_pretrained_path(spec, repo_id) → delegates to _processors.resolve_processor_dir (manifest-first per ADR-0013 / rSkill self-containment audit Gap 1+3, snapshot fallback for non-rSkill refs). Every load phase is wrapped in _pi05_phase(...) so the operator sees a per-phase wall-time + GPU footprint in the logs and in openral dashboard. Both nf4 and int8 take a fast meta-init path on CUDA that skips lerobot's slow PI05Policy.from_pretrained (~152 s for the 3.4 B-param backbone): nf4 loads from the prequant safetensors via load_prequantized_state_for_rskill; int8 loads the source bf16 safetensors via _load_bf16_state_for_int8 + _rebuild_int8_params_for_linear8bitlt. Combined effect on warm RTX 4070 cache: 95 s → 10 s (9×) for nf4 / 165 s → 11 s (14.6×) for int8. (L437)
  • _pi05_phase(name, **fields) -> ContextManager[None] — Adapter-local shortcut for phase_timer(name, prefix="pi05", gpu_mb=True, log=_log). (L389)
  • _targeted_reset_parameters, _tie_transformers_weights — module-level aliases re-importing _quantization.targeted_reset_parameters / tie_transformers_weights (promoted to the shared module so MolmoAct2's fast meta-init reuses them; the int8 fast path here still calls them via the alias).
  • _expand_covered_keys_via_tied_storage(policy, covered_keys) -> set[str] — Detects tied parameters via Tensor.untyped_storage().data_ptr() and extends covered_keys to include every key in a tied group whenever any member is already covered. Uses named_parameters(remove_duplicate=False) because the default dedups tied params away. (L225)
  • _load_bf16_state_for_int8(policy, repo_id, *, torch) -> None — Downloads <repo>/model.safetensors via _hf_download_cached_first and applies it via policy.load_state_dict(strict=False). The int8 fast meta-init path's substitute for lerobot's ~152 s PI05Policy.from_pretrained. (L323)
  • _rebuild_int8_params_for_linear8bitlt(policy) -> int — Re-wraps each Linear8bitLt.weight as a fresh bnb.nn.Int8Params(has_fp16_weights=False). to_empty(device=...) strips Parameter subclasses; without this rewrap the downstream policy.to(<cuda>) would never trigger bnb's int8 pack. (L270)
  • _resolve_pretrained_path(spec, repo_id) -> str — Returns a local directory containing the lerobot processor sidecars. Local path → verbatim; otherwise routes through _processors.resolve_processor_dir. (L403)

python/sim/src/openral_sim/policies/molmoact2.py

MolmoAct2 is a transformers custom-code model (trust_remote_code, loaded via AutoModelForImageTextToText + AutoProcessor), not a lerobot policy. The adapter drives its predict_action(...) continuous-action API and replays the returned chunk one step at a time (own queue, not lerobot's select_action). Model graph + processor + norm_stats.json load from the manifest's source_repo (hf://allenai/MolmoAct2-LIBERO); the NF4 weights overlay from the manifest's weights_uri prequant pack. Verified end-to-end on LIBERO-Spatial (NF4, 8 GiB RTX 4070, success on task 0).

  • _build_molmoact2(env_cfg) -> _MolmoAct2Adapter (L406, @POLICIES.register("molmoact2")) — Loads Ai2's MolmoAct2 (model_family: "molmoact2", ~5.49 B params; Molmo2-ER VLM + flow-matching action expert, arXiv:2605.02881). Resolves the manifest + dtype, delegates the load to _load_molmoact2_model, then wires replay cadence (clamped to config.max_action_horizon, LIBERO = 10), norm tag, image flips, state/action dims, and autocast. (L653)
  • _load_molmoact2_model(*, torch, auto_model_cls, auto_processor_cls, source_repo, spec, device, dtype_str, max_crops) -> tuple[model, processor, use_nf4, torch_dtype] (L382) — Always loads the processor (AutoProcessor.from_pretrained(..., trust_remote_code=True), optional image_processor.max_crops override). nf4-on-CUDA with a prequant pack takes a fast meta-init path (mirrors π0.5): detect_prequantized_nf4 → AutoConfig.from_pretrained → build on the meta device via accelerate.init_empty_weights() + AutoModelForImageTextToText.from_config(..., trust_remote_code=True) → quantize_nf4_in_place(new_modules_on_meta=True) → to_empty("cpu") → tie_transformers_weights → targeted_reset_parameters(covered_keys=peek_safetensors_keys(pack)) → load_prequantized_state_for_rskill → .to(device). Skips the ~200 s bf16 from_pretrained materialisation; measured 202 s → 14 s on a warm RTX 4070 cache. No manual buffer reconstruction needed — MolmoAct2RotaryEmbedding self-heals a meta/garbage inv_freq (persistent=True → restored by the pack). Falls back to the slow path (from_pretrained on CPU → precast_bf16 → quantize_nf4_in_place → prequant overlay → .to(device)) for bf16 / non-CUDA / no-pack. Supports quantization.dtype ∈ {nf4/int4, bf16, fp16, fp32}; nf4 is CUDA-only and the default on CUDA (bf16 ≈ 11 GiB → OOMs an 8 GiB GPU, nf4 ≈ 4 GiB). Every load phase wrapped in _molmoact2_phase(...). (L498)
  • _resolve_max_crops(spec, manifest) -> int | None (L305) — Resolve the image-processor max_crops override: vla.extra["image_max_crops"] → OPENRAL_MOLMOACT2_MAX_CROPS env → manifest.image_preprocessing.image_max_crops → None (checkpoint default 8). A secondary vision-activation lever: measured on an 8 GiB RTX 4070 (transformers 5.x) it does not by itself decide the 8 GiB fit — the inference peak is set by the LM token-embedding, and the fast MolmoAct2ImageProcessor largely ignores max_crops. The actual 8 GiB enabler is _enable_expandable_segments. (L471)
  • _enable_expandable_segments() -> None (L131) — os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") before the first CUDA allocation (called at the top of _build_molmoact2 when device is CUDA). MolmoAct2 NF4 is ~6 GiB resident and peaks ~7.63 GiB; on an 8 GiB card (~7.6 GiB usable) the first forward's ~1.5 GiB embedding cat OOMs without expandable segments and fits with them (verified). No-op if the operator already set the var. (L124)
  • _molmoact2_phase(name, **fields) -> ContextManager[None] (L105) — Adapter-local shortcut for phase_timer(name, prefix="molmoact2", gpu_mb=True, log=_log). (L150)
  • _import_transformers() -> tuple[Any, Any] (L118) — Imports AutoModelForImageTextToText + AutoProcessor behind a typed ROSConfigError install hint. Returns Any (transformers is an optional, unstubbed dep). (L163)
  • _strip_hf_uri(uri, *, field_name) -> str (L145) — Strip the hf:// prefix off a manifest URI, validating it is present. (L230)

python/sim/src/openral_sim/policies/_processors.py

Shared resolve_processor_dir(spec, repo_id) -> str helper used by the diffusion / xvla / pi05 adapters to fetch policy_preprocessor.json / policy_postprocessor.json. Mirrors the smolvla / modern-ACT pattern (ADR-0013) and closes the three sister TODOs on the rSkill self-containment audit (2026-05-18).

  • resolve_processor_dir(spec, repo_id) -> str — Manifest-first: when spec.weights_uri resolves to a manifest that declares a processors block, delegates to materialize_processor_dir(manifest) (per-file hf_hub_download). Otherwise falls back to snapshot_download(repo_id, ignore_patterns=["*.md"]) — the path legacy hf://lerobot/diffusion_pusht URIs still rely on. (L32)

python/sim/src/openral_sim/policies/rldx.py

Auto-managed sidecar adapter for RLWRLD/RLDX-1 (Qwen3-VL-8B + Multi-Stream Action Transformer, ~6.9 B params). Runs the upstream policy in an out-of-process Python 3.10 venv and speaks the server's native ZMQ + msgpack wire protocol — necessary because the rldx package pins requires-python = "==3.10.*" (incompatible with our 3.12 workspace) and ships a custom architectures=["RLDX"] class not in HF Transformers (the HF checkpoint does NOT include modeling_rldx.py, so trust_remote_code is not an escape). The adapter auto-spawns the sidecar on first observation (OPENRAL_RLDX_AUTO_SPAWN=1, default) so users run openral sim run once and never invoke the boot helper. Sidecar boot helper: tools/rldx_sidecar.py. Used as policy_id: "rldx" in rskills/rldx1-*/rskill.yaml. - class _RLDXSidecarAdapter — ZMQ-backed RLDX policy adapter. On __post_init__ it pings the server; if no answer and auto_spawn=True, it forks tools/rldx_sidecar.py (in its own start_new_session) with the manifest-resolved model id + port + quantization + embodiment tag, then polls ping until success or boot_timeout_s elapses (default 900 s — covers the first-run git clone + uv sync). Replays the upstream MSAT 16-action chunk. Replan precedence: vla.extra.replan_steps > manifest.n_action_steps > legacy _RLDX_CHUNK_LEN // 2 fallback — the rldx1-ft-{libero,gr1,rc365} manifests all ship n_action_steps: 16 (replay the full chunk; halves inference round-trips vs the old half-chunk RTC default at the cost of 16 open-loop env steps between observations). Manifest-driven state_layout dispatch ("libero" → LIBERO-flat keys; "gr1" → Fourier-native general_embodiment; "rc365" → PandaMobile general_embodiment; "simpler_widowx" → SimplerEnv WidowX bridge_orig with OXE_BRIDGE_ORIG embodiment_tag; "simpler_google" → SimplerEnv Google fractal20220817_data with OXE_FRACTAL embodiment_tag (the FT-SIMPLER-* checkpoints' processor_config.json only ships bridge_orig / fractal20220817_data modality buckets — the OXE_WIDOWX / OXE_GOOGLE enum names exist but crash PolicyLoader.load with KeyError because their .value strings are not registered modality buckets)). Public contract: reset/step/close/last_input_frame. close() tears down the spawned child via SIGTERM → SIGKILL fallback; no-op when we connected to a pre-existing server. Before adopting a pre-existing sidecar (mode="existing") it calls _verify_existing_identity, which cross-checks the on-disk identity record (family/model/embodiment_tag/quantization, written by run_sidecar) and raises ROSConfigError on a mismatch — closing the "two checkpoints share the default port → second run silently serves the first one's model" hole; a missing record is treated as unverifiable (warn + proceed) so operator-managed boots keep working. - _encode_ndarray(obj) -> Any — msgpack default hook; serialises ndarrays via np.save → BytesIO wrapped in {"__ndarray_class__": True, "as_npy": <bytes>} (mirrors MsgSerializer.encode_ndarray in rldx/policy/server_client.py). - _decode_ndarray(obj) -> Any — msgpack object_hook; reverse of _encode_ndarray. - Manifest resolution: the adapter resolves skill references in weights_uri through the shared openral_sim.policies._policy_loading.load_manifest_for_spec helper so it can read state_contract.layout for LIBERO / GR1 / RC365 dispatch, image_preprocessing.flip_180, and the canonical hf:// model id. (The private _load_manifest_for_spec copy that used to live here was removed in the 2026-05 cleanup.) - _RLDXSidecarAdapter._init_socket / _try_ping / _verify_existing_identity / _wait_for_boot / _spawn_sidecar / _terminate_child / _is_port_busy / _locate_sidecar_script / _resolve_model_id — auto-spawn lifecycle helpers. _verify_existing_identity reads the sidecar identity record via openral_sim._sidecar_common.read_sidecar_identity and fails closed on a checkpoint mismatch. _init_socket (re)creates the ZMQ REQ socket with LINGER=0 + RCV/SND timeouts and connects to tcp://host:port; called from __post_init__ AND from _try_ping on failure because a REQ socket whose recv() timed out is stuck in EFSM (strict REQ state machine: every send() must be followed by a matching recv()) and every subsequent _call would raise Operation cannot be accomplished in current state until the socket is reopened. _try_ping does one timeout-bounded ZMQ round-trip and resets the socket on failure so _wait_for_boot actually makes forward progress instead of looping against a dead socket for the full boot_timeout_s; _spawn_sidecar Popens the boot script (skipped if _is_port_busy reports a listener already); _wait_for_boot polls every 2 s until success or boot_timeout_s or child death; _terminate_child does best-effort SIGTERM → SIGKILL teardown; _locate_sidecar_script walks upwards from __file__ to find tools/rldx_sidecar.py (or honours OPENRAL_RLDX_SIDECAR_SCRIPT); _resolve_model_id picks vla.extra.model_id → manifest weights_uri → spec weights_uri. - _build_libero_obs / _build_gr1_obs / _build_rc365_obs / _build_simpler_widowx_obs / _build_simpler_google_obs / _pick_single_camera / _pick_images / _pick_state / _normalize_action_column / _assemble_libero_chunk / _assemble_gr1_chunk / _assemble_rc365_chunk / _assemble_simpler_chunk — wire-format builders / parsers split by embodiment. The SimplerEnv builders mirror the upstream rldx/eval/sim/SimplerEnv/simpler_env.py reference: WidowX feeds video.image_0 + 8 state scalars (bridge-rotated Euler state.roll/pitch/yaw + state.pad=0 sentinel + raw state.gripper); Google feeds video.image + position/xyzw-quat split state + state.gripper = 1 - raw_open. _assemble_simpler_chunk binarizes the WidowX gripper column (2*(g>0.5)-1) so MS3's bridge digital twin sees [-1, +1] per the upstream WidowXBridgeEnv._postprocess_gripper; Google's sticky-gripper state machine is intentionally NOT applied here (per-rollout state belongs in the env wrapper, not the chunk assembler). The LIBERO chunk assembler rescales the gripper column from the RLDS dataset convention ([0, 1], 0=close/1=open) to LIBERO/robosuite ([-1, +1], -1=open/+1=close) via _rldx_gripper_to_libero before returning — without this the Franka gripper never actuates (GH-133). The GR1 path concatenates Fourier-native general_embodiment action groups (right_arm + left_arm + waist + right_hand + left_hand) into the Fourier GR-1 BASIC 29-D composite. The RC365 path concatenates the 5 PandaMobile groups (eef_pos + eef_rot + gripper + base + control_mode) into 12-D and lets openral_sim.backends.robocasa trim to the 11-D BASIC env action. The matching unflatten path on the RoboCasa side is openral_sim.backends.robocasa.GrootRoboCasaEnv._split_gr1_action (shared validator + slicer) → _to_gr1_action_dict_gym (gymnasium action.{waist,right_arm,left_arm,right_hand,left_hand} keys) / _to_gr1_action_dict (raw robosuite robot0_{torso,right,left,right_gripper,left_gripper} keys); both helpers reuse the same _GR1_BASIC_DIM=29 constant. - _rldx_gripper_to_libero(gripper) -> NDArray[float32] — Maps an RLDS-convention gripper column ([0, 1], 0=close/1=open) to the LIBERO/robosuite convention ([-1, +1], -1=open/+1=close), via out = -sign(2*g - 1). Mirrors the two-step transform (normalize_gripper_action + invert_gripper_action) that the upstream rldx/eval/sim/LIBERO/libero_env.py::LiberoEnv.step applies before stepping the env. Called from _assemble_libero_chunk; consumed by LiberoEnv.step via the 7-D LIBERO action vector at index 6. Fixes GH-133 (Franka gripper stuck open). - _env_bool(name, default) -> bool — permissive boolean env-var parser (1 / true / yes / on). - _resolve_state_layout(manifest) -> str — module-level helper shared by the rldx and gr00t factories; maps manifest.state_contract.layout to one of gr1/rc365/simpler_widowx/simpler_google, else "libero". Single source of truth for obs/action dispatch so neither factory hardcodes an embodiment. - _derive_sidecar_port(*, family, model, embodiment_tag, quantization, layout) -> int / _resolve_sidecar_port(*, port_env, extra_port, …) -> int — per-identity default port (SHA-1-bucketed into 20000–39999, non-crypto) so two different checkpoints never collide on the old hard 5555; _resolve_sidecar_port applies precedence env-pin > vla.extra.port > derived default. - _build_rldx(env_cfg) -> _RLDXSidecarAdapter — @POLICIES.register("rldx") factory. Honours OPENRAL_RLDX_HOST / OPENRAL_RLDX_PORT / OPENRAL_RLDX_AUTO_SPAWN / OPENRAL_RLDX_BOOT_TIMEOUT_S / OPENRAL_RLDX_QUANTIZATION / OPENRAL_RLDX_EMBODIMENT_TAG / OPENRAL_RLDX_MODEL_ID / OPENRAL_RLDX_SIDECAR_SCRIPT env-var overrides; reads replan_steps / image_size / timeout_ms / camera_keys / auto_spawn / boot_timeout_s / quantization / embodiment_tag / model_id from vla.extra; dispatches the obs/action contract via _resolve_state_layout and the port via _resolve_sidecar_port (per-identity default when unpinned).

python/sim/src/openral_sim/policies/gr00t.py

NVIDIA Isaac GR00T policy adapter (ADR-0046) — reuses _Gr00tFamilySidecarAdapter with family="gr00t", forking tools/gr00t_sidecar.py over the same ZMQ + msgpack wire (RLDX-1 is a GR00T-N1.5 finetune sharing the PolicyServer contract). - _build_gr00t(env_cfg) -> _Gr00tFamilySidecarAdapter — @POLICIES.register("gr00t") factory. Reads the OPENRAL_GR00T_* env namespace + vla.extra; dispatches the obs/action layout via the shared rldx._resolve_state_layout (no longer hardcoded "libero" — a non-LIBERO GR00T finetune gets its own contract) and the port via rldx._resolve_sidecar_port. Embodiment tag stays GR00T-specific (LIBERO_PANDA default, overridable) since GR00T's tag enum differs from RLDX's GENERAL_EMBODIMENT family.

python/sim/src/openral_sim/_sidecar_common.py

Shared boot scaffolding for the out-of-process VLA sidecars (rldx / gr00t): clone, venv, install, env isolation, exec — plus the sidecar identity registry. - run_sidecar(*, label, family, repo_url, args, install_deps, make_wrapper) -> int — orchestrates a boot (uv → clone → install → wrapper → exec). Writes the identity record (write_sidecar_identity) just before exec_server so every sidecar this repo starts — auto-spawned or operator-launched — is identifiable. - sidecar_identity_path(port) -> Path / write_sidecar_identity(*, port, family, model, embodiment_tag, quantization) -> None / read_sidecar_identity(port) -> dict[str,str] | None — the per-port identity record under ~/.cache/openral/sidecars/port-<port>.json. The adapter reads it back in _verify_existing_identity to refuse reusing a sidecar serving a different checkpoint. None = no record (unverifiable, not a mismatch). - ensure_uv / ensure_source / make_isolated_env / exec_server / build_parser / run_cmd — uv resolver lookup, shallow clone, 3.10-venv env scrubbing, os.execvpe into the server, shared --model/--port/--quantization/--embodiment-tag/--home CLI, echoed subprocess runner.

python/sim/src/openral_sim/policies/__init__.py

  • _register_policies() -> None — Side-effect imports of the policy-adapter modules so each registers its factory in openral_sim.POLICIES at import time. (L15)

python/sim/src/openral_sim/backends/__init__.py

  • _register_backends() -> None — Side-effect imports of the scene-backend modules so each registers its factory in openral_sim.SCENES at import time. (L36)