Telemetry reference — spans, metrics, logs, instrumentation

Everything OpenRAL reports at runtime, where it comes from, how often it fires, and where it surfaces. If a signal is not here, it does not exist; if it is marked not emitted, the name exists but nothing produces it.

Names are normative in openral_observability.semconv; this page is its annotated index.

Contents: How it flows · Spans · Span events · Metrics · Logs · Load-phase instrumentation · Declared but not emitted · Configuration · Where it goes

How it flows

Every OpenRAL process calls configure_observability(service_name=…), which installs a TracerProvider, MeterProvider and LoggerProvider all pointed at one OTLP endpoint, plus a structlog→OTel log bridge. During openral deploy sim that endpoint is the in-tree dashboard, which is itself an OTLP receiver:

 nodes ──OTLP/HTTP──▶ openral dashboard :4318 ──▶ TelemetryStore ──▶ SSE ──▶ browser
   │                    /v1/traces  /v1/metrics  /v1/logs
   └──────────────▶ otel-collector ──▶ Jaeger (traces) / Prometheus (metrics)

So spans, metrics and every log line land in the same store that feeds the operator's Event Log, which is why the banding rules below matter: without them the panel is unreadable.

With --no-dashboard and no OTEL_EXPORTER_OTLP_ENDPOINT, the whole SDK is a no-op — no providers, no exporters, and structlog falls back to its console renderer.

Spans

Event Log band. dashboard/store.py::_is_headline_span decides the band: ERROR status → error; else a name in _HEADLINE_SPANS (or with a headline prefix) → info; else → debug. This is an allow-list, so a new span is quiet until deliberately promoted. Every span is still indexed in full for openral replay regardless of band — this only affects the operator's log view.

Span	Layer	Emitted at	Frequency	Band	Surfaces in
`cli.command`	CLI	`observability/cli.py`	1 / invocation	info	Event Log
`deploy.bringup`	all lifecycle nodes	`observability/lifecycle.py` (via `@log_lifecycle_errors`)	1 / transition	info	Event Log
`rskill.execute`	rSkill	`rskill/base.py` (`rSkillBase.step`), `rskill_runner_node.py`	1 / tick (`step`) + 1 / goal (runner node)	debug — not promoted while the per-tick emitter shares the name	rSkill card
`rskill.configure`	rSkill	`rskill/base.py`	per lifecycle	info	Event Log, rSkill card
`rskill.activate`	rSkill	`rskill/base.py`	per lifecycle	info	Event Log, rSkill card
`reasoner.tick`	Reasoning	`reasoner/core.py`	per LLM round-trip (~0.2 Hz cap)	info	Reasoner card
`world.scene_objects`	World state	`world_state/scene_objects_span.py`	on scene change (+60 s keepalive; checked at 0.2 Hz)	info	World-state card, SLAM overlay
`sim.run`	Sim	`sim/sim_runner.py`	1 / run (held open)	info	Event Log
`detect.probe.*`	Detect	`detect/detect.py`	1 / `openral detect`	info	Event Log
`hal.read_state`	HAL	`hal/lifecycle.py`, `deploy_runner.py`	30 Hz	debug	Robot-state card
`hal.send_action`	HAL	`hal/lifecycle.py`, `deploy_runner.py`	30 Hz	debug	Robot-state card
`sensors.read_latest`	Sensors	`deploy_runner.py`, `world_state_ros/lifecycle_node.py`, `sim_sensor_bridge.py`	30 Hz × cameras	debug	Perception card (thumbnails)
`world_state.snapshot`	World state	`world_state/aggregator.py`	30 Hz	debug	World-state card
`rskill.tick`	Runner	`runner/base.py`	30 Hz	debug	rSkill card
`rskill.chunk_inference`	rSkill	`rskill/_vla_core.py`, `rskill_runner_node.py`, 8 sim adapters	per chunk	debug	rSkill card (latency)
`safety.check`	Safety	`runner/safety.py`, `supervisor_node.py`, `lifecycle_kernel.cpp`	per candidate chunk	debug	Safety ledger
`reward.score`	Perception	`reward_monitor_node.py`	`score_period_s` (2 s)	debug	rSkill card reward bar
`slam.occupancy_grid`	SLAM	`runner/slam_bridge.py`	1 Hz (throttled)	debug	SLAM map card
`world.pointcloud`	World state	`runner/world_cloud_bridge.py`	throttled	debug	Pointcloud card
`sim.step` / `eval.step`	Sim	`sim/sim_runner.py`	per step	debug	—
`physics.step`	Sim	`sim/sim_runner.py`	per step	debug	—
`reasoner.skill_failure`	Reasoning	`reasoner_node.py`	per failure	error	Event Log, failure counter

Why the allow-list. A deny-list missed several 30 Hz spans ticking at the same rate. On one second of a real 30 Hz two-camera deploy, the Event Log took 121 info rows/s, cycling its 200-slot ring every 1.65 s — every lifecycle line an operator needed scrolled away before it could be read. Inverted, routine traffic contributes 0 info rows.

`rskill.chunk_inference` attributes

Every span opened by inference_span (directly, or through the _vla_core.run_inference seam) carries inference.kind and, when known, inference.chunk_index / inference.chunk_size / inference.engine / inference.device, plus inference.duration_ms recorded on close. Those six have semconv.py constants; the RTC attribute below rides the helper's generic inference.-prefixed **attrs path and has none.

Attribute	Recorded at	Meaning
`inference.rtc_delay`	`rskill/_vla_core.py::run_inference`	Real-Time Chunking only: how many actions the consumer popped out of the `ActionQueue` while the previous inference ran — the prefix length the guidance freezes. It is the queue's index delta, not a latency estimate, so it stays correct in fast-forward sim as well as wall-clock deploy. `0` on the cold-start chunk (no previous tail). The executor also logs each merge as `chunked_executor.rtc_merge` at DEBUG (`chunk_index`, `merge_tick` = actions served before the merge, measured `rtc_delay`), so switch points can be lined up with a bag of served actions (enable DEBUG on just that stdlib logger, `logging.getLogger("openral_rskill.executor").setLevel(logging.DEBUG)`, e.g. from a `sitecustomize.py`; a deploy's root logger stays at WARNING).

The attribute is written only when the executor threads an inference_delay into run_inference(call_kwargs=…), so its presence in a trace is also the "RTC is on for this skill" marker. It stays on the span and never becomes a metric label — the openral.inference.duration label set is closed to kind.

The three event rings

Demoting the noisy spans stopped info being generated, but the main ring is one FIFO shared with the debug stream, so the rows that remained were still evicted within seconds. The store therefore keeps two protected lanes beside it, and snapshot() merges all three deduped:

ring	size	holds
`_events`	200	everything, including debug
`_error_events`	64	errors + fatals, plus `skill_failure` even when downgraded to warn
`_headline_events`	64	the remaining non-debug rows (`deploy.bringup`, `reasoner.tick`, `world.scene_objects` …)

The two lanes are separate on purpose. Mirroring all non-debug traffic into a single lane lets routine info evict the safety events the protected 64 slots exist for: world.scene_objects alone ran at ~0.10/s (now change-gated to a 60 s keepalive, but the isolation argument still holds), enough to fully cycle a shared lane in ~11 minutes of an idle scene — long enough to evict a safety.violation before it is read. One budget each means neither class can starve the other.

Span events

Attached to whichever span is active. Counted in the dashboard's Command band where noted.

Event	Emitted at	Meaning	Band
`openral.event.safety_violation`	`runner/base.py`, `deploy_runner.py`, `lifecycle_kernel.cpp`	Safety check rejected an action	error, counted
`openral.event.deadline_missed`	`runner/base.py`	Tick overran its cadence budget	error, counted
`openral.event.skill_failure`	`reasoner_node.py`	Reasoner-published skill failure	error (→ warn while e-stop-latched), counted
`openral.event.sensor_stale`	`deploy_runner.py`	Sensor older than its age budget	counted
`openral.event.staleness_latched`	`world_state/aggregator.py`	World-state component went stale — only for a component that has had data. A never-received one is stale in `diag` and counted in `components_stale`, but has not latched: it never had a fresh state to fall from. Suppressing it removes a WARN that fired on every bringup (`world_state` subscribes before the HAL's first `joint_state`).	—
`openral.event.error_latched`	`world_state/aggregator.py`	World-state component latched an error	—
`openral.event.estop_requested`	`hal/lifecycle.py::_emit_estop_telemetry`	E-stop latched at the HAL	error, counted, protected lane
`openral.event.action_dropped`	—	not emitted	—
`openral.event.chunk_prefetch_hit`	—	not emitted	—
`openral.event.chunk_prefetch_miss`	—	not emitted	—
`openral.event.episode_closed`	—	not emitted	—

Metrics

Exported every OPENRAL_OTEL_METRIC_INTERVAL_MS (default 5000). Label vocabulary is a closed set (semconv.py); threshold hints ride as data-point attributes and are stripped by the store so they cannot fragment a series.

Metric	Type	Unit	Recorded at	Frequency
`openral.tick.duration`	Histogram	ms	`runner/base.py`	30 Hz — library-runner path only, see below
`openral.inference.duration`	Histogram	ms	`observability/tracing.py::inference_span`	per inference, every adapter; label `kind` ∈ `foreground` \| `prefetch` \| `single` (`InferenceKind`, mypy-enforced) — a timing axis; chunk shape rides the span's `inference.chunk_size`, not the label
`openral.hal.read_state.duration`	Histogram	ms	`hal/lifecycle.py::_hal_duration_metric` + `deploy_runner.py`	30 Hz, both paths
`openral.hal.send_action.duration`	Histogram	ms	`hal/lifecycle.py::_hal_duration_metric` + `deploy_runner.py`	per chunk, both paths
`openral.sensors.age_ms`	Histogram	ms	`deploy_runner.py`	30 Hz × camera
`openral.world_state.staleness_ms`	Histogram	ms	`world_state/aggregator.py`	30 Hz × component
`openral.tick.budget_violations`	Counter	—	`runner/base.py`	on breach
`openral.tick.deadline_misses`	Counter	—	`runner/base.py`	on breach — every miss, unlike the rate-limited log
`openral.safety.violations`	Counter	—	`runner/base.py`, `deploy_runner.py`	on violation
`openral.sensors.stale_reads`	Counter	—	`deploy_runner.py`	on stale read
`openral.sim.episode.count`	Counter	—	`sim/sim_runner.py`	per episode
`openral.sim.episode.success`	Counter	—	`sim/sim_runner.py`	per successful episode
`openral.world_state.components_stale`	UpDownCounter	—	`world_state/aggregator.py`	30 Hz
`openral.system.cpu.utilization_pct`	UpDownCounter	%	`system_metrics.py`	1 Hz
`openral.system.ram.used_mb` / `.total_mb`	UpDownCounter	MBy	`system_metrics.py`	1 Hz
`openral.system.gpu.memory_used_mb` / `.total_mb`	UpDownCounter	MBy	`system_metrics.py`	1 Hz
`openral.system.gpu.utilization_pct`	UpDownCounter	%	`system_metrics.py`	1 Hz
`openral.hal.estop.count`	Counter	—	`hal/lifecycle.py::_emit_estop_telemetry`	per e-stop (label `hal.adapter`)
`openral.observability.export_failures`	Counter	—	`_sdk._FailureCountingSpanExporter`	per failed span-export batch (label `signal_kind=trace`)

Latency metrics are emitted by the span helpers, not their callers. inference.duration comes from inference_span; the two hal.*.duration histograms from _hal_duration_metric, both paired with the spans in the shared HAL lifecycle base rather than recorded only inside openral_runner.InferenceRunnerBase / DeployRunner (which the ROS deploy graph never instantiates) — so the metric now works under both runtimes. openral.tick.duration is deliberately left alone: its unit is the library runner's whole tick (sensors + HAL + skill + safety), and no such unit exists on the ROS graph, where those are four separate processes.

InferenceRunnerBase used to also record inference.duration, from result.inference_ms — but that is Skill.step wall-time (the chunk dispatch cost, near-zero when a ChunkedExecutor replays a cached action), not the inference itself. Same instrument, two meanings, two disjoint label sets ({rskill.id} vs {kind}), which doubled the eval/sim path's sample count and computed p95 over a mixed population. It was removed; the dispatch cost now lives only on the tick span as rskill.inference_ms and in the run summary's avg/p99.

A single seam is only safe while every adapter uses it. tests/unit/test_every_adapter_is_instrumented.py is a source-level canary that fails if an adapter defines step() without reaching inference_span, run_inference, or build_chunk_executor (whose ChunkedExecutor calls run_inference for you) — it has already caught sidecar adapters that were reporting latency only through the runner's proxy.

The system metrics come from a 1 Hz daemon thread (start_system_metrics_collector) started automatically by configure_observability; it is a quiet no-op on hosts without psutil / pynvml.

Logs

All OpenRAL logging is structlog, bridged to OTLP. Records below OPENRAL_LOG_LEVEL (default INFO) are dropped before rendering or export.

Hot-path sites worth knowing

Site	Level	Fires at	Notes
`runner/base.py` `inference_runner.deadline_missed[.drop]`	warning	up to 30 Hz	Rate-limited to one line per 5 s carrying `suppressed_since_last` + `worst_tick_ms`. WARNING is the one band an operator cannot filter away, and a too-slow host misses on every tick.
`world_state/aggregator.py` `world_state.*.updated` (8 sites)	debug	30 Hz	Below the default floor.
`rskill/base.py` `skill.step`	debug	30 Hz	Below the default floor.
`runner/safety.py` `safety.null_check`	debug	30 Hz	Below the default floor.
`reward_monitor_node.py` reward score	info	0.5 Hz	Also on the rSkill card's reward bar.
`reasoner/core.py` `reasoner.tick.selected`	info	per LLM tick	Rare and high-value — the tool the model chose.
`reasoner_node.py` tick-suppressed reasons	debug	sub-second	Correctly demoted.
`ros_image_detector_node.py` "continuous leg alive"	info	per frame	Already `throttle_duration_sec=5.0`.
`dashboard/app.py` write-control audit trail	warning	per operator write	Audit-by-design; keep at WARNING.
`rskill/_diagnostics.py` `phase_timer`	info	start/done + 15 s heartbeat	The model-load progress signal — see below.
`hal/sim_attached.py` `sim.task_success` / `sim.task_success_final`	info	per verdict change / 1 per session	Deploy-sim ground truth — see below.
`hal/sim_attached.py` `sim.placement_grade` / `sim.placement_grade_final` / `sim.placement_grade_unavailable`	info (unavailable: warning when `set_task` raised)	per grade change / 1 per graded goal / 1 per ungraded goal	Deploy-sim placement grade of the dispatched goal — see below.
`hal/sim_attached.py` `sim.time_base` (logger `openral.sim.time_base`)	info	1 per connect	The deploy-sim time base: `sim_dt_per_tick_s` (sim seconds one `env.step` advances), `control_freq_hz`, `backend`, `robot` — emitted for a backend that declares its tick (the Isaac sidecar, RoboCasa, LIBERO) after `connect` has verified it equals one control period, and by the bare MuJoCo twin (`MujocoArmHAL`, plus `physics_dt_s` / `physics_steps`) once it has derived one control period per action. A trace replayed later knows which clock its velocities were measured on (issues #355, #358). Companions: `hal.time_base.pinned` / `hal.time_base.unresolved` (warning, logger `openral_hal._mujoco_arm`) mark a twin that is not timing-faithful (a pinned `settle_steps`, or a manifest without a rate); `hal.time_base.timestep_shrunk` / `robosuite.time_base.timestep_shrunk` (info) record a physics time step shrunk to divide the period.

The deploy-sim task-success signal

openral deploy sim runs its backend continuously, which suppresses the simulator's own per-step task evaluation. So that the deploy stack can still state whether the scene's task (e.g. "cup placed in the sink") was completed, SimAttachedHAL polls the backend's own predicate (RoboCasa's per-task env._check_success(); openral_sim.rollout.SimRollout's optional task_success extension) once per env.step and reports it:

Line	Fires	Carries
`sim.task_success`	every change of the verdict, both edges	`success`, `previous`, `first_success`, `scene_id`, `task_id`, `sim_time_ns`, `sim_time_s`, `step`
`sim.task_success_final`	once, at `disconnect` — reached by the lifecycle cleanup/shutdown transition and by `HALLifecycleNodeBase.shutdown_hal` on a signal teardown	`success`, `ever_succeeded`, `first_success_sim_time_ns`, `transitions`, `scene_id`, `task_id`, `sim_time_ns`, `steps`
`sim.task_success_probe_failed`	once, if the backend predicate raises	`error`, `error_type` — polling then latches off (no flood, no crash)

Both edges are logged because RoboCasa success is not latched: an object can be knocked back out of its receptacle, and a rising-edge-only record would claim a success no longer held. A backend with no success predicate emits nothing at all — "unknown" is never reported as failure.

An Isaac scene with placement_success is also graded per goal (see sim environments): the HAL follows /openral/reward/active_task and the sidecar grades the dispatched prompt's placements. Same logger, same stdout mirror:

Line	Fires	Carries
`sim.placement_grade`	every change of the graded prompt's grade (also while the object settles after the goal)	`prompt`, `grade`, `score`, `previous`, `goal_active`, `scene_id`, `task_id`, `sim_time_ns`, `sim_time_s`, `step`
`sim.placement_grade_final`	once per graded goal, at its end (`""` or the next prompt) or at `disconnect`	`prompt`, `grade`, `score`, `best_grade`, `best_score`, `scene_id`, `task_id`, `sim_time_ns`, `steps`
`sim.placement_grade_unavailable`	once per goal whose prompt the scene does not grade (`graded_prompts`), or once at WARNING if the backend's `set_task` raised (`error`, `error_type`; grading then latches off)	`prompt`

The grade never ends a deploy episode; it is a report.

The terminal line reaches both teardown paths. rclpy answers SIGINT by shutting the context down and raising out of spin without requesting the lifecycle shutdown transition — the way openral deploy sim ends every session (_terminate_launch_group SIGINTs the launch's process group). HALLifecycleNodeBase.shutdown_hal, called from the finally of both HAL main() factories, covers that path too; it is idempotent with the transition path, so exactly one line is emitted either way. A SIGKILLed process still emits nothing — the signal is uncatchable.

The logger name is openral.sim.task_success, deliberately dotted under openral. so it propagates to the stdlib logger the OTel bridge is attached to and reaches the dashboard Event Log. Each line is also mirrored to stdout as <event> <json> (matching sim.estop_ground_truth_snapshot), because the structlog record is exported but never printed in a launched HAL subprocess — and a verdict that exists only inside the collector is not recoverable from a validation run's artifacts.

Observability only: the verdict reaches the log and nothing else. It never feeds termination, reset, reward, or the action path (CLAUDE.md §1.4).

The E-stop ground-truth signals

SimSensorBridge records every kernel stop, carried payload or not. Three lines, all joined by stop_seq, all error-level, all emitted through the node logger (so they land in a launched HAL subprocess's own stdout):

Line	Fires	Carries
`sim.estop_ground_truth_snapshot`	every `/openral/estop`, when MuJoCo handles are live — or, on Isaac, when the env serves `contact_ground_truth`	`estop_ground_truth_snapshot(...)` (contacts, `nearest_*_pairs` probes + coverage, joint state, base TF) + `stop_seq` + the 3-deep `candidate_action_chunks` ring + the kernel's `CollisionEvidence` when it landed within 0.5 s. On Isaac: `isaac_estop_ground_truth_record(...)` instead (`backend: "isaacsim"`, PhysX `robot_world_contacts` with `min_separation_m`, `max_penetration_m`, no near-miss probes — see its `contacts_caveat`)
`sim.estop_ground_truth_evidence`	only when the evidence lost the publish race	`stop_seq`, `collision_evidence` — the snapshot is never delayed for it, since sim state must be captured at the stop instant
`sim.estop_initial_configuration`	only when the stop fired before any action reached the HAL	`violation: "initial_configuration"`, `stop_seq`, `candidate_chunks_seen`, `sim_time_s`, `nearest_robot_world_pair`, `detail`

The kernel's own CollisionEvidence additionally carries joint_positions_rad — the configuration forward kinematics was actually run on for the reported horizon_step, serialized at max_digits10 so it round-trips exactly. For a predicted step that configuration exists in no other artifact, so without it such a stop can only be adjudicated against the measured joints, which is a different pose.

Each pair inside nearest_*_pairs carries its own proof, not just a number: distance_m, distance_certified, distance_method, witness_a_xyz / witness_b_xyz, and distance_bracket_m when the geometry is bracketed rather than exact. Each coverage block adds certified_pairs, uncertified_pairs and distance_instrument. This is because mujoco.mj_geomDistance — which every probe before 2026-08-25 used — returns confidently wrong values for RoboCasa fixture geoms against panda_mobile collision meshes in two silent modes, and a reader must be able to tell a defensible distance from an undefendable one without re-running the round. A verdict may rest only on a certified pair (tools/validation_matrix.py::probe_is_distance_certified); see the 2026-08-25 correction.

The third line exists because the first one, read alone, is indistinguishable from an ordinary mid-task stop. SimAttachedHAL.last_action_ns is 0 until the first send_action — the single choke point every real action passes, stamped before any early return — so last_action_ns == 0 at a stop means the configuration the kernel refused is the one the scene reset produced. The robot is not doing something unsafe; it was spawned somewhere unsafe, and no policy, chunk, or margin change can clear it: the remedy is a scene-config one (a different seed, or pinned backend_options.layout_ids / style_ids).

candidate_chunks_seen is reported but deliberately does not gate the classification: a chunk the kernel rejected never reached send_action, so that stop is still at the initial configuration.

Diagnostics only (CLAUDE.md §1.4). The stop itself is correct and is neither suppressed, delayed, nor altered — an initial pose that interpenetrates the scene is exactly what the kernel exists to refuse. All the line does is name it.

Graded velocity scaling

Off by default (collision_scale_proximity_m: 0.0). When armed, an accepted chunk near an obstacle is republished with its rate columns scaled, and that is disclosed three ways:

Signal	Where	Carries
`safety.velocity_scale`	`safety.check` span attribute	the scale in `(0, 1]` applied to this chunk
`safety.scale_slack_m`	`safety.check` span attribute	clearance above the gate margin that produced it
`safety.collision_scaled`	kernel log, transition-gated	scale, slack, band, control mode, `rskill_id` — emitted when the scale moves by ≥ 0.1 or the band is cleared, not per chunk (the accept path runs at 30–200 Hz)
`scaled`	`/diagnostics` key-value, 1 Hz	count of chunks republished slower. A subset of `passed`, never of `dropped` — a scaled chunk was accepted and executed.

The /diagnostics counter is the continuous signal; the log line is the transition. Reading only the log undercounts, by design.

The RoboCasa scene composition

robocasa_scene_composition — structlog, info, once per reset, emitted by openral_sim.backends.robocasa._RoboCasaSim.

Field	Meaning
`scene_id`	the scene that was built, e.g. `robocasa/PickPlaceFridgeShelfToDrawer`
`effective_layout_id`	the kitchen floor plan RoboCasa actually composed, 1..60
`effective_style_id`	the style it actually composed, 1..60, or `"custom"` for a style dict
`pinned_layout_id` / `pinned_style_id`	what the scene YAML pinned, or `null` when it left the pool open

RoboCasa redraws (layout_id, style_id) from env.rng inside Kitchen._load_model on every reset, so the composition is a property of the episode, not of the built env — which is why the line is emitted at reset rather than at scene-factory time.

It exists because the seed alone does not identify the kitchen. Layout, style, fixtures, rack_index, door-open amounts and the robot spawn deviation are all drawn from one shared env.rng, so any upstream change to the draw order reshuffles the scene at the same seed. Without this line a round's artifacts never state which of the 60 kitchens ran, and two rounds are not comparable even when both say seed: 1.

When the YAML pinned exactly one layout or style, a mismatch between pinned and effective is a ROSConfigError, not a log line. The pin can be silently overridden two ways upstream — the task's EXCLUDE_LAYOUTS / EXCLUDE_STYLES filters the pinned id out of the pool, or a replayed _ep_meta carries its own layout_id / style_id and wins outright — and both would otherwise run a different kitchen than the scene file describes while every artifact still shows the declared one.

Load-phase instrumentation

openral_rskill._diagnostics.phase_timer(name, *, prefix, gpu_mb=…) wraps every model-load phase and emits <prefix>_<name>_{start,heartbeat,done} with elapsed_s, plus rss_mb and a major-fault delta, and optionally gpu_mb. Heartbeat every 15 s so a long load is visibly progressing rather than hung.

It has a process-global side effect. Entering the context manager raises sys.setswitchinterval to 50 ms and restores it on exit. Load phases were being starved by 30 fps camera threads; peers still get the GIL ~20×/s, well inside the safety kernel's 1 s staleness deadline. Do not wrap a steady-state phase with it.

Standard phase names: imports, from_pretrained, to_device, processor_dir, make_processors, plus family-specific quantisation/compile phases. Per-adapter shortcuts live in the sim policies (_smolvla_phase, _pi05_phase, _molmoact2_phase, _groot_phase).

runtime_node additionally emits prewarm_vla_framework_* for the one-time torch + lerobot import, which must run before any camera reader thread exists — see docs/methods/11-ros2-nodes.md.

Offline phase breakdown for a single skill:

uv run tools/profile_policy_load.py --rskill rskills/<dir>

Other instrumentation

Mechanism	Where	Default
`DiagnosticsHeartbeat` → `/diagnostics`	`observability/diagnostics.py`	1 Hz per lifecycle node
FailureTrigger bus → `/openral/failure/*`	`observability/failure_bus.py`	token-bucket rate-limited
LTTng tracepoints (8 pairs)	`observability/tracing_lttng.py`	off; `OPENRAL_ROS2_TRACING=1`, <300 ns when off
rosbag2/mcap ↔ OTel join	`observability/replay/`	`openral replay`
W3C traceparent propagation	`observability/propagation.py`	cross-process, parent trace preserved

Declared but not emitted

These names exist in semconv / metrics but nothing produces them. They are kept because each records an intended contract, and annotated in-place so the modules do not read as an inventory of what actually works. A source-scanning test (python/observability/tests/test_declared_not_emitted.py) pins this list both ways, so wiring one up — or adding another — fails until this table is updated.

Signal	Why it is not just a missing `add(1)`
`openral.event.action_dropped`	`DeadlineOverrunPolicy.DROP` does drop the action, but reports via `deadline_missed` instead.
`openral.event.chunk_prefetch_hit` / `_miss`	Action-chunk prefetch runs but never reports its hit rate, so "is the prefetch helping?" cannot be answered from a trace.
`openral.event.episode_closed`	Intended to let a Jaeger query pivot from a skill execution to the produced dataset row; `RolloutRecorder` closes episodes without it, so that pivot does not work.

Removed: openral.safety.clamps / safety.clamped. The attribute was always False — no code path ever set it True. That is not an oversight: the envelope layer is deny-by-default, and compute_intersection "rejects (never clamps)" any envelope field that would loosen the robot ceiling. The counter measured an operation the system never performs, at a constant cost on every safety.check span at 30 Hz. (HAL adapters do saturate commands into an actuator's physical range — the Franka gripper maps [0, 1] onto its travel — but that is device range-mapping, not a safety correction.)

With graded velocity scaling, the blanket claim "OpenRAL never clamps" is no longer true, and the replacement is deliberately a different signal rather than a revival of this one. Distance-graded velocity scaling does modify a commanded value: an accepted chunk near an obstacle is republished with its rate columns multiplied by a scale in (0, 1], reported by safety.velocity_scale / safety.scale_slack_m (see Graded velocity scaling), which carry how much and why — safety.clamped carried only a boolean. The envelope layer still never clamps; a second, separately-named mechanism now sits beside it.

The e-stop latch self-clears. topics.safety.estopped clears only on a clean pass from a real kernel (safety.severity == "ok"; the C++ kernel sends info on a pass, warn while latched, violation on a drop) — proof the kernel itself is not latched (it returns early with estop_latched while fault_latch_ is set). The null client is excluded: it emits info unconditionally without checking anything, so treating that as evidence of a clear would let a no-op client unlatch the UI. Dashboard-side only — no safety code changed.

Configuration

Variable	Default	Effect
`OTEL_EXPORTER_OTLP_ENDPOINT`	unset → full no-op	Where traces, metrics and logs go
`OTEL_EXPORTER_OTLP_PROTOCOL`	gRPC	`http/protobuf` for the in-tree dashboard
`OPENRAL_LOG_LEVEL`	`INFO`	Log-record floor. Names or integers; unparseable falls back to the default rather than raising. Does not affect span bands.
`OPENRAL_OTEL_SAMPLE_RATIO`	always-on	Head-based ratio; `deploy`/`connect` set 0.1 in the CLI process
`OPENRAL_OTEL_METRIC_INTERVAL_MS`	5000	Metric export interval
`OPENRAL_OTEL_SPAN_SCHEDULE_DELAY_MS`	30	`BatchSpanProcessor` flush — ~1.3× the thumbnail rate so the dashboard does not alias
`OPENRAL_ROS2_TRACING`	off	LTTng tracepoints
`OPENRAL_JAEGER_UI_URL`	unset	Enables the trace deep-link in the dashboard header
`OPENRAL_DASHBOARD_WRITE_CONTROLS`	off	Enables `POST /api/skill/execute` and `/api/param/set`

Sampling note: the 0.1 hardware ratio applies to the CLI process only. HAL and other nodes call configure_observability() without a ratio, so their spans are always-on even on real hardware.

Where it goes

Sink	Transport	Notes
In-tree dashboard	OTLP/HTTP → `127.0.0.1:4318`	Default for `deploy sim`; SSE to the browser; MJPEG re-serve of thumbnails
otel-collector-contrib	OTLP → `:4317`	`docker-compose.dev.yml`; traces→Jaeger, metrics→Prometheus, logs→debug
Jaeger	via collector	UI on `:16686`
Prometheus	scrapes `otelcol:8889`	5 s interval, 2 h retention
C++ safety kernel	its own OTLP/HTTP exporter	service `openral_safety_kernel`; no sampler configured (always-on)
rosbag2 / mcap	`openral replay`	joins bag ↔ `/api/traces`

Service names on the wire: openral (CLI), openral.runtime, openral.hal.<robot>, openral.world_state, openral.reasoner, openral.safety, openral.prompt_router, openral.reward_monitor, openral_safety_kernel.