Run a benchmark and publish the number

openral sim run answers "does this pairing work?". The benchmark tier answers "how well, under a protocol someone else can repeat?", and it is the only tier that writes a result back into an rSkill package.

Prerequisite: your first sim rollout.

1. Which tier writes what

DeployScene  ⊆  SimScene  ⊆  BenchmarkScene
 deploy sim      sim run      benchmark scene  →  benchmark run --suite

A BenchmarkScene is a SimScene plus the two things a claim needs: a non-None seed and n_episodes, and a metadata block naming the paper and an honest_scope statement. Per-tier loaders reject a wrong-tier YAML at parse time, so you cannot accidentally publish a number from an ad-hoc config.

Command	Input	Writes
`openral sim run`	`scenes/sim/*.yaml`	a summary to `--save-dir`; never the rSkill package
`openral benchmark scene`	`scenes/benchmark/*.yaml`	`rskills/<dir>/eval/scene_<scene_id>.json`
`openral benchmark run --suite`	`benchmarks/*.yaml`	`rskills/<dir>/eval/<suite_id>.json`
`openral benchmark report`	`rskills//eval/.json`	nothing — reads and validates

A suite aggregates N scenes under uniform protocol invariants; a scene is one of them. The suite id is the YAML filename stem.

2. Find a suite

openral benchmark list        # suite ids from benchmarks/*.yaml
openral rskill list           # paste-able --rskill names

The rSkill is a suite's only free axis; robot, scene, task, seed and episode count are pinned by the YAML, so two rSkills on one suite are directly comparable.

3. Dry-run the matrix

openral benchmark run --suite pusht --rskill rskills/diffusion-pusht --dry-run

Prints the planned (task × seed) matrix and exits without a rollout: confirms wiring in CI and shows the episode count (benchmarks/pusht.yaml pins 50). benchmark scene has the same flag.

4. Run the suite

openral benchmark run --suite pusht --rskill rskills/diffusion-pusht

The runner iterates scenes × range(seed, seed + n_episodes), delegating each rollout to the same SimRunner that backs openral sim run, so the compatibility gate, OpenTelemetry spans and latency-budget reporting are identical. Two tracked files change on success; review the diff before committing:

rskills/diffusion-pusht/eval/pusht.json — the full RSkillEvalResult.
rskills/diffusion-pusht/rskill.yaml — the aggregate written back to benchmarks.pusht, as a surgical edit that preserves comments.

When you do not want either write

# Read-only: still write the eval JSON, leave the manifest alone.
openral benchmark run --suite pusht --rskill rskills/diffusion-pusht \
  --no-update-manifest

# Fast iteration: 2 episodes instead of the suite's 50.
openral benchmark run --suite pusht --rskill rskills/diffusion-pusht \
  --n-episodes 2 --no-update-manifest

--n-episodes overrides every scene in the suite. It is for iteration only: a number produced under it is not paper-comparable, because the published protocol is the count in the YAML.

benchmark scene adds a third opt-out, --no-write-eval: the rollout prints its score and nothing is written to the package. It implies --no-update-manifest.

5. The two guardrails

The suite must be internally consistent

openral_core.raise_on_invalid_suite rejects a suite unless every scene shares one non-None robot_id, the same n_episodes and seed, and a byte-identical metadata block, and every task.id is unique. Otherwise an "aggregate" would average across different protocols.

task.success_key and max_steps may differ per scene, because suites like ManiSkill3 legitimately span tasks with different step budgets.

The rSkill must have been trained for the task

A suite is auto-filtered to the tasks the rSkill declares in evaluated_tasks; scenes it does not cover are skipped with a logged benchmark_suite_task_filter summary. Pick a task explicitly with --task that the manifest does not cover and the filter leaves nothing to run, so you get a typed ROSCapabilityMismatch instead of a rollout. benchmark scene applies the same gate to its single scene, and its message says why:

the checkpoint was trained/validated for a different task; running it here yields a plausible-looking rollout that cannot succeed

A policy on the wrong task still emits smooth, confident actions but can never satisfy the success condition, so the 0% reads like a bad model rather than a mispairing. If the checkpoint genuinely covers the task, add it to evaluated_tasks.

A manifest with an empty evaluated_tasks skips the gate and logs rskill_task_compat_undeclared. Declare the field.

6. Read the result

openral benchmark report              # rich table across every rSkill
openral benchmark report --json       # same rows, machine-readable

It validates every rskills/<id>/eval/*.json against RSkillEvalResult (the schema the loader uses at install time), so a rotted file fails the command rather than being skipped. Rows carry the benchmark, robot, simulator, model_variant, status, the results block, and reproduced_locally.

`reproduced_locally` is the honesty switch

A result you produced carries reproduced_locally: true. A paper number you have not reproduced is allowed but must carry reproduced_locally: false plus a reproduction_cli so a reader can run it. The in-tree rskills/diffusion-pusht/eval/pusht.json shows the reproduced shape:

"source": {
  "paper": "https://arxiv.org/abs/2303.04137",
  "model_variant": "diffusion",
  "evaluated_by": "OpenRAL:openral benchmark run",
  "reproduced_locally": true,
  "reproduction_cli": "openral benchmark run --suite pusht --rskill diffusion-pusht",
  "status": "reproduced"
}

RSkillEvalResult also carries a trace_id that deep-links a number to the trace that produced it (paste it into the dashboard's /api/spans/<id>, Jaeger or Tempo). It is the run's cli.command root span, so it resolves to the whole tree, sim.run and every tick underneath. A run with nowhere to send spans leaves it null; pass --dashboard or point OTEL_EXPORTER_OTLP_ENDPOINT at your own collector. Older eval files predate the field and carry null.

Never hand-edit a success rate into a manifest or an eval JSON; the commands write both, and the number must be reproducible from the trace alone.

7. Add a new suite

A suite YAML is a bare list of BenchmarkScenes at the document root; a {id, tasks, metadata} wrapper is rejected with a redirect. YAML anchors keep per-scene fields DRY:

- scene: &scene
    id: pusht
    backend: mujoco
    observation_height: 96
    observation_width: 96
  task:
    id: pusht/0
    scene_id: pusht
    max_steps: 300
    success_key: is_success
  robot_id: pusht_2d
  n_episodes: 50
  seed: 0
  metadata:
    paper: "https://arxiv.org/abs/2303.04137"
    display_name: "PushT (gym-pusht)"
    simulator: "gym-pusht (pymunk 2-D)"
    honest_scope: >
      50 evaluation rollouts on the single PushT-v0 task, matching the
      Diffusion Policy paper's protocol.

honest_scope is required: it states what the number does not cover. A file at benchmarks/<id>.yaml is suite <id>; see benchmarks/README.md for the full invariant list.