---
name: reviewing-a-submission
description: Review a Reactor Bench submission from reactorbench.com - claim it, download it, verify it stage by stage with reactor-bench's verifying skill while reporting each stage's progress, and upload verification.json. Use when given a submission ID or a reactorbench.com/admin/submissions link to verify or review.
---

# Reviewing a submission

You verify one submission on reactorbench.com and report each stage as you go.
A person publishes or rejects it; you never do.

The verifying itself is reactor-bench's
[verifying skill](https://github.com/reactor-team/reactor-bench/blob/main/skills/verifying-a-submission/SKILL.md).
Read it first: it says what each check means and how to reproduce a full
submission. This skill wraps it with the site's review API.

- **Input:** a submission ID (a UUID), and a reviewer API key in
  `REACTOR_BENCH_REVIEW_KEY`. If it is not set, ask the person to mint one on
  https://reactorbench.com/admin. Never print it, log it or write it to a file.
- **Output:** the submission's stages reported, `verification.json` uploaded,
  and a summary for the person.

```sh
API=https://reactorbench.com/api/review
AUTH="Authorization: Bearer $REACTOR_BENCH_REVIEW_KEY"
ID=<submission id>
stage() {  # stage <id> <status> [detail] [done] [total]
  curl -fsS -X PUT "$API/submissions/$ID/stages/$1" -H "$AUTH" -H 'content-type: application/json' \
    -d "$(jq -n --arg s "$2" --arg d "${3:-}" --argjson n "${4:-null}" --argjson t "${5:-null}" \
          '{status: $s, detail: $d, done: $n, total: $t}')"
}
```

A stage is `pending`, `running`, `completed`, `failed` or `skipped`. Report
`running` when you start it and `completed` or `failed` when it ends, with what
you found in `detail`; add `done` and `total` for anything counted in cases.

## 0. Check the key, read the submission, and claim it

```sh
curl -fsS "$API/me" -H "$AUTH"                       # who the key belongs to
curl -fsS "$API/submissions/$ID" -H "$AUTH" | jq '.submission | {type, status, benchmark, track, scorer, source_kind}'
curl -fsS "$API/submissions/$ID" -H "$AUTH" | jq '[.stage_order[].id]'
curl -fsS -X POST "$API/submissions/$ID/claim" -H "$AUTH"
```

The stages, in order, are in `stage_order`: a results-only submission has
`download`, `consistency`, `revisions`, `scrutiny`, `re-scoring-sample`,
`re-scoring-all` and `report`; a full submission also has `serve-model`,
`reproduce-generate` and `reproduce-score` before `report`.

- **Check before moving on:** the claim answered with your email as
  `claimed_by`. A 409 means another reviewer has it: stop and tell the person.

## 1. Download

```sh
stage download running
curl -fsS -X POST "$API/submissions/$ID/download" -H "$AUTH" > /tmp/download.json   # recorded in the history
```

- **A Hugging Face submission:** use `.link` and, for a private dataset,
  `.token` with reactor-bench's `verify(link, token=...)`, which downloads it.
- **An upload:** download every URL in `.downloads` into a work folder on a
  large disk, check each file's SHA-256 against `.files[].sha256` from step 0,
  and unzip `submission.zip`; the folder it holds is what you verify. A full
  submission's `model/workspace.tar.gz` and `model/weights.tar`, when present,
  are what you serve in step 5. A Hugging Face submission can upload them too.
- **Private code or weights:** a full submission whose pinned repositories are
  private carries `.workspaceToken` (reads the code's repository) and
  `.weightsToken` (reads the weights'). Use them only to fetch the model for
  step 5, and never print them.

```sh
stage download completed "<what you downloaded, and the hashes matched>"
```

Delete `/tmp/download.json` once you no longer need its tokens.

## 2. Consistency, revisions and scrutiny

Run `verify(<link or folder>, token=..., rescore=False, data_root=...)`, as
step 1 of the verifying skill says, and report its rows:

- `consistency`: completed when the rows `entry`, `result`, `rollouts`,
  `scored-input` and `served-model` all passed; failed otherwise, with the
  failed rows' detail. On a failure, report it, upload the report (step 6) and
  stop.
- `revisions`: the `revisions` row. When it failed, follow the verifying skill:
  a separate checkout at the submission's Reactor Bench commit with its own
  dataset folder, then run this step again from there.
- `scrutiny`: completed, with every `scrutiny:*` row that did not pass in the
  detail. Scrutiny never fails a submission.

## 3. Re-scoring

```sh
stage re-scoring-sample running "" 0 10
```

Run `verify(..., scoring={"gpus": (...)})` as the verifying skill's step 3 says
(for a full submission, its step 4 does this and reproducing in one call).
Report what the `re-scoring` row says:

- "N sampled cases agree": `re-scoring-sample` completed, `re-scoring-all`
  skipped.
- "The sample of N disagreed, so every case was re-scored": `re-scoring-sample`
  failed with the differing cases, and `re-scoring-all` completed or failed
  with its detail.

Report the scorer's phases in `detail` while they run (WBench: precompute, gpu,
vlm, visual_plausibility, report).

## 4. Skip this step for a results-only submission

A results-only submission pins no model, so it is not reproduced.

## 5. Reproducing a full submission

Follow the verifying skill's step 4, reporting:

- `serve-model`: running while you start the pinned workspace with its weights,
  completed when `curl <endpoint>/health` reports `"state": "available"`.
- `reproduce-generate`: running with `done` and `total` cases as rollouts are
  generated.
- `reproduce-score`: completed when the `reproducing` row passed, failed with its
  detail otherwise.

Stop the served model afterwards.

## 6. Upload verification.json

```python
report.write("verification.json")
```

```sh
curl -fsS -X POST "$API/submissions/$ID/verification" -H "$AUTH" -H 'content-type: application/json' \
  -d @verification.json
```

The answer says whether it was accepted, and why not. It is accepted when it
verified exactly this submission's `entry.json` and `result.json`, it passed,
it was re-scored, and a full submission was reproduced; the submission is then
`verified`, and the site marks `report` completed.

- **Check before moving on:** the answer's `accepted` and `problems`.

## 7. Tell the person

Give them the submission's page (https://reactorbench.com/admin/submissions/<ID>),
whether `verification.json` was accepted, every stage that failed or was
skipped and why, and every scrutiny difference. They publish or reject it on
that page. If you stop before step 6, release the claim:

```sh
curl -fsS -X POST "$API/submissions/$ID/release" -H "$AUTH"
```