ReasonIR Difficulty Curriculum

Choose how a fixed pool of easy and hard retrieval examples is weighted and ordered to improve ReasonIR-8B, without weakening its general retrieval ability.

The climb

Both agent runs share one plot. Dots are Judge submissions and the step lines show each run's best score so far. The vertical scale is zoomed to make improvements readable. Hollow downward triangles mark the negative general-retrieval penalties below the displayed range; they are valid scores, not rejected submissions. Exact values remain in the tables below.

The task

Can a better training curriculum strengthen reasoning-intensive retrieval without sacrificing everyday retrieval? The agent starts from ReasonIR-8B and a fixed pool of hard-query (HQ) and varied-length-query (VL) examples. It changes which existing examples the model learns from, how often, and in what order. The source figure explains the original ReasonIR data recipe; this task uses its already-prepared HQ/VL pool and does not generate new data.

Environment

• A pinned ReasonIR-8B checkpoint, tokenizer, and immutable HQ/VL training pool are provided.
• Each example contains a query, a relevant document, and a hard negative: a plausible but irrelevant document.
• The supplied trainer produces a fixed-schema LoRA adapter, a small set of additional weights applied to the unchanged base model.
• The deliverable is a completed adapter and a declarative manifest with its required training records.
• Training and Judge evaluation each use four GPUs, with the same four GPUs reusable in sequence.

Reference baseline

The dashed 0.21141 baseline is the unchanged ReasonIR-8B checkpoint, submitted through the same adapter interface using an all-zero LoRA adapter. Its reasoning score is R0 = 0.21141 and its general retrieval score is G0 = 0.36126. An operator calibration and a separate formal Judge reproduced both values under this task's current protocol.

Every candidate is scored against these fixed references; Judge does not retrain or rerun the baseline beside it. Neither agent's first submission is the reference adapter. The original paper's full-BRIGHT result uses a different evaluation scope and is not this baseline.

Research loop

1. Form a hypothesis about how example difficulty should affect learning.
2. Define a policy over the fixed training pool and complete a matched-budget adapter-training run.
3. Validate and submit the completed adapter with its manifest.
4. Compare reasoning and general retrieval feedback, then retain or revise the policy.

A claim about why a curriculum works should be supported by matched policy trials, not just the best score. This reporting requirement is separate from Judge's scoring rules.

What the agent may change

• How training-example difficulty is estimated from the supplied pool.
• How examples are grouped into difficulty buckets or filtered.
• The nonnegative weights and sampling frequencies assigned to existing examples.
• The deterministic ordering of examples and how the curriculum changes during training.
• Policy implementations, trial configurations, comparisons, and research notes.
• The completed LoRA adapter and its required manifest and training records.

What stays fixed

• The base checkpoint, tokenizer, architecture, and LoRA adapter schema.
• The training pool's text, labels, and example identities; no new or rewritten examples.
• The trainer, objective, optimizer schedule, seed, and fixed budget: 1,000 completed updates and 64,000 attempted triplet exposures.
• The held-out evaluation subjects and datasets, retrieval protocol, and scoring rules.
• The offline environment and the separation between candidate artifacts and task-owned evaluation code.

Evaluation

Judge validates the finished adapter and manifest, then evaluates the candidate without retraining it. Retrieval quality is measured with nDCG@10, which rewards placing relevant documents near the top of the first ten results.

Reasoning score R: the equal-weight mean over three BRIGHT subjects: Biology, Pony, and TheoremQA-theorems.

General retrieval score G: the equal-weight mean over four BEIR datasets: NFCorpus, SciFact, FiQA-2018, and ArguAna.

If G ≥ 0.36126, the reward is R. If general retrieval falls below that reference, the reward is G − 0.36126, a negative penalty. Higher is better in both cases. The means and reward use five decimal places.

The agent receives these aggregate scores, changes from the reference, and per-subject/dataset metrics. The evaluation uses the complete fixed query and candidate sets for the selected subjects; it is a three-subject reasoning proxy, not a full-BRIGHT score. Invalid or incomplete artifacts receive no score, distinct from a valid candidate's negative regression penalty.

Agent runs

Each figure shows one agent's submissions and running best. Hollow downward triangles mark negative general-retrieval penalties outside the zoomed range. Open the row beneath each figure for all submissions, including the exact negative scores.

Links

• Task package (instruction, policy, tests) ↗
• Run logs — trajectories, submissions, judge feedback ↗
• Source project ↗
• Paper ↗