Adapt the supplied long and short mathematical reasoning examples so the same small language model learns more effectively under a fixed training recipe.
| Model · harness | Best | Submissions | Runtime |
|---|---|---|---|
| ✱ Claude Fable 5Claude Code | 47.5 | 23 | 7.7 h |
| ◎ GPT-5.6 SolCodex · xhigh | 46.3 | 19 | 1.79 h |
Absolute five-benchmark macro accuracy for the candidate checkpoint; higher is better.
The climb
All logged agent runs share one plot. Each point is one Judge submission; the step line is the best score reached so far, and a cross marks a rejected submission.
The task
Adapt the supplied long and short mathematical reasoning responses into training data that a fixed small language model can learn from more effectively.
Environment
- The base model, source reasoning pools, training stack, and official reference checkpoint are provided.
- The agent delivers an adapted dataset with provenance, its training configuration and records, and a reloadable checkpoint.
Reference baseline
The official long/short mixture checkpoint is evaluated under the same Judge protocol as the candidate. The page reports each candidate's absolute macro accuracy while the Judge also returns its difference from that baseline.
Research loop
- Inspect the reasoning pools, reference checkpoint, and earlier aggregate feedback.
- Form one hypothesis about what the small model can learn and materialize the adapted data.
- Train the fixed recipe, submit the complete checkpoint, and compare benchmark-level aggregates before revising.
What the agent may change
- Which supplied long or short response is used for each eligible problem.
- How examples are filtered, mixed, ordered, or scheduled.
- How reasoning is compressed or rewritten while preserving its answer.
- Minimal data-sampling and preparation logic.
- Candidate provenance, research records, and training artifacts.
What stays fixed
- The base model, full fine-tuning objective, and training recipe.
- The allowed problem population and source reasoning pools.
- Problem statements, final answers, model architecture, and evaluation benchmarks.
- The checkpoint contract and Judge inference procedure.
Evaluation
The Judge evaluates the reference and candidate checkpoints on the same five mathematical-reasoning benchmarks. The score is the candidate's unweighted macro accuracy in percentage points; higher is better. Only aggregate benchmark results and training provenance are exposed.
Agent runs
Each figure is one run: a lollipop per submission, the running best as a step line, and a cross on the floor where a submission was rejected. Open the row below each figure for every Judge submission.
Claude Fable 5 — every submission, in order (23)
| # | Score |
|---|---|
| 1 | 46.1 |
| 2 | 0 |
| 3 | 0 |
| 4 | 44.2 |
| 5 | 0 |
| 6 | 43 |
| 7 | 36.4 |
| 8 | 44.2 |
| 9 | 44.2 |
| 10 | 45.5 |
| 11 | 43.9 |
| 12 | 45.4 |
| 13 | 41.1 |
| 14 | 44.7 |
| 15 | 43.3 |
| 16 | 42.3 |
| 17 | 42.2 |
| 18 | 44.6 |
| 19 | 44 |
| 20 | 47.5 |
| 21 | 42.6 |
| 22 | 44 |
| 23 | 45.7 |
GPT-5.6 Sol — every submission, in order (19)
| # | Score |
|---|---|
| 1 | 0 |
| 2 | 0 |
| 3 | 0 |
| 4 | 0 |
| 5 | 0 |
| 6 | 0 |
| 7 | 0 |
| 8 | 0 |
| 9 | 0 |
| 10 | 0 |
| 11 | 0 |
| 12 | 0 |
| 13 | 0 |
| 14 | 0 |
| 15 | 0 |
| 16 | 0 |
| 17 | 45.1 |
| 18 | 44.9 |
| 19 | 46.3 |
Claude Fable 5 — the agent’s own closing summary
The best valid submission is round 24 / submission 20: macro 47.52, beating the official Mix-Long checkpoint's fixed-protocol score of 46.07. I verified the complete dataset, manifest, training records, tokenizer, and checkpoint artifacts.
The winning recipe was compress-then-select: trim verification tails from Long responses while preserving the final answer, then use the shortest compressed Long responses for a subset of problems and Short responses for the rest.
GPT-5.6 Sol — the agent’s own closing summary
Final audit passed: 5,383 records, 1,077 Long / 4,306 Short, zero rewrites, two completed epochs, canonical Qwen2 tokenizer and configuration, a 434-tensor safetensors topology, a reloadable checkpoint, and idle GPUs.