01 Benchmarks
Measured, not promised.
We ran production regna-1 against RouterBench, the public routing benchmark introduced by Martian. Every number below is a measurement from 2026-07-13, with the methodology and its limits stated in full.
94.2%
overall accuracy (n=400)
$0.01
average cost per question
0
API errors across the run
02 MMLU
MMLU sample, 57 subjects.
200 questions sampled across all 57 MMLU subjects in RouterBench. Reference scores are the original benchmark's recorded results for its 2024-era models, computed on the same 200 samples.
| Model | Accuracy |
|---|---|
| regna-1 (2026-07) | 93.0% |
| gpt-4-1106-preview | 86.0% |
| gpt-3.5-turbo-1106 | 73.5% |
| claude-v2 | 68.5% |
| mixtral-8x7b-chat | 68.0% |
03 ARC-Challenge
ARC-Challenge sample.
200 questions from the ARC-Challenge science reasoning set, same protocol.
| Model | Accuracy |
|---|---|
| gpt-4-1106-preview | 97.5% |
| regna-1 (2026-07) | 95.5% |
| claude-v2 | 90.0% |
| mixtral-8x7b-chat | 85.0% |
| gpt-3.5-turbo-1106 | 83.0% |
04 Methodology
How we measured.
01 · Dataset
RouterBench 0-shot (withmartian/routerbench). We sampled 200 MMLU and 200 ARC-Challenge prompts with a fixed random seed (42).
02 · Grading
RouterBench does not ship gold answers, so we recovered them by majority vote over the reference models' responses that the benchmark scored as correct. Gold was recoverable for 95.5% of MMLU and 99.0% of ARC items; unrecoverable items were excluded before sampling, which removes the hardest tail for all models equally.
03 · Run
Production regna-1 API, temperature 0, max_tokens 1500, 2026-07-13. All 400 requests succeeded. Average billed cost: 1 credit ($0.01) per question. Answers were parsed as letter choices; parse failure counted as wrong (there were none).
04 · What this does and does not show
The reference models are the ones RouterBench shipped with, from 2024. regna-1 orchestrates newer-generation models, so this is not an era-adjusted model comparison. What it shows is what you actually buy: the delivered accuracy and the billed cost of one regna-1 call on standard benchmark questions, measured end to end through the production gateway, including routing overhead.
05 · Reproducibility
Evaluation scripts and raw per-question results are archived in our repository and available on request: support@regnax.ai.
05 Try it