Parallel Web Systems has open-sourced parallel-search-evals, the harness it uses to score language models on web search and to run its Search Capability Leaderboard. The framework code is under the MIT licence. The benchmark data isn’t included and keeps its own terms.
Parallel announced the release on X on 17 November, US time, with an article that’s also on its blog, written by James Liounis, Vlad Shulman and Lukas Levert. The GitHub repo was created that day and has a single commit.
What a run does
Each run tests one model on one benchmark in one mode. The modes are Parallel’s four Search settings (turbo, fast, basic and advanced) or a closed-book baseline, which gives the model no tools. Run the baseline and a search mode on the same questions and the gap between them is the lift from search, Parallel says.
In a search run the model gets two Parallel tools. Search returns up to 10 results with dense excerpts, and Extract pulls evidence from up to 20 given URLs. The methodology page allows 20 tool calls and 25 research turns per question, then makes one last call with tools switched off if the model still owes an answer. A judge model, GPT-6 Sol, grades the answer against the reference. Reference answers go to the judge only, never to the model.

Two benchmarks ship as adapters. DeepSearchQA, from Google, has 900 multi-step web research questions across 17 fields and is scored with a fractional F1, which gives partial credit for an incomplete answer set. Humanity’s Last Exam, or HLE, uses 2,158 text-only, expert-written academic questions, with its 342 image questions left out, and is scored on strict correctness. Parallel says you can add your own questions, reference answers and grading rules while the search setup, budgets and reporting stay as they are on its leaderboard.
The model side supports the native OpenAI, Anthropic and Google APIs, OpenRouter, Unbiased and any OpenAI-compatible endpoint. The search side is Parallel only: the README says every search-enabled route uses the same local Parallel tools, and Parallel says the harness can be modified to compare other providers.
The licence, exactly
The repo’s LICENSE file is the standard MIT text, copyright 2026 Parallel. It lets you use, copy, modify, merge, publish, distribute, sublicense and sell the software, as long as the copyright and permission notice stay with copies. It comes with no warranty.
That covers the framework code only. The README says dataset terms and third-party notices apply separately, and the third-party notes spell out what that means. DeepSearchQA’s data is Apache-2.0 on Hugging Face, and the harness doesn’t download it: you fetch the 900-question CSV yourself and the loader checks its checksum. HLE is gated on Hugging Face, and its dataset card asks users not to publicly share, re-upload or distribute it. Run folders hold the selected questions and reference answers, so the notes say they mustn’t be committed, uploaded or distributed. The two judge prompts carry their own licences: Apache-2.0 for the DeepSearchQA prompt, from a Google DeepMind starter notebook, and MIT for the HLE prompt, from the Center for AI Safety’s repo.

What you need, and the bill
You need Python 3.12 or 3.13 and uv, the Python package manager. Credentials go in the environment: one for the model provider, PARALLEL_API_KEY for search runs and OPENAI_API_KEY for the grader. The README says the grader needs the OpenAI key on every route, so a Claude run is still graded by GPT-6 Sol. The operations notes say every model, tool and judge call is paid.
The leaderboard, last updated on 14 November, gives a sense of what Parallel’s own runs cost. It lists Claude Fable 5.2, its top scorer at 80.1, at US$2,426 per 1,000 tasks, and GPT-6.1 Terra, which it calls the most cost-efficient model, at US$43.7. It puts the average lift from search across all models at 34.4 points and names Hy4 as the biggest gainer, at 45.3.

Where the post and the README disagree
Parallel’s blog post and X article show this smoke test:
uv run --frozen parallel-search-evals run \
--provider anthropic --model claude-sonnet-5 \
--benchmark dsqa --mode fast --limit 3 --output runs/dsqa-smoke
The README shows a different command for the same job:
uv run --frozen python -m parallel_search_evals.run_eval \
provider=anthropic model=claude-sonnet-5 suite=dsqa search_engine=parallel-fast \
source=/absolute/path/DSQA-full.csv limit=3 max_workers=2 \
output_dir=runs/dsqa-smoke
The repo’s pyproject.toml declares no parallel-search-evals command, and the README’s version is the only one that names a dataset path, which a run needs because nothing is downloaded for you. I haven’t run either.
What it doesn’t reproduce
The README says a three-question smoke test isn’t a reproduction of published full-benchmark scores. The third-party notes call the repo a lean version of a pinned Parallel reference, with its differences documented, and say it isn’t “a claim of full main parity”. Source-order question selection is one difference they name: it replaces main’s seeded cohort.
The methodology page covers two more limits. A default filter drops Search and Extract results whose URLs match known benchmark source paths. It runs locally after Parallel responds, so it doesn’t stop the service retrieving those sources, and it misses mirrors it doesn’t know about. The page says not to claim a contamination-free evaluation, and setting filter_benchmarks=False turns the filter off. The judge, GPT-6 Sol at medium reasoning, is the harness’s own choice, not a reproduction of the original upstream judge model.
Parallel sells the search product the harness measures, and says so in its own guide to how it evaluates search APIs. That guide listed what its benchmarks page hadn’t published as of 28 September, including the judge model and a tool-call budget. The repo now states both: GPT-6 Sol and 20 calls.
The repo includes an agent skill, under .agents/skills/run-search-evals, that walks Claude Code or Codex through a run, and Parallel asks for bug reports as GitHub issues. It’s at version 0.1.0, and the README’s citation note asks you to record the exact source revision and saved configuration when you report results.
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.




