|
03 · KEENABLE AI · SEARCH BENCHMARKS
Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
A search agent has a fetch tool. If the gold labels sit in a public dataset, the agent can download the answer key mid-evaluation and skip retrieval entirely. NEEDLE’s fix is to never hold still: news queries regenerate hourly from RSS feeds and Google Trends, and finance, scholar, legal and rare-entity queries regenerate daily from SEC XBRL, arXiv, Europe PMC, CourtListener and public agent logs.
Fifteen search APIs run the same query text under one protocol: one request at a time, a 2,000-character evidence cap, no page fetching, no re-ranking, and a judge that never sees the engine name. Every score is read against ultimate, a pooled oracle built from what all fifteen engines together managed to retrieve — which separates a ranking problem from a retrieval problem the whole market shares. Seven-day means ending 2026-08-28: finance is close to solved, Exa at 0.910 against a 0.965 ceiling; deep-tail, the vertical closest to real agent traffic, tops out at 0.557 of its ceiling. MIT licensed, runs execute in public GitHub Actions.
READ ON MARKTECHPOST >>
· Live dashboard
· GitHub
· Artifacts
|