Microsoft pushes a cyber model to 95.95% on CyberGym, Moonshot open-sources the RL sandbox behind Kimi K3, Perplexity puts search in the terminal, and Black Forest Labs unifies image, video, audio, and robot actions in one flow model. The cyber race and multimodality both leveled up. Datalab's Marker 2 vs MinerU, Docling & LiteParse: The Doc-Parsing Benchmark
Microsoft's first cyber model, MAI-Cyber-1-Flash, is a sparse MoE with 137B total / 5B active parameters that pushes its MDASH security agent to 95.95% on CyberGym. It's built to handle up to 90% of MDASH tasks, reserving GPT-5.4 for the hardest 10%.
So what: A week after Sakana's Fugu-Cyber, Microsoft answers with an even higher CyberGym score — the specialist-cyber-model race is now a weekly leaderboard. The 5B-active MoE design is the tell: cheap enough to run on the 90% of security work that's routine, escalating only the hard cases.
A head-to-head of open PDF-to-markdown parsers. Marker 2 scores 76.0 on olmOCR-bench at 2.9 pages/sec — matching Gemini/MinerU quality while running ~5x faster, beating MinerU and Docling on score and throughput at once.
So what: Document parsing is the silent failure point of most RAG stacks — bad extraction poisons everything downstream. Winning quality and speed simultaneously makes Marker 2 the new default front door for turning PDFs into clean model input.
sponsored
Moonshot's Kimi team and kvcache-ai open-sourced AgentENV (AENV) under MIT — a distributed platform for running agent environments at scale. It targets the overlooked half of agentic RL: environment throughput, and it powered the agentic RL training for Kimi K3.
So what: Everyone talks about the policy model; almost nobody ships the environment infra that makes agentic RL feasible. Open-sourcing the exact sandbox that trained a frontier model hands the community the missing piece — this is how open agentic RL actually catches up.
Perplexity shipped pplx, a single-binary CLI that puts its Search API results in the terminal as JSON — designed to be invoked by coding agents, not just humans. Priced at $5 per 1,000 requests.
So what: Agents live in the terminal, so meeting them there — JSON out, one binary, no SDK — is the right primitive. This is Perplexity positioning its search as the default retrieval tool inside coding-agent loops, competing on ergonomics rather than a chat UI.
A hands-on build: a stateful agent that runs a venue-operations workflow end to end. MongoDB Atlas for data + vector search, Voyage embeddings for retrieval, and LangGraph to orchestrate the planning / tool-calling loop into a durable graph.
So what: A clean reference for the emerging "database + embeddings + graph orchestration" agent stack — the pattern most production agents are quietly converging on.
sponsored
FLUX 3 is a 32B multimodal flow model (a diffusion transformer) that jointly learns from images, video, and audio into one world representation — generating 20-second video with native audio, 4MP images, and — via FLUX 3 x mimic — predicting robot actions. (model overview)
So what: The generative and world-model tracks are converging. A single flow model that produces media and predicts actions means the same backbone that makes video can drive a robot — the clearest sign yet that "generation" and "control" are becoming one modality.
|
AI Dev Signals · Short Course How to Evaluate an LLM Feature A 4-part course for PMs and researchers shipping to real users |
|
Short Courses › Evaluating an LLM Feature › Ch. 1 of 4 — Benchmarks vs. Your Eval Set |
|
|
Why MMLU tells you nothing about your product Every benchmark score is a measurement. Every measurement has a noise floor. On MMLU, that floor is now wider than the gap between the models you are choosing between. Start there, and the rest of this course follows. |
|
01 What MMLU actually measures MMLU is a multiple-choice exam, published by Hendrycks et al. at ICLR 2021. Four options per question. Scored by exact match on the letter.
Non-expert humans on Mechanical Turk scored 34.5%. In 2020 this was a genuinely hard test. |
|
02 The answer key has bugs in it In 2024 an Edinburgh-led team re-annotated MMLU by hand. They released the corrections as MMLU-Redux. They re-checked 5,700 questions across all 57 subjects. Their estimate: 6.49% of MMLU questions contain errors.
|
|
03 Do the arithmetic once The test split holds 14,079 questions. So one percentage point is about 141 questions. The pool of questions with suspect labels is about 914. Now compare the two on the same scale.
The noise is roughly six times the signal. A vendor slide showing a one-point lead is showing you measurement error. This is not an argument against benchmarks. It is an argument for knowing a benchmark's resolution before you quote it. |
|
04 Now assume the labels were perfect The deeper problem survives the cleanup. MMLU measures recall of academic facts under a four-choice format. Your product is not that.
That last row is the one that matters. A benchmark averages all errors together. Your users do not. |
|
05 Before Chapter 2 Next week you build a golden set. Answer these four first — they are the spec for it.
Question 4 before question 3 is the discipline. Picking the threshold after seeing the score is how teams talk themselves into shipping. |
|
| |||||||
$ ./partner --with us
# Reach 1M+ AI developers, researchers & builders.
# Sponsor a future issue of the Intelligence Brief.
> apply: forms.gle/CY1eqZzuWFQBp7dH9