"Trillions of dollars are lost each year due to system outages," the researchers write. The benchmark tests whether AI can actually help change that.
“Despite the central role of such question-driven analysis in incident response, it remains unclear whether modern foundation models can reliably answer the kinds of time series questions engineers ask in practice,” the paper reads.
Questions come in three tiers. Tier I: Does an anomaly exist in this chart? Tier II: When did it start, how severe is it, what type?
The Tier III—the hardest—requires cross-metric reasoning: Is this chart causing the problem in that other chart? That's where AI falls apart. GPT-5 scores just 47.5% F1 on Tier III questions, a metric that penalizes models for gaming answers by picking the most common class.
"Despite the central role of such question-driven analysis in incident response, it remains unclear whether modern foundation models can reliably answer the kinds of time series questions engineers ask in practice," the researchers write.
How every model stacked upGPT-5 led all existing models at 62.7% accuracy—on a test where random guessing gets 24.5%. Gemini 3 Pro scored 58.1%. Claude Opus 4.6: 54.8%. Claude Sonnet 4.5: 47.2%.
Domain experts scored 72.7% accuracy. Non-domain experts—time series researchers at Datadog without extensive observability experience—still hit 69.7%.
No AI model beat either human baseline.
Image built by Decrypt based on the ARFBench leaderboard CSVThe model that actually topped the full leaderboard was Datadog's own hybrid: Toto—their internal time series forecasting model—combined with Qwen3-VL 32B. Toto-1.0-QA-Experimental scored 63.9% accuracy, edging past GPT-5 while using a fraction of its parameters. On anomaly identification specifically, it outperformed every other model by at least 8.8 percentage points in F1.
A purpose-built domain model, trained on observability data, outperforming a frontier general-purpose system at this specific task is the expected outcome. That's the point.
The most valuable finding isn't which model scored highest.
"We observe substantially different error profiles between leading models and human experts, suggesting that their strengths are complementary," the researchers write. Models hallucinate, miss metadata, and lose domain context. Humans misread precise timestamps and occasionally fail on complex instructions. The mistakes barely overlap.
Model a theoretical "Model-Expert Oracle"—a perfect judge that always picks the right answer between the AI and the human—and you get 87.2% accuracy and 82.8% F1. Way above either alone.
