LLMOps and evals

Peak · emerged March 2023 · high July 2026 · +5% on the month before · 5 mentions on Trending in 14 days

Measuring, tracing and monitoring model-backed software: eval suites, LLM observability and prompt management. Named by OpenAI: OpenAI Evals: a framework for evaluating LLMs (Mar 14, 2023). Also called LLMOps, LLM evals, Evals, LLM observability, LLM evaluation, LLM-as-a-judge.

Index 0 to 100, monthly

Signals

Signal	Value	Reads as
Hacker News stories, last 12 months227 stories, +67% on the 12 months before	227 stories, +67% on the 12 months before	Adoption
arXiv papers, last 12 months1,374 papers, +85% on the 12 months before	1,374 papers, +85% on the 12 months before	Adoption
Product releases naming it8 since Sep 1, 2025	8 since Sep 1, 2025	Adoption
GitHub repositories4,191	4,191	Adoption

Companies

Name	Mentions
Vercel	1
ClickHouse	1
Palantir	1
CoreWeave	1
Anthropic	1

Releases

8 releases since Sep 1, 2025

Name	Date
Langfuse 4.44.0Langfuse · Sep 24, 2026 yesterday	Sep 24, 2026yesterday
Langfuse 4.43.0Langfuse · Sep 23, 2026 2 days ago	Sep 23, 20262 days ago
Run Terminal-Bench and other Harbor evals on Vercel SandboxVercel · Sep 17, 2026 8 days ago	Sep 17, 20268 days ago
What's new in ClickStack - Aug ’26ClickHouse · Sep 16, 2026 9 days ago	Sep 16, 20269 days ago
LLM evaluation suites in Pipeline Builder are generally availablePalantir · Sep 10, 2026 2 weeks ago	Sep 10, 20262 weeks ago
W&B Weave 0.53.7CoreWeave · Aug 27, 2026 4 weeks ago	Aug 27, 20264 weeks ago
🧪 Introducing EvalsHex · Aug 4, 2026 7 weeks ago	Aug 4, 20267 weeks ago
Legacy Claude Console Workbench sunsets on August 17, 2026Anthropic · Jul 17, 2026 2 months ago	Jul 17, 20262 months ago

GitHub projects

4,191 repositories

Name	Created	Stars
confident-ai/deepevalThe LLM Evaluation Framework · Aug 10, 2023 3 years ago	Aug 10, 20233 years ago	18,433
open-compass/opencompassOpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, acros · Jun 15, 2023 3 years ago	Jun 15, 20233 years ago	7,472
langwatch/langwatchThe platform for LLM evaluations and AI agent testing · Sep 9, 2023 3 years ago	Sep 9, 20233 years ago	4,872
huggingface/evaluation-guidebookSharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and · Oct 9, 2024 1 year ago	Oct 9, 20241 year ago	2,147
tjunlp-lab/Awesome-LLMs-Evaluation-PapersThe papers are organized according to our survey: Evaluating Large Language Models: A Comprehensive Survey. · Oct 29, 2023 2 years ago	Oct 29, 20232 years ago	810
allenai/olmesReproducible, flexible LLM evaluations · Nov 2, 2024 1 year ago	Nov 2, 20241 year ago	395

Movement	Stage
AI agentsModels that plan and act in steps, calling tools and other services until a task is done.	Peak
Synthetic dataTraining and evaluation data generated by models or simulators instead of collected from people.	Declining
Context engineeringDeciding what goes into a model's context window, and in what order: instructions, retrieved documents, tools and memory.	Plateau

History

Date	What changed
Sep 24, 2026	Tracking started: emerged March 2023, peak, 6 other names recorded

Sources: Curve: Hacker News story titles, Wikipedia pageviews and arXiv papers by month, refreshed Sep 24, 2026. Related entities from the fru.dev sites' public APIs, matched by name. How stages work.

Signals

Companies

Releases

GitHub projects

Related movements

History

Trending terms by email