Mintlify open-sources agent-traffic, the classifier it uses to tell agents from people

Mintlify has open-sourced agent-traffic, the dataset and classifier it uses to tell which agent, crawler or person is behind a request to documentation hosted on its platform. The official Mintlify account says agents made 489 million requests last month to docs it hosts, and that agent-traffic is how the company tells those requests apart. Official docs are at agent-traffic.mintlify.site. The 489 million figure is Mintlify’s.

Introducing agent-traffic: the dataset and classifier we use to tell which agent, crawler or person is behind every request.

Last month, agents made 489M requests to docs hosted on Mintlify. agent-traffic is how we know which ones.

Today we’re open-sourcing it for everyone.

— Mintlify (@mintlify) November 10, 2026

What the classifier does

The project is a curated dataset of user agents, headers and official verification sources for real agents and crawlers, plus a small function that applies it. Mintlify says ordinary bot lists often match a user-agent substring, put all automated traffic in one bucket, and take a request at its word. agent-traffic is meant to go further: it checks crawlers against their operators’ published IP ranges and flags impersonators, splits AI traffic into separate categories, and looks at headers when an agent hides behind a stock browser user agent. Every result includes a list of reasons.

The official categories are agent (an AI product fetching a page for a person, with kinds such as assistant or coding), ai-crawler (an AI company collecting pages ahead of time), crawler (a search engine), bot (other automation, including impersonators and HTTP clients), human, and unknown when there’s nothing to classify. The package can also load IP range files published by OpenAI, Anthropic, Google, Perplexity, Apple, Microsoft, Common Crawl and DuckDuckGo.

Mintlify notes some limits. Some products share an identity: the docs say Codex has been reported to fetch as ChatGPT-User, and Cursor through Exa as ExaUser, and that results name only what can be proven. Agentic browsers that run inside a person’s own Chrome, it says, still look like that person at the HTTP level.

How to get it

The package is @mintlify/agent-traffic on npm, under an MIT licence. Official docs say it has no dependencies and runs anywhere fetch does, including Node, Bun, Deno, Cloudflare Workers and Vercel Edge. You can add it with:

npm i @mintlify/agent-traffic

There’s also an agent skill, which Mintlify says you can install in Claude Code, Cursor, Codex or any agent the skills CLI supports:

npx skills add mintlify/agent-traffic