Sulat.com / Writing on AIFriday, November 20, 2026
Agents/20 November 2026

Nectiv’s check of 300 tech sites finds 22% serve markdown to agents, and 54% of dev tools do

Nectiv's Chris Long says 22% of 300 technology sites serve markdown when asked, led by dev tools at 54%. His check covered home pages only and doesn't test whether markdown wins AI citations.

Glossy 3D Markdown logo turning a stack of cluttered HTML pages into one clean page of text that a robot hand reaches for

At a glance

  • Nectiv co-founder Chris Long says 22% of 300 technology sites served markdown when a request asked for it, in a home-page-only check he posted on 19 November US time.
  • Long says dev tools lead at 54% and data tools follow at 34%. His chart names Sentry, PagerDuty, Redis, Auth0 and Prisma among the 14 dev-tool sites that answered with markdown.
  • Cloudflare's Markdown for Agents converts HTML to markdown for clients that send Accept: text/markdown. Bloomberry counted 16,342 companies using it in an update on 15 November.
  • Evil Martians logged coding agents getting markdown on 88% of about 38,000 requests and chat assistants on 0.9% of about 23,000 in September and October.
  • Long's post doesn't say how the 300 sites were chosen or claim a citation benefit, and Google's John Mueller and Microsoft's Fabrice Canel have argued against separate pages for bots.

Chris Long, co-founder of Nectiv, says 22% of the 300 technology sites his team tested serve a markdown version of their home page when a request asks for one. In the post, dated 19 November US time (20 November in New Zealand), he says dev tools lead at 54% and data tools follow at 34%. Nectiv does answer engine optimisation, known as AEO, for B2B and technology companies, and Long checked home pages only.

Long explains the mechanism in the post. A site reads the headers on each request, so a request for text/html returns the full page and a request for text/markdown returns a markdown version. To check a competitor, he says, you tell Claude Code which sites to look at, and it sends requests without a browser and groups the results. On whether serving markdown counts as cloaking, he says that as long as the content is the same, he doesn’t see a problem.

What the chart shows

Long attached a chart that lists, by category, the sites that answered with markdown. Dev tools account for 14 sites, ahead of AI companies at nine and data and analytics at eight. The screenshot cuts off after the Consumer row, which shows one site. The list still holds up in at least one case: a request with the header to redis.io on 20 November returned content-type: text/markdown, with Vary: Accept.

Bar chart of tech sites serving markdown to agents: dev tools 54%, data tools 34%, all 300 sites 22%, home pages only
Dev tools are the only group where a majority serves markdown.
CategorySitesExamples in the chart
Dev tools14Sentry, PagerDuty, Redis, Auth0, Prisma
AI9Jasper, Together AI, Fireworks AI, Replicate, Pinecone
Data and analytics8Snowplow, Airbyte, ClickHouse, MotherDuck, Domo
Sales and marketing7Pipedrive, ZoomInfo, Hootsuite
Productivity4Wrike, Zapier, Craft
HR and ops4Eightfold, Workable, Zip
Security4Illumio, Tanium, Orca Security
Fintech3Checkout.com, Novo, Increase
Health tech3Function Health, Ambience, Garner Health
Consumer1Not shown
Counts and names come from the chart in Long’s post, which cuts off after the Consumer row. Source: Chris Long on X.

How a site serves markdown

The technique is HTTP content negotiation, the mechanism a browser uses to say which formats it can take. An agent sends Accept: text/markdown, and a server that supports it returns markdown from the same address, adding Accept to its Vary header so caches keep the two versions apart. Cloudflare turned that into a switch on 12 February with a feature it calls Markdown for Agents. Cloudflare says its network fetches the original HTML from the site’s server, converts it on the fly and returns markdown to any client that asks.

One URL answers text/html with 16,180 tokens and text/markdown with 3,150 tokens, from Cloudflare's own blog post
The request header picks the version; the address stays the same.

Cloudflare’s documentation, last updated 9 November, lists the feature on Pro, Business and Enterprise plans and for SSL for SaaS customers at no cost. It caps conversion at 6 MiB of decompressed HTML. If a site’s own server sends a content-signal header, Cloudflare keeps that value. If not, it adds ai-train=yes, search=yes, ai-input=yes, which signals that the content can be used for AI training, search and AI input.

Bloomberry, which tracks which companies use which web services, counted 16,342 companies using the feature in an update on 15 November, plus 488 that have stopped. That count covers Cloudflare’s feature only, so sites that serve markdown from their own servers aren’t in it.

The payoff Cloudflare points to is size. It says its own announcement post takes 16,180 tokens as HTML and 3,150 as markdown, an 80% cut. Checkly, a monitoring company, measured one docs page at 615.4 KB and 180,573 tokens as HTML against 2.3 KB and 478 tokens as markdown. Tokens are the chunks of text a model reads and bills by.

Who asks for it

Serving markdown only helps if something requests it. Checkly’s February test of seven coding agents found three that send text/markdown in the Accept header: Claude Code, Cursor and OpenCode. Cloudflare’s announcement names Claude Code and OpenCode as agents it already sees sending it.

Server logs show the same split. Evil Martians, a software consultancy, logged requests to its own site in September and October 2026. Coding agents made about 38,000 requests and got markdown on 88% of them. Chat assistants made about 23,000 and got markdown on 0.9%. Claude Code alone accounted for 94% of all markdown reads on the site.

Coding agents got markdown on 88% of about 38,000 requests; chat assistants on 0.9% of about 23,000, one site's logs
On one site’s logs, coding agents ask for markdown and chat assistants don’t.

Ethan at Gauge replied to Long with similar numbers from Gauge’s data: about 25% of coding agents’ fetches to docs pages are attempts to get the markdown version. He says Gauge recommends that its devtool customers add markdown versions of their pages.

What the check doesn’t show, and the pushback

Long’s post says nothing about how the 300 sites were chosen, and it doesn’t say whether each markdown version matched its HTML page. It doesn’t claim that serving markdown wins citations in AI answers. The one controlled test of bot traffic found no significant change: Brandon Punturo’s randomised experiment covered 381 pages on six sites for 20 days from 19 January 2026, using Profound’s Agent Analytics. Median visits per page were 6.0 for HTML and 7.0 for markdown, and the difference wasn’t statistically significant. It measured bot visits, not citations.

Google and Bing have pushed back on separate pages for bots. Google’s John Mueller said in November 2025 that large language models have always read normal web pages, and asked “Why would they want to see a page that no user sees? And, if they check for equivalence, why not use HTML?” Microsoft’s Fabrice Canel asked whether publishers want to double the crawl load, since Bing would crawl anyway to check similarity. SEO consultant David McSweeney raised cloaking when Cloudflare launched the feature: because the Accept header reaches a site’s server, a site could send an agent different content from what a person sees, Search Engine Land reported.

Mueller has also drawn a line around developer docs. In May he said Google publishes markdown versions of its developer documentation for AI coding tools and that it isn’t done for search. In August he said that on his test sites the only crawlers claiming to accept markdown were SEO tools.

Check a site yourself

The check Long describes takes one command. It sends the header and prints the response headers.

Code
curl -s -o /dev/null -D - -H "Accept: text/markdown" https://example.com/

A site that serves markdown answers with content-type: text/markdown and a Vary header that includes Accept. One that doesn’t returns text/html. Mueller’s advice is to log the Accept header on your own server before building anything, since many server setups don’t record it.

The full list of 300 sites isn’t in the post, and neither is the date range of the check.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.