tldr; Serve raw, clean Markdown to AI coding agents and beautifully styled HTML to humans from the exact same URL using HTTP content negotiation.
Serving Markdown instead of fully rendered HTML can reduce page weight by 74%. For example, fetching the Git cheatsheet on tools wiki returns 22KB of compressed HTML to a browser request, but only 5.8KB of compressed Markdown to an AI agent request. That is a massive saving on token costs and context windows, completely skipping the lossy HTML scraping step in agents.
This post shows you how to set up content negotiation at the edge to serve both audiences from a single URL and a single source of truth.
# The standard way
Hint: It’s a bottleneck
When we point an AI agent or a browser to a page, this is what normally happens under the hood:
[ AI Agent ]
│
│ GET /docs/git/
▼
[ Docs Site ]
│
│ Returns HTML loaded with UI chrome (nav, footer, scripts)
▼
[ HTML Scraping ]
│
│ - Strips tags & wastes tokens
│ - Fragile; breaks on layout changes
│ - Loses semantic tables & lists
▼
[ AI Agent (Frustrated) ]
To solve this, we have three different approaches. Let’s weigh them and look at the pros and cons of each.
# Approach 1: Let the agent scrape the HTML
This is the “do nothing” approach. We just give the agent the URL and hope its parsing engine can make sense of the webpage.
Pros:
- Zero configuration or setup on your side.
Cons:
- Expensive: Agents pay for every single byte of HTML. A doc with 8KB of actual text can easily balloon to 80KB of HTML because of headers, footers, script tags, and syntax highlighting spans.
- Fragile: A minor change in your site’s layout can break the agent’s extraction parser.
- Loss of context: Beautiful markdown tables or nested code block structures get flattened or mangled during the HTML-to-text conversion.
# Approach 2: Build an alternate markdown route
We can spin up an alternate route like /docs/git.md that serves the raw markdown content instead of HTML.
Pros:
- Super clean, token-efficient payload for agents.
Cons:
- Drift risk: Maintaining two different routes means they might drift over time.
- Maintenance: You have to set up extra routes, and figure out how to let agents know about those routes without confusing your human users or search crawlers.
# Approach 3: Serve both from one route
This is based on content negotiation at the edge. Instead of maintaining separate routes, we keep a single URL for both humans and AI agents. When a request comes in, our edge router checks who is asking and serves either the fully styled HTML or the raw markdown from the exact same link.
For static pages, build both .html and .md pages and rewrite the url based on the request at the edge. For dynamic pages, render the desired text/html or text/markdown page based on the request. Although it’s easier for dynamic pages since content negotiation can be done in the page handler itself.
Pros:
- Same hyperlink: Humans get a beautiful web page. Agents get a super-efficient markdown.
- Agents don’t need to be aware: It works out of the box with existing agents without any code changes on their side.
Cons:
- Requires a lightweight edge router (or a L7 reverse proxy) in front of your static assets.
You can use whichever approach suits your need best, but for modern developer documentation, content negotiation is the clear winner.
# Understanding content negotiation
Before we get to the code, let’s understand how content negotiation works. The HTTP spec actually has a built-in mechanism for “serve me the best format you have.” It uses two headers:
Accept: The client tells the server what media type it wants. For example,Accept: text/markdownmeans “Hey, I want markdown if you have it.” This is the spec-compliant, elegant way.User-Agent: The client identifies what software it is running. This acts as our fallback, because most AI agents today don’t bother setting theAcceptheader correctly yet. Although a few modern AI agents have started explicitly sending theAccept: text/markdown.
Here is how the request flow looks:
[ Incoming Request ]
│
▼
[ Edge Worker ]
/ \
/ \
No Agent Header Agent Header Detected
(Browser client) (Accept/User-Agent matching)
/ \
▼ ▼
[ Rendered HTML ] [ Raw Markdown ]
│ │
22KB Gzip 5.8KB Gzip
# Step-by-step implementation
For a website serving static content, follow these three steps to make your site agent-friendly.
You don’t need any ahead-of-time setup like this for dynamic pages since you can handle this logic in the page handler during request-time.
# 1. Export raw markdown files during your build
To make this work, we need our build pipeline to emit a .md file for every single webpage. For example, if you have a /docs/git/ page, the build should also output /docs/git.md alongside it.
If you are using Astro, you can achieve this easily using a dynamic route. Astro already reads your Markdown files, so we can export them directly like this:
1// src/pages/[...path].md.ts
2import { getCollection } from "astro:content";
3import { rawMarkdownByPath } from "../md-map"; // A custom map built from your collection
4
5export async function getStaticPaths() {
6 const docs = await getCollection("docs");
7 return docs.map((doc) => ({ params: { path: doc.slug } }));
8}
9
10export async function GET({ params }: APIContext) {
11 return new Response(rawMarkdownByPath[params.path], {
12 headers: { "content-type": "text/markdown; charset=utf-8" },
13 });
14}
Note: If you aren’t using Astro, almost any modern static site generator (like Next.js, Eleventy, or Hugo) can be configured to export raw text or markdown files during the build process.
Now we have both .html and .md files sitting ready in our build directory. Step 1 is complete!
# 2. Write the edge function to handle content negotiation
Next, we need a lightweight edge worker that intercepts incoming requests, checks who is calling, and routes them to the correct file.
We’ll set up a Cloudflare Worker (or reverse proxy should work) to sit in front of our static assets. Here is the entire implementation in just about ~20 lines of clean TypeScript:
1// Detect agent traffic in two ways:
2// 1. Explicit Accept header (the spec-correct way)
3// 2. A User-Agent matching known agents (the pragmatic fallback)
4const AGENT_UA =
5 /bot|crawler|claude|gptbot|chatgpt|curl|wget|python-requests|go-http-client|opencode|pi/i;
6
7async function handleRequest(request: Request, env: Env): Promise<Response> {
8 const url = new URL(request.url);
9 const ua = request.headers.get("user-agent") ?? "";
10 const accept = request.headers.get("accept") ?? "";
11
12 const wantsMarkdown = accept.includes("text/markdown") || AGENT_UA.test(ua);
13
14 if (wantsMarkdown) {
15 // Convert /docs/git/ to /docs/git.md
16 const mdUrl = `${url.origin}${url.pathname.replace(/\/$/, ".md")}`;
17 const md = await env.ASSETS.fetch(mdUrl);
18
19 if (md.ok) {
20 return new Response(md.body, {
21 headers: {
22 "content-type": "text/markdown; charset=utf-8",
23 "cache-control": "public, max-age=3600, stale-while-revalidate=86400",
24 // Crucial: Keep the caches separated!
25 vary: "Accept, User-Agent",
26 },
27 });
28 }
29 }
30
31 // Everyone else gets the normal rendered HTML page
32 return env.ASSETS.fetch(request);
33}
The beauty of this approach is that it is fully progressive. If a markdown version of a specific route doesn’t exist, we fall back gracefully to serving the standard HTML page. Your site will never break.
With our edge worker handling traffic, Step 2 is complete!
# 3. Configure cache hygiene with the Vary header
This is the most critical step. If you miss this, you’ll run into serious caching issues.
Because we are serving two different representations from the exact same URL, CDN caches can get confused. If a human visits /docs/git/ first, the CDN caches the HTML. If an AI agent then visits /docs/git/, the CDN might mistakenly serve the cached HTML to the agent! Or worse, if the agent visits first, the human gets the raw Markdown in their browser.
To solve this, we must include the Vary header:
1Vary: Accept, User-Agent
This tells the CDN and browsers: “Hey, this URL has multiple versions. Only serve the cached version if the incoming Accept and User-Agent headers match perfectly.”
Once you have your Vary headers properly configured, you are good to go. All done.
# Why this is worth doing
If you’re building a docs site, it’s more likely an agent is reading your content than a human.
- Token and context efficiency: A markdown file is typically 74% smaller than its rendered HTML page under compression. Saving over 3x on every fetch means faster agent response times and lower API costs for your users.
- No code drift: Since you write Markdown once, there is only one source of truth. There is absolutely zero chance of your API docs drifting away from your main website.
- Out-of-the-box support: Today’s coding tools like
curl, Python requests,pi, or custom LLM scrapers are caught automatically by the user-agent check. When future tools start supporting properAcceptheaders, this is already future proof. - Graceful fallback: If your SSG fails to generate a
.mdfile, the edge worker gracefully drops back to the HTML site.
How amazing is that! One source of truth, two perfectly optimized representations.
# Wrapping up
Implementing HTTP content negotiation for documentation makes your site ready for the next decade of the web. It is low overhead, easy to set up, and highly beneficial for the growing ecosystem of AI tools.
If you are planning to make your documentation more AI-friendly, don’t think twice. The token savings and developer experience gains are worth it.
If you have any questions or want to chat about edge functions, hit me up on X @dpandiyan.