# What Makes an AI-Ready Website? The 6 Layers We Build *By Joshua Cogar · 2026-10-01 · Updated 2026-10-07 · Web Design* > See the 6 layers that make an AI-ready website readable by ChatGPT, Claude, and AI Overviews, from robots.txt and llms.txt to schema, and how we build them. - **What This Covers:** The six technical layers that let ChatGPT, Claude, Gemini, and Google AI Overviews read, trust, and cite a business website - **Who It's For:** Business owners whose website was built for Google years ago and now needs to show up in AI answers - **Key Takeaway:** An AI-ready website has to be built in from the start, and we handle the whole build so you never have to touch the technical side More of your customers are asking AI tools who to hire before they open a search results page. When someone asks ChatGPT for "the best HVAC company in Henderson" or "a financial advisor who works with small business owners," the AI recommends businesses it can read and understand. If your website is hard for AI to read, you don't get recommended. An AI-ready website is a website built so AI crawlers and AI agents can read every page, understand exactly who the business is, and pull accurate facts into their answers. Our version has six layers: a fast static foundation, correctly configured robots.txt and llms.txt files, clean Markdown versions of every page, connected schema markup, machine-readable business endpoints, and a server that recognizes AI crawlers. In our own scans of small business websites, the AI layers are the ones most often missing. Our own site runs all six, and we build the same stack for our clients. ![Diagram of the six layers of an AI-ready website, stacked from the foundation up and read by ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews: 1 fast static foundation on Astro and Cloudflare, 2 robots.txt and llms.txt, 3 a Markdown twin of every page, 4 connected schema markup, 5 agent endpoints, 6 AI crawler recognition](/images/ai-ready-website-six-layers.svg) ## Why Does Your Website Need to Be Readable by AI? Your website needs to be readable by AI because AI tools only cite pages they can actually read, and many AI crawlers read less than Google does. In [Vercel's December 2024 analysis of AI crawler traffic](https://vercel.com/blog/the-rise-of-the-ai-crawler), OpenAI's GPTBot made 569 million requests across Vercel's network in one month and Anthropic's Claude crawler made 370 million. Both fetched JavaScript files but didn't run them. Google's Gemini was the exception, because it uses Googlebot, which does render JavaScript. So if your page builder loads text, prices, and service details with JavaScript, ChatGPT's and Claude's crawlers may see a mostly empty shell. The stakes keep rising. A [Pew Research Center study from July 2025](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/) found that 58% of Google users saw at least one AI-generated summary in March 2025. Users who saw a summary clicked a regular search result in only 8% of visits, compared with 15% for users who didn't. When the answer appears on the results page itself, being the cited source is worth as much as ranking. ## Layer 1: A Fast, Static Foundation Built on Astro and Cloudflare The foundation of an AI-ready website is a site where every word is already in the page's HTML before any script runs. We build on Astro, a modern web framework, and serve finished pages from Cloudflare's global network. Visitors and crawlers both get a complete page right away, with nothing to assemble. Speed is the second benefit. When we moved our own site off WordPress, its mobile PageSpeed Insights scores went from 41 for performance, 88 for accessibility, 69 for best practices, and 69 for SEO to 97, 100, 100, and 100. Speed affects revenue as well as scores: [Deloitte's "Milliseconds Make Millions" study](https://www.deloitte.com/ie/en/services/consulting/research/milliseconds-make-millions.html) found that a 0.1 second mobile speed improvement raised retail conversions by 8.4% and travel conversions by 10.1%. WordPress runs 40.1% of all websites, according to [W3Techs](https://w3techs.com/technologies/details/cm-wordpress) (October 2026), and it isn't a bad tool. But plugin-heavy WordPress sites are often slow and depend on scripts. For most service businesses, a static site is the better WordPress alternative when the goal is speed and AI visibility. We explain that decision in detail in [Websites In 2026: Why I Moved From WordPress To Astro And Cloudflare](/blog/websites-in-2026/). ## Layer 2: robots.txt and llms.txt, the Gate and the Guide robots.txt and llms.txt are two small text files at the root of a website, and they do very different jobs. robots.txt is the gate: it tells crawlers which parts of your site they're allowed to visit. llms.txt is the guide: it tells AI tools what your business is and which pages count most. You need the gate set correctly before the guide can do anything. robots.txt has been around since 1994 and became an official internet standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), in 2022. Every major search and AI crawler checks it first. One wrong line can shut ChatGPT or Claude out of your entire site. Blocking can also happen at the network level: since July 1, 2025, [Cloudflare asks every new domain whether to allow AI crawlers](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), with blocking as the default, so a business that clicked through setup may be blocking AI tools without knowing it. Our own [robots.txt](https://seomarketingadvisors.com/robots.txt) names GPTBot, ClaudeBot, OAI-SearchBot, and other AI crawlers individually and allows each one in. llms.txt is much newer. Jeremy Howard published the [llms.txt proposal](https://llmstxt.org/) on September 3, 2024. It works like a table of contents written for AI: a short summary of your services, your location, and the pages that answer the questions customers ask. You can see ours at [seomarketingadvisors.com/llms.txt](https://seomarketingadvisors.com/llms.txt).
robots.txt llms.txt
Job Controls which crawlers can access which pages Summarizes the business and points AI to key pages
Status Official standard (RFC 9309, 2022) Proposal (September 2024)
Who reads it Google, Bing, and every major AI crawler Rarely read today; no major AI platform has committed to it
Cost of getting it wrong High: one bad line can hide your whole site from AI Low: a missing file means AI works out your site on its own
Proven benefit Yes Not yet
Here's the honest part. There's very little data showing that llms.txt does anything measurable today. [Ahrefs studied 137,000 sites in 2026](https://ahrefs.com/blog/llmstxt-study/) and found that 97% of llms.txt files received zero requests in May 2026, and that no major AI platform has committed to reading the file. Anyone who promises results from llms.txt alone is overselling it. We add llms.txt anyway, for a simple reason: it costs almost nothing to create and keep current, it can't hurt, and if AI tools start relying on it the way search engines came to rely on sitemaps, your site is already prepared. Getting robots.txt right is what has real impact today. llms.txt is a low-cost bet on where AI could head. ## Layer 3: A Clean Markdown Version of Every Page A Markdown twin is a clean, text-only copy of a web page. It strips out menus, cookie banners, scripts, and design code and keeps only the content. On our site, every page has one. For example, our About page also exists at [/about.md](https://seomarketingadvisors.com/about.md). The llms.txt proposal itself recommends offering Markdown versions of pages at the same address with ".md" added. The benefit is accuracy. AI models read in chunks called tokens, and a typical web page wastes many of them on layout code. A Markdown twin gives the AI only the facts: your services, your prices, your answers. Less clutter means fewer misreadings, and fewer misreadings mean AI tools are less likely to quote your business incorrectly. ## Layer 4: Connected Schema Markup, One Knowledge Graph Instead of Scattered Snippets Schema markup is structured code that tells search engines and AI tools exactly what each fact on your site means. It uses the [Schema.org](https://en.wikipedia.org/wiki/Schema.org) vocabulary, written in a format called [JSON-LD](https://en.wikipedia.org/wiki/JSON-LD). Most sites have a few separate schema snippets added by plugins. We build one connected graph instead, where your business, your people, your services, your prices, and your pages all point to each other. A connected graph removes guesswork. It states in machine-readable terms that this business offers this service at this price in this city, and that this person runs it. Schema alone won't get you cited. What it does is make the facts AI tools pull from your site consistent and unambiguous, and that consistency is part of what makes a source trustworthy. ## Layer 5: Machine-Readable Business Endpoints for AI Agents Machine-readable endpoints are files that tell AI agents what your business offers and what actions they can take, in a format software reads directly. Our site publishes an [agents.json file](https://seomarketingadvisors.com/agents.json), an AI plugin manifest, and an [OpenAPI](https://en.wikipedia.org/wiki/OpenAPI_Specification) description of the actions available. This layer looks ahead. AI agents are starting to go beyond answering questions: they compare providers, request quotes, and book appointments for their users. We cover this shift in our guide to [agentic commerce for service businesses](/blog/agentic-commerce-for-service-businesses/). The standards are still early, but a business that agents can already understand is better placed than one that has to catch up later. ## Layer 6: A Server That Recognizes AI Crawlers The final layer is a server that recognizes when an AI crawler is visiting and responds in the format that crawler reads best. We use a Cloudflare Worker for this. When GPTBot, ClaudeBot, or PerplexityBot requests a page, it gets the clean Markdown version. When a person visits, they get the fully designed page. The content is identical; only the format changes. We tested this on our own site. When GPTBot requests our About page, the server returns Markdown instead of HTML. Recognizing AI crawlers also means deciding which ones to let in. Training crawlers such as GPTBot, ClaudeBot, and Google-Extended collect content for future models. Search and retrieval crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot fetch pages in real time so an AI tool can cite you. Block the second group and you drop out of AI answers. We set these rules bot by bot, and on our own site we allow both groups. Cloudflare's [Is It Agent Ready](https://isitagentready.com/) scanner, launched in April 2026, can push you toward blanket blocking, so read our [Cloudflare Agent Readiness Scan review](/blog/cloudflare-agent-readiness-scan-review/) before following its bot access advice. ## What's the Difference Between a Standard Website and an AI-Ready Website? A standard website is built for human visitors and Google's search results. An AI-ready website is built for those two audiences plus every AI tool and agent that might recommend you. The gap shows up in our own scan data. In May and June 2026 our prospecting scanner checked about 460 small business websites. The median overall marketing score was 38 out of 100, and AI search readiness was the weakest of five categories for 83% of them. Local businesses look the same. Between August 11 and October 7, 2026, our Local Visibility Scanner checked 377 local businesses in Las Vegas, Henderson, and North Las Vegas. For 261 of them (69%), AEO and AI visibility, meaning how easily AI search tools can find and cite the business, was their lowest-scoring category. Both groups are businesses our prospecting tools pulled, not a random sample, but the pattern is consistent: the AI layers are the ones most sites are missing. ## Why We Build It for You Instead of Handing You a Checklist We build the AI-ready stack for clients because the six layers only work when they agree with each other, and a single mismatch can undo the rest. A blocked crawler makes your llms.txt invisible. Schema that says one price while the page says another makes AI tools trust you less. A Markdown twin that wasn't updated with the page sends AI an old version of your business. Getting it right means planning, building, and testing everything together, then keeping it in sync each time content changes. That's the work we do. Every site launches with an AI Readiness score so you can see where you stand from day one. Our [website design](/services/website-design/) packages start at $3,200 one-time for a brochure site, $7,499 for e-commerce, and $15,000 for a custom web app. If you want to know where your current site stands first, run it through our free [AI Readiness Tool](/ai-readiness-tool/). It checks whether AI search engines can find and cite your site. You don't need to learn any of this to benefit from it. If you'd like a website that AI tools can read, trust, and recommend, book a [Free Marketing Consultation](https://api.leadconnectorhq.com/widget/bookings/30-minute-marketing-consult). We'll review your current site, show you which of the six layers are missing, and give you a straight answer on what it would take to fix them. ## AI-Ready Website FAQs ### What is an AI-ready website? An AI-ready website is built so AI crawlers and AI agents can read every page, understand the business, and quote accurate facts. That usually means fast static pages, correct robots.txt and llms.txt files, Markdown versions of pages, connected schema markup, agent-readable endpoints, and a server that recognizes AI crawlers. ### Do AI crawlers render JavaScript? Most don't. Vercel's December 2024 study found that OpenAI's and Anthropic's crawlers fetch JavaScript files but don't run them, while Gemini uses Googlebot, which does. Content your site loads with scripts may be invisible to ChatGPT, Claude, and similar tools. ### What's the difference between robots.txt and llms.txt? robots.txt controls which crawlers can access your site, and it's an official standard that every major search and AI crawler follows. llms.txt is a newer proposal that summarizes your business for AI tools. robots.txt is the gate and llms.txt is the guide, and the guide only helps once the gate is open. ### Does llms.txt actually help? Not measurably yet. Ahrefs found that 97% of llms.txt files received zero requests in May 2026. We add it because it costs almost nothing, can't hurt, and prepares your site in case AI tools adopt it more widely. Your robots.txt file counts for far more right now, because one wrong setting there can block AI tools completely. ### Is WordPress bad for AI search? WordPress isn't bad, but plugin-heavy WordPress sites are often slow and depend on JavaScript, and both of those hurt AI visibility. When our own site moved from WordPress to Astro, its mobile performance score went from 41 to 97. ### Can you make my existing website AI-ready without a rebuild? Some layers can be added to an existing site, including correct robots.txt settings, llms.txt, and better schema. The full stack, especially static delivery and Markdown versions of every page, usually requires a rebuild. We'll tell you which path makes sense during a consultation. ### How much does an AI-ready website cost? Our website packages start at $3,200 one-time for a brochure site, $7,499 for e-commerce, and $15,000 for a custom web app. Final cost depends on page count, integrations, and content needs. #### Additional Resources - [How To Optimize For LLM Search](/blog/how-to-optimize-for-llm-search/) - [How AI Is Changing SEO: Rankings to Relevance In 2025](/blog/how-ai-is-changing-seo/) - [Website Design: Best Practices](/blog/website-design-best-practices/) - [What Is A Google Business Profile?](/blog/what-is-a-google-business-profile/) - [Website Design & Development](/services/website-design/) - [Digital Advertising](/services/digital-advertising/) --- Canonical page: https://seomarketingadvisors.com/blog/ai-ready-website/ Site index: https://seomarketingadvisors.com/llms.txt · Agent directory: https://seomarketingadvisors.com/agents.json