How to Set Up WordPress llms.txt (Control Which AI Crawlers Read Your Site)

Setting up wordpress llms.txt lets you control how AI crawlers like ChatGPT, Claude, and Perplexity understand your site by providing a structured markdown file at your domain root that tells LLMs what your site is about and which pages matter most.

What Is llms.txt and Why WordPress Needs It

The wordpress llms.txt file is a new standard created by Jeremy Howard at Answer.AI that gives large language models a human-readable guide to your website. Unlike robots.txt, which tells crawlers what NOT to index, llms.txt tells AI systems what your site IS and which content matters most.

Here’s the thing: AI crawlers are already scraping your WordPress site. ChatGPT’s GPTBot, Claude’s ClaudeBot, and Perplexity’s PerplexityBot are all reading your content right now. But they’re doing it blind, with no context about what your site does or which pages represent your best work. The llms.txt specification fixes that problem by providing a structured markdown file that LLMs can parse instantly.

Think of wordpress llms.txt as your site’s executive summary for AI systems. It’s a single file at https://yourdomain.com/llms.txt that contains:

• A one-line H1 title describing your site
• A blockquote summary (1-3 sentences) explaining what you do
• Organized sections with H2 headings and prioritized links
• Optional details about your products, services, or key content

The difference between wordpress llms.txt and robots.txt is fundamental. Robots.txt is a permission system written in a proprietary syntax that tells crawlers which directories to avoid. It’s binary: allowed or disallowed. The llms.txt format uses plain markdown and focuses on positive guidance — here’s what matters, here’s what we do, here’s where to look first.

WordPress sites need wordpress llms.txt because AI-powered search is replacing traditional SEO. When someone asks ChatGPT or Perplexity a question, the AI doesn’t just return ten blue links. It synthesizes an answer from multiple sources. If your site has a properly configured wordpress llms.txt file, you’re giving those AI systems a roadmap to understand and cite your content accurately.

How to Create Your wordpress llms.txt File Manually

Creating a wordpress llms.txt file is straightforward because it’s just markdown. Open any text editor and follow this exact structure from the llmstxt.org specification:

# AI Or Die Now

> A no-BS guide to WordPress, AI integration, and building sites that don't suck. We hate SaaS lock-in and corporate fluff.

## Key Pages

- WordPress Hooks Explained
- WordPress REST API Guide
- Add AI to WordPress Without SaaS

## Products

- AODN WordPress Anti Spam
- Typography Pro

## About

We're pirates fighting against predatory SaaS pricing and WordPress bloat. Read the manifesto.

The wordpress llms.txt format requires three elements:

1. H1 title (one line starting with #) — your site name or primary focus
2. Blockquote summary (starting with >) — a concise description of what your site does
3. Organized sections (H2 headings with ##) — grouped links to your most important content

The beauty of wordpress llms.txt is its simplicity. You’re not writing XML or JSON. You’re writing the same markdown you’d use in a README file. AI systems can parse it instantly because it’s structured but human-readable.

When you’re building your wordpress llms.txt content, prioritize ruthlessly. This isn’t your sitemap with 500 URLs. Include your homepage, your about page, your 5-10 best articles, and your core product or service pages. Think of it as the content you’d show someone if they only had 60 seconds to understand your site.

Where to Upload the wordpress llms.txt File

Your wordpress llms.txt file must live at the root of your domain: https://yourdomain.com/llms.txt. Not in a subdirectory, not in /wp-content/, not anywhere else. AI crawlers expect it at the root, just like robots.txt.

Here’s where WordPress gets tricky. Unlike a static HTML site where you just drop a file in your public root, WordPress uses a complex rewrite system that routes most requests through index.php. Some hosting environments won’t serve plain .txt files from the root without extra configuration because WordPress intercepts those requests.

If you’re using FTP or SFTP, connect to your server and navigate to the directory where wp-config.php lives. That’s your WordPress root. Upload your llms.txt file there, next to wp-config.php, .htaccess, and robots.txt.

If you’re using cPanel File Manager, navigate to public_html (or www or httpdocs depending on your host). Upload the wordpress llms.txt file there. Make sure the file is named exactly llms.txt — no .md extension, no capital letters.

After uploading, visit https://yourdomain.com/llms.txt in your browser. If you see your markdown content displayed as plain text, you’re done. If you get a 404 error or WordPress tries to load a theme template, you need the dynamic approach in the next section.

How to Serve wordpress llms.txt Dynamically (The Better Way)

The most reliable way to serve wordpress llms.txt is dynamically through WordPress itself. This method works on any hosting environment, survives plugin conflicts, and lets you update your llms.txt content without FTP access.

Add this code to your theme’s functions.php file or create a custom plugin. This uses the template_redirect action hook, which you can learn more about in our WordPress Hooks Explained guide:

function aodn_serve_llms_txt() {
    if ( $_SERVER['REQUEST_URI'] === '/llms.txt' ) {
        header( 'Content-Type: text/plain; charset=utf-8' );
        status_header( 200 );
        
        $content = "# " . get_bloginfo( 'name' ) . "\n\n";
        $content .= "> " . get_bloginfo( 'description' ) . "\n\n";
        $content .= "## Key Pages\n\n";
        $content .= "- [Home](" . home_url() . ")\n";
        $content .= "- [About](" . home_url( '/about/' ) . ")\n";
        $content .= "- [Blog](" . home_url( '/blog/' ) . ")\n\n";
        $content .= "## Contact\n\n";
        $content .= "- [Contact Page](" . home_url( '/contact/' ) . ")\n";
        
        echo $content;
        exit;
    }
}
add_action( 'template_redirect', 'aodn_serve_llms_txt' );

This wordpress llms.txt implementation intercepts requests to /llms.txt before WordPress tries to load a template. It sets the correct Content-Type: text/plain header (critical for AI crawlers), returns a 200 status code, and outputs your markdown content directly.

The example above uses get_bloginfo() to pull your site name and tagline dynamically, but you should customize the content sections to match your actual site structure. Replace the placeholder links with your real pages.

This approach is superior to a static file because you can programmatically generate links to your most recent posts, pull content from custom fields, or even query your WordPress database to build a dynamic priority list based on page views or engagement metrics.

What to Put in Your wordpress llms.txt File

Your wordpress llms.txt content strategy should focus on signal over noise. AI crawlers don’t need every page on your site — they need the pages that best represent what you do and why someone should care.

Start with these sections:

• Key Pages — Homepage, about page, your 5-10 best articles or cornerstone content
• Products/Services — What you sell or offer, with direct links
• Resources — Guides, tutorials, documentation if applicable
• Contact/About — How to reach you, who you are

For a WordPress tutorial site like ours, the wordpress llms.txt file might prioritize technical guides about WordPress REST API integration or permalink structure over generic blog posts. For an e-commerce site, you’d highlight product category pages and your best-sellers.

Each section should have 3-10 links maximum. If you’re listing more than that, you’re not prioritizing. Remember: this file exists to help AI systems understand your site quickly. If a human can’t scan your wordpress llms.txt in 30 seconds and understand what you do, it’s too long.

Write your blockquote summary like you’re explaining your site to someone at a bar. Skip the corporate mission statement. “We help WordPress developers build faster sites without SaaS bloat” beats “We leverage synergistic solutions to empower digital transformation.”

wordpress llms.txt vs robots.txt vs sitemap.xml — What Each One Does

Understanding how wordpress llms.txt fits into your site’s crawler ecosystem requires knowing what each file actually does. They’re not interchangeable, and you need all three.

File	Purpose	Audience	Format
robots.txt	Block/allow crawler access	Search engine crawlers	Proprietary syntax
sitemap.xml	List all URLs for indexing	Search engines	XML
llms.txt	Guide AI understanding	LLM crawlers	Markdown
Example	Disallow: /wp-admin/	<url><loc>https://…</loc></url>	# Site Name > Description

Your robots.txt file tells crawlers what they’re NOT allowed to access. It’s a bouncer at the door. Your sitemap.xml file is a comprehensive directory of every page you want indexed. It’s the building directory. Your wordpress llms.txt file is a guided tour for AI systems — here’s what matters, here’s what we do, here’s where to start.

All three files serve different purposes and work together. A properly configured WordPress site has all three at the root level. You can’t replace robots.txt with wordpress llms.txt, and you shouldn’t try to cram your entire sitemap into your llms.txt file.

The wordpress llms.txt format is specifically designed for large language models that need context, not just URLs. When ChatGPT’s crawler hits your site, it can read your llms.txt file and immediately understand “this is a WordPress tutorial site focused on AI integration” instead of having to infer that from analyzing 500 blog posts.

How to Test Your wordpress llms.txt File

After setting up your wordpress llms.txt file, you need to verify it’s working correctly. Testing is straightforward — you’re just checking that the file is accessible and returns the right content type.

First, visit https://yourdomain.com/llms.txt directly in your browser. You should see plain text markdown, not a styled WordPress page. If WordPress loads your theme around the content, your wordpress llms.txt implementation isn’t working correctly.

For a more technical test, use curl from your terminal:

# Check HTTP headers
curl -I https://yourdomain.com/llms.txt

# Should return:
# HTTP/1.1 200 OK
# Content-Type: text/plain; charset=utf-8

# Check actual content
curl https://yourdomain.com/llms.txt

The -I flag shows you just the headers. You’re looking for two things: a 200 OK status code and Content-Type: text/plain. If you see Content-Type: text/html, your server is treating the wordpress llms.txt file as an HTML page, which will confuse AI crawlers.

If you’re using the dynamic PHP approach from earlier, test that your markdown formatting is correct. Copy your wordpress llms.txt output and paste it into a markdown preview tool. The H1 should render as a heading, the blockquote should be indented, and your links should be clickable.

You can also test whether AI systems can actually access your wordpress llms.txt file by asking ChatGPT or Claude directly: “What does the llms.txt file at [yourdomain.com] say?” If they can read it, they’ll summarize your content. If they can’t, they’ll tell you the file isn’t accessible or doesn’t exist.

“The llms.txt file is a small thing that makes a big difference. It’s the difference between AI systems guessing what your site does and actually knowing.” — Jeremy Howard, Founder of Answer.AI and creator of llms.txt

Common wordpress llms.txt Mistakes to Avoid

Setting up wordpress llms.txt is simple, but people still screw it up. Here are the most common mistakes that break your implementation:

• Wrong Content-Type header — Serving llms.txt as text/html instead of text/plain confuses parsers. AI crawlers expect plain text, not HTML.
• Missing the blockquote summary — The llms.txt specification requires a blockquote after the H1. Without it, you’re not following the standard.
• Linking to dead pages — If your wordpress llms.txt file links to URLs that return 404 errors, you’re wasting AI crawlers’ time and damaging your credibility.
• Using HTTP instead of HTTPS — All your links should use https:// URLs. Mixed content warnings and security flags hurt your site’s reputation with both humans and bots.
• Putting the file in the wrong location — Your wordpress llms.txt file must be at https://yourdomain.com/llms.txt, not /wp-content/llms.txt or any subdirectory.
• Including too many links — This isn’t a sitemap. If you’re listing 50+ URLs, you’re not prioritizing. Keep it focused.
• Using relative URLs — Always use absolute URLs with the full domain. Relative paths like /about/ can break in different contexts.
• Forgetting to test — Don’t assume your wordpress llms.txt file works. Test it with curl and by visiting the URL directly.

The biggest mistake is treating wordpress llms.txt like a robots.txt file. They’re fundamentally different. Robots.txt is about permission and access control. The wordpress llms.txt format is about communication and context. You’re not blocking anything — you’re guiding AI systems toward your best content.

Another common error is overcomplicating the content. Your wordpress llms.txt file should be readable by a human in under a minute. If you’re writing paragraphs of explanation or including every single blog post you’ve ever published, you’re missing the point. Think executive summary, not comprehensive documentation.

Do AI Crawlers Actually Read wordpress llms.txt?

Here’s the honest answer: some do, some don’t, and the landscape changes monthly. Setting up wordpress llms.txt is still worth doing, but you need realistic expectations about adoption.

As of early 2025, the llms.txt standard is gaining traction but isn’t universally implemented. Anthropic’s Claude and Perplexity have both shown interest in the specification. ChatGPT’s GPTBot crawler can read and parse wordpress llms.txt files, though OpenAI hasn’t made an official statement about prioritizing sites that have them.

Google’s AI Overview system doesn’t officially use the llms.txt file yet, though Google Search Central documentation suggests they’re monitoring the standard. The challenge is that Google has decades of infrastructure built around traditional SEO signals — structured data, sitemaps, robots.txt. Adding a new markdown-based standard requires significant engineering work.

The reality is that your llms.txt is an emerging standard, not an established one. That doesn’t mean you should skip it. Early adopters of robots.txt in the 1990s had the same uncertainty, and now it’s universal. The sites that implement this file today are positioning themselves for the AI-first search landscape that’s coming.

Even if AI crawlers don’t explicitly prioritize llms.txt files yet, having one doesn’t hurt. It’s a lightweight text file that costs nothing to serve. And as more AI systems adopt the standard, you’ll already be ahead of competitors who are still figuring out what llms.txt even is.

The bigger question is whether AI crawlers should honor the file files. The answer is yes, because the current alternative is worse. Right now, AI systems scrape everything and try to figure out what matters through algorithmic analysis. That’s inefficient and often wrong. A properly configured the llms.txt file file gives AI crawlers direct information from the source — you, the site owner — about what your site does and which content represents your best work.

FAQ: wordpress llms.txt Setup

Conclusion: Set Up Your wordpress llms.txt Today

Implementing wordpress llms.txt on your site gives you control over how AI systems understand and cite your content in an era where AI-powered search is replacing traditional SEO. Whether you upload a static file or serve it dynamically via PHP, the wordpress llms.txt standard is a lightweight, future-proof way to guide AI crawlers toward your best work. The specification is simple, the implementation takes minutes, and early adopters will have a significant advantage as AI search continues to grow.

Stop letting AI systems guess what your WordPress site does. Create your wordpress llms.txt file, prioritize your best content, and take ownership of your AI visibility before your competitors do.

What’s in your llms.txt file? Drop a comment with your domain and let’s see how you’re guiding AI crawlers.