robots.txt Setup for AI Crawlers: What Actually Works

Published May 22, 2026 · 5 min read · By the HighRank team

The Short Answer Before We Get Into It

Your robots.txt file is probably blocking AI crawlers right now. Not on purpose. Just because the defaults were written before these bots existed.

This is fixable in about ten minutes. Here's how.

---

Why robots.txt Matters for AI Citation

If an AI crawler can't read your site, it can't cite your site. That's the whole equation.

ChatGPT, Claude, Perplexity, and the others all pull from crawled content. If your robots.txt tells their bots to stay out, you're invisible. You could have the best answer on the internet and it doesn't matter. You're shouting into a room with the door closed.

---

The AI Crawlers You Actually Need to Know

Each major AI platform sends its own bot. Here are the ones worth caring about right now.

• GPTBot: OpenAI's crawler. Powers ChatGPT's browsing and training data.
• ClaudeBot: Anthropic's crawler. Used for Claude's knowledge.
• PerplexityBot: Perplexity AI's crawler. Directly affects what Perplexity cites in answers.
• Google-Extended: Google's AI-specific crawler, separate from Googlebot. Feeds Gemini and AI Overviews.
• Cohere-AI: Cohere's crawler. Smaller reach but growing in enterprise tools.
• Bytespider: ByteDance's crawler. Powers some AI features in TikTok and other products.

---

What a Default robots.txt Gets Wrong

A lot of site generators, WordPress installs, and older templates ship with something like this:

User-agent: *
Disallow: /wp-admin/

That looks fine. And for traditional SEO, it mostly is. But it doesn't explicitly allow the AI bots, and some hosting setups or security plugins add blanket disallow rules that catch everything.

The honest catch here: some people have User-agent: * with Disallow: / buried in their file from a staging-site lockdown they forgot to remove. Check your file. Actually check it.

---

The robots.txt Setup for AI Crawlers That Works

Here's a clean, working example you can adapt directly.

# Standard search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI crawlers: allow all
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Cohere-AI
Allow: /

User-agent: Bytespider
Allow: /

# Block everyone from admin areas
User-agent: *
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /private/

Sitemap: https://yourdomain.com/sitemap.xml

That's it. No tricks. The explicit Allow: / entries for each AI bot make sure nothing gets caught by a broad disallow rule lower in the file.

---

Should You Block Any AI Crawlers?

Honest answer: it depends on what you're protecting.

If your content is your product, like a paid newsletter or a members-only course, you probably don't want GPTBot training on it. That's a legitimate call. You can block specific bots while allowing others.

# Block OpenAI from training data
User-agent: GPTBot
Disallow: /

# Still allow Perplexity to cite you
User-agent: PerplexityBot
Allow: /

But if you're a founder or freelancer trying to get your expertise cited in AI answers, blocking these bots works against you. You want to be in the room.

---

Where to Put Your robots.txt File

It lives at the root of your domain. Always.

https://yourdomain.com/robots.txt

Not /blog/robots.txt. Not /public/robots.txt. The root. If it's anywhere else, crawlers won't find it.

You can check yours right now by typing your domain into a browser and adding /robots.txt at the end. If you get a 404, you don't have one. That's not the same as having an open file. Some bots treat a missing file as open access, others don't. Don't leave it to chance.

---

The Actual Pain Most People Miss

The actual pain isn't usually a wrong robots.txt. It's a correct robots.txt that gets overridden.

Here's what I mean. Some WordPress security plugins, like Wordfence or iThemes Security, can add their own crawl rules on top of your file. Some CDN configurations, like Cloudflare with certain WAF rules, block bots before they even reach your robots.txt. Some hosting panels auto-generate a robots.txt that conflicts with yours.

If you've set everything up correctly and you're still not getting cited, check those layers. Use a tool like Google Search Console's robots.txt tester, or just fetch yourdomain.com/robots.txt in a browser and read every line.

---

One More Thing: Pair It With a Sitemap

Your robots.txt should always point to your sitemap. AI crawlers use it the same way Google does, as a map of what's worth reading.

Add this line at the bottom of your robots.txt:

Sitemap: https://yourdomain.com/sitemap.xml

If you're on WordPress, Yoast and Rank Math both generate sitemaps automatically. If you're on a static site, most generators like Astro or Eleventy have sitemap plugins. Running on autopilot means both files are in place and pointing at each other.

---

FAQ

Q: Does robots.txt actually affect whether ChatGPT cites me?

Yes. GPTBot respects robots.txt directives. If you disallow GPTBot, OpenAI won't crawl your content, which means it can't appear in ChatGPT's training data or browsing results. Allowing GPTBot is a prerequisite, not a guarantee, of being cited.

Q: What happens if I have no robots.txt file at all?

Most crawlers treat a missing robots.txt as full access. But that's not a reliable setup. Some bots behave differently, and you lose control over what gets crawled. A minimal, explicit file is always better than nothing.

Q: Can I allow AI crawlers but block Google?

Yes. Each User-agent block is independent. You can write User-agent: Googlebot with Disallow: / and still have User-agent: GPTBot with Allow: /. The rules don't bleed into each other as long as you're explicit.

Q: How often do AI crawlers update their bot names?

Not often, but it happens. OpenAI has published GPTBot's official user-agent string in their documentation. It's worth checking the official docs for each platform once or twice a year. Perplexity, Anthropic, and Google all publish their bot names publicly.

Q: My robots.txt looks right but I'm still not getting cited. What else could it be?

Robots.txt is just the door. Once a bot is inside, your content still needs to answer questions clearly, use structured headings, and cover a topic well enough to be worth citing. In our voice: getting the door open is step one. Having something worth reading inside is the rest of it.