Server firewall returning 403 to AI crawlers

Requesting any page with the user-agent of an AI crawler returns 403 Forbidden. Requesting the exact same URL as Googlebot, as Bingbot, or from an ordinary Chrome browser returns 200 OK. The 403 is returned for every path we tested, including the homepage, robots.txt, sitemap.xml and static CSS files, so it is not specific to any page or file type.

If crawlers being refused are GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot. The crawlers being allowed are Googlebot and Bingbot.

When response carries the header server: LiteSpeed and returns the default LiteSpeed 403 page. Our own robots.txt allows all of these crawlers explicitly, and there are no bot rules, user-agent rules or rewrite conditions in any .htaccess file under our accounts. This indicates the block is applied at the server or WAF layer, above our document root, so we cannot change it ourselves.

Add following to top of your .htaccess file to allow known crawlers:

SetEnvIfNoCase User-Agent .. !bad_bot
SetEnvIfNoCase User-Agent .*ChatGPT-User.* !bad_bot
SetEnvIfNoCase User-Agent .*GPTBot.* !bad_bot
SetEnvIfNoCase User-Agent .*OpenAI.* !bad_bot
SetEnvIfNoCase User-Agent .*ClaudeBot.* !bad_bot
SetEnvIfNoCase User-Agent .*GrokBot.* !bad_bot
SetEnvIfNoCase User-Agent .*OAI-SearchBot.* !bad_bot
SetEnvIfNoCase User-Agent .*ChatGPT.* !bad_bot
SetEnvIfNoCase User-Agent .*Perplexity.* !bad_bot
SetEnvIfNoCase User-Agent .*Screaming Frog SEO Spider.* !bad_bot
SetEnvIfNoCase User-Agent .*AhrefsBot.* !bad_bot
SetEnvIfNoCase User-Agent .*SemrushBot.* !bad_bot
SetEnvIfNoCase User-Agent .*Ahrefs.* !bad_bot 
SetEnvIfNoCase User-Agent .*Semrush.* !bad_bot

You can also adjust it as per your requirements.