Skip to main content

​
robots.txt

Purpose: Control crawler access, explicitly allow AI crawlers
Location: shop.anitone.com.au/robots.txt

​
File Contents

# Anitone Shop - robots.txt
# Optimized for AEO (Answer Engine Optimization) - January 2026

# ===========================================
# STANDARD CRAWLERS
# ===========================================

User-agent: Googlebot
Allow: /
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /admin/

User-agent: Bingbot
Allow: /
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

# ===========================================
# AI/LLM CRAWLERS - EXPLICITLY ALLOWED
# Per AEO best practices: Allow GPTBot, ClaudeBot, PerplexityBot
# ===========================================

User-agent: GPTBot
Allow: /
Allow: /collections/
Allow: /products/
Allow: /pages/
Allow: /llms.txt
Allow: /llms-full.txt
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

User-agent: ChatGPT-User
Allow: /
Allow: /collections/
Allow: /products/
Allow: /pages/
Allow: /llms.txt
Allow: /llms-full.txt
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

User-agent: ClaudeBot
Allow: /
Allow: /collections/
Allow: /products/
Allow: /pages/
Allow: /llms.txt
Allow: /llms-full.txt
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

User-agent: anthropic-ai
Allow: /
Allow: /collections/
Allow: /products/
Allow: /pages/
Allow: /llms.txt
Allow: /llms-full.txt
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

User-agent: PerplexityBot
Allow: /
Allow: /collections/
Allow: /products/
Allow: /pages/
Allow: /llms.txt
Allow: /llms-full.txt
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

User-agent: Bytespider
Allow: /
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/

# ===========================================
# OTHER SEARCH ENGINES
# ===========================================

User-agent: DuckDuckBot
Allow: /

User-agent: Slurp
Allow: /

User-agent: facebookexternalhit
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: LinkedInBot
Allow: /

# ===========================================
# DEFAULT RULE
# ===========================================

User-agent: *
Allow: /
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Disallow: /admin/
Disallow: /orders/
Disallow: /services/
Disallow: /challenge/

# ===========================================
# SITEMAP
# ===========================================

Sitemap: https://shop.anitone.com.au/sitemap.xml

# ===========================================
# LLM CONTEXT FILES
# ===========================================

# Token-efficient content for LLM ingestion
# llms.txt - Concise summary (recommended first read)
# llms-full.txt - Complete documentation in markdown

​
Key Features

​
AI Crawlers Allowed

User-agentPurpose
GPTBotOpenAI’s crawler
ChatGPT-UserChatGPT browsing
ClaudeBotAnthropic’s crawler
anthropic-aiAnthropic AI
PerplexityBotPerplexity AI
BytespiderByteDance AI

​
Protected Paths

All crawlers are blocked from:
  • /checkout/ - Checkout process
  • /cart/ - Shopping cart
  • /account/ - Customer accounts
  • /admin/ - Admin areas
  • /orders/ - Order information
  • /services/ - Internal services
  • /challenge/ - Bot challenges

​
LLM-Specific Allows

AI crawlers have explicit access to:
  • /collections/ - Product collections
  • /products/ - Product pages
  • /pages/ - Content pages
  • /llms.txt - Concise summary
  • /llms-full.txt - Full documentation

​
Implementation Note

Shopify generates its own robots.txt. To customize:
  1. Go to Online Store → Themes → Edit code
  2. Find robots.txt.liquid in Templates
  3. Add the AI crawler rules