# Robots.txt for TrackaBeast.com # Purpose: LLM Friendly, Scraper Hostile, Security Focused # ========================================== # SECTION 1: AI & SEO FRIENDLY ACCESS # ========================================== # We explicitly allow reputable AI and SEO crawlers to index public content # to improve TrackaBeast's visibility in search and AI assistants. User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: AhrefsBot Allow: / User-agent: SEMrushBot Allow: / # ========================================== # SECTION 2: BLOCKING MALICIOUS / FAKE BOTS # ========================================== # Blocking aggressive scrapers, data-hoarders, and fake bots # that often ignore crawl delays or attempt to exploit vulnerabilities. User-agent: CCBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: Bytespider Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / # ========================================== # SECTION 3: GENERAL SECURITY & CRAWL RULES # ========================================== User-agent: * # Standard Directories to protect Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /config/ Disallow: /cache/ Disallow: /tmp/ Disallow: /search?* Disallow: /?author= Disallow: /wp-admin/ Disallow: /login/ # Optimization: Set a crawl delay for generic/unknown bots to save server resources Crawl-delay: 5 # ========================================== # SECTION 4: SITEMAP # ========================================== Sitemap: https://www.trackabeast.com/sitemap.xml