User-agent: AhrefsBot User-agent: ezooms User-agent: sistrix User-agent: MJ12bot User-agent: MegaIndex.ru User-agent: megaindex.com User-agent: PetalBot Disallow: / # AI Data Scrapers # ---------------- # This list of bots based on https://darkvisitors.com/ and https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ # Info on the different bots is possible at https://darkvisitors.com/ User-agent: CCBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: ChatGPT-User User-agent: GPTBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: anthropic-ai User-agent: Omgilibot User-agent: Omgili User-agent: FacebookBot User-agent: Diffbot User-agent: Bytespider User-agent: ImagesiftBot User-agent: PerplexityBot User-agent: cohere-ai Disallow: / User-agent: * Disallow: /*/*/account/* Disallow: /*/*/search/ Disallow: /*/*/print/ Disallow: /*/*/download/* Disallow: /*/*/basket/* Disallow: /*/enhanced/ Disallow: /Content/Compiled/ Disallow: /Content/JQueryUI/ Disallow: /Content/Flash/ Disallow: /Content/Fonts/ Disallow: /Scripts/ Disallow: /Style/ Crawl-delay: 2
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)