# For all robots User-agent: * # Block access to specific groups of pages Disallow: */cart/ Disallow: */checkout/ Disallow: */my-account/ Disallow: */search/ # Temporary Disallow: /fi_FI/ Disallow: /et_EE/ # Allow search crawlers to discover the sitemap Sitemap: https://shop.spandex.com/cs_CZ/sitemap.xml Sitemap: https://shop.spandex.com/de_AT/sitemap.xml Sitemap: https://shop.spandex.com/de_CH/sitemap.xml Sitemap: https://shop.spandex.com/de_DE/sitemap.xml Sitemap: https://shop.spandex.com/en_AU/sitemap.xml Sitemap: https://shop.spandex.com/en_GB/sitemap.xml Sitemap: https://shop.spandex.com/es_ES/sitemap.xml Sitemap: https://shop.spandex.com/fr_BE/sitemap.xml Sitemap: https://shop.spandex.com/fr_CH/sitemap.xml Sitemap: https://shop.spandex.com/fr_FR/sitemap.xml Sitemap: https://shop.spandex.com/it_IT/sitemap.xml Sitemap: https://shop.spandex.com/it_CH/sitemap.xml Sitemap: https://shop.spandex.com/nl_BE/sitemap.xml Sitemap: https://shop.spandex.com/nl_NL/sitemap.xml Sitemap: https://shop.spandex.com/sk_SK/sitemap.xml Sitemap: https://shop.spandex.com/sv_SE/sitemap.xml Sitemap: https://shop.spandex.com/no_NO/sitemap.xml Sitemap: https://shop.spandex.com/tr_TR/sitemap.xml Sitemap: https://shop.spandex.com/en_US/sitemap.xml # Block CazoodleBot as it does not present correct accept content headers User-agent: CazoodleBot Disallow: / # Block MJ12bot as it is just noise User-agent: MJ12bot Disallow: / # Block dotbot as it cannot parse base urls properly User-agent: dotbot/1.0 Disallow: / # Block Gigabot User-agent: Gigabot Disallow: / # Block Screaming Frog SEO Spider User-agent: Screaming Frog SEO Spider Disallow: / User-agent: OnPageBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: TMM Crawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: CCBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: YandexBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Nutch Disallow: / User-agent: ia_archiver Disallow: / User-agent: WebCopier Disallow: / User-agent: WebStripper Disallow: / User-agent: Teleport Disallow: / User-agent: Yandex Disallow: / User-agent: Offline Explorer Disallow: / User-agent: SiteSnagger Disallow: / User-agent: psbot Disallow: / User-agent: TeleportPro Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailCollector Disallow: / User-agent: larbin Disallow: / User-agent: MSIECrawler Disallow: / User-agent: moget Disallow: /
Folgende Keywords wurden erkannt. Überprüfe die Optimierung dieser Keywords für Deine Seite.
(Nice to have)