# For all robots User-agent: * # Block access to specific groups of pages Disallow: */cart Disallow: */checkout* Disallow: */my-account* Disallow: */produktvergleich Disallow: */Comparaison-des-produits Disallow: /api/* Disallow: */c/su_m_99* Disallow: */c/sale_su_m_* Crawl-delay: 10 # 10 seconds between page requests # Allow search crawlers to discover the sitemap Sitemap: https://www.allchemet.ch/sitemap.xml # Block CazoodleBot as it does not present correct accept content headers User-agent: CazoodleBot Disallow: / # Block MJ12bot as it is just noise User-agent: MJ12bot Disallow: / # Block dotbot as it cannot parse base urls properly User-agent: dotbot/1.0 Disallow: / # Block Gigabot User-agent: Gigabot Disallow: / #Definition for FacebookBot because it doesnt respect settings for Useragent * User-agent: facebookexternalhit/1.1 # Block access to specific groups of pages Disallow: */cart Disallow: */checkout* Disallow: */my-account* Disallow: */produktvergleich Disallow: */Comparaison-des-produits Disallow: /api/* Disallow: */c/SU_H_CUSTOM* Crawl-delay: 10 # 10 seconds between page requests
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)