# For all robots User-agent: * # Block access to specific groups of pages Disallow: /account/* Disallow: /lukas/* # Restrict FacebookBot/facebookexternalhit User-agent: FacebookBot Crawl-delay: 5 User-agent: facebookexternalhit Crawl-delay: 5 # Block CazoodleBot as it does not present correct accept content headers User-agent: CazoodleBot Disallow: /* # Block MJ12bot as it is just noise User-agent: MJ12bot Disallow: /* # Block dotbot as it cannot parse base urls properly User-agent: dotbot/1.0 Disallow: /* # Block Gigabot User-agent: Gigabot Disallow: /*
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)