# robots-prod.txt # Production Robots File # 20190824 0749 # ----- DEFAULT CRAWLER RULES ----- User-agent: * # - RESOURCE PATHS SS - Disallow: /api/ Disallow: /admin/ Disallow: /dev/ Disallow: /health/check/ Disallow: /Security/ Disallow: /CMSSecurity/ Disallow: /RemoveOrphanedPagesTask/ Disallow: /SiteTreeMaintenanceTask/ Disallow: /UserDefinedFormController/ Disallow: /InstallerTest/ Disallow: /SapphireInfo/ Disallow: /SapphireREPL/ Disallow: /farefinder/ # - RESOURCE PATHS ALACRITY - Disallow: /_proxy # - CONTENT EDITION PATHS - Disallow: /*/utilities/search/ Disallow: /*/utilities/product-overview-transport/ Disallow: /*/listing/*/ Crawl-delay:5 # ----- DISABLED CRAWLERS ----- User-agent: *Fetch* User-agent: http://www.almaden.ibm.com/cs/crawler User-agent: BorderManager* User-agent: webcollage* User-agent: Java* User-agent: grub-client User-agent: lwp* User-agent: *TunnelPro* User-agent: LinkWalker User-agent: Offline Explorer User-agent: larbin User-agent: 008 User-agent: MJ12bot User-agent: BLEXBot User-agent: dotbot Disallow: / # ----- Swiftype specific config User-agent: Swiftbot Crawl-delay: 0.25 # ----- Yeti (Korean SEO crawler bot) User-agent: Yeti Disallow: /api/ Crawl-delay: 2 # ----- AdsBot (Google bot) User-agent: AdsBot-Google Disallow: /api/ # ----- AhrefsBot (Ahrefs crawler) User-agent: AhrefsBot Disallow: /api/ Crawl-delay: 5 # ----- SITEMAP Sitemap: https://www.newzealand.com/int/sitemap.xml Sitemap: https://www.newzealand.com/au/sitemap.xml Sitemap: https://www.newzealand.com/ca/sitemap.xml Sitemap: https://www.newzealand.com/de/sitemap.xml Sitemap: https://www.newzealand.com/in/sitemap.xml Sitemap: https://www.newzealand.com/nz/sitemap.xml Sitemap: https://www.newzealand.com/sg/sitemap.xml Sitemap: https://www.newzealand.com/uk/sitemap.xml Sitemap: https://www.newzealand.com/us/sitemap.xml Sitemap: https://www.newzealand.com/jp/sitemap.xml Sitemap: https://www.newzealand.com/kr/sitemap.xml Sitemap: https://www.newzealand.com/cn/sitemap.xml
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)