# Block all robots going to the following directories User-agent: * Disallow: /search/ Disallow: /newsletter/ Disallow: */pf/api/ # OneTrustBot can go anywhere User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0;OneTrustBot; Disallow: # AI bots specific crawl-delay User-agent: anthropic-ai User-agent: Bytespider User-agent: ChatGPT User-agent: ClaudeBot User-agent: GPTBot User-agent: OAI-SearchBot Crawl-delay: 1 Disallow: /search/ Disallow: /newsletter/ Disallow: */pf/api/ # Other bots with specific crawl-delay User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-agent: Amazonbot User-agent: Archive-It User-agent: Audacy-Podcast-Scraper User-agent: Discordbot User-agent: facebookexternalhit User-agent: FeedBurner User-agent: Feedly User-agent: feedparser User-agent: FeedValidator User-agent: flipboard User-agent: Gofeed User-agent: HubSpot Crawler User-agent: NetNewsWire User-agent: Pinterestbot User-agent: Twitterbot User-agent: WhatsApp Crawl-delay: 1 Disallow: /search/ Disallow: /newsletter/ Disallow: */pf/api/ # Unwanted bots User-agent: 2ip bot User-agent: 360Spider User-agent: 8LEGS User-agent: AdsBot User-agent: AliyunSecBot User-agent: ArchiveBot User-agent: AwarioBot User-agent: BacklinksExtendedBot User-agent: Baiduspider User-agent: Barkrowler User-agent: Birdcrawlerbot User-agent: BitSightBot User-agent: blackboardally User-agent: BLEXBot User-agent: Blogtrottr User-agent: BluechipBacklinks User-agent: BPImageWalker User-agent: Bravebot User-agent: CCBot User-agent: Cincraw User-agent: cis5550-crawler User-agent: CloudServerMarketSpider User-agent: coccocbot User-agent: COIBotParser User-agent: CrsspxlBot User-agent: CyberFindCrawler User-agent: CyotekWebCopy User-agent: Darwin User-agent: DataForSeoBot User-agent: DeepCrawl User-agent: Diffbot User-agent: domainsbot User-agent: DomainStatsBot User-agent: DomCopBot User-agent: DotBot User-agent: Dragonbot User-agent: Dubbotbot User-agent: EmailWolf User-agent: equellaurlbot User-agent: ev-crawler User-agent: Exabot User-agent: Facebot User-agent: Faraday User-agent: feedbot User-agent: FileMaker User-agent: GLMSLinkAnalysis User-agent: GrapeshotCrawler User-agent: harsilbot User-agent: iaskspider User-agent: ImagesiftBot User-agent: INETDEX-BOT User-agent: Informa Java API User-agent: Intently User-agent: io_bot User-agent: ISSCyberRiskCrawler User-agent: istellabot User-agent: kazbtbot User-agent: Linguee Bot User-agent: LinkAnalyser User-agent: linkdexbot User-agent: linkfluence User-agent: LivelapBot User-agent: Magus Bot User-agent: Maxthon User-agent: MixrankBot User-agent: MJ12bot User-agent: MJ12Bot User-agent: MojeekBot User-agent: Monsidobot User-agent: MoodleBot User-agent: NetcraftSurveyAgent User-agent: netEstate NE Crawler User-agent: Neticle Crawler User-agent: Netpeak User-agent: Newsify User-agent: Nicecrawler User-agent: NodePing User-agent: online-webceo-bot User-agent: OpenindexSpider User-agent: Orbbot User-agent: Paqlebot User-agent: PetalBot User-agent: PocketParser User-agent: PubMatic Crawler Bot User-agent: RankurBot User-agent: Scrapy User-agent: SearchBot User-agent: SEBot User-agent: Seekport Crawler User-agent: SeekportBot User-agent: SeekrBot User-agent: SemanticScholarBot User-agent: SentiBot User-agent: seo-audit-check-bot User-agent: SeobilityBot User-agent: serpstatbot User-agent: SeznamBot User-agent: SeznamBot User-agent: SimplePie User-agent: SiteAuditBot User-agent: Sitebulb User-agent: SiteOne-Crawler User-agent: SMTBot User-agent: Sprinklr User-agent: SrceDAMP User-agent: Staffbase User-agent: StartmeBot User-agent: Storebot User-agent: SuperBot User-agent: superpagesbot2 User-agent: SurdotlyBot User-agent: Tentacles User-agent: Timpibot User-agent: trendictionbot0 User-agent: Turnitin User-agent: uk_lddc_bot User-agent: ultimate_sitemap_parser User-agent: vebidoobot User-agent: VelenPublicWebCrawler User-agent: WallabyupBot User-agent: WebwikiBot User-agent: WellKnownBot User-agent: WepchSearchEngine User-agent: Xbot User-agent: Yeti User-agent: YisouSpider User-agent: ZoominfoBot Disallow: / # Sitemaps Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/news-sitemap/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-section/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/videos/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/promotional-features/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/product-innovations/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/suppliers/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/events/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/News/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Restaurant-Openings/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Trends-Reports/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Business-Legislation/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Product-Launches/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Latest/Events/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Casual-Dining/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Fine-Dining/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/QSR-Street-Food/ Sitemap: https://www.restaurantonline.co.uk/arc/outboundfeeds/sitemap-index/category/Chef/
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)