# Essential Search Engine Bots - Allow User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: AdsBot-Google Allow: / User-agent: Bingbot Allow: / User-agent: YandexBot Allow: / User-agent: DuckDuckBot Allow: / # Block AI Training & Large Language Model Bots User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / # Block Social Media & Content Aggregator Bots User-agent: FacebookBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / # Block Aggressive SEO & Analysis Bots User-agent: dotbot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: serpstatbot Disallow: / User-agent: GeedoBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Riddler Disallow: / User-agent: PanguBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: YouBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: AI2Bot Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: SeekportBot Disallow: / # Global Rules for All Other Bots User-agent: * # WordPress specific paths # Blocks access to all PHP files (including xmlrpc.php, wp-*.php, etc.) Disallow: /*.php$ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /readme.html Disallow: /license.txt Disallow: /wp-json/ Disallow: */feed/ Disallow: */trackback/ Disallow: */page/*/? Disallow: /tag/ Disallow: /category/*/* Disallow: /*? Disallow: /*?s= Disallow: /*&s= Disallow: /search/ Disallow: /author/ Disallow: /archive/ Disallow: */wp-login.php Disallow: */wp-json/* Disallow: */wp-admin/* Disallow: */?attachment_id=* Disallow: */?s=* Disallow: */?taxonomy=nav_menu* Disallow: */?eventDisplay=past* Disallow: */?eventDisplay=photo* Disallow: */?post_type=tribe_events&eventDisplay=day* Disallow: */?post_type=tribe_events&eventDisplay=week* Disallow: */?post_type=tribe_events&eventDisplay=month* Disallow: */?tribe-bar-date=* Disallow: *&eventDisplay=past* Disallow: *&eventDisplay=photo* Disallow: *&tribe-bar-date=* Disallow: */2009/* Disallow: */2010/* Disallow: */2011/* Disallow: */2012/* Disallow: */2013/* Disallow: */2014/* Disallow: */2015/* Disallow: */2016/* Disallow: */2017/* Disallow: */2018/* Disallow: */2019/* Disallow: */2020/* Disallow: */2021/* Disallow: */2022/* Disallow: */2023/* Disallow: */author/* Disallow: */category/* Disallow: */events/* Disallow: */organizer/* Disallow: */scripts/webalert.js?__ver=* Disallow: */tag/* Disallow: */venue/* # Allow specific content directories Allow: /wp-content/uploads/ Allow: /wp-content/themes/*/assets/ Allow: /wp-content/themes/*/images/ Allow: /wp-content/themes/*/css/ Allow: /wp-content/themes/*/js/ # Allow pagination Allow: /*?p=* Allow: /*?paged=* # Crawl-delay for remaining bots Crawl-delay: 10 # Allow Yoast SEO sitemap patterns Allow: /*?sitemap= Allow: /page-sitemap.xml Allow: /post-sitemap.xml Allow: /category-sitemap.xml Allow: /author-sitemap.xml Allow: /sitemap_index.xml Allow: /news-sitemap.xml Allow: /product-sitemap.xml Allow: /portfolio-sitemap.xml # Site Improve blocking User-agent: SiteimproveBot-Crawler Disallow: */wp-login.php Disallow: */wp-json/* Disallow: */wp-admin/* Disallow: */?attachment_id=* Disallow: */?s=* Disallow: */?taxonomy=nav_menu* Disallow: */?eventDisplay=past* Disallow: */?eventDisplay=photo* Disallow: */?post_type=tribe_events&eventDisplay=day* Disallow: */?post_type=tribe_events&eventDisplay=week* Disallow: */?post_type=tribe_events&eventDisplay=month* Disallow: */?tribe-bar-date=* Disallow: *&eventDisplay=past* Disallow: *&eventDisplay=photo* Disallow: *&tribe-bar-date=* Disallow: */2009/* Disallow: */2010/* Disallow: */2011/* Disallow: */2012/* Disallow: */2013/* Disallow: */2014/* Disallow: */2015/* Disallow: */2016/* Disallow: */2017/* Disallow: */2018/* Disallow: */2019/* Disallow: */2020/* Disallow: */2021/* Disallow: */2022/* Disallow: */2023/* Disallow: */author/* Disallow: */category/* Disallow: */events/* Disallow: */organizer/* Disallow: */scripts/webalert.js?__ver=* Disallow: */tag/* Disallow: */venue/* Crawl-delay: 3 # START YOAST BLOCK # --------------------------- User-agent: * Disallow: Sitemap: https://universitypolicy.unc.edu/sitemap_index.xml # --------------------------- # END YOAST BLOCK
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)