Show your SEO score to your visitors using the Seobility widget. The widget always displays the current SEO score of your homepage.
# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/robotstxt.html User-agent: Bad-Robot Disallow: * User-agent: GPTBot Disallow: / User-agent: PiplBot Disallow: / User-agent: CCbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: trendictionbot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: PetalBot Disallow: / User-agent: viberbot Disallow: / User-agent: YaK Disallow: / User-agent: Yandex Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-Agent: FacebookBot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: SemrushBot-BA Disallow: / User-agent: SemrushBot-SI Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: SplitSignalBot Disallow: / User-agent: SemrushBot-OCOB Disallow: / User-Agent: PerplexityBot Disallow: / User-agent: Mail.Ru Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: MojeekBot Disallow: / User-agent: PetalBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Seekr Disallow: / User-agent: SeznamBot Disallow: / User-agent: keys-so-bot Disallow: / # AwarioSmartBot and AwarioRssBot are legitimate web crawlers sent by Awario to discover AwarioSmartBot Disallow: / AwarioRssBot Disallow: / User-agent: SummalyBot Disallow: / User-agent: Applebot-Extended Disallow: / User-Agent: ZoomBot Disallow: / User-agent: dotbot Disallow: / User-agent: Gabanzabot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: YandexBot Disallow: / User-agent: YandexUserproxy Disallow: / User-agent: PubMatic Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: coccocbot-web Disallow: / #scraper SummalyBot Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/bytespider User-agent: Bytespider Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/ccbot User-agent: CCBot Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/claudebot User-agent: ClaudeBot Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/diffbot User-agent: Diffbot Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/meta-externalagent User-agent: Meta-ExternalAgent Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/omgili User-agent: omgili Disallow: / # AI Data Scraper # https://darkvisitors.com/agents/timpibot User-agent: Timpibot Disallow: / # Undocumented AI Agent # https://darkvisitors.com/agents/claude-web User-agent: Claude-Web Disallow: / # Undocumented AI Agent # https://darkvisitors.com/agents/cohere-ai User-agent: cohere-ai Disallow: / User-agent: Googlebot-Image Disallow: /path/to/visitor/jpg/files/*.jpg$ Disallow: /path/to/visitor/gif/files/*.gif$ Disallow: /path/to/visitor/png/files/*.png$ User-agent: * Crawl-delay: 5 # CSS, JS, Images Allow: /core/*.css$ Allow: /core/*.css? Allow: /core/*.js$ Allow: /core/*.js? Allow: /core/*.gif Allow: /core/*.jpg Allow: /core/*.jpeg Allow: /core/*.png Allow: /core/*.svg Allow: /profiles/*.css$ Allow: /profiles/*.css? Allow: /profiles/*.js$ Allow: /profiles/*.js? Allow: /profiles/*.gif Allow: /profiles/*.jpg Allow: /profiles/*.jpeg Allow: /profiles/*.png Allow: /profiles/*.svg # Directories Disallow: /core/ Disallow: /profiles/ # Files Disallow: /README.txt Disallow: /web.config # Paths (clean URLs) Disallow: /admin/ Disallow: /comment/reply/ Disallow: /filter/tips Disallow: /node/add/ Disallow: /search/ Disallow: /user/register Disallow: /user/password Disallow: /user/login Disallow: /user/logout Disallow: /media/oembed Disallow: /*/media/oembed # Paths (no clean URLs) Disallow: /index.php/admin/ Disallow: /index.php/comment/reply/ Disallow: /index.php/filter/tips Disallow: /index.php/node/add/ Disallow: /index.php/search/ Disallow: /index.php/user/password Disallow: /index.php/user/register Disallow: /index.php/user/login Disallow: /index.php/user/logout Disallow: /index.php/media/oembed Disallow: /index.php/*/media/oembed Disallow: /search Disallow: /wp-content/ Disallow: /wp-includes/
Following keywords were found. You can check the keyword optimization of this page for each keyword.
Please select one of the following options for your request.
None of these options suit your need?
You can also download our Quick Start Guide, visit the FAQ section, our Knowledge Base or Blog.
Meta Title
(Nice to have)