Bsmall.co.uk - SEO Check

Meta Tags

Medienliste

Überschriftenstruktur

Gefundene Links auf dieser Seite

HTTP-Header

Links von Wikipedia

Robots.txt

# START BOOKSWARM ROBOTS.TXT TEMPLATE
# -----------------------------------
#
# Set the crawl delay to 7 seconds - not all search engines will honour this
User-agent: *
Crawl-delay: 7
#
# Tell all user agents to ignore wp-admin
User-agent: *
Disallow: /wp-admin/
#
# Block AI bots
User-agent: GPTBot
User-agent: CCBot
User-agent: ChatGPT-User
Disallow: /
#
# Block these other bots
User-agent: Scrapy
User-agent: Buck
User-agent: TinyTestBot
User-agent: SEMrushBot
User-agent: AhrefsBot
User-agent: PetalBot
User-agent: MJ12Bot
User-agent: DotBot
User-agent: MauiBot
User-agent: YandexBot
User-agent: Baiduspider
User-agent: Barkrowler
User-agent: Bytespider
User-agent: WhatStuffWhereBot
User-agent: Applebot
User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
User-agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
User-agent: Swiftbot
User-agent: Slurp
User-agent: CCBot/2.0 (https://commoncrawl.org/faq/)
User-agent: CCBot/2.0
User-agent: CCBot/2.0 (http://commoncrawl.org/faq/)
Disallow: /
#
# Allow Googlebot and other specific bots
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
Disallow:
#
# ---------------------------------
# END BOOKSWARM ROBOTS.TXT TEMPLATE
#        _
#       [ ]
#      (   )
#       |>|
#    __/===\__
#   //| o=o |\\
# <]  | o=o |  [>
#     \=====/
#    / / | \ \
#   <_________>

Suchvorschau