Ausdauerleistung.de - SEO Checker

Meta tags

Media list

Heading structure

Found links on this page

HTTP Response Header

Links from Wikipedia

Robots.txt

User-agent: *
Crawl-delay: 30
Disallow: /address_book_process.php 
Disallow: /address_book_process.php/ 
Disallow: /account.php 
Disallow: /account.php/ 
Disallow: /account_edit.php 
Disallow: /account_edit.php/ 
Disallow: /account_edit_process.php 
Disallow: /account_edit_process.php/ 
Disallow: /account_history.php 
Disallow: /account_history.php/ 
Disallow: /account_history_info.php 
Disallow: /account_history_info.php/
Disallow: /address_book.php 
Disallow: /address_book.php/ 
Disallow: /ajax_handler.php 
Disallow: /ajax_handler.php/ 
Disallow: /banned.php
Disallow: /banned.php/
Disallow: /checkout_process.php 
Disallow: /checkout_process.php/ 
Disallow: /advanced_search.php 
Disallow: /advanced_search.php/ 
Disallow: /advanced_search_result.php 
Disallow: /advanced_search_result.php/ 
Disallow: /checkout_address.php 
Disallow: /checkout_address.php/ 
Disallow: /checkout_shipping.php 
Disallow: /checkout_shipping.php/
Disallow: /checkout_payment.php
Disallow: /checkout_payment.php/
Disallow: /checkout_confirmation.php 
Disallow: /checkout_confirmation.php/ 
Disallow: /checkout_success.php 
Disallow: /checkout_success.php/
Disallow: /create_account.php
Disallow: /create_account.php/
Disallow: /create_ebay_account.php
Disallow: /create_ebay_account.php/
Disallow: /create_guest_account.php 
Disallow: /create_guest_account.php/
Disallow: /login.php 
Disallow: /login.php/
Disallow: /logoff.php
Disallow: /logoff.php/
Disallow: /renew_pwd.php
Disallow: /renew_pwd.php/
Disallow: /popup_image.php 
Disallow: /popup_image.php/ 
Disallow: /product_notifications.php 
Disallow: /product_notifications.php/ 
Disallow: /product_reviews.php 
Disallow: /product_reviews.php/ 
Disallow: /product_reviews_info.php 
Disallow: /product_reviews_info.php/ 
Disallow: /reviews.php 
Disallow: /reviews.php/ 
Disallow: /shopping_cart.php 
Disallow: /shopping_cart.php/ 
Disallow: /admin/
Disallow: /cache/
Disallow: /cgi-bin/
Disallow: /download/ 
Disallow: /export/
Disallow: /includes/ 
Disallow: /pub/ 
Disallow: /media/

##########################################################################################################
############# unerwuenschte bots, die aber die robots.txt abfragen
##########################################################################################################
#Despicable and evil robots to keep out :) 

User-agent: 008
User-agent: Afilias Web Mining Tool
User-agent: AhrefsBot
User-agent: aiHitBot
User-agent: Aqua_Products
User-agent: asterias
User-agent: b2w/0.1
User-agent: BackDoorBot/1.0
User-agent: BacklinkCrawler
User-agent: Baiduspider
User-agent: BlowFish/1.0
User-agent: Bookmark search tool
User-agent: BotALot
User-agent: BPImageWalker
User-agent: BPImageWalker*
User-agent: bdbrandprotect
User-agent: Birubot
User-agent: bixolabs
User-agent: BotOnParade
User-agent: BUbiNG
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: Bullseye
User-agent: BunnySlippers
User-agent: CatchBot
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerSE/1.0
User-agent: CherryPickerElite/1.0
User-agent: COMODO SSL Checker
User-agent: Comodo-Certificates-Spider
User-agent: Content Crawler
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: Crescent
User-agent: DCPbot
User-agent: discobot
User-agent: DittoSpyder
User-agent: EC2LinkFinder
User-agent: EdisterBot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: Eurobot
User-agent: Exabot
User-agent: ExDomain
User-agent: ExtractorPro
User-agent: EZooms
User-agent: FairAd Client
User-agent: findfiles.net
User-agent: findlinks
User-agent: Foobot
User-agent: Gaisbot
User-agent: GetRight/4.2
User-agent: Gigabot
User-agent: gonzo
User-agent: grub
User-agent: grub-client
User-agent: Harvest/1.5
User-agent: hloader
User-agent: htdig
User-agent: httplib
User-agent: HuaweiSymantecSpider
User-agent: humanlinks
User-agent: ia_archiver/1.6
User-agent: ia_archiver
User-agent: ICCrawler - iCjobs
User-agent: ichiro
User-agent: iCjobs
User-agent: InfoNaviRobot
User-agent: ips-agent 
User-agent: Iron33/1.0.2
User-agent: JennyBot
User-agent: JikeSpider
User-agent: KaloogaBot
User-agent: Kenjin Spider
User-agent: Keyword Density/0.9
User-agent: larbin
User-agent: lb-spider
User-agent: lex
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: linkdex.com
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkScan
User-agent: LinkWalker
User-agent: LNSpiderguy
User-agent: looksmart
User-agent: lwp-trivial/1.34
User-agent: lwp-trivial
User-agent: magpie-crawler
User-agent: Mata Hari
User-agent: Microsoft URL Control - 6.00.8169
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control
User-agent: MIIxpc/4.2
User-agent: MIIxpc
User-agent: Mister PiX
User-agent: MJ12bot
User-agent: MLBot
User-agent: moget/2.1
User-agent: moget
User-agent: MSIECrawler
User-agent: NerdByNature.Bot
User-agent: NetAnts
User-agent: netEstate NE Crawler
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Nutch
User-agent: oBot
User-agent: Offline Explorer
User-agent: OneRiot
User-agent: Openbot
User-agent: Openfind data gathere
User-agent: Openfind 
User-agent: OpenindexSpider
User-agent: OpidooBOT 
User-agent: Oracle Ultra Search
User-agent: PagePeeker
User-agent: PerMan
User-agent: picmole
User-agent: Pixray-Seeker
User-agent: plukkie
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: Purebot
User-agent: Python-urllib
User-agent: Qualidator*
User-agent: QueryN Metasearch
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RepoMonkey
User-agent: ReverseGet
User-agent: RMA
User-agent: schrein
User-agent: Scooter
User-agent: ScoutJet
User-agent: Screaming Frog SEO Spider
User-agent: search17
User-agent: searchpreview
User-agent: SemrushBot
User-agent: SeznamBot
User-agent: SEOkicks-Robot
User-agent: SEOkicks
User-agent: sistrix
User-agent: Sitebot
User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SpankBot
User-agent: spanner
User-agent: spbot
User-agent: Speedy
User-agent: Spinn3r
User-agent: suggybot
User-agent: suzuran
User-agent: SWEBot
User-agent: Szukacz/1.4 
User-agent: Teleport
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The IntraformantUser-agent: *thunderstone*
User-agent: TheNomad
User-agent: TinEye
User-agent: True_Robot/1.0
User-agent: True_Robot
User-agent: toCrawl/UrlDispatcher
User-agent: turingos
User-agent: TurnitinBot
User-agent: UnisterBot
User-agent: Unister*
User-agent: UnwindFetchor
User-agent: Updownerbot
User-agent: URL Control
User-agent: URL_Spider_Pro
User-agent: URLy Warning
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VCI
User-agent: VoilaBot
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebCopier
User-agent: WebEnhancer
User-agent: Webinator
User-agent: WebmasterCoffee
User-agent: WebmasterWorldForumBot
User-agent: WebReaper
User-agent: WebRipper
User-agent: WebSauger
User-agent: WBSearchBot
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebZip/4.0
User-agent: WebZip
User-Agent: weneobot
User-agent: Wget/1.6
User-agent: Wget/1.5.3
User-agent: Wget
User-agent: WWW-Collector-E
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Xenu's
User-agent: yacybot
User-Agent: Yandex
User-agent: Yeti
User-agent: Yeti-Mobile
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: Zeus
User-agent: Zookabot
User-agent: ZyBORG
Crawl-delay: 3600
Disallow: /


################
##### monitor:
################

#"ssearch_bot (sSearch Crawler; http://www.semantissimo.de)"
#"Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)"
#"Mozilla/5.0 (compatible; lemurwebcrawler [email protected]; +http://boston.lti.cs.cmu.edu/crawler_12/)"

########################################################################################################
############# unerwünschte bots, die die robots.txt NICHT abfragen, gehören ggf. per Rewrite gesperrt:
########################################################################################################

#"Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)" IGNORIERT ROBOTS.TXT
#"Mozilla/5.0 (compatible; TweetmemeBot/2.11; +http://tweetmeme.com/)" IGNORIERT ROBOTS.TXT


#########################################################################################################
##### in der Regel okay:
##########################################################################################################

#"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
#"Googlebot-Image/1.0"
#"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
#"Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)"
#"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
#"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
#"Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
#"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
#"msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)"
#"Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)"
#"CloudACL/Nutch-1.4"
#"webcrawler (compatible; heritrix/1.14.4 ++http://www.onb.ac.at/about/webarchivierung.htm)"
#"Mail.RU/2.0" (russ. Suchmaschine)
#"Sosospider+(+http://help.soso.com/webspider.htm)" (chin. Suchmaschine)
#"ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])" (hängt auch mit archive.org zusammen)
#"Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
#"Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)"
#"Eurobot/1.1 (http://eurobot.ayell.eu)"
#"Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0" (koreanische Suchmaschine)
#"Acoon v4.10.3 (www.acoon.de)"
#"DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)" (jap. Suchmaschine)
#"ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)"
#"frogl-bot (Version: 1.06, powered by www.frogl.de +http://www.frogl.de/pfadzurbotseite/bot.html)"
#"Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)"
#"Agent-SharewarePlazaBot/3.0+(+http://www.SharewarePlaza.com)" IGNORIERT ROBOTS.TXT
#"Wotbox/2.0 ([email protected]; http://www.wotbox.com)" IGNORIERT ROBOTS.TXT
#"www.freefileszone.com PadPollbot/1.1b (+http://www.freefileszone.com/)" IGNORIERT ROBOTS.TXT
#"Mozilla/5.0 (compatible; Sitedomain-Bot 1.0; Headers only; +http://www.sitedomain.de/sitedomain-bot/)" IGNORIERT ROBOTS.TXT - checkt auf gelöschte Domains - ruft nur Hauptseite auf
#"emefgebot/beta (+http://emefge.de/bot.html)" IGNORIERT ROBOTS.TXT

Search preview