Übersicht der SEO Analyse
Metaangaben
80% 
Seitenqualität
85% 
Seitenstruktur
35% 
Verlinkung
25% 
Server
74% 
Externe Faktoren
100% 
SEO Score
Antwortzeit
0,76 s
Dateigröße
60,90 kB
Wörter
1361
Medien
39
Anzahl Links
138 Intern / 3 Extern

To-do Liste mit SEO Optimierungen

  • Sehr wichtig! Vermeiden Sie interne Links mit dynamischen URL-Parametern.
  • Sehr wichtig! Reduzieren Sie die Anzahl der eingebundenen CSS-Dateien.
  • Sehr wichtig! Reduzieren Sie die Anzahl der eingebundenen JavaScript-Dateien.
  • Wichtig! Verbessern Sie den Seitentitel.
  • Wichtig! Verbessern Sie den Text der Meta-Description.
Weitere To-dos anzeigen

Meta-Angaben im HTML

Titel
(Extrem wichtig)
Pferde kaufen, Pferde verkaufen und Pferde suchen - Westernpferde kaufen - HorseBase Pferdemarkt
Der Titel sollte kürzer als 580 Pixel sein. Er ist insgesamt 814 Pixel lang.
Im Titel werden Worte wiederholt.
Meta-Description
(Extrem wichtig)
Pferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.
Die Meta-Description ist zu lang. (1631 Pixel von maximal 1000 Pixel)
Crawlbarkeit
(Extrem wichtig)
Es gibt keine Probleme beim Zugriff auf die Webseite.
Canonical Link
(Wichtig)
Es ist kein Canonical Link angegeben.
Sprache
(Wenig wichtig)
Im Text erkannte Sprache: de
Im HTML angegebene Sprache: de-de
Serverstandort: Finnland
Die Sprache wird im HTML Code wie folgt angegeben: de-de
Alternate/Hreflang Links
(Wenig wichtig)
Die Seite nutzt keine Alternate Links.
Weitere Metatags
(Wenig wichtig)
Es gibt keinen rel next Meta Tag auf der Seite.
Es gibt keinen rel prev Meta Tag auf der Seite.
Domain
(Wenig wichtig)
Die Domain ist keine Subdomain.
Die Länge der Domain ist gut.
Die Domain enthält keine Umlaute.
Seiten URL
(Wenig wichtig)
In der URL wurden keine Parameter entdeckt.
In der URL wurde keine Session ID entdeckt.
Die URL hat nicht zu viele Unterverzeichnisse.
Zeichensatzkodierung
(Wenig wichtig)
Die Angaben zur Zeichensatzkodierung (UTF-8) sind fehlerfrei.
Doctype
(Nice to have)
Die Doctype Angabe XHTML 1.0 Transitional ist korrekt angegeben.
Die Doctype Angabe befindet sich an erster Stelle im HTML-Code.
Favicon
(Nice to have)
Das Favoriten Icon (Favicon) ist korrekt verlinkt.

Meta Tags

NameWert
robotsindex, follow
keywordsPferde kaufen, Pferde verkaufen, Pferde, Pferdemarkt, Pferdeangebote, Pferdeanzeigen, Pferdebörse, Pferdeboerse, Pferdeinserate, Pferdeverkauf, Pferdekauf, Pferde inserieren, Pferde kaufen, Pferdehandel, Pferd kaufen, Pferdekauf, Deckhengste, Deckanzeigen
titlePferdemarkt
descriptionPferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.
generatorHorseBase CMS
langde-de
content-typetext/html; charset=utf-8

Analysieren Sie jetzt kostenlos bis zu 1.000 Unterseiten von horsebase.de!

Kostenlos registrieren
Die Nutzung des Basis Accounts ist zeitlich unbegrenzt möglich

Seitenqualität

Inhalt
(Extrem wichtig)
Auf der Seite wurden Tippfehler entdeckt:
  • grossen => großen
Der Inhalt ist mit 1361 Wörtern in Ordnung.
Der Text besteht zu 27.1% aus Füllwörtern.
Worte aus dem Titel werden im Text wiederholt.
Im Text befindet sich eine Aufzählung, dies deutet auf eine gute Textstruktur hin.
Es wurden 29 Fließtextblöcke auf der Seite gefunden.
Der Text auf der Seite ist optimal.
Es befinden sich keine Duplikate auf der Seite.
Die durchschnittliche Satzlänge ist mit 10.8 Wörtern gut.
Frames
(Extrem wichtig)
Die Seite hat kein Frameset.
Mobile
(Wenig wichtig)
Es ist kein Viewport angegeben.
Die Webseite lädt 6 Javascript Dateien, dies kann die Ladezeit negativ beeinträchtigen.
Mindestens ein Apple-Touch Icon ist definiert.
Bold- und Strongtags
(Wenig wichtig)
Die Nutzung von Strong- und Bold-Tags ist optimal. Wir empfehlen für diese Webseite die Verwendung von bis zu 27 Tags.
Bilder Optimierung
(Wenig wichtig)
Es gibt keine Bilder ohne ALT-Attribute.
Soziale Vernetzung
(Nice to have)
Es befinden sich wenige Social-Sharing Möglichkeiten auf der Seite. Mit Plugins zum Teilen, kann die Reichweite der Seite in sozialen Netzwerken erhöht werden.
Zusätzliches Markup
(Nice to have)
Es wurde kein zusätzliches Markup gefunden.
Werbung
(Wenig wichtig)
Es befindet sich etwas Werbung auf der Seite.
HTTPS
(Wenig wichtig)
Die Seite verwendet HTTPS um Daten sicher zu übertragen.
Alle eingebundenen Dateien werden ebenfalls über HTTPS ausgeliefert.

Medienliste

URLALT-AttributeTitel
...mages/cads/159374/130x100/0000495321.jpgSprunggewaltiger Lehrmeister für S
...mages/cads/157906/130x100/0000488890.jpgsehr schicke Stute mit viel Potenzial
...mages/cads/159302/130x100/0000495034.jpgTraumpferd mit sehr guten Grundgangarten
...mages/cads/144672/130x100/0000433101.jpgPassier A Tempi 33 KW , 17 Zoll guter Zustand !!
...mages/cads/159376/130x100/0000495332.jpg3-jähriger Dressurnachwuchs zuverkaufen !
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
i2.ytimg.com/vi/YzgikkbtTB4/default.jpgInserat Nr. 159383
i2.ytimg.com/vi/ip4GkGZluM4/default.jpgInserat Nr. 159343
....com/media/system/images/cads/gratis.jpgGratis inserieren
...mages/cads/158545/130x100/0000491537.jpgBewegungsstarkes Diacontinus Stut Fohlen
....horsebase.com/images/cads/kein_bild.pngKein Bild
...mages/cads/157631/130x100/0000487807.jpgHaflinger im sportlichem look stehend
....horsebase.com/images/cads/kein_bild.pngKein Bild
...mages/cads/154775/130x100/0000475475.jpgStiefelschaft aus Leder; Größe L
...mages/cads/154773/130x100/0000475468.jpgSporen mit gezacktem Rad
...mages/cads/154772/130x100/0000475463.jpgQuarter Wallch von Tinsel Jac Futurity Prospect
...mages/cads/154771/130x100/0000475457.jpgSicherheitsweste DMS Performance 36M
...mages/cads/154768/130x100/0000475441.jpgHübsche QH Hengst für reining, cowhorse, cutting! Sehr brav mit super Pedigree!
...mages/cads/154767/130x100/0000475436.jpgCremello QH Stütchen, für reining, cutting, cowhorse! Super hübsche und brav!
...mages/cads/154766/130x100/0000475431.jpgHingucker im Viereck und im Parcours
...mages/cads/154763/130x100/0000475421.jpgImposanter Wallach von Sir Donnerhall
...mages/cads/152618/130x100/0000466017.jpgMinishetlandstute zu verkaufen in gute Hände
...mages/cads/154769/130x100/0000475446.jpgSuper brav 2jährige Stute für reining, cowhorse, cutting! Exclusive pedigree, Super brav!
...mages/cads/152611/130x100/0000465991.jpgDressursattel Maejestoso de Luxe KW31 SF17,5
...mages/cads/152610/130x100/0000465990.jpgReithose kariert fast neu Vollbesatz
...mages/cads/152608/130x100/0000465988.jpgReithose weiß Knielederbesatz NEU
...mages/cads/152607/130x100/0000465986.jpgTekna Maximus Sattel Haflinger Kaltblut
...mages/cads/152599/130x100/0000465938.jpgDaw Mag Springsattel 17 Kammerweite L
...mages/cads/150301/130x100/0000455755.jpgHaflingerstute mit guten Papier sucht neue herrausforderung

Seitenstruktur

H1 Überschrift
(Extrem wichtig)
Pferdemarkt
Die H1-Überschrift besteht nur aus einem Wort. Es sollten mehr Informationen angegeben werden.
Die H1-Überschrift ist zu kurz (11 Zeichen). Sie sollte mindestens 20 Zeichen lang sein.
Überschriften
(Wichtig)
Einige Überschriftentexte kommen doppelt auf der Seite vor.
Die Überschriftenstruktur ist fehlerhaft. Es sollte keine Hierarchie (H1-H6) ausgelassen werden.
Es befinden sich 49 Überschriften auf der Seite. Die Anzahl der Überschriften sollte in einem besseren Verhältnis zum Text stehen.

Überschriftenstruktur

Überschriften HierarchieInhalt
H1 Pferdemarkt
H2 Westernpferde kaufen
H2 Pferde online kaufen
H2 Pferde online verkaufen
H2 Bewegungsstarkes Diacontinus Stut Fohlen
H2 chicer, neuwertiger Bucks Islandsattel mit kurzer Auflagefläche
H2 Haflinger im sportlichem look stehend
H2 Röther Dressursattel Direkt 17.5 Zoll
H2 Stiefelschaft aus Leder; Größe L
H2 Sporen mit gezacktem Rad
H2 Quarter Wallch von Tinsel Jac Futurity Prospect
H2 Sicherheitsweste DMS Performance 36M
H2 Hübsche QH Hengst für reining, cowhorse, cutting! Sehr brav mit super Pedigree!
H2 Cremello QH Stütchen, für reining, cutting, cowhorse! Super hübsche und brav!
H2 Hingucker im Viereck und im Parcours
H2 Imposanter Wallach von Sir Donnerhall
H2 Minishetlandstute zu verkaufen in gute Hände
H2 Super brav 2jährige Stute für reining, cowhorse, cutting! Exclusive pedigree, Super brav!
H2 Dressursattel Maejestoso de Luxe KW31 SF17,5
H2 Reithose kariert fast neu Vollbesatz
H2 Reithose weiß Knielederbesatz NEU
H2 Tekna Maximus Sattel Haflinger Kaltblut
H2 Daw Mag Springsattel 17 Kammerweite L
H2 Haflingerstute mit guten Papier sucht neue herrausforderung
H3 Direkt zu
H3 Unsere neuesten Videos
H3 HorseBase Pferdemarkt
H3 Pferde, Hannoveraner, Buntes Pferd, Fohlen, Remonte, Springen, Vielseitigkeit, Zucht
H3 Sättel, Sonstige Sattelmarken, Dressur, Vielseitigkeit, Wanderreiten
H3 Pferde, Haflinger, Pony, Dressur, Freizeit, Springen
H3 Sättel, Sonstige Sattelmarken, Dressur
H3 Bekleidung
H3 Bekleidung Text-Duplikat
H3 Pferde, Quarter Horse, Sportpferd, Reining
H3 Bekleidung Text-Duplikat
H3 Pferde, American Quarter Horse, Fohlen, Jährling, Sportpferd, Cutting, Freizeit, Gangpferde, Reining, Western
H3 Pferde, American Quarter Horse, Buntes Pferd, Kleinpferd, Sportpferd, Cutting, Freizeit, Reining, Show, Western
H3 Pferde, Deutsches Reitpony, Pony, Dressur, Springen, Vielseitigkeit
H3 Pferde, -- keine Angabe --, Großpferd, Dressur, Fahren, Freizeit, Schulpferd, Springen
H3 Pferde, -- keine Angabe --, Pony, Zucht
H3 Pferde, American Quarter Horse, Buntes Pferd, Sportpferd, Cutting, Freizeit, Reining, Show, Western
H3 Sättel, Stübben, Dressur
H3 Bekleidung Text-Duplikat
H3 Bekleidung Text-Duplikat
H3 Sättel, Sonstige Sattelmarken, Dressur, Springen, Vielseitigkeit
H3 Sättel, Sonstige Sattelmarken, Springen
H3 Pferde, Haflinger, Fohlen, Jährling, Pony, Dressur
H6 Achtung beim Pferdekauf und -verkauf
H6 Die 20 neuesten Inserate im Pferdemarkt
Baselink: https://www.horsebase.de/
Die internen Links haben teilweise dynamische Parameter. Alle internen URLs, welche nicht als nofollow markiert sind, sollten keine dynamischen Parameter aufweisen.
Einige der Linktexte wiederholen sich.
Die Anzahl an internen Links ist ok.
Keiner der Linktexte ist zu lang.
Es befinden sich 3 externe Links auf der Seite.
LinkAttributeLinktext
/anmelden/reset.htmlA-TITLE Ich habe mein Passwort vergessen.
/anmelden/remind.htmlA-TITLE Ich habe meinen Benutzernamen vergessen.
/anmelden/mitglied-werden.htmlA-TITLE Mitglied werden
https://www.horsebase.de/Start
https://www.horsebase.de/Pferdemarkt
/pferde.htmlPferde
/alle-pferde.html?cf=1Alle Pferde
/freizeitpferde.html?cf=1Freizeitpferde
/springpferde.html?cf=1Springpferde
/dressurpferde.html?cf=1Dressurpferde
/westernpferde.html?cf=1Westernpferde
/barockpferde.html?cf=1Barockpferde
/gangpferde.html?cf=1Gangpferde
/vielseitigkeitspferde.html?cf=1Vielseitigkeitspferde
/sportpferde.html?cf=1Sportpferde
/fohlen.html?cf=1Fohlen
/ponies.html?cf=1Ponies
/deckhengste.htmlDeckhengste
/sattelmarkt.htmlSattelmarkt
/saettel.htmlSättel
/sattelzubehoer.htmlSattelzubehör
/reitbekleidung.htmlReitbekleidung
/pferdezubehoer.htmlPferdezubehör
/anhaenger-und-kutschen.htmlAnhänger und Kutschen
/reiterferien.htmlReiterferien
/boxen-und-weideplaetze.htmlBoxen und Weideplätze
/immobilien.htmlImmobilien
/gesuche.html?cf=1Gesuche/Tausch
/dienstleistungen.html?cf=1Dienstleistungen
/anzeige-aufgeben.htmlAnzeige aufgeben
/merkzettel/merkzettel.htmlMerkzettel
/pferde/159374/sprunggewaltige...IMG-ALT Sprunggewaltiger Lehrmeister für S
/pferde/159374/sprunggewaltige...Textduplikat Sprunggewaltiger Lehrmeister für S
/pferde/157906/sehr-schicke-st...IMG-ALT sehr schicke Stute mit viel Potenzial
/pferde/157906/sehr-schicke-st...Textduplikat sehr schicke Stute mit viel Potenzial
/pferde/159302/traumpferd-mit-...IMG-ALT Traumpferd mit sehr guten Grundgangarten
/pferde/159302/traumpferd-mit-...Textduplikat Traumpferd mit sehr guten Grundgangarten
/saettel/144672/passier-a-temp...IMG-ALT Passier A Tempi 33 KW , 17 Zoll guter Zustand !!
/saettel/144672/passier-a-temp...Textduplikat Passier A Tempi 33 KW , 17 Zoll guter Zustand !!
/pferde/159376/3-jaehriger-dre...IMG-ALT 3-jähriger Dressurnachwuchs zuverkaufen !
/pferde/159376/3-jaehriger-dre...Textduplikat 3-jähriger Dressurnachwuchs zuverkaufen !
/alle-pferde.html?cf=1Textduplikat Alle Pferde
/freizeitpferde.html?cf=1Textduplikat Freizeitpferde
/springpferde.html?cf=1Textduplikat Springpferde
/dressurpferde.html?cf=1Textduplikat Dressurpferde
/westernpferde.html?cf=1Textduplikat Westernpferde
/barockpferde.html?cf=1Textduplikat Barockpferde
/gangpferde.html?cf=1Textduplikat Gangpferde
/vielseitigkeitspferde.html?cf=1Textduplikat Vielseitigkeitspferde
/sportpferde.html?cf=1Textduplikat Sportpferde
/ponies.html?cf=1Textduplikat Ponies
/fohlen.html?cf=1Textduplikat Fohlen
/deckhengste.htmlTextduplikat Deckhengste
/saettel.htmlTextduplikat Sättel
/gesuche.html?cf=1Textduplikat Gesuche/Tausch
https://www.youtube.com/v/Yzgi...Neues Fenster Extern IMG-ALT Inserat Nr. 159383
A-TITLE Inserat Nr. 159383
/pferde/159383/grossrahmiger-w...Zum Inserat
https://www.youtube.com/v/ip4G...Neues Fenster Extern IMG-ALT Inserat Nr. 159343
A-TITLE Inserat Nr. 159343
/pferde/159343/pre-hengst-7-ja...Textduplikat Zum Inserat
/achtung-betrueger.htmlNeues Fenster Betrugswarnungen
A-TITLE Betrugswarnung
/warum-sie-auf-horsebase-inser...Neues Fenster Warum Sie auf HorseBase inserieren sollten.
A-TITLE Warum Sie auf HorseBase inserieren sollten
/anzeige-aufgeben.htmlPferde zum Verkauf anbieten
/pferde-suchen.htmlPferde suchen
/pferdemarkt.htmlHorseBase-Pferdemarkt
/achtung-betrueger.htmlsicheren
/pferde-kaufen.htmlKauf
/pferde-verkaufen.htmlVerkauf
/pferdezubehoer.htmlPferdezubehörs
/saettel.htmlTextduplikat Sättel
/reitbekleidung.htmlTextduplikat Reitbekleidung
/pferdekauf.htmlPferdekauf
/pferdeverkauf.htmlPferdeverkauf
/pferdemarkt.htmlTextduplikat Pferdemarkt
/pferdelinksPferdelinkliste
/pferdebuecher.htmlPferdebuchshop
/anzeige-aufgeben.htmlIMG-ALT Gratis inserieren
A-TITLE Jetzt gratis inserieren
http://base.google.com/support...Neues Fenster Extern Subdomain Javascript aktivieren
/pferde/158545/bewegungsstarke...IMG-ALT Bewegungsstarkes Diacontinus Stut Fohlen
A-TITLE Diarado, Contendro, Contender, Caprilli, Don Carlos, Vielseitigkeit
/pferde/158545/bewegungsstarke...Textduplikat Bewegungsstarkes Diacontinus Stut Fohlen
A-TITLE Diarado, Contendro, Contender, Caprilli, Don Carlos, Vielseitigkeit
/saettel/157632/chicer-neuwert...IMG-ALT Kein Bild
/saettel/157632/chicer-neuwert...chicer, neuwertiger Bucks Islandsattel mit kurzer Auflagefläche
/saettel/157632/chicer-neuwert...[...]
A-TITLE Zur Detailansicht
/pferde/157631/haflinger-im-sp...IMG-ALT Haflinger im sportlichem look stehend
A-TITLE Haflinger, Freizeitpferd, Familienpferd
/pferde/157631/haflinger-im-sp...Textduplikat Haflinger im sportlichem look stehend
A-TITLE Haflinger, Freizeitpferd, Familienpferd
/saettel/157630/roether-dressu...Textduplikat IMG-ALT Kein Bild
/saettel/157630/roether-dressu...Röther Dressursattel Direkt 17.5 Zoll
/saettel/157630/roether-dressu...Textduplikat [...]
A-TITLE Zur Detailansicht
/bekleidung/154775/stiefelscha...IMG-ALT Stiefelschaft aus Leder; Größe L
/bekleidung/154775/stiefelscha...Textduplikat Stiefelschaft aus Leder; Größe L
/bekleidung/154773/sporen-mit-...IMG-ALT Sporen mit gezacktem Rad
/bekleidung/154773/sporen-mit-...Textduplikat Sporen mit gezacktem Rad
/pferde/154772/quarter-wallch-...IMG-ALT Quarter Wallch von Tinsel Jac Futurity Prospect
/pferde/154772/quarter-wallch-...Textduplikat Quarter Wallch von Tinsel Jac Futurity Prospect
/pferde/154772/quarter-wallch-...Textduplikat [...]
A-TITLE Zur Detailansicht
/bekleidung/154771/sicherheits...IMG-ALT Sicherheitsweste DMS Performance 36M
/bekleidung/154771/sicherheits...Textduplikat Sicherheitsweste DMS Performance 36M
/bekleidung/154771/sicherheits...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/154768/huebsche-qh-hen...IMG-ALT Hübsche QH Hengst für reining, cowhorse, cutting! Sehr brav mit super Pedigree!
A-TITLE AQH, QH, Hengst, reining, cutting, amateur, cowhorse, Wallach
/pferde/154768/huebsche-qh-hen...Textduplikat Hübsche QH Hengst für reining, cowhorse, cutting! Sehr brav mit super Pedigree!
A-TITLE AQH, QH, Hengst, reining, cutting, amateur, cowhorse, Wallach
/pferde/154768/huebsche-qh-hen...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/154767/cremello-qh-stu...IMG-ALT Cremello QH Stütchen, für reining, cutting, cowhorse! Super hübsche und brav!
A-TITLE AQH, QH, Stute, reining, cutting, amateur, cowhorse, cremello, super
/pferde/154767/cremello-qh-stu...Textduplikat Cremello QH Stütchen, für reining, cutting, cowhorse! Super hübsche und brav!
A-TITLE AQH, QH, Stute, reining, cutting, amateur, cowhorse, cremello, super
/pferde/154767/cremello-qh-stu...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/154766/hingucker-im-vi...IMG-ALT Hingucker im Viereck und im Parcours
A-TITLE Deutsches, Reitpony
/pferde/154766/hingucker-im-vi...Textduplikat Hingucker im Viereck und im Parcours
A-TITLE Deutsches, Reitpony
/pferde/154766/hingucker-im-vi...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/154763/imposanter-wall...IMG-ALT Imposanter Wallach von Sir Donnerhall
/pferde/154763/imposanter-wall...Textduplikat Imposanter Wallach von Sir Donnerhall
/pferde/154763/imposanter-wall...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/152618/minishetlandstu...IMG-ALT Minishetlandstute zu verkaufen in gute Hände
/pferde/152618/minishetlandstu...Textduplikat Minishetlandstute zu verkaufen in gute Hände
/pferde/152618/minishetlandstu...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/154769/super-brav-2jae...IMG-ALT Super brav 2jährige Stute für reining, cowhorse, cutting! Exclusive pedigree, Super brav!
A-TITLE AQH, QH, Stute, reining, cutting, amateur, cowhorse
/pferde/154769/super-brav-2jae...Textduplikat Super brav 2jährige Stute für reining, cowhorse, cutting! Exclusive pedigree, Super brav!
A-TITLE AQH, QH, Stute, reining, cutting, amateur, cowhorse
/pferde/154769/super-brav-2jae...Textduplikat [...]
A-TITLE Zur Detailansicht
/saettel/152611/dressursattel-...IMG-ALT Dressursattel Maejestoso de Luxe KW31 SF17,5
/saettel/152611/dressursattel-...Textduplikat Dressursattel Maejestoso de Luxe KW31 SF17,5
/saettel/152611/dressursattel-...Textduplikat [...]
A-TITLE Zur Detailansicht
/bekleidung/152610/reithose-ka...IMG-ALT Reithose kariert fast neu Vollbesatz
/bekleidung/152610/reithose-ka...Textduplikat Reithose kariert fast neu Vollbesatz
/bekleidung/152608/reithose-we...IMG-ALT Reithose weiß Knielederbesatz NEU
/bekleidung/152608/reithose-we...Textduplikat Reithose weiß Knielederbesatz NEU
/saettel/152607/tekna-maximus-...IMG-ALT Tekna Maximus Sattel Haflinger Kaltblut
/saettel/152607/tekna-maximus-...Textduplikat Tekna Maximus Sattel Haflinger Kaltblut
/saettel/152607/tekna-maximus-...Textduplikat [...]
A-TITLE Zur Detailansicht
/saettel/152599/daw-mag-spring...IMG-ALT Daw Mag Springsattel 17 Kammerweite L
/saettel/152599/daw-mag-spring...Textduplikat Daw Mag Springsattel 17 Kammerweite L
/saettel/152599/daw-mag-spring...Textduplikat [...]
A-TITLE Zur Detailansicht
/pferde/150301/haflingerstute-...IMG-ALT Haflingerstute mit guten Papier sucht neue herrausforderung
A-TITLE Pferde, Jungpferd, Fohlen, Haflinger
/pferde/150301/haflingerstute-...Textduplikat Haflingerstute mit guten Papier sucht neue herrausforderung
A-TITLE Pferde, Jungpferd, Fohlen, Haflinger
/pferde/150301/haflingerstute-...Textduplikat [...]
A-TITLE Zur Detailansicht
https://www.horsebase.de/Anchor Kein Text
/agb-nutzungsbedingungen.htmlNutzungsbedingungen
/datenschutz.htmlDatenschutz
/regeln.htmlRegeln
/achtung-betrueger.htmlBetrugswarnung
https://www.horsebase.de/faq.htmlFAQ
/kontakt.htmlImpressum
/partner.htmlPartner
/abos.htmlAbos
/datenschutz.htmlMehr erfahren

Serverkonfiguration

HTTP-Weiterleitungen
(Extrem wichtig)
Die Seite leitet weiter auf "https://www.horsebase.de/"
HTTP-Header
(Wichtig)
Es wird kein X-Powered HTTP-Header mitgesendet.
Der Webserver nutzt GZip zur komprimierten Übertragung der Webseite (HTML).
Performance
(Wenig wichtig)
Die Antwortzeit der HTML-Seite ist mit 0,76 Sekunden länger als die empfohlene Zeit von maximal 0,4 Sekunden. Eine hohe Antwortzeit verlängert unnötig das Crawling und sorgt für eine schlechte User Experience.
Die Webseite lädt 4 CSS Dateien, dies kann die Ladezeit negativ beeinträchtigen.
Die Webseite lädt 6 Javascript Dateien, dies kann die Ladezeit negativ beeinträchtigen.
Die Dateigröße des HTML-Dokuments ist mit 61 kB in Ordnung.

HTTP-Header

NameWert
servernginx
dateSat, 22 Dec 2018 03:00:29 GMT
content-typetext/html; charset=utf-8
content-length12607
connectionkeep-alive
p3pCP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
link</media/system/js/jquery.imageloader.js>; rel=preload; as=script, </media/system/css/jquery-ui-1.7.2.custom.pack.css>; rel=preload; as=style, <https://img2.horsebase.com/images/cads/158545/130x100/0000491537.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/157631/130x100/0000487807.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/154775/130x100/0000475475.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/154773/130x100/0000475468.jpg>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/154772/130x100/0000475463.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/154771/130x100/0000475457.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/154768/130x100/0000475441.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/154767/130x100/0000475436.jpg>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/154766/130x100/0000475431.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/154763/130x100/0000475421.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/152618/130x100/0000466017.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/154769/130x100/0000475446.jpg>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/152611/130x100/0000465991.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/152610/130x100/0000465990.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/152608/130x100/0000465988.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/152607/130x100/0000465986.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/152599/130x100/0000465938.jpg>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/150301/130x100/0000455755.jpg>; rel=preload; as=image, </templates/waybeyond/css/cads.pack.css>; rel=preload; as=style, <https://www.google.com/jsapi?key=ABQIAAAAxGowbxTXCepwj4r7re0VAhThmdyYcW5uTnKhqD3LqtG8qxFv3hQqDhnd3pIvuLw8uAECbAnA_lmquQ>; rel=preload; as=script, </media/system/js/jquery.min.js>; rel=preload; as=script, </media/system/js/jquery-ui.min.js>; rel=preload; as=script, </media/system/css/colorbox.pack.css>; rel=preload; as=style, </media/system/js/jquery.colorbox.pack.js>; rel=preload; as=script, <https://img2.horsebase.com/images/cads/159374/130x100/0000495321.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/157906/130x100/0000488890.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/159302/130x100/0000495034.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/144672/130x100/0000433101.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/159376/130x100/0000495332.jpg>; rel=preload; as=image, </modules/mod_yoo_login/styles/style.pack.css>; rel=preload; as=style
expiresMon, 1 Jan 2001 00:00:00 GMT
cache-controlpost-check=0, pre-check=0
pragmano-cache
set-cookie67 Zeichen
last-modifiedSat, 22 Dec 2018 03:00:29 GMT
varyAccept-Encoding
content-encodinggzip
x-frame-optionsSAMEORIGIN
x-xss-protection1; mode=block
x-content-type-optionsnosniff
referrer-policyno-referrer-when-downgrade
content-security-policydefault-src * data: 'unsafe-eval' 'unsafe-inline'
strict-transport-securitymax-age=31536000; includeSubDomains; preload
statuscode200
http_versionHTTP/1.1

Externe Faktoren

Blacklists
(Extrem wichtig)
Die Seite wird von Webwiki nicht als "nur für Erwachsene" eingestuft.
Die Seite ist nicht auf der Shallalist verzeichnet.
Die Seite ist exzellent von anderen Webseiten verlinkt.
Die Seite hat Backlinks von 435 verweisenden Domains.
Die Seite hat insgesamt 3.157 Backlinks.
Die Seite hat Backlinks von 133 verschiedenen IP Adressen.
Verbreitung bei Facebook
(Wenig wichtig)
Die Seite ist bei Facebook kaum relevant.
Eintrag bei Webwiki
(Nice to have)
Die Seite ist bei Webwiki verzeichnet.

Links von Wikipedia

Es wurden keine Links von Wikipedia gefunden.

Robots.txt

User-agent: WebReaper
User-agent: AlphaSeoBot
User-agent: AlphaSeoBot-SA
User-agent: WebCopier
User-agent: Offline Explorer
User-agent: HTTrack
User-agent: Microsoft.URL.Control
User-agent: EmailCollector
User-agent: penthesilea
User-agent: MJ12bot
User-Agent: Topsy
User-agent: spbot
User-agent: dotbot
User-agent: SeznamBot
Disallow: /

User-agent: OrangeBot
Disallow: /


User-agent: Slurp
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-Agent: Linkdexbot/2.0
Disallow: /
User-Agent: Linkdexbot/2.1
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-agent: Sogou web spider Sogou web spider/4.0
Disallow: /
User-agent: sogou web spider
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: Opoint
Disallow: /
User-agent: 008
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: ClarityDailyBot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; ClarityDailyBot)
Disallow: /

User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
Disallow: /
User-agent: PAD-bot/9.0
Disallow: /
User-agent: PAD-bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)
Disallow: /
User-agent: SoftListBot/2.2
Disallow: /
User-agent: SoftListBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: NativeHost
Disallow: /
User-agent: newsme/1.0; feedback@news.me
Disallow: /
User-agent: newsme/1.0
Disallow: /
User-agent: newsme
Disallow: /
User-agent: Niki-Bot
Disallow: /
User-agent: NING/1.0
Disallow: /
User-agent: NING
Disallow: /

User-agent: peerindex
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Disallow: /
User-agent: Diffbot/0.1
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net)
Disallow: /
User-agent: elefent
Disallow: /
User-agent: elefent/1.2
Disallow: /
User-agent: crawler.elefent.net
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 4.0
Disallow: /
User-agent: MS Search 5.0
Disallow: /
User-agent: MS Search 6.0
Disallow: /
User-agent: MS Search 7.0
Disallow: /
User-agent: MS Search
Disallow: /


User-agent: CCBot/2.0 (http://commoncrawl.org/faq/)
Disallow: /
User-agent: CCBot/2.0
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)
Disallow: /
User-agent: Mail.RU_Bot/Robots/2.0
Disallow: /
User-agent: AOL_Daniel_Clark_Spider/Nutch-0.9 (AOL Search; danielaclark1@aol.com)
Disallow: /
User-agent: AolOnScraper/1.0
Disallow: /


User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html; gzip)
Disallow: /

User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html)
Disallow: /

User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /



User-agent: Apache Sites Search Facet
Disallow: /
User-agent: bigdata
Disallow: /

User-agent: Fast corporate crawler
Disallow: /
User-agent: 008
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: magpie-crawler/1.1
Disallow: /
User-agent: MagpieRSS
Disallow: /
User-agent: psnsearch
Disallow: /
User-agent: SLCC2
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: GuzzleHttp
Disallow: /
User-agent: Go 1.1 package http
Disallow: /
User-agent: Go 1.1
Disallow: /
User-agent: Go 1.2
Disallow: /
User-agent: Go 1.2 package http
Disallow: /
User-agent: gocrawl v0.4
Disallow: /
User-agent: gocrawl
Disallow: /
User-agent: Google-HTTP-Java-Client
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: Hatena Star
Disallow: /
User-agent: PagePeeker.com
Disallow: /
User-agent: PagePeeker
Disallow: /
User-agent: _agentname
Disallow: /
User-agent: agentname
Disallow: /
User-agent: 1 in Business Networking, www.MarcusWalker.ws
Disallow: /
User-agent: Business Networking
Disallow: /
User-agent: AntiOpensourceBot/1.1
Disallow: /
User-agent: AntiOpensourceBot
Disallow: /
User-agent: antibot-V1.1.5/i586-linux-2.2
Disallow: /
User-agent: Anti-XBOCT/2007 [xxx] (OS; X; JU)
Disallow: /
User-agent: AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Disallow: /
User-agent: Anti-XBOCT
Disallow: /
User-agent: AntBot
Disallow: /
User-agent: Ant-Nutch
Disallow: /
User-agent: AntBot/1.0
Disallow: /
User-agent: AnzwersCrawl/2.0
Disallow: /

User-agent: Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Disallow: /

User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /

User-agent: Wotbox/2.01 (+http://www.wotbox.com/bot/)
Disallow: /

User-agent: Wotbox
Disallow: /

User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPicker /1.0
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-Agent: litefinder
Disallow: /
User-agent: Friendica
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Flamingo
Disallow: /
User-agent: Faraday v0.8.8
Disallow: /
User-agent: Faraday
Disallow: /
User-agent: Barkrowler 0.1.6
Disallow: /
User-agent: Barkrowler
Disallow: /
User-Agent: ExperiBot_v1
Disallow: /
User-Agent: ExperiBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-Agent: omgilibot
Disallow: /
User-Agent: omgili
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: hloader
Disallow: /
User-agent: ConveraCrawler/0.9e
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: YahooFeedSeeker/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://publisher.yahoo.com/rssguide)
Disallow: /
User-agent: YahooFeedSeeker/2.0+
Disallow: /
User-agent: YahooFeedSeeker
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: JennyBot
Disallow: /


User-agent: Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Disallow: /

User-agent: ExtLinksBot/1.5
Disallow: /

User-agent: ExtLinksBot
Disallow: /

User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /

User-agent: GrapeshotCrawler/2.0
Disallow: /

User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /

User-agent: NCBot
Disallow: /
User-agent: Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html)
Disallow: /
User-agent: Testbot/1.0
Disallow: /
User-agent: Testbot
Disallow: /
User-agent: Bot !Susie
Disallow: /
User-agent: Slurp Disallow: /
User-agent: !Susie
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /


User-agent: Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Disallow: /

User-agent: GroupHigh/1.0
Disallow: /


User-agent: G-i-g-a-b-o-t
Disallow: /

User-agent: SeznamBot/3.2
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Disallow: /

User-agent: Helloo/1.0.4
Disallow: /

User-agent: Mozilla/5.0 (compatible; Helloo/1.0.4; +http://www.helloo.io/;)
Disallow: /

User-agent: SiteExplorer/1.1b
Disallow: /

User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /



User-agent: Mail.RU_Bot/Robots
Disallow: /


User-agent: attach
Disallow: /
User-agent: ASPSeek
Disallow: /
User-agent: appie
Disallow: /
User-agent: AbachoBOT
Disallow: /
User-agent: autoemailspider
Disallow: /
User-agent: anarchie
Disallow: /
User-agent: antibot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: B2w
Disallow: /
User-agent: BackWeb
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: Bandit
Disallow: /
User-agent: BatchFTP
Disallow: /
User-agent: Black\ Hole
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bot\ mailto
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: Buddy
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: bumblebee
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: ClariaBot
Disallow: /
User-agent: curl
Disallow: /
User-agent: clsHTTP
Disallow: /
User-agent: ChinaClaw
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: Collector
Disallow: /
User-agent: COAST\ WebMaster
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: ColdFusion
Disallow: /
User-agent: Copier
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DA
Disallow: /
User-agent: DTS\ Agent
Disallow: /
User-agent: DISCo\ Pump
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Diamond
Disallow: /
User-agent: Download\ Demon
Disallow: /
User-agent: Download\ Wonder
Disallow: /
User-agent: Downloader
Disallow: /
User-agent: dloader
Disallow: /
User-agent: Drip
Disallow: /
User-agent: eCatch
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: Express\ WebPictures
Disallow: /
User-agent: Extreme\ Picture\ Finder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EasyDL
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: EyeNetIE
Disallow: /
User-agent: FAST\ WebCrawler
Disallow: /


User-agent: masscan/1.0 (https://github.com/robertdavidgraham/masscan)
Disallow: /

User-agent: masscan/1.0
Disallow: /

User-agent: masscan
Disallow: /

User-agent: semantic-visions.com crawler
Disallow: /





User-agent: MJ12bot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: ADmantX Platform Semantic Analyzer
Disallow: /

User-agent: proximic
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DeuSu
Disallow: /

User-agent: SMTBot
Disallow: /

User-agent: GetintentCrawler
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: XoviBot
Disallow: /

User-agent: zgrab
Disallow: /

User-agent: Pigafetta
Disallow: /

User-agent: plukkie
Disallow: /

User-agent: postano
Disallow: /

User-agent: Prlog
Disallow: /

User-agent: PulseCrawler/1.1
Disallow: /

User-agent: QuerySeekerSpider
Disallow: /

User-agent: quipu
Disallow: /

User-agent: Raven
Disallow: /

User-agent: RavenCrawler
Disallow: /

User-agent: RBot
Disallow: /

User-agent: Recorded Future
Disallow: /

User-agent: Riddler
Disallow: /

User-agent: Ruby
Disallow: /

User-agent: RukiCrawler
Disallow: /

User-agent: SafeDNSBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: Searchie
Disallow: /

User-agent: semantic-visions.com crawler
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: seocharger-robot
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SEOlyticsCrawler
Disallow: /

User-agent: seoscanners
Disallow: /

User-agent: seoscanners.net
Disallow: /

User-agent: ShowyouBot (http://showyou.com/crawler)
Disallow: /

User-agent: simplecrawler
Disallow: /

User-agent: SISTRIX Crawler
Disallow: /

User-agent: SiteExplorer
Disallow: /

User-Agent: siteluxbot
Disallow: /

User-agent: SmeshBot
Disallow: /

User-agent: SMTBot
Disallow: /

User-agent: srmse/Nutch
Disallow: /

User-agent: SoftListBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: squirrly
Disallow: /

User-agent: ssearch_bot
Disallow: /

User-agent: Stratagems Kumo
Disallow: /

User-agent: tbot-nutch
Disallow: /

User-agent: TWMBot
Disallow: /

User-agent: Typhoeus - https://github.com/typhoeus/typhoeus
Disallow: /

User-agent: uipbot
Disallow: /

User-agent: uMBot-LN
Disallow: /

User-agent: UptimeRobot/2.0
Disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: WebIndex
Disallow: /

User-agent: VeBot
Disallow: /

User-agent: VeriCiteCrawler
Disallow: /

User-agent: WeSEE_Bot
Disallow: /

User-agent: WiseGuys Robot
Disallow: /

User-agent: voltron
Disallow: /

User-agent: woobot
Disallow: /

User-agent: woriobot
Disallow: /

User-agent: wotbox
Disallow: /

User-agent: X-CAD-SE
Disallow: /

User-agent: XoviBot
Disallow: /

User-agent: yacybot
Disallow: /

User-agent: Yahoo Pipes 1.0
Disallow: /

User-agent: YioopBot
Disallow: /

User-agent: crawler.vs.uni-duisburg-essen.de
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: FileHound
Disallow: /
User-agent: Fetch\ API\ Request
Disallow: /
User-agent: FlashGet
Disallow: /
User-agent: FlickBot
Disallow: /
User-agent: FrontPage
Disallow: /
User-agent: FreeFind.com
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: GetSmart
Disallow: /
User-agent: Generic
Disallow: /
User-agent: Go!Zilla
Disallow: /
User-agent: Go-Ahead-Got-It
Disallow: /
User-agent: gotit
Disallow: /
User-agent: Grabber
Disallow: /
User-agent: GrabNet
Disallow: /
User-agent: Grafula
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: HMView
Disallow: /
User-agent: Heretrix
Disallow: /
User-agent: HitboxDoctor
Disallow: /
User-agent: HTTPapp
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTPTrack
Disallow: /
User-agent: HTTPviewer
Disallow: /
User-agent: httplib
Disallow: /
User-agent: httpfetcher
Disallow: /
User-agent: httpscraper
Disallow: /
User-agent: hloader
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: InterGET
Disallow: /
User-agent: Internet\ Ninja
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: InternetSeer.com
Disallow: /
User-agent: Iria
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: JetCar
Disallow: /
User-agent: JOC
Disallow: /
User-agent: JOC\ Web\ Spider
Disallow: /
User-agent: JoBo
Disallow: /
User-agent: Java
Disallow: /
User-agent: JustView
Disallow: /
User-agent: Jonzilla
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin\ Spider
Disallow: /
User-agent: Keyword\ Density
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LeechFTP
Disallow: /
User-agent: Lachesis
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb
Disallow: /
User-agent: Libby_
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: lftp
Disallow: /
User-agent: likse
Disallow: /
User-agent: Link
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp\ request
Disallow: /
User-agent: Magnet
Disallow: /
User-agent: Mag-Net
Disallow: /
User-agent: Mass\ Downloader
Disallow: /
User-agent: MIIxpc
Disallow: /

User-agent: Missigua
Disallow: /
User-agent: Mewsoft\ Search\ Engine
Disallow: /
User-agent: moget
Disallow: /
User-agent: Mata\ Hari
Disallow: /
User-agent: Memo
Disallow: /
User-agent: Metacarta
Disallow: /
User-agent: Mercator
Disallow: /
User-agent: MIDown\ tool
Disallow: /
User-agent: MFC_Tear_Sample
Disallow: /
User-agent: Mirror
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Mister\ PiX
Disallow: /
User-agent: NationalDirectory\ WebSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Nikto
Disallow: /
User-agent: Navroad
Disallow: /
User-agent: NearSite
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NetResearchServer
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: Net\ Vampire
Disallow: /
User-agent: Net\ Probe
Disallow: /
User-agent: NetZip
Disallow: /
User-agent: nexuscache
Disallow: /
User-agent: Ninja
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: our\ agent
Disallow: /
User-agent: onestop
Disallow: /
User-agent: oBot
Disallow: /
User-agent: Octopus
Disallow: /
User-agent: Offline\ Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind\ data\ gatherer
Disallow: /
User-agent: PageGrabber
Disallow: /
User-agent: Papa\ Foto
Disallow: /

User-agent: Xaldon\ WebSpider
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: Xara
Disallow: /
User-agent: Y!TunnelPro
Disallow: /
User-agent: YahooYSMcm
Disallow: /
User-agent: Zade
Disallow: /
User-agent: ZBot
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Rogerbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Dotbot
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: AhrefsBot
Disallow: /

User-agent: Xaldon\ WebSpider
Disallow: /

User-agent: Screaming Frog SEO Spider/8.1
Disallow: /

User-agent: DoBot
Disallow: /

User-agent: Laserlikebot
Disallow: /
User-agent: oBot
Disallow: /

User-agent: Neliveto
Disallow: /
User-Agent: ahrefs
Disallow: /
User-Agent: AHC/1.0
Disallow: /
User-Agent: AHC
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Disallow: /

User-agent: evc-batch/2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Disallow: /

User-agent: SemrushBot/1.2~bl
Disallow: /

User-agent: evc-batch
Disallow: /

User-agent: 007ac9 Crawler
 disallow: /

User-agent: 1 2 3 Submit PRO
 disallow: /

User-agent: 200PleaseBot
 disallow: /

User-agent: 2ADAMbot
 disallow: /

User-agent: 2ADAMbot/1.0
 disallow: /

User-agent: 360Spider
 disallow: /

user-agent: Abonti
 disallow: /

user-agent: Abonti/0.92
 disallow: /

user-agent: abot v1.0
 disallow: /

user-agent: aboutthedomain
 disallow: /

user-agent: Add Catalog
 disallow: /

user-agent: Add Catalog/2.1
 disallow: /

user-agent: AdvBot
 disallow: /

user-agent: AdvBot/2.0
 disallow: /

user-agent: AhrefsBot
 disallow: /

user-agent: Ahrefs-Bot
 disallow: /

user-agent: AhrefsBot/1.0
 disallow: /

user-agent: Ahrefs-Bot/1.0
 disallow: /

user-agent: Ahrefs-Bot/2.0
 disallow: /

user-agent: Ahrefs-Bot/3.0
 disallow: /

user-agent: Ahrefs-Bot/4.0
 disallow: /

user-agent: Ahrefs-Bot/5.0
 disallow: /

user-agent: aiHitBot
 disallow: /

user-agent: aiHitBot/2.9
 disallow: /

user-agent: Anonymous/0.0
 disallow: /

user-agent: Arachnida
 disallow: /

user-agent: Associative Spider
 disallow: /

User-agent: Baiduspider
 disallow: /

User-agent: Baidu Spider
 disallow: /

User-agent: Battleztar Bazinga
 disallow: /

User-agent: Battleztar Bazinga/0.01
 disallow: /

User-agent: BDFetch
 disallow: /

User-agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
Disallow: /

User-agent: ZmEu
Disallow: /

User-agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
Disallow: /


User-agent: betaBot
 disallow: /

User-agent: bieshu
 disallow: /

User-agent: Bigli SEO
 disallow: /

User-agent: Blackboard Safeassign
 disallow: /

User-agent: Blazer 1.0
 disallow: /

User-agent: BLEXBot
 disallow: /

User-agent: BLEXBot/1.0
 disallow: /

User-agent: BLP_bbot
 disallow: /

User-agent: BLP_bbot/0.1
 disallow: /

User-agent: BOIA-Accessibility-Agent/PR 1.0
 disallow: /

User-agent: BOT for JCE
 disallow: /

User-agent: BOT/0.1 (BOT for JCE)
 disallow: /

User-agent: BPImageWalker
 disallow: /

User-agent: BPImageWalker/2.0
 disallow: /

User-agent: BUbiNG
 disallow: /

User-agent: BuiBui-Bot
 disallow: /

User-agent: BuiBui-Bot/1.0
 disallow: /

User-agent: ca-crawler
 disallow: /

User-agent: ca-crawler/1.0
 disallow: /

User-agent: CakePHP
 disallow: /

User-agent: Calypso v/0.01
 disallow: /

User-agent: Calypso
 disallow: /

User-agent: CB/Nutch-1.7
 disallow: /

User-agent: CCBot
 disallow: /

User-agent: CCBot/2.0
 disallow: /

User-agent: Checkbot
 disallow: /

User-agent: checkgzipcompression.com
 disallow: /

User-agent: chushou
 disallow: /

User-agent: CloudServerMarketSpider
 disallow: /

User-agent: CloudServerMarketSpider/1.0
 disallow: /

User-agent: Clushbot/3.x-BinaryFury
 disallow: /

User-agent: CMS Crawler
 disallow: /

User-agent: CMS Crawler: http://www.cmscrawler.com
 disallow: /

User-agent: coccoc
 disallow: /

User-agent: CoinCornerBot
 disallow: /

User-agent: CoinCornerBot/1.1
 disallow: /

User-agent: Copyscape
 disallow: /

User-agent: crawler4j
 disallow: /

User-agent: CRAZYWEBCRAWLER 0.9.0
 disallow: /

User-agent: CRAZYWEBCRAWLER 0.9.1
 disallow: /

User-agent: CRAZYWEBCRAWLER 0.9.7
 disallow: /

User-agent: CrazyWebCrawler
 disallow: /

User-agent: CrazyWebCrawler-Spider
 disallow: /

User-agent: Crowsnest
 disallow: /

User-agent: Crowsnest/0.5
 disallow: /

User-agent: Curious George - www.analyticsseo.com/crawler
 disallow: /

User-agent: Curious George
 disallow: /

User-agent: cuwhois
 disallow: /

User-agent: cuwhois/1.0
 disallow: /

User-agent: dahoms
 disallow: /

User-agent: datagnionbot
 disallow: /

User-agent: DeuSu/5.0.2
 disallow: /

User-agent: Digincore
 disallow: /

User-agent: Digincore bot
 disallow: /

User-agent: Dispatch/0.11.0
 disallow: /

User-agent: Domain Re-Animator Bot
 disallow: /

User-agent: DomainAppender /1.0
 disallow: /

User-agent: DomainAppender
 disallow: /

 User-agent: DomainCrawler/3.0
 disallow: /

User-agent: DomainSigmaCrawler
 disallow: /

User-agent: DomainSigmaCrawler/0.1
 disallow: /

User-agent: Domnutch
 disallow: /

User-agent: Domnutch-Bot
 disallow: /

User-agent: Domnutch-Bot/Nutch
 disallow: /

User-agent: Domnutch-Bot/Nutch-1.0
 disallow: /

User-agent: dotbot
 disallow: /

User-agent: ECCP/1.2.1
 disallow: /

User-agent: eCommerceBot
 disallow: /

User-agent: enlle punto com/Nutch-1.9
 disallow: /

User-agent: EPiServer Link Checker
 disallow: /

User-agent: EuripBot
 disallow: /

User-agent: EuripBot/2.0
 disallow: /

User-agent: evc/2.0
 disallow: /

User-agent: evc-batch
 disallow: /

User-agent: evc-batch/2.0
 disallow: /

User-agent: Express WebPictures
 disallow: /

User-agent: Faraday v0.8.8
 disallow: /

User-agent: Faraday
 disallow: /

User-agent: Findxbot
 disallow: /

User-agent: Findxbot/1.0
 disallow: /

User-agent: Flamingo_SearchEngine
 disallow: /

User-agent: Flipboard Robot
 disallow: /

User-agent: GetProxi.es-bot
 disallow: /

User-agent: GetProxi.es-bot/1.1
 disallow: /

User-agent: GigablastOpenSource
 disallow: /

User-agent: GigablastOpenSource/1.0
 disallow: /

User-agent: Girafabot
 disallow: /

User-agent: Gluten Free Crawler
 disallow: /

User-agent: Gluten Free Crawler/1.0
 disallow: /

User-agent: GriffinBot
 disallow: /

User-agent: GrifinBot/0.01
 disallow: /

User-agent: GWPImages
 disallow: /

User-agent: GWPImages/1.0
 disallow: /

User-agent: Haiula
 disallow: /

User-agent: Haiula/1.4
 disallow: /

User-agent: HaosouSpider
 disallow: /

User-agent: Hivemind
 disallow: /

User-agent: HostHarvest
 disallow: /

User-agent: HostHarvest/0.4.28
 disallow: /

User-agent: HRCrawler
 disallow: /

User-agent: HRCrawler/2.0
 disallow: /

User-agent: http://git.io/tl_S2w
 disallow: /

User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
 disallow: /

User-agent: HubSpot Links Crawler 1.0
 disallow: /

User-agent: HubSpot Webcrawler
 disallow: /

User-agent: HubSpot
 disallow: /

User-agent: hunchan
 disallow: /

User-agent: HyperCrawl
 disallow: /

User-agent: HyperCrawl/0.2
 disallow: /

User-agent: ICAP-IOD
 disallow: /

User-agent: ICC-Crawler
 disallow: /

User-agent: ICC-Crawler/2.0
 disallow: /

User-agent: Ichiro Robot
 disallow: /

User-agent: image.coccoc/1.0
 disallow: /

User-agent: Image2play
 disallow: /

User-agent: Image2play/0.1
 disallow: /

User-agent: Indy Library
 disallow: /

User-agent: InsightsCollector
 disallow: /

User-agent: InsightsCollector/0.1
 disallow: /

User-agent: InsightsCollector/0.1beta
 disallow: /

User-agent: integrity/5
 disallow: /

User-agent: InterNaetBoten
 disallow: /

User-agent: InterNaetBoten/0.99
 disallow: /

User-agent: Barkrowler/0.7 (+http://www.exensa.com/crawl)
 disallow: /

User-agent: Barkrowler
 disallow: /

User-agent: IRL Crawler
 disallow: /

User-agent: James BOT - WebCrawler
 disallow: /

User-agent: James BOT
 disallow: /

User-agent: JamesBOT
 disallow: /

User-agent: JetBrains 5.0
 disallow: /

User-agent: JetBrains
 disallow: /

User-agent: Kraken
 disallow: /

User-agent: Kraken/0.1
 disallow: /

User-agent: Kyoto-Tohoku-Crawler/v1
 disallow: /

User-agent: larbin
 disallow: /

User-agent: lechenie
 disallow: /

User-agent: libwww-perl
 disallow: /

User-agent: link checker
 disallow: /

User-agent: Link/1.0
 disallow: /

User-agent: linkCheck
 disallow: /

User-agent: linkCheckV3.0
 disallow: /

User-agent: Linkdex
 disallow: /

User-agent: linkdex.com/v2.0
 disallow: /

User-agent: linkdex.com/v2.1
 disallow: /

User-agent: LinkdexBot
 disallow: /

User-agent: linkdexbot/2.0
 disallow: /

User-agent: linkdexbot/2.1
 disallow: /

User-agent: linkdexbot-mobile/2.1
 disallow: /

User-agent: LinkpadBot
 disallow: /

User-agent: LinkpadBot/1.06
 disallow: /

User-agent: LinqiaScrapeBot
 disallow: /

User-agent: LinqiaScrapeBot/1.0
 disallow: /

User-agent: Lipperhey SEO Service
 disallow: /

User-agent: Lipperhey
 disallow: /

User-agent: Lipperhey-Kaus-Australis
 disallow: /

User-agent: Lipperhey-Kaus-Australis/5.0
 disallow: /

User-agent: listicka
 disallow: /

User-agent: LSSRocketCrawler
 disallow: /

User-agent: LSSRocketCrawler/1.0 LightspeedSystems
 disallow: /

User-agent: LSSRocketCrawler/1.0
 disallow: /

User-agent: ltx71
 disallow: /

User-agent: LWNutch/Nutch-1.4
 disallow: /

User-agent: Mail.RU
 disallow: /

User-agent: Mail.RU_Bot
 disallow: /

User-agent: Mail.RU_Bot/2.0
 disallow: /

User-agent: Mail.RU_Bot/Fast/2.0
 disallow: /

User-agent: md5sum
 disallow: /

User-agent: md5sum\x22
 disallow: /

User-agent: meanpathbot
 disallow: /

User-agent: MegaIndex.ru
 disallow: /

User-agent: MegaIndex.ru/2.0
 disallow: /

User-agent: mezhpozvonochnoi
 disallow: /

User-agent: Mike-Crawler
 disallow: /

User-agent: MixBot
 disallow: /

User-agent: MixrankBot
 disallow: /

User-agent: MJ12bot
 disallow: /

User-agent: Monkeybot/0.1
 disallow: /

User-agent: my crawler
 disallow: /

User-agent: My Nutch Spider/Nutch-1.9
 disallow: /

User-agent: mycrowl/Nutch-1.9
 disallow: /

User-agent: MyGreatUA/2.0
 disallow: /

User-agent: MyIPTest
 disallow: /

User-agent: NameProtect Robot
 disallow: /

User-agent: NerdyBot
 disallow: /

User-agent: Netcraft Spider
 disallow: /

User-agent: netEstate NE Crawler
 disallow: /

User-agent: NetLyzer FastProbe
 disallow: /

User-agent: NetResearchServer
 disallow: /

User-agent: NetResearchServer/4.0
 disallow: /

User-agent: Nmap Scripting Engine
 disallow: /

User-agent: node.io
 disallow: /

User-agent: node.js
 disallow: /

User-agent: Node/simplecrawler 0.5.2
 disallow: /

User-agent: Node/simplecrawler
 disallow: /

User-agent: oBot/2.3.1
 disallow: /

User-agent: omgilibot
 disallow: /

User-agent: omgilibot/0.4
 disallow: /

User-agent: Online Domain Tools - Online Website Link Checker
 disallow: /

User-agent: Online Domain Tools - Online Website Link Checker/1.2
 disallow: /

User-agent: Openfind Robot
 disallow: /

User-agent: OpenHoseBot
 disallow: /

User-agent: OpenHoseBot/2.1
 disallow: /

User-agent: Openstat
 disallow: /

User-agent: Openstat/0.1
 disallow: /

User-agent: OptimizationCrawler
 disallow: /

User-agent: OptimizationCrawler/0.2
 disallow: /

User-agent: Page Analyzer v4.0
 disallow: /

User-agent: Page Analyzer
 disallow: /

User-agent: PageAnalyzer
 disallow: /

User-agent: PageAnalyzer/1.1
 disallow: /

User-agent: PageAnalyzer/1.5
 disallow: /

User-agent: PagesInventory
 disallow: /

User-agent: Pagespeed/1.1 Fetcher
 disallow: /

User-agent: Pagespeed/1.1
 disallow: /

User-agent: Pagespeedbot
 disallow: /

User-agent: Perl LWP
 disallow: /

User-agent: PHPCrawl
 disallow: /

User-agent: phpSiteCheck 1.0
 disallow: /

User-agent: phpSiteCheck
 disallow: /

User-agent: Plukkie
 disallow: /

User-agent: POGS/2.0
 disallow: /

User-agent: Powermarks
 disallow: /

User-agent: PowerPivot
 disallow: /

User-agent: PRIVACY_ENFAQ.jsp
 disallow: /

User-agent: Prlog
 disallow: /

User-agent: Prlog/1.0
 disallow: /

User-agent: publiclibraryarchive.org
 disallow: /

User-agent: publiclibraryarchive.org/1.0
 disallow: /

User-agent: Pu_iN Crawler
 disallow: /

User-agent: Putin
 disallow: /

User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /

User-agent: Putin spider
 disallow: /

User-agent: qingdao
 disallow: /

User-agent: QlikView
 disallow: /

User-agent: quipu
 disallow: /

User-agent: quipu/1.0
 disallow: /

User-agent: quipu/2.0
 disallow: /

User-agent: R6_CommentReader
 disallow: /

User-agent: R6_FeedFetcher
 disallow: /

User-agent: Riddler
 disallow: /

User-agent: RivalSeek.com-Bot
 disallow: /

User-agent: rogerbot
 disallow: /

User-agent: rogerbot/1.0
 disallow: /

User-agent: rootlink
 disallow: /

User-agent: RU_Bot/2.0
 disallow: /

User-agent: Scopia
 disallow: /

User-agent: Scopia crawler
 disallow: /

User-agent: Scopia crawler 1.0
 disallow: /

User-agent: Scopia crawler 1.1
 disallow: /

User-agent: Scopia crawler 1.2
 disallow: /

User-agent: Scrapy
 disallow: /

User-agent: Scrapy/0.16.5
 disallow: /

User-agent: Scrapy/0.24.4
 disallow: /

User-agent: Scrapy/0.24.5
 disallow: /

User-agent: Scrapy/0.24.6
 disallow: /

User-agent: Scrapy/1.0.1
 disallow: /

User-agent: Screaming Frog SEO Spider
 disallow: /

User-agent: Screaming Frog SEO Spider/2,55
 disallow: /

User-agent: Screaming Frog SEO Spider/2.55
 disallow: /

User-agent: Screaming Frog SEO Spider/3.1
 disallow: /

User-agent: Screaming Frog SEO Spider/3.3
 disallow: /

User-agent: Screaming Frog SEO Spider/4.1
 disallow: /

User-agent: Screaming Frog SEO Spider/5.0
 disallow: /

User-agent: Screaming Frog SEO Spider/5.1
 disallow: /

User-agent: Screaming Frog SEO Spider/5.1 Beta 2
 disallow: /

User-agent: scrutiny/4
 disallow: /

User-agent: SemrushBot
 disallow: /

User-agent: SemrushBot-SA
 disallow: /

User-agent: SEOdiver/1.0
 disallow: /

User-agent: SEOkicks
 disallow: /

User-agent: SEOkicks-Robot
 disallow: /

User-agent: SEOlyticsCrawler
 disallow: /

User-agent: SEOlyticsCrawler/3.0
 disallow: /

User-agent: seoscanners
 disallow: /

User-agent: seoscanners.net/1
 disallow: /

User-agent: SEOstats 2.1.0
 disallow: /

User-agent: Seosys/Nutch-2.3
 disallow: /

User-agent: SetCronJob/1.0
 disallow: /

User-agent: R6_FeedFetcher_(www.radian6.com/crawler)
disallow: /

User-agent: SeznamBot
 disallow: /

User-agent: SheerBoredom.Experimental.Robot
 disallow: /

User-agent: SheerBoredom.Experimental.Robot/0.2
 disallow: /

User-agent: ShowyouBot
 disallow: /

User-agent: Simplecrawler
 disallow: /

User-agent: SISTRIX Crawler
 disallow: /

User-agent: sistrix
 disallow: /

User-agent: SiteBot
 disallow: /

User-agent: SiteBot/0.1
 disallow: /

User-agent: SiteExplorer
 disallow: /

User-agent: SiteExplorer/1.0
 disallow: /

User-agent: SiteExplorer/1.0b
 disallow: /

User-agent: Siteluxbot
 disallow: /

User-agent: Siteluxbot/1.0
 disallow: /

User-agent: SkimBot
 disallow: /

User-agent: RSSingBot (http://www.rssing.com)
disallow: /

User-agent: SkimBot/1.0
 disallow: /

User-agent: sky nutch crawler/Nutch-1.9
 disallow: /

User-agent: SMTBot
 disallow: /

User-agent: SMTBot/1.0
 disallow: /

User-agent: SNK Screenshot Bot
 disallow: /

User-agent: SNK Screenshot Bot/0.20
 disallow: /

User-agent: Sogou Spider
 disallow: /

User-agent: Sogou web spider
 disallow: /

User-agent: SpamBayes
 disallow: /

User-agent: SpamBayes/1.1a3+
 disallow: /

User-agent: spbot
 disallow: /

User-agent: spbot/4.4.2
 disallow: /

User-agent: spiderbot
 disallow: /

User-agent: SpiderLing
 disallow: /

User-agent: Spiderbot/Nutch-1.7
 disallow: /

User-agent: spray-can
 disallow: /

User-agent: spray-can/1.2.1
 disallow: /

User-agent: SSG/3.0
 disallow: /

User-agent: Statastico
 disallow: /

User-agent: Statastico/4.0
 disallow: /

User-agent: Steeler
 disallow: /

User-agent: Steeler/3.5
 disallow: /

User-agent: Stratagems Kumo
 disallow: /

User-agent: Stratagems
 disallow: /

User-agent: StudioFACA Search
 disallow: /

User-agent: StudioFACA
 disallow: /

User-agent: sukibot
 disallow: /

User-agent: sukibot_heritrix
 disallow: /

User-agent: sukibot_heritrix/3.1.1
 disallow: /

User-agent: SurveyBot
 disallow: /

User-agent: Synapse
 disallow: /

User-agent: Synthesio Crawler release MonaLisa
 disallow: /

User-Agent: tbot-nutch/Nutch-1.10
 disallow: /

User-Agent: Traackr.com bot
 disallow: /

User-Agent: trendictionbot
 disallow: /

User-Agent: Trendiction-Bot
 disallow: /

User-Agent: TrueBot
 disallow: /

User-Agent: TrueBot/1.0
 disallow: /

User-Agent: TulipChain/5.xx
 disallow: /

User-Agent: TWMBot/0.1
 disallow: /

User-Agent: Typhoeus
 disallow: /

User-Agent: UCMore Crawler App
 disallow: /

User-Agent: uMBot-LN
 disallow: /

User-Agent: uMBot-LN/1.0
 disallow: /

User-Agent: updown_tester
 disallow: /

User-Agent: URLChecker
 disallow: /

User-agent: V1.0/1.2
 disallow: /

User-agent: w3af.org
 disallow: /

User-agent: WASALive
 disallow: /

User-agent: WASALive-Bot
 disallow: /

User-agent: vBSEO
 disallow: /

User-agent: WBSearchBot
 disallow: /

User-agent: WBSearchBot/1.1
 disallow: /

User-agent: WeAreNotEvil
 disallow: /

User-agent: WebAlta
 disallow: /

User-agent: WebAlta Crawler
 disallow: /

User-agent: Web corpus crawler
 disallow: /

User-agent: WebCookies
 disallow: /

User-agent: WebCookies/1.0
 disallow: /

User-agent: WebCopier vx.xa
 disallow: /

User-agent: Webnest 0.9
 disallow: /

User-agent: WebQL
 disallow: /

User-agent: Webscout
 disallow: /

User-agent: Webscout/1.0
 disallow: /

User-agent: Web-sniffer
 disallow: /

User-agent: Web-sniffer/1.1.0
 disallow: /

User-agent: Webster Pro V3.4
 disallow: /

User-agent: WebTarantula.com Crawler
 disallow: /

User-agent: WeCrawlForThePeace
 disallow: /

User-agent: VegeBot
 disallow: /

 User-agent: Vegi bot
 disallow: /

User-agent: WeLikeLinks
 disallow: /

User-agent: VeriCiteCrawler
 disallow: /

User-agent: VeriCiteCrawler/Nutch-1.9
 disallow: /

User-agent: WhatWeb
 disallow: /

User-agent: WhatWeb/0.4.8-dev
 disallow: /

User-agent: Visited by http://tools.geek-tools.org
 disallow: /


User-agent: voltron
 disallow: /

User-agent: woobot
 disallow: /

User-agent: woobot/1.1
 disallow: /

User-agent: woobot/2.0
 disallow: /

User-agent: Vorboss Web Crawler
 disallow: /

User-agent: Vorboss Web Crawler/Nutch-2.3
 disallow: /

User-agent: WorldBrewBot
 disallow: /

User-agent: WorldBrewBot/2.1
 disallow: /

User-agent: worldwebheritage.org
 disallow: /

User-agent: worldwebheritage.org/1.0
 disallow: /

User-agent: wscheck.com
 disallow: /

User-agent: wscheck.com/1.0.0
 disallow: /

User-agent: www.deadlinkchecker.com
 disallow: /

User-agent: www.petitsage.fr site detector 0.4
 disallow: /

User-agent: WWW-Mechanize
 disallow: /

User-agent: WWW-Mechanize/1.74
 disallow: /

User-agent: Xenu Link Sleuth
 disallow: /

User-agent: Xenu's Link Sleuth
 disallow: /

User-agent: XoviBot
 disallow: /

User-agent: XoviBot/2.0
 disallow: /

User-agent: XSpider
 disallow: /

User-agent: Yandex Robot
 disallow: /

User-agent: Yandex
 disallow: /

User-agent: Yetibot
 disallow: /

User-agent: YisouSpider
 disallow: /

User-agent: yoozBot
 disallow: /

User-agent: yoozBot-2.2
 disallow: /

User-agent: zgrab/0.x
 disallow: /

User-agent: zzabmbot
 disallow: /

User-agent: zzabmbot/1.0
 disallow: /



User-agent: Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-2.3)
Disallow: /
User-agent: Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Disallow: /




User-agent: Gigabot/2.0
Disallow: /
User-agent: GameSpyHTTP/1.0
Disallow: /
User-agent: Baiduspider ( http://www.baidu.com/search/spider.htm)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Disallow: /

User-Agent: istellabot/t.1.13
Disallow: /
User-Agent: MagpieRSS/0.72 (+http://magpierss.sf.net)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /


User-Agent: Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
Disallow: /
User-Agent: Liferea
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-Agent: ahrefsbot
Disallow: /
User-Agent: Ahrefs
Disallow: /
User-Agent: Ahrefsbot
Disallow: /
User-Agent: BoardReader Blog Indexer(http://boardreader.com)
Disallow: /
User-Agent: BoardReader
Disallow: /
User-Agent: YOUBot
Disallow: /
User-Agent: AhrefsBot.com
Disallow: /
User-Agent: yandexBot
Disallow: /
User-Agent: Dow Jones Searchbot
Disallow: /
User-Agent: TurnitinBot
Disallow: /
User-Agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /
User-Agent: Almaden
Disallow: /
User-Agent: eCairn-Grabber/1.0
Disallow: /
User-Agent: eCairn-Grabber
Disallow: /


User-Agent: Amazon
Disallow: /

User-Agent: Gluten Free Crawler/1.0
Disallow: /

User-Agent: DotBot/1.1
Disallow: /

User-Agent: LinqiaCrawlerBot/1.0
Disallow: /

User-Agent: istellabot/t.1.13
Disallow: /

User-Agent: nutch-1.4/Nutch-1.4
Disallow: /

User-Agent: nutch-1.4
Disallow: /

User-Agent: DoBot
Disallow: /

User-Agent: AsapBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-Agent: amazonbot
Disallow: /


User-Agent: Amazon-bot
Disallow:/

User-Agent: Meditoolkitbot
Disallow: /


User-agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Disallow: /

User-agent: Yeti/1.1
Disallow: /

User-agent: Yeti
Disallow: /


User-agent: Yahoo! Slurp
Disallow: /

User-agent: HaoSouSpider
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: Applebot/0.1; +http://www.apple.com/go/applebot
Disallow: /


User-agent: Applebot/0.1
Disallow: /

User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
Disallow: /


User-agent: ia_archiver
Disallow: /

User-agent: Naverbot
Disallow: /

User-agent: seznambot
Disallow: /

User-agent: teoma
Disallow: /



User-agent: Yeti
Disallow: /

User-agent: EtaoSpider
Disallow: /

User-Agent: swebot
Disallow: /

User-Agent: Newsbrain
Disallow: /

User-Agent: Feedreader 3.14
Disallow: /

User-Agent: Feedreader
Disallow: /

User-Agent: Feedreader 3.14 (Powered by Newsbrain)
Disallow: /

User-agent: aihitbot
Disallow: /

User-agent: Exabot
Disallow: /


User-agent: Exabot-Thumbnails
Disallow: /

User-agent: sitebot
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: TwengaBot-Discover
Disallow: /

User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: Bender
Disallow: /

User-agent: discobot
Disallow: /

User-Agent: panscient.com
Disallow: /

User-Agent: searchwebengine.net
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: NerdByNature.Bot
disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: sistrix
Disallow: /

User-agent: SindiceBot
Disallow: /

User-agent: plukkie
Disallow: /

User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
Disallow: /

User-agent: SBIder
Disallow: /

User-agent: IWAgent/1.0
Disallow: /

User-agent: IWAgent
Disallow: /

User-agent: BotALot
Disallow:/

User-agent: findfiles.net
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Goodzer
Disallow: /

User-agent: DJEcoBot
Disallow: /

User-agent: DJEcoBot/1.0
Disallow: /

User-agent: YandexBot/3.0
Disallow: /

User-agent: AhrefsBot/5.2
Disallow: /

User-agent: Sogou inst spider
Disallow: /

User-agent: Sogou spider2
Disallow: /

User-agent: Sogou blog
Disallow: /

User-agent: Sogou News Spider
Disallow: /

User-agent: Sogou Orion spider
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: spray-can/1.3.4
Disallow: /

User-agent: spray-can
Disallow: /

User-agent: spray can
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: 360spider
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: BacklinkCrawler
Disallow: /

User-agent: lemurwebcrawler
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: discobot Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /

User-agent: Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /

User-agent: Cliqzbot
Disallow: /


User-agent: FAST Enterprise Crawler 6
Disallow: /

User-agent: Sensis.com.au Web Crawler
Disallow: /

User-agent: Baiduspider
Disallow: /



User-agent: Teoma
Disallow: /

User-agent: XilopixBot/0.1
Disallow: /

User-agent: XilopixBot
Disallow: /

User-agent: twiceler
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Arachmo
Disallow: /

User-agent: Acoon
Disallow: /

User-agent: adbeat_bot
Disallow: /

User-agent: AgentLinkSpammer
Disallow: /

User-agent: aiHitBot
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: Scrubby
Disallow: /

User-agent: Robozilla
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: worio bot heritrix
Disallow: /

User-agent: trovitBot
Disallow: /

User-agent: MojeekBot/0.6
Disallow: /

User-agent: rogerbot/1.0
Disallow: /

User-agent: istellabot/t.1.13
Disallow: /

User-agent: SeznamBot/3.2
Disallow: /



User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: careerbot
Disallow: /

User-agent: SogouBot
Disallow: /

User-agent: Sogou spider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: Sogouspider
Disallow: /

User-agent: Qihoo360Bot
Disallow: /

User-agent: EasouSpider
Disallow: /

User-agent: COMODOSpider
Disallow: /

User-agent: findlinks
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: KaloogaBot
Disallow: /

User-agent: Mail.RU_Bot
Disallow: /

User-agent: Mail.RU
Disallow: /

User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Disallow: /

User-agent: SurdotlyBot/1.0
Disallow: /

User-agent: SurdotlyBot
Disallow: /

User-agent: Apexoo Spider (http://www.apexoo.com/spider/)
Disallow: /


User-agent: Apexoo Spider 1.0
Disallow: /

User-agent: Apexoo Spider 1.1
Disallow: /

User-agent: AportCatalogRobot/2.0
Disallow: /

User-agent: Apostrophe/0.2.1.93 (compatible) filecache
Disallow: /

User-agent: Apostrophe
Disallow: /

User-agent: Aplix HTTP/1.0.1
Disallow: /

User-agent: moget
Disallow: /
User-agent: Ichiro
Disallow: /
User-agent: Ichiro 3.0
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: NetcraftSurveyAgent/1.0
Disallow: /
User-agent: OpenWebIndex/Nutch-1.6
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: panoptaStudyBot
Disallow: /
User-agent: checks.panopta.com
Disallow: /
User-agent: psbot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: SeznamBot/1.0
Disallow: /
User-agent: SeznamBot/1.1
Disallow: /
#SeznamBot/3.0
User-agent: SeznamBot/3.0
Disallow: /
User-agent: SistrixCrawler
Disallow: /
User-agent: Sistrix
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SISTRIX
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sosospider+
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sosospider/2.0
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: Wada.vn
Disallow: /
User-agent: Wada.vn Vietnamese Search
Disallow: /
User-agent: Wada.vn Vietnamese Search/2.1
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: YandexBot/3.0-MirrorDetector
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: YandexSomething
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: YoudaoBot/1.0
Disallow: /
User-agent: Zao
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs
Disallow: /
User-agent: aipbot
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: asterias
Disallow:/
User-agent: Aqua_Products
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: becomebot
Disallow: /
User-agent: Black Hole
Disallow:/
User-agent: BlowFish
Disallow:/
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite
Disallow:/
User-agent: CherryPickerSE
Disallow:/
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: FairAd Client
Disallow: /
User-agent: Fasterfox
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow:/
User-agent: Gaisbot
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: Harvest
Disallow:/
User-agent: Go!Zilla
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: HTTrack
Disallow: /
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: Iron33
Disallow: /
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density
Disallow:/
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: Mata Hari
Disallow: /
User-agent: MIIxpc
Disallow:/
User-agent: Mister PiX
Disallow:/

User-agent: moget
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NetMechanic
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: Oracle Ultra Search
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: ProPowerBot
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: psbot
Disallow: /

User-agent: QueryN Metasearch
Disallow:/
User-agent: Radiation Retriever
Disallow: /
User-agent: RepoMonkey
Disallow:/
User-agent: Bait
Disallow:/
User-agent: RMA
Disallow:/
User-agent: searchpreview
Disallow: /
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz
Disallow:/
User-agent: SurveyBot
Disallow: /
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: turingos
Disallow:/
User-agent: TurnitinBot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URLy Warning
Disallow:/
User-agent: URL_Spider_Pro
Disallow: /
User-agent: VCI
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebCapture
Disallow: /
User-agent: scalaj-http/1.0
Disallow: /
User-agent: WebCollage
Disallow: /
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: Wget
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
User-agent: Meltawer
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Cision
Disallow: /

User-agent: Talkwater
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /

User-agent: Zite
Disallow: /

User-agent: flipboard
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: trendybuzz
Disallow: /

User-agent: spotter
Disallow: /

User-agent: scoop.it
Disallow: /

User-agent: linkfluence
Disallow: /

User-agent: 5emeRue
Disallow: /

User-agent: Augure
Disallow: /

User-agent: Corporama
Disallow: /

User-agent: readability.com
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: libwww
Disallow: /

User-agent: wget
Disallow: /

User-agent: 5erue
Disallow: /

User-agent: Sysomos/1.0
Disallow: /

User-agent: adequat
Disallow: /

User-agent: adequat-systems
Disallow: /

User-agent: auramundi
Disallow: /

User-agent: coexel
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: leadbox
Disallow: /

User-agent: mention
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: mytwip
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: mixnode.com
Disallow: /

User-agent: opinion-tracker
Disallow: /

User-agent: proxem
Disallow: /

User-agent: score3
Disallow: /

User-agent: trendeo
Disallow: /

User-agent: vecteurplus
Disallow: /

User-agent: verticalsearch
Disallow: /

User-agent: vsw
Disallow: /

User-agent: winello
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: infoseek
Disallow: /


User-agent: Offline Explorer
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: asknread.com
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: spotter
Disallow: /

User-agent: LexxeBot/1.0
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: Mozilla/5.0 (compatible; spbot/2.0; http://www.seoprofiler.com/bot/ )
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: Nuzzel
Disallow: /

User-agent: Nuzzel Bot
Disallow: /

User-agent: Sosospider+(+http://help.soso.com/webspider.htm)
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: SiteBot/0.1
Disallow: /

User-agent: SiteBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: crystalsemanticsbot
Disallow: /

User-agent: CrystalSemanticsBot
Disallow: /

User-agent: NetSeer crawler
Disallow: /

User-agent: trovitBot
Disallow: /

User-agent: LexxeBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: discobot
Disallow: /

User-agent: Jyxobot
Disallow: /

User-agent: sogou
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: Sogoubot
Disallow: /

User-agent: Sogou.com
Disallow: /

User-agent: sogouspider-106-120-173-138.crawl.sogou.com
Disallow: /

User-agent: crawl13.exabot.com
Disallow: /

User-agent: sogouspider-106-120-173-138
Disallow: /

User-agent: Sogou web spider/4.0
Disallow: /


User-agent: Sogouwebspider
Disallow: /

User-agent: sogouspider
Disallow: /

User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Disallow: /

User-agent: Exabot/3.0
Disallow: /


User-agent: Sogou web spider
Disallow: /

User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: sistrix
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: NerdByNature.Bot
Disallow: /


User-agent: psbot
Disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: AddThis.com robot tech.support@clearspring.com
Disallow: /

User-agent: AddThis.com
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: proximic
Disallow: /

User-agent: discoverybot
Disallow: /

User-agent: bl.uk_lddc_bot
Disallow: /

User-agent: IstellaBot
Disallow: /

User-agent: seokicks
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: UnisterBot
Disallow: /

User-agent: Bender
Disallow: /

User-agent: wotbox
Disallow: /

User-agent: Yasni
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: netEstate NE Crawler
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: Pixray-Seeker
Disallow: /

User-agent: Linguee
Disallow: /

User-agent: integromedb
Disallow: /

User-agent: SearchmetricsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: BDCbot
Disallow: /

User-agent: GrapeshotCrawler
Disallow: /

User-agent: WeSEE:Search
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: admantx
Disallow: /

User-agent: spbot
Disallow: /

User-agent: BUbiNG
Disallow: /

User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: Harvest/1.5
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Nutch
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: Spiderbot
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: Xenu Link Sleuth/1.3.8
User-agent: yacy
User-agent: yandex
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Spiderbot/Nutch-1.7
User-agent: g2reader-bot/1.0
User-agent: SocialShare/1.0
User-agent: Owlin Bot
User-agent: trendictionbot
Disallow: /

# Disallow: Sistrix
User-agent: sistrix
Disallow: /

# Disallow: Sistrix
User-agent: SISTRIX Crawler
Disallow: /

# Disallow: Sistrix
User-agent: SISTRIX
Disallow: /

# Disallow: SEOkicks-Robot
User-agent: SEOkicks-Robot
Disallow: /

# Disallow: jobs.de-Robot
User-agent: jobs.de-Robot
Disallow: /

# Backlink Analysis
user-agent: AhrefsBot
disallow: /

# Bot der Leipziger Unister Holding GmbH
user-agent: UnisterBot
disallow: /

# http://moz.com/products
User-agent: dotbot
Disallow: /

# http://www.searchmetrics.com
User-agent: SearchmetricsBot
Disallow: /

# http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
User-agent: MJ12bot
Disallow: /

# http://www.domaintools.com/webmasters/surveybot.php
User-agent: SurveyBot
Disallow: /

# http://www.seodiver.com/bot
user-agent: SEOdiver
disallow: /

# http://openlinkprofiler.org/bot
User-agent: spbot
Disallow: /

# http://www.wotbox.com/bot/
User-agent: wotbox
Disallow: /

# http://www.opensiteexplorer.org/dotbot
# http://moz.com/researchtools/ose/dotbot
User-agent: dotbot
Disallow: /

# http://www.meanpath.com/meanpathbot.html
User-agent: meanpathbot
Disallow: /

# http://www.backlinktest.com/crawler.html
User-agent: BacklinkCrawler
Disallow: /

# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler
Disallow: /

# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Disallow: /

# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1
Disallow: /

# http://filterdb.iss.net/crawler/
User-agent: oBot
Disallow: /

User-agent: fr-crawler
Disallow: /

# http://webmeup-crawler.com
User-agent: BLEXBot
Disallow: /

# https://megaindex.com/crawler
User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

# http://www.cloudservermarket.com
User-Agent: CloudServerMarketSpider
Disallow: /

# http://www.trendiction.de/de/publisher/bot
User-Agent: trendictionbot
Disallow: /

# http://www.exalead.com
User-agent: Exabot
Disallow: /

# http://www.career-x.de/bot.html
User-agent: careerbot
Disallow: /

# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /

User-agent: seoscanners.net/1
Disallow: /

User-agent: MetaJobBot
Disallow: /

User-agent: Spiderbot
Disallow: /

User-agent: LinkStats
Disallow: /

User-agent: JobboerseBot
Disallow: /

User-agent: ICCrawler
Disallow: /

User-agent: Plista
Disallow: /

User-agent: Domain Re-Animator Bot
Disallow: /

User-agent: Aboundexbot
Disallow: /

User-agent: acapbot
Disallow: /

User-agent: AdnormCrawler
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: AOLbot/4.0
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: Attribot
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Bot.AraTurka.com
Disallow: /

User-agent: Butterfly/1.0
Disallow: /

User-agent: BuzzSumo
Disallow: /

User-agent: cis455crawler
Disallow: /

User-agent: ClarityBot
Disallow: /

User-agent: ClarityDailyBot
Disallow: /

User-agent: CMS Crawler
Disallow: /

User-agent: CommonCrawler
Disallow: /

User-agent: crawler4j (http://code.google.com/p/crawler4j/)
Disallow: /

User-agent: CrazyWebCrawler
Disallow: /

User-agent: CrazyWebCrawler-Spider
Disallow: /

User-agent: DataparkSearch
Disallow: /

User-agent: deepcrawl
Disallow: /

User-agent: demandbase-bot
Disallow: /

User-agent: DeuSu
Disallow: /

User-agent: dialogsearch.com
Disallow: /

User-agent: diffbot
Disallow: /

User-agent: dj-research
Disallow: /

User-agent: DomainAppender
Disallow: /

User-agent: DomainMacroCrawler
Disallow: /

User-agent: DomainSigmaCrawler
Disallow: /

User-agent: Domain Re-Animator Bot
Disallow: /

User-agent: Ezooms Robot
Disallow: /

User-agent: Experibot
Disallow: /

User-agent: FavOrg
Disallow: /

# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /

# https://turnitin.com/robot/crawlerinfo.html
User-agent: turnitinbot
Disallow: /

# http://help.coccoc.com/
User-agent: coccoc
Disallow: /

# ubermetrics-technologies.com
User-agent: um-IC
Disallow: /

User-agent: scanbot
Disallow: /

# datenbutler.de
User-agent: mindUpBot
Disallow: /

# http://searchgears.de/uber-uns/crawling-faq.html
User-agent: sg-Orbiter
Disallow: /

# http://commoncrawl.org/faq/
User-agent: CCBot
Disallow: /


# http://linkfluence.net/
User-agent: Kraken
Disallow: /

# http://www.botje.com/plukkie.htm
User-agent: plukkie
Disallow: /

# https://www.safedns.com/searchbot
User-agent: SafeDNSBot
Disallow: /

# http://www.haosou.com/help/help_3_2.html
User-agent: 360Spider
Disallow: /

# http://www.haosou.com/help/help_3_2.html
User-agent: HaosouSpider
Disallow: /

# http://www.moz.com/dp/rogerbot
User-agent: rogerbot
Disallow: /

# http://www.openhose.org/bot.html
User-agent: OpenHoseBot
Disallow: /

# http://www.screamingfrog.co.uk/seo-spider/
User-agent: Screaming Frog SEO Spider
Disallow: /

# http://thumbsniper.com
User-agent: ThumbSniper
Disallow: /

# http://www.radian6.com/crawler
User-agent: R6_CommentReader
Disallow: /

User-agent: ImplisenseBot
Disallow: /

# http://cliqz.com/company/cliqzbot
User-agent: Cliqzbot
Disallow: /

# https://www.aihitdata.com/about
User-agent: aiHitBot
Disallow: /

# http://www.trendiction.com/en/publisher/bot
User-Agent: trendictionbot
Disallow: /

# Riddler
#	Riddler (http://riddler.io/about)
#	http://riddler.io/about
User-agent: Riddler
Disallow: /

# SMTBot
#	http://www.similartech.com/smtbot
#	Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
User-agent: SMTBot
Disallow: /

# ScoutJet
#	http://www.scoutjet.com
#	Mozilla/5.0+(compatible;+ScoutJet;++http://www.scoutjet.com/)
#	Mozilla/5.0+(compatible;+Blekkobot;+ScoutJet;++http://blekko.com/about/blekkobot)
User-agent: ScoutJet
Disallow: /

# SeeBot
#	http://www.seegnify.com/bot
#	seebot/2.0 (+http://www.seegnify.com/bot)
User-agent: SeeBot
Disallow: /

# Archive.org Bot
#	http://archive.org/details/archive.org_bot
#	Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot)
#	Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot)
User-agent: archive.org_bot
Disallow: /

# AdBeat Bot
#	https://www.adbeat.com/operation_policy
#	adbeat_bot
User-agent: adbeat_bot
Disallow: /

# MeanPathBot
#	http://www.meanpath.com/meanpathbot.html
#	Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
User-agent: meanpathbot
Disallow: /

# Mojeek Bot
#	https://www.mojeek.com/bot.html
#	Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /

# Speedy Spider / EntireWeb Search
#	http://www.entireweb.com
#	Speedy+Spider+(http://www.entireweb.com)
User-agent: Speedy Spider
Disallow: /

# GenieO
#	http://www.genieo.com/webfilter.html
#	Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
User-agent: genieo
Disallow: /

# scrapy.org
# by default ROBOTSTXT_OBEY is set to false
#	http://doc.scrapy.org/en/latest/topics/settings.html
User-agent: Scrapy
Disallow: /

# Gigablast Open Source Search Engine/Crawler/Spider
# You can ignore robots.txt
#	http://gigablast.com/admin.html
#	GigablastOpenSource/1.0
User-agent: GigablastOpenSource
Disallow: /

# Semrush Bot
#	http://www.semrush.com/bot.html
#	Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html)
User-agent:	SemrushBot
Disallow: /

# Proximic
#	http://www.proximic.com/info/spider.php
#	Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
User-agent: proximic
Disallow: /

# AppleBot
#	Used by Siri/Spotlight suggestions (commented out for now)
#	http://www.apple.com/go/applebot
#	Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot)
#	Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot)
#	Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot)
User-agent: applebot
Disallow: /

# CCBot
#	http://commoncrawl.org/faq/
#	CCBot/2.0+(http://commoncrawl.org/faq/)
User-agent: CCBot
Disallow: /

# Plukkie
#	http://www.botje.com/plukkie.htm
#	Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
User-agent: Plukkie
Disallow: /

# Lipperhey Kaus Australis
#	https://www.lipperhey.com/en/about/
#	Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
User-agent: lipperhey
Disallow: /

# oBot
#	http://filterdb.iss.net/crawler/
#	Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
#	Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
User-agent: oBot
Disallow: /

# Screaming Frog SEO Spider
#	Downloaded software, robots.txt can be ignored
#	http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
#	Screaming+Frog+SEO+Spider/2.55
#	Screaming+Frog+SEO+Spider/3.3
User-agent: Screaming Frog SEO Spider
Disallow: /

# VegeBot
#	http://www.exactbot.com/vegebot/index.html
User-agent: VegeBot
Disallow: /

# ScreenerBot
#	ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
User-agent: ScreenerBot
Disallow: /

# Garlik from Experian
#	GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
User-agent: Garlik
Disallow: /

#	BusinessBot: Nathan@lead-caddy.com
User-agent: BusinessBot
Disallow: /


#	FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
User-agent: FAST Enterprise
Disallow: /

# Nutch
#	http://nutch.apache.org/bot.html
#	Nutch+Spider/Nutch-1.5
#	mycrowl/Nutch-1.9
#	nutch+crawler/Nutch-1.9
#	sky+nutch+crawler/Nutch-1.9
#	Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
User-agent: Nutch
Disallow: /





# Open Link Profiler
#	Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
#	http://OpenLinkProfiler.org/bot
User-agent: spbot
Disallow: /


# James Bot
#	Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
#	http://cognitiveseo.com/bot.html
User-agent: JamesBOT
Disallow: /



# Kraken
#	Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;+bot@linkfluence.net)
#	http://linkfluence.net/
User-agent: Kraken
Disallow: /



#	Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
#	http://www.linkdex.com/bots/
User-agent: linkdex
Disallow: /


#	Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
#	Siteimprove.com
User-agent: SiteCheck-sitecrawl by Siteimprove.com
Disallow: /


#	Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
#	http://megaindex.com/crawler
User-agent: MegaIndex
Disallow: /


#	Prlog
#	Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
#	http://prlog.ru/
User-agent: Prlog
Disallow: /

#	Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
#	https://www.aihitdata.com/about
User-agent: AiHitBot
Disallow: /


#	KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
#	http://www.kingland.com
User-agent: KinglandSystemsCorp
Disallow: /


#	Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;+SitemapBot@stmp.com)
#	http://www.sitemapbot.com
User-agent: SitemapBot
Disallow: /


#	Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
#	https://www.deepcrawl.com/bot
User-agent: deepcrawl
Disallow: /


#	Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21
#	http://www.alexa.com/toolbar
#User-agent: AlexaToolbar
#Disallow: /


#	Mozilla/5.0+(compatible;+heritrix/3.3.0-SNAPSHOT-20150302-2206++http://127.0.0.1)
#	https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
User-agent: heritrix
Disallow: /


#	Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
#	http://help.naver.com/robots/
User-agent: Yeti
Disallow: /


#	Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,+contact+abuse@inpwrd.com+for+information)
User-agent: NetShelter
Disallow: /

#	VSE/1.0+(rabraham@multiview.com)

#	LSSRocketCrawler/1.0 LightspeedSystems
#	http://lightspeedsystems.com
User-agent: LSSRocketCrawler
Disallow: /


#	Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
#	http://www.idmarch.org/bot.html
User-agent: idmarch
Disallow: /


#	Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0
#	Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0+(Following+Mediapartners-Google)
User-agent: Daumoa
Disallow: /




#	Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
#	https://www.companiesintheuk.co.uk/bot.html
User-agent: CukBot
Disallow: /


#	Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
#	http://www.expertsearch.nl/spider
User-agent: ExpertSearchSpider
Disallow: /


#	Mozilla/5.0+(compatible;+Goodzer/2.0;+crawler@goodzer.com)
User-agent: Goodzer
Disallow: /


#	WeSEE
#	http://www.wesee.com/
User-agent: WeSEE
Disallow: /


#	http://ltx71.com/
#	ltx71 - (http://ltx71.com/)
User-agent: ltx71
Disallow: /

#	rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,+rogerbot-crawler+shiny@moz.com)
#	http://moz.com/help/pro/what-is-rogerbot-
User-agent: rogerbot
Disallow: /


#	bitlybot
#	http://bit.ly ??
User-agent: bitlybot
Disallow: /


#	Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
#	http://www.trendiction.de/bot
User-agent: trendictionbot
Disallow: /


#	SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
User-agent: SearchMassive
Disallow: /


#	spiderbot
User-agent: spiderbot
Disallow: /


#	yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
#	yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
User-agent: yacybot
Disallow: /


#	AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;+help@aboutus.org)
#	http://www.aboutus.org/Aboutus:Bot
User-agent: AboutUsBot
Disallow: /




#	Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
#	http://www.opensiteexplorer.org/dotbot
User-agent: DotBot
Disallow: /


#	Mozilla/5.0+(compatible;+BLEXBot/1.0;++http://webmeup-crawler.com/)
#	http://webmeup-crawler.com/
User-agent: BLEXBot
Disallow: /





#	Xenu Link Sleuth 1.3.4
#	Xenu Link Sleuth/1.3.8
User-agent: Xenu Link Sleuth
Disallow: /


# Go language
#	Go+1.1+package+http
User-agent: Go 1.1
Disallow: /


# wotbox.com
#	http://www.wotbox.com/bot/
#	Wotbox/2.01+(+http://www.wotbox.com/bot/)
User-agent: Wotbox
Disallow: /

# yandex
#	http://yandex.com/bots
#	Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
User-agent: YandexBot
Disallow: /


#	Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
#	http://help.yahoo.com/help/us/ysearch/slurp
# User-agent: Yahoo! Slurp
# Disallow: /


# Yahoo Japan
#	http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
#	Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
User-agent: Y!J-ASR
Disallow: /

# domainreanimator.com
User-agent: Domain Re-Animator Bot
Disallow: /

# crazywebcrawler.com
User-agent: CrazyWebCrawler-Spider
Disallow: /

# crazywebcrawler.com
User-agent: CrazyWebCrawler
Disallow: /

# Dow Jones Searchbot
#	Mozilla/5.0+(compatible;+Dow+Jones+Searchbot)
User-agent: Dow Jones SearchBot
Disallow: /

# OpenVAS.org
#User-agent: OpenVAS
#Disallow: /

# WeCrawlForThePeace - We are not Evil
User-agent: WeCrawForThePeace
Disallow: /

# WeLikeLinks - WeAreNotEvil
User-agent: WeLikeLinks
Disallow: /

# icevikatam/1.0
#User-agent: icevikatam
#Disallow: /

User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Disallow: /
User-agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Disallow: /
User-agent: SocialRankIOBot; http://socialrank.io/about
Disallow: /
User-agent: SocialRankIOBot
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: Curious George
Disallow: /
User-agent: CybEye.com
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: ExB Language Crawler
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Genio
Disallow: /
User-agent: LWNutch
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: RediffNewsBot
Disallow: /
User-agent: SEOENGWorldBot
Disallow: /
User-agent: Scanmine
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShowyouBot
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: WocBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: daumoa
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: libcrawl
Disallow: /
User-agent: linkdex
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: repparser
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: sindice-site-manager
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: sogou
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: yolinkBot
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: wget
Disallow: /

# Block Sogou
User-agent: sogou spider
Disallow: /

# Block SEOkicks
User-agent: SEOkicks-Robot
Disallow: /

# Block BlexBot
User-agent: BLEXBot
Disallow: /

# Block SISTRIX
User-agent: SISTRIX Crawler
Disallow: /

# Block Uptime robot
User-agent: UptimeRobot/2.0
Disallow: /

User-agent: 008
Disallow: /

# Block Ezooms Robot
User-agent: Ezooms Robot
Disallow: /

# Block Perl LWP
User-agent: Perl LWP
Disallow: /

# Block BlexBot
User-agent: BLEXBot
Disallow: /

# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /

# Block WiseGuys Robot
User-agent: WiseGuys Robot
Disallow: /

# Block Turnitin Robot
User-agent: Turnitin Robot
Disallow: /


User-agent: HuaweiSymantecSpider
Disallow: /

User-agent: Purebot
Disallow: /

# Block Heritrix
User-agent: Heritrix
Disallow: /

User-agent: eContext/1.0 (eContext Classification Engine)
Disallow: /

User-agent: eContext/1.0
Disallow: /

User-agent: eContext
Disallow: /

# Block pricepi
User-agent: pimonster
Disallow: /
User-agent: Pimonster
Disallow: /

# Block Searchmetrics Bot
User-agent: SearchmetricsBot
Disallow: /

# Block Eniro
User-agent: ECCP/1.0 (search@eniro.com)
Disallow: /

# Block YandexBot
User-agent: Yandex
Disallow: /

# Block Baidu
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /

# Block SoGou
User-agent: Sogou Spider
Disallow: /

# Block Youdao
User-agent: YoudaoBot
Disallow: /

# Block  MegaIndex.ru
User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: Yandex
Disallow: /
user-agent: AhrefsBot
disallow: /
User-Agent: trendictionbot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: search:robot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: xovi
Disallow: /
User-agent: Cutbot
Disallow: /



User-agent: DotBot
Disallow: /

User-agent: LivelapBot
Disallow: /

User-agent: LS Session
Disallow: /

User-agent: MagicBot
Disallow: /

User-agent: AliveAdvisorCrawler/1.7.1
Disallow: /

User-agent: AliveAdvisorCrawler/1.7.1 (whole web crawler; http://search.aliveadvisor.com; crawler@aliveadvisor.com)
Disallow: /

User-agent: madaali.de
Disallow: /

User-agent: memoryBot
Disallow: /

User-agent: whole web crawler
Disallow: /




User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: MegaIndex.ru/
Disallow: /

User-agent: SputnikBot
Disallow: /

User-agent: UbiCrawler
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: wget
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: berlin-fu-cow
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: semager
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: InAGist URL Resolver
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Insitesbot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: IndeedBot
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaIndex.ru
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: ltx71
Disallow: /

User-agent: SputnikBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: seoscanners.net
Disallow: /

User-agent: DeuSu
Disallow: /

User-agent: meds-online24.com
Disallow: /

User-agent: seoscanners
Disallow: /

User-agent: GuzzleHttp
Disallow: /

User-agent: IDBot
Disallow: /

User-agent: PRTGCloudBot
Disallow: /

User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: WebLyzard Crawler
Disallow: /

User-agent: WebCopier v4.3
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: Shim-Crawler
Disallow: /

User-agent: OmniExplorer_Bot
Disallow: /

User-agent: NimbleCrawler
Disallow: /

User-agent: hl_ftien_spider_v1.1
Disallow: /

User-agent: EMPAS_ROBOT
Disallow: /

User-agent: EDI/1.4.1 (Edacious & Intelligent Web Crawler)
Disallow: /

User-agent: (compatible; hetrix/1.8.0+http://www.accelobot.com)
Disallow: /

User-agent: WebReaper [info@webreaper.net]
Disallow: /

User-agent: WebReaper [webcreaper@otway.com]
Disallow: /

User-agent: Mozilla/5.0 (compatible; Vagabondo/Kliksafe; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Disallow: /

User-agent: Vagabondo/4.0 (webagent at wise-guys dot nl)
Disallow: /

User-agent: Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Disallow: /

# http://www.wise-guys.nl/
User-agent: webcrawler
Disallow: /

User-agent: ilse
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: Bilbo
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: lwp-trivial/5.810
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: discoverybot
Disallow: /

User-agent: SimplePie
Disallow: /

User-agent: SimplePie/1.1.1
Disallow: /

User-agent: SimplePie/1.0.1
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Ocelli/1.4
Disallow: /

User-agent: Ocelli
Disallow: /

User-agent: ezooms.bot@gmail.com
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: Ezooms/1.0
Disallow: /

User-agent: Ezooms/1.0; ezooms.bot@gmail.com
Disallow: /

#ad Crawler: http://www.a6corp.com/a6-web-scraping-policy/
User-agent: A6-Indexer/1.0
Disallow: /

#ad Crawler: http://www.grapeshot.co.uk/crawler.php
User-agent: grapeshot
Disallow: /

# http://www.proximic.com/info/spider.php
User-agent: proximic
Disallow: /

# http://www.jobdigger.nl/spider/
User-agent: JobdiggerSpider
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: rogerbot
Disallow:/

User-agent: SEOkicks-Robot
Disallow:/

User-agent: SMTBot
Disallow:/

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: EnaBot/1.2 (http://www.enaball.com/crawler.html)
Disallow: /

User-agent: EnaBot
Disallow: /

User-agent: Go-http-client/1.1
Disallow: /


User-agent: SMTBot/1.0
Disallow: /

User-agent: e-SocietyRobot
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Disallow: /

User-agent: FreeFind
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Caliperbot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: SimplePie
Disallow: /

User-agent: SimplePie/1.1.3 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20081219
Disallow: /

User-Agent: trendictionbot
Disallow: /

# http://sentibot.eu/
User-agent: SentiBot
Disallow: /

User-agent: SentiBot
Disallow: /

User-agent: omgili/0.5 +http://omgili.com
Disallow: /

User-agent: Mediatoolkitbot (complaints@mediatoolkit.com)
Disallow: /

User-agent: Mediatoolkitbot
Disallow: /

User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /

User-agent: GrapeshotCrawler/2.0
Disallow: /

User-agent: GrapeshotCrawler
Disallow:/

User-agent: Grapeshot Crawler
Disallow:/

User-agent: Grapeshot
Disallow:/

User-agent: omgili/0.5 +http://omgili.com
Disallow: /

User-agent: omgili
Disallow: /


User-agent: musobot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SimplePie
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: Mozilla/5.0 ResearchBot/0.6.2 (project: flattr-id-meta-tokens) This/1
Disallow: /

User-agent: ResearchBot/0.6.2
Disallow: /

User-agent: ResearchBot
Disallow: /

User-agent: jobs.de-Robot
Disallow: /

User-agent: FreeFind
Disallow: /

User-agent: xingbot
Disallow: /

User-agent: xing bot
Disallow: /

User-agent: xing
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: ChinasoSpider
Disallow: /

User-agent: yisouspider
Disallow: /

User-agent: EasouSpider
Disallow: /


User-agent: SeznamBot
Disallow: /

User-agent: Caliperbot
Disallow: /

User-agent: spbot
Disallow: /

user-agent: EveryoneSocialBot
Disallow: /
user-agent: something
Disallow: /

user-agent: ABCdatos BotLink
user-agent: Acme.Spider
user-agent: Ahoy! The Homepage Finder
user-agent: Alkaline
user-agent: Anthill
user-agent: Walhello appie
user-agent: Arachnophilia
user-agent: Arale
user-agent: Araneo
user-agent: AraybOt
user-agent: ArchitextSpider
user-agent: Aretha
user-agent: ARIADNE
user-agent: arks
user-agent: AskJeeves
user-agent: ASpider (Associative Spider)
user-agent: ATN Worldwide
user-agent: Atomz.com Search Robot
user-agent: AURESYS
user-agent: BackRub
user-agent: Bay Spider
user-agent: Big Brother
user-agent: Bjaaland
user-agent: BlackWidow
user-agent: Die Blinde Kuh
user-agent: Bloodhound
user-agent: Borg-Bot
user-agent: BoxSeaBot
user-agent: bright.net caching robot
user-agent: BSpider
user-agent: CACTVS Chemistry Spider
user-agent: Calif
user-agent: Cassandra
user-agent: Digimarc Marcspider/CGI
user-agent: Checkbot
user-agent: ChristCrawler.com
user-agent: churl
user-agent: cIeNcIaFiCcIoN.nEt
user-agent: CMC/0.01
user-agent: Collective
user-agent: Combine System
user-agent: Conceptbot
user-agent: ConfuzzledBot
user-agent: CoolBot
user-agent: Web Core / Roots
user-agent: XYLEME Robot
user-agent: Internet Cruiser Robot
user-agent: Cusco
user-agent: CyberSpyder Link Test
user-agent: CydralSpider
user-agent: Desert Realm Spider
user-agent: DeWeb(c) Katalog/Index
user-agent: DienstSpider
user-agent: Digger
user-agent: Digital Integrity Robot
user-agent: Direct Hit Grabber
user-agent: DNAbot
user-agent: DownLoad Express
user-agent: DragonBot
user-agent: DWCP (Dridus' Web Cataloging Project)
user-agent: e-collector
user-agent: EbiNess
user-agent: EIT Link Verifier Robot
user-agent: ELFINBOT
user-agent: Emacs-w3 Search Engine
user-agent: ananzi
user-agent: esculapio
user-agent: Esther
user-agent: Evliya Celebi
user-agent: FastCrawler
user-agent: Fluid Dynamics Search Engine robot
user-agent: Felix IDE
user-agent: Wild Ferret Web Hopper #1, #2, #3
user-agent: FetchRover
user-agent: fido
user-agent: Hämähäkki
user-agent: KIT-Fireball
user-agent: Fish search
user-agent: Fouineur
user-agent: Robot Francoroute
user-agent: Freecrawl
user-agent: FunnelWeb
user-agent: gammaSpider, FocusedCrawler
user-agent: gazz
user-agent: GCreep
user-agent: GetBot
user-agent: GetURL
user-agent: Golem
user-agent: Grapnel/0.01 Experiment
user-agent: Griffon
user-agent: Gromit
user-agent: Northern Light Gulliver
user-agent: Gulper Bot
user-agent: HamBot
user-agent: Harvest
user-agent: havIndex
user-agent: HI (HTML Index) Search
user-agent: Hometown Spider Pro
user-agent: ht://Dig
user-agent: HTMLgobble
user-agent: Hyper-Decontextualizer
user-agent: iajaBot
user-agent: IBM_Planetwide
user-agent: Popular Iconoclast
user-agent: Ingrid
user-agent: Imagelock
user-agent: IncyWincy
user-agent: Informant
user-agent: InfoSeek Robot 1.0
user-agent: Infoseek Sidewinder
user-agent: InfoSpiders
user-agent: Inspector Web
user-agent: IntelliAgent
user-agent: I, Robot
user-agent: Iron33
user-agent: Israeli-search
user-agent: JavaBee
user-agent: JBot Java Web Robot
user-agent: JCrawler
user-agent: Jeeves
user-agent: JoBo Java Web Robot
user-agent: Jobot
user-agent: JoeBot
user-agent: The Jubii Indexing Robot
user-agent: JumpStation
user-agent: image.kapsi.net
user-agent: Katipo
user-agent: KDD-Explorer
user-agent: Kilroy
user-agent: KO_Yappo_Robot
user-agent: LabelGrabber
user-agent: larbin
user-agent: legs
user-agent: Link Validator
user-agent: LinkScan
user-agent: LinkWalker
user-agent: Lockon
user-agent: logo.gif Crawler
user-agent: Lycos
user-agent: Mac WWWWorm
user-agent: Magpie
user-agent: marvin/infoseek
user-agent: Mattie
user-agent: MediaFox
user-agent: MerzScope
user-agent: NEC-MeshExplorer
user-agent: MindCrawler
user-agent: mnoGoSearch search engine software
user-agent: moget
user-agent: MOMspider
user-agent: Monster
user-agent: Motor
user-agent: Muncher
user-agent: Muninn
user-agent: Muscat Ferret
user-agent: Mwd.Search
user-agent: Internet Shinchakubin
user-agent: NDSpider
user-agent: Nederland.zoek
user-agent: NetCarta WebMap Engine
user-agent: NetMechanic
user-agent: NetScoop
user-agent: newscan-online
user-agent: NHSE Web Forager
user-agent: Nomad
user-agent: The NorthStar Robot
user-agent: nzexplorer
user-agent: ObjectsSearch
user-agent: Occam
user-agent: HKU WWW Octopus
user-agent: OntoSpider
user-agent: Openfind data gatherer
user-agent: Orb Search
user-agent: Pack Rat
user-agent: PageBoy
user-agent: ParaSite
user-agent: Patric
user-agent: pegasus
user-agent: The Peregrinator
user-agent: PerlCrawler 1.0
user-agent: Phantom
user-agent: PhpDig
user-agent: PiltdownMan
user-agent: Pimptrain.com's robot
user-agent: Pioneer
user-agent: html_analyzer
user-agent: Portal Juice Spider
user-agent: PGP Key Agent
user-agent: PlumtreeWebAccessor
user-agent: Poppi
User-agent: iskanie
user-agent: PortalB Spider
user-agent: psbot
user-agent: GetterroboPlus Puu
user-agent: The Python Robot
user-agent: Raven Search
user-agent: RBSE Spider
user-agent: Resume Robot
user-agent: RoadHouse Crawling System
user-agent: RixBot
user-agent: Road Runner: The ImageScape Robot
user-agent: Robbie the Robot
user-agent: ComputingSite Robi/1.0
user-agent: RoboCrawl Spider
user-agent: RoboFox
user-agent: Robozilla
user-agent: Roverbot
user-agent: RuLeS
user-agent: SafetyNet Robot
user-agent: Scooter
user-agent: Sleek
user-agent: Search.Aus-AU.COM
user-agent: SearchProcess
user-agent: Senrigan
user-agent: SG-Scout
user-agent: ShagSeeker
user-agent: Shai'Hulud
user-agent: Sift
user-agent: Simmany Robot Ver1.0
user-agent: Site Valet
user-agent: Open Text Index Robot
user-agent: SiteTech-Rover
user-agent: Skymob.com
user-agent: SLCrawler
user-agent: Inktomi Slurp
user-agent: Smart Spider
user-agent: Snooper
user-agent: Solbot
user-agent: Spanner
user-agent: Speedy Spider
user-agent: spider_monkey
user-agent: SpiderBot
user-agent: Spiderline Crawler
user-agent: SpiderMan
user-agent: SpiderView(tm)
user-agent: Spry Wizard Robot
user-agent: Site Searcher
user-agent: Suke
user-agent: suntek search engine
user-agent: Sven
user-agent: Sygol
user-agent: TACH Black Widow
user-agent: Tarantula
user-agent: tarspider
user-agent: Tcl W3 Robot
user-agent: TechBOT
user-agent: Templeton
user-agent: TeomaTechnologies
user-agent: TITAN
user-agent: TitIn
user-agent: The TkWWW Robot
user-agent: TLSpider
user-agent: UCSD Crawl
user-agent: UdmSearch
user-agent: UptimeBot
user-agent: URL Check
user-agent: URL Spider Pro
user-agent: Valkyrie
user-agent: Verticrawl
user-agent: Victoria
user-agent: vision-search
user-agent: void-bot
user-agent: Voyager
user-agent: VWbot
user-agent: The NWI Robot
user-agent: W3M2
user-agent: WallPaper (alias crawlpaper)
user-agent: the World Wide Web Wanderer
user-agent: w@pSpider by wap4.com
user-agent: WebBandit Web Spider
user-agent: WebCatcher
user-agent: WebCopy
user-agent: webfetcher
user-agent: The Webfoot Robot
user-agent: Webinator
user-agent: weblayers
user-agent: WebLinker
user-agent: WebMirror
user-agent: The Web Moose
user-agent: WebQuest
user-agent: Digimarc MarcSpider
user-agent: WebReaper
user-agent: webs
user-agent: Websnarf
user-agent: WebSpider
user-agent: WebVac
user-agent: webwalk
user-agent: WebWalker
user-agent: WebWatch
user-agent: Wget
user-agent: whatUseek Winona
user-agent: WhoWhere Robot
user-agent: Wired Digital
user-agent: Weblog Monitor
user-agent: w3mir
user-agent: WebStolperer
user-agent: The Web Wombat
user-agent: The World Wide Web Worm
user-agent: WWWC Ver 0.2.5
user-agent: WebZinger
user-agent: XGET
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /

User-agent: Zite
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: trendybuzz
Disallow: /

User-agent: spotter
Disallow: /

User-agent: linkfluence
Disallow: /

User-agent: 5emeRue
Disallow: /

User-agent: Augure
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Corporama
Disallow: /

User-agent: readability.com
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: libwww
Disallow: /

User-agent: 5erue
Disallow: /

User-agent: adequat
Disallow: /

User-agent: adequat-systems
Disallow: /

User-agent: auramundi
Disallow: /

User-agent: coexel
Disallow: /

User-agent: leadbox
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: mytwip
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: opinion-tracker
Disallow: /

User-agent: proxem
Disallow: /

User-agent: score3
Disallow: /

User-agent: trendeo
Disallow: /

User-agent: vecteurplus
Disallow: /

User-agent: verticalsearch
Disallow: /

User-agent: vsw
Disallow: /

User-agent: winello
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: infoseek
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: asknread.com
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: spotter
Disallow: /

User-agent: 008
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: AlvinetSpider
Disallow: /

User-agent: Antenne Hatena
Disallow: /

User-agent: ApocalXExplorerBot
Disallow: /

User-agent: asterias
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: BizInformation
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: DISCo Pump 3.1
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: DotBot/1.1
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: Flamingo_SearchEngine
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: hloader
Disallow: /

User-agent: httplib
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: HTTrack 3.0
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: Igentia
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: libWeb/clsHTTP
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: InfoPath
Disallow: /

User-agent: InfoPath.3
Disallow: /

User-agent: Cyberwatcher
Disallow: /

User-agent: SafeDNSBot (https://www.safedns.com/searchbot)
Disallow: /



User-agent: Mata Hari
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: moget
Disallow: /

User-agent: moget/2.1
Disallow: /

User-agent: MS Search 4.0 Robot
Disallow: /

User-agent: MS Search 5.0 Robot
Disallow: /

User-agent: Naverbot
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NetAttache
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: OpenindexSpider
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: psbot
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RMA
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SightupBot
Disallow: /

User-agent: SiteBot
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SiteSucker
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: sosospider
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: spanner
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: suggybot
Disallow: /

User-agent: SuperBot
Disallow: /

User-agent: SuperBot/2.6
Disallow: /

User-agent: suzuran
Disallow: /

User-agent: Szukacz/1.4
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: TightTwatBot
Disallow: /

User-agent: Titan
Disallow: /

User-agent: toCrawl/UrlDispatcher
Disallow: /

User-agent: TosCrawler
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: turingos
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: UrlPouls
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: VCI
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: webcopy
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: webmirror
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: website extractor
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebStripper/2.02
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: wget
Disallow: /

User-agent: WikioFeedBot
Disallow: /

User-agent: WinHTTrack
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: yacy
Disallow: /

User-agent: YRSPider
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zookabot
Disallow: /

## EXCLUSION BOTS
User-agent: ConveraCrawler
User-agent: Jetbot
User-agent: Meltwater
User-agent: Digimind
User-agent: Knowings
User-agent: kbcrawl
User-agent: AmiSoftware
User-agent: Ask n read
User-agent: Qwam content intelligence
User-agent: Zite
User-agent: FlipboardProxy
User-agent: Youmag
User-agent: Synthesio
User-agent: trendybuzz
User-agent: spotter
User-agent: linkfluence
User-agent: 5emeRue
User-agent: Augure
User-agent: Sindup
User-agent: Corporama
User-agent: readability.com
User-agent: grub-client
User-agent: ia_archiver
User-agent: ia_archiver-web.archive.org
User-agent: k2spider
User-agent: libwww
User-agent: 5erue
User-agent: adequat
User-agent: adequat-systems
User-agent: auramundi
User-agent: coexel
User-agent: leadbox
User-agent: Moreover
User-agent: mytwip
User-agent: NewsNow
User-agent: Newzbin
User-agent: opinion-tracker
User-agent: proxem
User-agent: score3
User-agent: trendeo
User-agent: vecteurplus
User-agent: verticalsearch
User-agent: vsw
User-agent: winello
User-agent: Fetch
User-agent: infoseek
User-agent: MSIECrawler
User-agent: sitecheck.internetseer.com
User-agent: TeleportPro
User-agent: Zealbot
User-agent: asknread.com
User-agent: ellisphere
User-agent: spotter
User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: yacy
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Linguee Bot
Disallow: /


User-agent: BLP_bbot/0.1
Disallow: /




User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
Disallow: /

User-agent: YandexSomething/1.0
Disallow: /

User-agent: Java/1.6.0_04
Disallow: /

User-agent: 8484 Boston Project v 1.0
Disallow: /

User-agent: Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; m)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; H)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; P)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; I)
Disallow: /

User-agent: +BecomeBot/3.0
Disallow: /

User-agent: AISearchBot
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: AmazonCloudServices
Disallow: /

User-agent: Apexoo Spider 1.0
Disallow: /

User-agent: Aqua_Products
Disallow: /

User-agent: Arachnoidea
Disallow: /

User-agent: ArchitextSpider
Disallow: /



User-agent: BackDoorBot/1.0
Disallow: /

User-agent: BackDoorBot
Disallow: /



User-agent: DotBot/1.1
Disallow: /




User-agent: YandexBot
Disallow: /

User-agent: YandexBot/3.0
Disallow: /





User-agent: SemrushBot-SA/0.97
Disallow: /




User-agent: archive.org_bot
Disallow: /


User-agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Disallow: /


User-agent: AsapBot
Disallow: /


User-agent: semantic-visions.com crawler
Disallow: /

User-agent: MetaURI API/2.0 +metauri.com
Disallow: /

User-agent: MetaURI
Disallow: /


User-agent: citeseerxbot
Disallow: /

User-agent: autokrawl
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /

User-agent: becomebot
Disallow: /

User-agent: BecomeBot/3.0
Disallow: /

User-agent: Bitacle
Disallow: /

User-agent: Bitacle bot
Disallow: /

User-agent: Bitacle bot/1.1
Disallow: /

User-agent: Bitacle*
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: BobCrawl
Disallow: /

User-agent: boitho
Disallow: /

User-agent: boitho.com
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: bot/1.0
Disallow: /

User-agent: bot/1.0+(bot;+http://;+bot@bot.bot)
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: botmobi
Disallow: /

User-agent: bsalsa
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Butterfly
Disallow: /

User-agent: Cazoodle-Bot
Disallow: /

User-agent: CazoodleBot
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: Charlotte
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: classroombot
Disallow: /

User-agent: ConveraCrawler/0.9d
Disallow: /

User-agent: ConveraCrawler/0.9e
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: crawl13.exabot.com
Disallow: /

User-agent: crawl6.exabot.com
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: cyberpatrol
Disallow: /

User-agent: cyberpatrolcrawler
Disallow: /

User-agent: datacha0s/2.0
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /


User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
Disallow: /

User-agent: dcbot
Disallow: /

User-agent: del.icio.us
Disallow: /

User-agent: del.icio.us-thumbnails/1.0
Disallow: /

User-agent: DepSpid
Disallow: /

User-agent: discobot
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: DoCoMo
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: DTAAgent
Disallow: /

User-agent: eagle
Disallow: /

User-agent: ee:
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: envolk
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: EUDORA
Disallow: /

User-agent: Everest-Vulcan
Disallow: /

User-agent: exabot
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: Exabot-Images/1.0
Disallow: /

User-agent: Exabot-Images/3.0
Disallow: /

User-agent: Exabot/2.0
Disallow: /

User-agent: Exabot/3.0
Disallow: /

User-agent: exooba
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: FairShare
Disallow: /

User-agent: fast
Disallow: /

User-agent: fast Enterprise crawler
Disallow: /

User-agent: fast metaweb crawler
Disallow: /

User-agent: FAST-WebCrawler
Disallow: /

User-agent: fast-Webcrawler
Disallow: /

User-agent: findlinks/1.1.1-a5
Disallow: /

User-agent: findlinks/1.1.3-beta9
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: flatlandbot
Disallow: /

User-agent: jack
Disallow: /

User-agent: JS-Kit URL Resolver
Disallow: /

User-agent: KomodiaBot
Disallow: /

User-agent: FollowSiteBot
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: FunWebProducts
Disallow: /

User-agent: FunWebProducts f39-2350
Disallow: /

User-agent: FyberSpider
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: genieBot
Disallow: /

User-agent: genieBot (wgao@genieknows.com)
Disallow: /

User-agent: genieBot+(wgao@genieknows.com)
Disallow: /

User-agent: genieBot+wgao@genieknows.com
Disallow: /

User-agent: GetRight/4.2
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: Gigabot*
Disallow: /

User-agent: Gigabot/2.0
Disallow: /

User-agent: GingerCrawler
Disallow: /

User-agent: grub
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: gsa-crawler
Disallow: /

User-agent: Gulliver
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: HenryTheMiragoRobot
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: heritrix/1.12.1 +http://www.page-store.com)
Disallow: /

User-agent: hloader
Disallow: /

User-agent: HMSE_Robot
Disallow: /

User-agent: HornySexSearch
Disallow: /

User-agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /

User-agent: http://www.cuill.com/twiceler/robot.html
Disallow: /

User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/)
Disallow: /

User-agent: httplib
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: ia_archive
Disallow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

User-agent: ia_archiver/1.6
Disallow: /

User-agent: IBM
Disallow: /

User-agent: IBM_Planetwide
Disallow: /

User-agent: ichiro/2.0
Disallow: /

User-agent: ichiro/2.0 (ichiro@nttr.co.jp)
Disallow: /

User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
Disallow: /

User-agent: IconSurf
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: InfoPath
Disallow: /

User-agent: InfoPath.1
Disallow: /

User-agent: InfoPath.2
Disallow: /

User-agent: IntegraTelecom
Disallow: /

User-agent: internetseer
Disallow: /

User-agent: InternetSeer.com
Disallow: /

User-agent: ips-agent
Disallow: /

User-agent: IRLbot
Disallow: /

User-agent: IRLbot/3.0
Disallow: /

User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: Isidorus
Disallow: /

User-agent: Java Bot 2.0
Disallow: /

User-agent: Java/1.5.0_05
Disallow: /

User-agent: Java/1.5.0_10
Disallow: /

User-agent: JavaBot
Disallow: /

User-agent: JavaBot 2.0
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: Jyxobot
Disallow: /

User-agent: kalooga
Disallow: /

User-agent: KaloogaBot
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Keyword Density/0.9
Disallow: /

User-agent: Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Disallow: /


User-agent: kyluka
Disallow: /

User-agent: LaBot
Disallow: /

User-agent: larbin
Disallow: /

User-agent: larbin_2.6.1
Disallow: /

User-agent: larbin_2.6.1+(larbin2.6.1@unspecified.mail)
Disallow: /

User-agent: laycat
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: LexiBotWeb Image Collector
Disallow: /

User-agent: libcurl-agent
Disallow: /



User-agent: LinkAider
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LinkWalker/2.0
Disallow: /

User-agent: LinkWalker/2.0 - www.seventwentyfour.com
Disallow: /

User-agent: LMQueueBot/0.2
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: LocalBot
Disallow: /

User-agent: LocalcomBot
Disallow: /

User-agent: looksmart
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: Mail.Ru
Disallow: /

User-agent: MantraAgent
Disallow: /

User-agent: Mata Hari
Disallow: /


User-agent: Microsoft URL Control
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: MJ12bot/v1.2.0
Disallow: /

User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? )
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: Modified GMMBotRightHere
Disallow: /

User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Disallow: /

User-agent: Mozilla/5.0+(compatible;+BecomeBot/3.0;++http://www.become.com/site_owners.html)
Disallow: /


User-agent: MSR-ISRCCrawler
Disallow: /

User-agent: nambu
Disallow: /

User-agent: NationalDirectory-SuperSpider
Disallow: /

User-agent: NaverBot
Disallow: /

User-agent: naverbot
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: nexen
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: NG/2.0
Disallow: /

User-agent: nicebot
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: noxtrumbot/1.0
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: obot
Disallow: /

User-agent: Ocelli
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Offline Explorer/1.5
Disallow: /

User-agent: Offline Explorer/1.9
Disallow: /

User-agent: Offline Explorer/2.5
Disallow: /

User-agent: OmniExplorer
Disallow: /

User-agent: OmniExplorer_Bot
Disallow: /

User-agent: OmniExplorer_Bot/1.07
Disallow: /

User-agent: OmniExplorer_Bot/1.09
Disallow: /

User-agent: OmniExplorer_Bot/1.10
Disallow: /

User-agent: OnTownsBot
Disallow: /

User-agent: OOZBOT
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openfind data gathere
Disallow: /

User-agent: OpenX
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: page_prefetcher
Disallow: /


User-agent: PerMan
Disallow: /

User-agent: Piffany_Web_Cacher_v0.91
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: psbot
Disallow: /

User-agent: psbot/0.1
Disallow: /

User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html)
Disallow: /

User-agent: PSIBots
Disallow: /


User-agent: QEAVis
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: Radiation Retriever 1.1
Disallow: /

User-agent: Rankivabot
Disallow: /

User-agent: REAP-Crawler
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /

User-agent: RiceComputerArchitecture
Disallow: /

User-agent: RMA
Disallow: /

User-agent: robotgenius
Disallow: /

User-agent: Robozilla
Disallow: /

User-agent: ru_com_viewer
Disallow: /

User-agent: RufusBot
Disallow: /

User-agent: SapphireWebCrawler
Disallow: /

User-agent: Twiceler www.cuill.com/robots.html
Disallow: /


User-agent: k2spider
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: wget
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /


User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: IsraBot
Disallow: /

User-agent: Orthogaffe
Disallow: /

User-agent: Zealbot
Disallow: /


User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: fast
Disallow: /

User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Disallow: /

User-agent: Twiceler www.cuill.com/twiceler/robot.html
Disallow: /

User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Disallow: /



User-agent: sbider
Disallow: /

User-agent: SBIder
Disallow: /

User-agent: SBIder/SBIder-0.8.2-dev
Disallow: /

User-agent: scooter
Disallow: /

User-agent: Scooter/2.0
Disallow: /

User-agent: scoutjet
Disallow: /

User-agent: Scrubby
Disallow: /

User-agent: searchme
Disallow: /

User-agent: searchpreview
Disallow: /

User-agent: Sensis Web Crawler
Disallow: /

User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)
Disallow: /

User-agent: Sensis+Web+Crawler+
Disallow: /

User-agent: Servage
Disallow: /

User-agent: ServageRobot
Disallow: /

User-agent: setooz
Disallow: /

User-agent: ShopWiki
Disallow: /

User-agent: Sidewinder
Disallow: /

User-agent: Sika
Disallow: /

User-agent: sitecat
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /


User-agent: SiteSnagger
Disallow: /

User-agent: Slicehost
Disallow: /

User-agent: Slurp.so/1.0
Disallow: /

User-agent: slurp@inktomi
Disallow: /

User-agent: slurp@inktomi.com
Disallow: /

User-agent: Slurp/2.0-KiteHourly
Disallow: /

User-agent: Slurp/2.0-OwlWeekly
Disallow: /

User-agent: Slurp/2.0j
Disallow: /

User-agent: Slurp/3.0-AU
Disallow: /

User-agent: SlySearch
Disallow: /

User-agent: SlySearch/1.0
Disallow: /

User-agent: SlySearch/1.x
Disallow: /

User-agent: Snapbot
Disallow: /

User-agent: Snapbot/1.0
Disallow: /

User-agent: SnapPreviewBot
Disallow: /

User-agent: Snappy
Disallow: /

User-agent: Snappybot
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: sogou+spider
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: spanner
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: speedyspider
Disallow: /

User-agent: sphsearch
Disallow: /

User-agent: sproose
Disallow: /

User-agent: StackRambler/2.0
Disallow: /

User-agent: StackRambler/2.0+(MSIE+incompatible)
Disallow: /

User-agent: SuperBot/4.4.0.60 (Windows XP)
Disallow: /

User-agent: SuperPages
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: SurveyBot/2.3
Disallow: /

User-agent: SurveyBot/2.3 (Whois Source)
Disallow: /

User-agent: uberbot
Disallow: /

User-agent: UltraSeek
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
Disallow: /



User-agent: ViolaBot
Disallow: /



User-agent: WebCopier
Disallow: /

User-agent: WebCopier v3.2a
Disallow: /

User-agent: WebCopier v3.3
Disallow: /

User-agent: WebCopier v3.3.0
Disallow: /

User-agent: WebCopier v3.3.2
Disallow: /

User-agent: WebCrawler
Disallow: /

User-agent: WebDataCentreBot
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebmasterWorld Extractor
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: Websense
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webspider
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebViewer
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: WebZIP/3.5 (http://www.spidersoft.com)
Disallow: /


User-agent: whois.de
Disallow: /

User-agent: whoisde.de
Disallow: /

User-agent: WillyBot
Disallow: /

User-agent: Win32
Disallow: /

User-agent: worio
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: Xenu Link Sleuth
Disallow: /

User-agent: Xenu Link Sleuth 1.2i
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Xenu Link Sleuth/1.3.8
Disallow: /

User-agent: Y!J
Disallow: /

User-agent: y!j-bsc
Disallow: /

User-agent: Y!J-BSC
Disallow: /

User-agent: Y!J-BSC/1.0
Disallow: /

User-agent: TunitinBot
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings D
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: Wget
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /



User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Disallow: /

User-agent: y!j/1.0
Disallow: /

User-agent: Yahoo-Blogs
Disallow: /

User-agent: Yahoo-Blogs/v3.9
Disallow: /

User-agent: Yahoo-MMCrawler/3.x
Disallow: /

User-agent: YahooSeeker
Disallow: /

User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
Disallow: /

User-agent: YahooYSMcm
Disallow: /

User-agent: Zite
Disallow: /

User-agent: flipboard
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: YahooYSMcm/2.0.0
Disallow: /

User-agent: yandex
Disallow: /

User-agent: YandexSomething
Disallow: /

User-agent: Yanga
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: yoogliFetchAgent/0.1
Disallow: /

User-agent: Zao
Disallow: /

User-agent: zermelo
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: zibber
Disallow: /

User-agent: zibber v0.1
Disallow: /

User-agent: zibber-v0.1(www.zibb.com/crawler/)
Disallow: /

User-agent: ZupeeCrawler
Disallow: /

User-agent: zyborg
Disallow: /

User-agent: ZyBorg/1.0
Disallow: /

User-agent: BLEXBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Snoothbot
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Daum
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: WeSEE
Disallow: /
User-agent: ShopWiki
Disallow: /
user-agent: crystalsemantics
disallow: /
user-agent: f-six
disallow: /
user-agent: sitebot
disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: scrapybot
Disallow: /
User-agent: VocusBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: ezooms.bot
Disallow: /
User-agent: skimlinks.com
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: wikiwix-bot
Disallow: /
User-agent: semantissimo
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: LemonSources
Disallow: /
User-agent: discobot
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: ltbot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: language-tools.com
Disallow: /
User-agent: TREC-KBA-Bot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: Socialradarbot
Disallow: /
User-agent: Havij
Disallow: /
User-agent: proximic
Disallow: /
User-agent: MireoBot
Disallow: /
User-agent: brandwatch
Disallow: /
User-agent: seoprofiler
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Bot clicker
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: EC2LinkFinder
Disallow: /
User-agent: ExaleadCloudview
Disallow: /
User-agent: AdMedia
Disallow: /
User-agent: grepnetstat.com
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: NerdByNature
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: netseer
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: netseer.com
Disallow: /
User-agent: nagios
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: FeedRetriever
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: Argus Presse
Disallow: /
User-agent: IBM ICA Crawler
Disallow: /
User-agent: nutch agent
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: PaperLiBot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: ip-web-crawler.com
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: LinksCrawler
Disallow: /
User-agent: archive.org
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: netEstate
Disallow: /
User-agent: MyNutchTest
Disallow: /
User-agent: admantx
Disallow: /
User-agent: genieo.com
Disallow: /
User-agent: mixdata.com
Disallow: /
User-agent: ubermetrics-technologies
Disallow: /
User-agent: grapeshot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: WASALive
Disallow: /
User-agent: URLAppendBot
Disallow: /
User-agent: MaxPointCrawler
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Aboundexbot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: psbot
Disallow: /


User-Agent: Slurp
User-agent: Yahoo! Slurp
User-agent: Baiduspider
User-agent: TweetedTimes Bot
User-agent: TweetmemeBot
User-agent: PaperLiBot
User-agent: BomboraBot
User-agent: WF search
User-agent: heritrix
User-agent: expo9
User-agent: NetSeer
User-agent: AddThis.com
User-agent: Feed
User-agent: Sogou
User-agent: ZumBot
User-agent: Feedspot
User-agent: acebookexternalhit
User-agent: StorygizeBot
User-agent: Scrapy
User-agent: GermCrawler
User-agent: Netvibes
User-agent: Vagabondo
User-agent: Genieo
User-agent: um-IC
User-agent: Typhoeus
User-agent: newspaper
User-agent: weborama-fetcher
User-agent: ScooperBot
User-agent: BYO-2
User-agent: SimplePie
User-agent: GroupHigh
User-agent: feedparser
User-agent: UniversalFeedParser
User-agent: Pulsepoint
User-agent: Steeler
User-agent: GarlikCrawler
User-agent: Flipboard
User-agent: FlipboardRSS
User-agent: Moreover
User-agent: Mercury
User-agent: FeedBurner
User-agent: Yahoo
User-agent: omgili
User-agent: Bloglovin
User-agent: YahooCacheSystem
User-agent: mTalk
User-agent: EveryoneSocialBot
User-agent: EzineArticlesLinkScanner
User-agent: dlvr.it
User-agent: pfblogs
User-agent: Aol
User-agent: ToutiaoSpider
User-agent: Photon
User-agent: PicoFeed
User-agent: um-FC
User-agent: Digg
User-agent: Synaptive
User-agent: YisouSpider
User-agent: BYO-4
User-agent: Feedbin
User-agent: Twingly
User-agent: Tiny
User-agent: WWW-Mechanize
User-agent: CommaFeed
User-agent: TkBot
User-agent: TinEye-bot
User-agent: Exabot
User-agent: coccoc
User-agent: BLEXBot
User-agent: uMBot-LN
User-agent: QuerySeekerSpider
User-agent: Yandex
User-agent: YandexBot
User-agent: YandexMobileBot
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: sogou spider
User-agent: Sogou web spider
User-agent: Baiduspider
User-agent: YoudaoBot
User-agent: MJ12bot
User-agent: spbot
User-agent: 360Spider
User-agent: SeznamBot
User-Agent: AhrefsBot
User-Agent: Linkdex
User-agent: linkdexbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: CFNetwork
User-agent: SputnikBot
User-agent: TurnitinBot
User-agent: proximic
User-agent: grapeshot
User-agent: VigLink
User-agent: Clickagy
User-agent: YaK
User-agent: DeuSu
User-agent: Taboolabot
User-agent: Pinterestbot
User-agent: psbot
User-agent: Plukkie
User-agent: GetIntent
User-agent: Facebot
User-agent: linkdexbot
User-agent: Vegi
User-agent: istellabot
User-agent: Leikibot
User-agent: Claritybot
User-agent: PiplBot
User-agent: Feedspotbot
User-agent: moatbot
User-agent: Yeti
User-agent: contxbot
User-agent: SEOkicks-Robot
User-agent: rogerbot
User-agent: Discordbot
User-agent: Tapatalk
User-agent: Mail.RU_Bot
User-agent: coccocbot
User-agent: HybridBot
User-agent: Slack-ImgProxy
User-agent: adbeat_bot
User-agent: Kraken
User-agent: MojeekBot
User-agent: calculon
User-agent: mcsbot
User-agent: avkzarabotok
User-agent: SearchmetricsBot
User-agent: Uptimebot
User-agent: yacybot
User-agent: Applebot
User-agent: Gigabot
User-agent: g2reader-bot
User-agent: Seoterritory
User-agent: AhrefsBot
User-agent: feedbot
User-agent: Findxbot
User-agent: Wotbox
User-agent: SMTBot
User-agent: Ocarinabot
User-agent: Dragonbot
User-agent: Zombiebot
User-agent: metabot.ru
User-agent: RSSMicro
User-agent: _zbot
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Cliqzbot
User-agent: DotBot
User-agent: Exabot
User-agent: moatbot
User-agent: Tapatalk
Disallow: /


User-agent: GingerCrawler
Disallow: /

User-agent: nutch
Disallow: /

User-agent: Urlbot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: ScoutJet
Disallow: /

User-agent: 008
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Speedy Spider
Disallow: /

User-agent: Accelobot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: discobot
Disallow: /

User-agent: AboutUsBot
Disallow: /

User-agent: Vocus
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: plukkie
Disallow: /

User-agent: AhrefsBot
Disallow: /


User-agent: http://ahrefs.com/robot
Disallow: /

User-agent: http://www.genieo.com/webfilter.html
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: linkdex.com
Disallow: /

User-agent: linkdex.com/v2.0
Disallow: /

User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Disallow: /

User-agent: NetcraftSurveyAgent/1.0; +info@netcraft.com
Disallow: /

User-agent: NetcraftSurveyAgent/1.0
Disallow: /

User-agent: NetcraftSurveyAgent
Disallow: /

User-agent: NetcraftSurveyAgent/2.0
Disallow: /

User-agent: SemrushBot/0.95
Disallow: /

User-agent: MJ12bot/v1.4.3
Disallow: /
User-agent: Java/1.6.0_04
Disallow: /
User-agent: JS-Kit URL Resolver, http://js-kit.com/
Disallow: /
User-agent: JS-Kit
Disallow: /
User-agent: NetShelter ContentScan
Disallow: /

User-agent: Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Disallow: /
User-agent: Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Disallow: /
User-agent: Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Disallow: /
User-agent: YYSpider
Disallow: /
User-agent: YisouSpider
Disallow: /

User-agent: PagePeeker2
Disallow: /

User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /

User-agent: Ruby
Disallow: /
User-agent: Abonti
Disallow: /
User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: Image\ Sucker
Disallow: /


User-agent: SuperBot
Disallow: /

User-agent: Web\ Image\ Collector
Disallow: /
User-agent: Web\ Sucker
Disallow: /

User-agent: UbiCrawler
Disallow: /
User-agent: zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: linko
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /


Crawl-delay: 10
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/

Popularität bei Facebook

Shares / Likes / Kommentare
0

Es werden nur die Daten zu der angegebenen URL abgefragt und nicht zu einer eventuell vorhandenen und auf der Seite verlinkten Facebook Seite.

Suchvorschau

Pferde kaufen, Pferde verkaufen und Pferde suchen - Westernpferde k...
https://www.horsebase.de/
Pferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.

Wichtigste Suchbegriffe

Folgende Keywords hat Seobility erkannt. Überprüfen Sie die Optimierung dieser Keywords für Ihre Seite.

KeywordErgebnisPrüfen
Pferdemarkt71%Check
HorseBase Pferdemarkt66%Check
Pferde63%Check
verkaufen61%Check
kaufen61%Check
Kauf und Verkauf61%Check
Pferde kaufen58%Check
Pferde verkaufen58%Check
verkaufen und Pferde57%Check
HorseBase54%Check

Analysieren Sie jetzt kostenlos bis zu 1.000 Unterseiten von horsebase.de!

Kostenlos registrieren
Die Nutzung des Basis Accounts ist zeitlich unbegrenzt möglich