Horsebase.de - SEO Checker

Overview of the SEO Check
Meta information
80% 
Page quality
85% 
Page structure
35% 
Link structure
25% 
Server
79% 
External factors
100% 
SEO Score
Response time
0.65 s
File size
59.80 kB
Words
1297
Media files
39
Number of links
137 internal / 3 external

Task list of SEO Improvements

Meta specifications

Title
(Critically important)
Pferde kaufen, Pferde verkaufen und Pferde suchen - Pferdezubehör - HorseBase Pferdemarkt
The page title should be shorter than 580 pixels. It is 851 pixels long. Optimize title
There are word repetitions in the page title.
Meta description
(Critically important)
Pferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.
The meta description is too long: 1698 pixels from max. 1000 pixels. Optimize description
Crawlability
(Critically important)
There are no problems in accessing the website.
Canonical URL
(Important)
No canonical link is specified.
Language
(Somewhat important)
Language detected in text: de
Language defined in HTML: de-de
Server location: Finland
The following language is defined by HTML: de-de
Alternate/Hreflang Links
(Somewhat important)
There are no alternate links specified on this page.
Other meta tags
(Somewhat important)
There is no rel next meta tag on this page.
There is no rel prev meta tag on this page.
Domain
(Somewhat important)
The domain is no subdomain.
The domain length is good.
The domain does not contain non-latin characters.
Page URL
(Somewhat important)
No parameters were found in the URL.
No session ID was found in the URL.
The URL does not have too many subdirectories.
Charset encoding
(Somewhat important)
The charset encoding (UTF-8) is set correctly.
Doctype
(Nice to have)
The doctype XHTML 1.0 Transitional is set correctly.
The doctype is placed at first in the HTML code.
Favicon
(Nice to have)
The favicon is linked correctly.

Meta tags

NameValue
robotsindex, follow
keywordsPferde kaufen, Pferde verkaufen, Pferde, Pferdemarkt, Pferdeangebote, Pferdeanzeigen, Pferdebörse, Pferdeboerse, Pferdeinserate, Pferdeverkauf, Pferdekauf, Pferde inserieren, Pferde kaufen, Pferdehandel, Pferd kaufen, Pferdekauf, Deckhengste, Deckanzeigen
titlePferdemarkt
descriptionPferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.
generatorHorseBase CMS
langde-de
content-typetext/html; charset=utf-8

Test up to 1.000 webpages of horsebase.de with our free plan!

Try for free
No trial. It's just free!

Page quality

Content
(Critically important)
This page contains 1297 words. That's ok.
30.3% of the text are stop words.
Keywords used in the page title are also used in the page content. That's good!
The page contains a listing, which indicates a good text layout.
29 paragraphs were found on this page.
The text content is perfect.
No placeholders texts or images were found.
There are no duplicates on the site.
The average number of words per sentence of 13.02 words is good.
Frames
(Critically important)
This page does not use a frameset.
Mobile optimization
(Somewhat important)
No viewpoint tag is provided.
This page loads 6 JavaScript files. This may affect the load time negatively.
At least one Apple touch icon is specified.
Bold and strong tags
(Somewhat important)
The usage of strong and bold tags is perfect. We recommend the use of up to 26 tags for this page.
Image SEO
(Somewhat important)
Alt text (alternative text) is correctly used on all found images.
Social Networks
(Nice to have)
There are only a few social sharing widgets on the page. Make your website popular in social networks with social sharing widgets.
Additional markup
(Nice to have)
No additional page markup was found.
HTTPS
(Somewhat important)
This website uses HTTPS to protect privacy and integrity of the exchanged data.
All included files are also transferred via HTTPS.

Media list

URLAlt attributeTitle
...mages/cads/147271/130x100/0000443831.jpgBarefoot Sattel London Größe2 mit Zubehör
...mages/cads/107088/130x100/0000304258.jpgbewegungsstarker wunderschöner Reitponyhengst von FS Dont Worry
...mages/cads/146483/130x100/0000440439.jpgLieber, unverbrauchter Fuchswallach zu verkaufen
...mages/cads/156898/130x100/0000484717.jpgBayrische Warmblutstite zu verkaufen
...mages/cads/145574/130x100/0000436770.jpgSchöner dunkelfuchs Pony-Wallach, lieb, verschmust, 3 Jahre
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
...m/templates/waybeyond/images/indent1.pngPferde kaufen
i2.ytimg.com/vi/-wF2e1a8wjM/default.jpgInserat Nr. 162524
i2.ytimg.com/vi/6KSKSNIBWUk/default.jpgInserat Nr. 162516
....com/media/system/images/cads/gratis.jpgGratis inserieren
....horsebase.com/images/cads/kein_bild.pngKein Bild
....horsebase.com/images/cads/kein_bild.pngKein Bild
...mages/cads/141947/130x100/0000423039.jpgTop Longhorn Allround Westernsattel
....horsebase.com/images/cads/kein_bild.pngKein Bild
....horsebase.com/images/cads/kein_bild.pngKein Bild
....horsebase.com/images/cads/kein_bild.pngKein Bild
...mages/cads/153706/130x100/0000470964.jpgVerkaufe Halfter aus einer Colection
...mages/cads/153703/130x100/0000470954.jpg7 Jährige Hannoveraner Stute sucht erfahrenen Reiter
...mages/cads/143776/130x100/0000429857.jpgEuroriding Jade GP, 32 Kammer
...mages/cads/141948/130x100/0000423042.jpgTop Tom Winter 56 Sattel
...mages/cads/139846/130x100/0000414674.jpgMarathonwagen 1 und 2 spännig
...mages/cads/137944/130x100/0000406891.jpgDressursattel Spirig St. Gallen für Großpferd und KleinpferdPony
...mages/cads/133518/130x100/0000390605.jpgDressursattel Kieffer Wien DL Professionell
...mages/cads/124694/130x100/0000360440.jpgVictoria kutschen für Hochzeiten und Feiern
...mages/cads/155464/130x100/0000478577.jpgWunderschöne Nachwuchs-Dressurstute aus eigener Zucht
...mages/cads/143777/130x100/0000429861.jpgSommer Lancelot DS, 32 Kammerweite
...mages/cads/124676/130x100/0000360669.jpgZukünftiger Stern am Dressurhimmel
...mages/cads/124667/130x100/0000360365.jpgdiverse Reitartikel NEUWERTIG zu verkaufen
...mages/cads/117137/130x100/0000335382.jpgwunderschönes Reitpferd und Zuchtstute zu verkaufen
...mages/cads/117136/130x100/0000335380.jpgerfolgreiche Zuchtstute macht sehr schöne Fohlen

Page structure

H1 heading
(Critically important)
Pferdemarkt
The H1 heading consists of only one word. There should be more information given.
The H1 heading is too short (11 characters). It should be at least 20 Characters long.
Headings
(Important)
Some headings occur twice on the page.
The structure of headings is missing one or more levels. Do not skip heading levels.
There are 49 headings on the page. The amount of headings should be in a more proper relation to the amount of text.

Heading structure

Heading levelContent
H1 Pferdemarkt
H2 Pferdezubehör
H2 Pferde online kaufen
H2 Pferde online verkaufen
H2 Verkaufe mehrere Reitstiefel in versch. Größen
H2 Verkaufe mehrere Bandagen in verschiedenen Farben.
H2 Top Longhorn Allround Westernsattel
H2 Verkaufe Hannoveraner Stute mit Springportenzial
H2 Verkaufe Bandagen in 4er Set
H2 Verkaufe Halfter in Braun
H2 Verkaufe Halfter aus einer Colection
H2 7 Jährige Hannoveraner Stute sucht erfahrenen Reiter
H2 Euroriding Jade GP, 32 Kammer
H2 Top Tom Winter 56 Sattel
H2 Marathonwagen 1 und 2 spännig
H2 Dressursattel Spirig St. Gallen für Großpferd und KleinpferdPony
H2 Dressursattel Kieffer Wien DL Professionell
H2 Victoria kutschen für Hochzeiten und Feiern
H2 Wunderschöne Nachwuchs-Dressurstute aus eigener Zucht
H2 Sommer Lancelot DS, 32 Kammerweite
H2 Zukünftiger Stern am Dressurhimmel
H2 diverse Reitartikel NEUWERTIG zu verkaufen
H2 wunderschönes Reitpferd und Zuchtstute zu verkaufen
H2 erfolgreiche Zuchtstute macht sehr schöne Fohlen
H3 Direkt zu
H3 Unsere neuesten Videos
H3 HorseBase Pferdemarkt
H3 Bekleidung
H3 Pferdezubehör Duplicate text
H3 Sättel, Longhorn, Western
H3 Pferde, Hannoveraner, Großpferd, Dressur, Schulpferd, Springen
H3 Pferdezubehör Duplicate text
H3 Pferdezubehör Duplicate text
H3 Pferdezubehör Duplicate text
H3 Pferde, Hannoveraner, Großpferd, Dressur, Freizeit, Springen
H3 Sättel, Sonstige Sattelmarken, Vielseitigkeit
H3 Sättel, Sonstige Sattelmarken, Western
H3 Anhänger und Kutschen
H3 Sättel, Sonstige Sattelmarken, Dressur
H3 Sättel, Kieffer, Dressur
H3 Anhänger und Kutschen Duplicate text
H3 Pferde, Hannoveraner, Sportpferd, Dressur, Zucht
H3 Sättel, Sommer, Vielseitigkeit
H3 Pferde, Deutsches Sportpferd, Großpferd, Jährling, Sportpferd, Dressur
H3 Pferdezubehör Duplicate text
H3 Pferde, Westfale, Großpferd, Freizeit
H3 Pferde, -- keine Angabe --, Großpferd, Zucht
H6 Achtung beim Pferdekauf und -verkauf
H6 Die 20 neuesten Inserate im Pferdemarkt
Baselink:
Some internal links have dynamic parameters. All internal URLs, which are not marked as nofollow, should not contain dynamic parameters.
Some anchor texts are used more than once.
The number of internal links is ok.
None of the anchor texts is too long.
There are 3 external links on this page.
LinkAttributesAnchor text
/anmelden/reset.htmlA-TITLE Ich habe mein Passwort vergessen.
/anmelden/remind.htmlA-TITLE Ich habe meinen Benutzernamen vergessen.
/anmelden/mitglied-werden.htmlA-TITLE Mitglied werden
https://www.horsebase.de/Start
https://www.horsebase.de/Pferdemarkt
/pferde.htmlPferde
/alle-pferde.html?cf=1Alle Pferde
/freizeitpferde.html?cf=1Freizeitpferde
/springpferde.html?cf=1Springpferde
/dressurpferde.html?cf=1Dressurpferde
/westernpferde.html?cf=1Westernpferde
/barockpferde.html?cf=1Barockpferde
/gangpferde.html?cf=1Gangpferde
/vielseitigkeitspferde.html?cf=1Vielseitigkeitspferde
/sportpferde.html?cf=1Sportpferde
/fohlen.html?cf=1Fohlen
/ponies.html?cf=1Ponies
/deckhengste.htmlDeckhengste
/sattelmarkt.htmlSattelmarkt
/saettel.htmlSättel
/sattelzubehoer.htmlSattelzubehör
/reitbekleidung.htmlReitbekleidung
/pferdezubehoer.htmlPferdezubehör
/anhaenger-und-kutschen.htmlAnhänger und Kutschen
/reiterferien.htmlReiterferien
/boxen-und-weideplaetze.htmlBoxen und Weideplätze
/immobilien.htmlImmobilien
/gesuche.html?cf=1Gesuche/Tausch
/dienstleistungen.html?cf=1Dienstleistungen
/anzeige-aufgeben.htmlAnzeige aufgeben
/merkzettel/merkzettel.htmlMerkzettel
/saettel/147271/barefoot-satte...IMG-ALT Barefoot Sattel London Größe2 mit Zubehör
/saettel/147271/barefoot-satte...Text duplicate Barefoot Sattel London Größe2 mit Zubehör
/pferde/107088/bewegungsstarke...IMG-ALT bewegungsstarker wunderschöner Reitponyhengst von FS Dont Worry
/pferde/107088/bewegungsstarke...Text duplicate bewegungsstarker wunderschöner Reitponyhengst von FS Dont Worry
/pferde/146483/lieber-unverbra...IMG-ALT Lieber, unverbrauchter Fuchswallach zu verkaufen
/pferde/146483/lieber-unverbra...Text duplicate Lieber, unverbrauchter Fuchswallach zu verkaufen
/pferde/156898/bayrische-warmb...IMG-ALT Bayrische Warmblutstite zu verkaufen
/pferde/156898/bayrische-warmb...Text duplicate Bayrische Warmblutstite zu verkaufen
/pferde/145574/schoener-dunkel...IMG-ALT Schöner dunkelfuchs Pony-Wallach, lieb, verschmust, 3 Jahre
/pferde/145574/schoener-dunkel...Text duplicate Schöner dunkelfuchs Pony-Wallach, lieb, verschmust, 3 Jahre
/alle-pferde.html?cf=1Text duplicate Alle Pferde
/freizeitpferde.html?cf=1Text duplicate Freizeitpferde
/springpferde.html?cf=1Text duplicate Springpferde
/dressurpferde.html?cf=1Text duplicate Dressurpferde
/westernpferde.html?cf=1Text duplicate Westernpferde
/barockpferde.html?cf=1Text duplicate Barockpferde
/gangpferde.html?cf=1Text duplicate Gangpferde
/vielseitigkeitspferde.html?cf=1Text duplicate Vielseitigkeitspferde
/sportpferde.html?cf=1Text duplicate Sportpferde
/ponies.html?cf=1Text duplicate Ponies
/fohlen.html?cf=1Text duplicate Fohlen
/deckhengste.htmlText duplicate Deckhengste
/saettel.htmlText duplicate Sättel
/gesuche.html?cf=1Text duplicate Gesuche/Tausch
https://www.youtube.com/v/-wF2...New window External IMG-ALT Inserat Nr. 162524
A-TITLE Inserat Nr. 162524
/pferde/162524/grosses-jaehrli...Zum Inserat
https://www.youtube.com/v/6KSK...New window External IMG-ALT Inserat Nr. 162516
A-TITLE Inserat Nr. 162516
/pferde/162516/reduziert-dress...Text duplicate Zum Inserat
/achtung-betrueger.htmlNew window Betrugswarnungen
A-TITLE Betrugswarnung
/warum-sie-auf-horsebase-inser...New window Warum Sie auf HorseBase inserieren sollten.
A-TITLE Warum Sie auf HorseBase inserieren sollten
/anzeige-aufgeben.htmlPferde zum Verkauf anbieten
/pferde-suchen.htmlPferde suchen
/pferdemarkt.htmlHorseBase-Pferdemarkt
/achtung-betrueger.htmlsicheren
/pferde-kaufen.htmlKauf
/pferde-verkaufen.htmlVerkauf
/pferdezubehoer.htmlPferdezubehörs
/saettel.htmlText duplicate Sättel
/reitbekleidung.htmlText duplicate Reitbekleidung
/pferdekauf.htmlPferdekauf
/pferdeverkauf.htmlPferdeverkauf
/pferdemarkt.htmlText duplicate Pferdemarkt
/pferdelinksPferdelinkliste
/pferdebuecher.htmlPferdebuchshop
/anzeige-aufgeben.htmlIMG-ALT Gratis inserieren
A-TITLE Jetzt gratis inserieren
http://base.google.com/support...New window External Subdomain Javascript aktivieren
/bekleidung/153711/verkaufe-me...IMG-ALT Kein Bild
/bekleidung/153711/verkaufe-me...Verkaufe mehrere Reitstiefel in versch. Größen
/pferdezubehoer/153710/verkauf...Text duplicate IMG-ALT Kein Bild
/pferdezubehoer/153710/verkauf...Verkaufe mehrere Bandagen in verschiedenen Farben.
/saettel/141947/top-longhorn-a...IMG-ALT Top Longhorn Allround Westernsattel
/saettel/141947/top-longhorn-a...Text duplicate Top Longhorn Allround Westernsattel
/saettel/141947/top-longhorn-a...[...]
A-TITLE Zur Detailansicht
/pferde/139851/verkaufe-hannov...Text duplicate IMG-ALT Kein Bild
A-TITLE suche, gutes, zuhause
/pferde/139851/verkaufe-hannov...Verkaufe Hannoveraner Stute mit Springportenzial
A-TITLE suche, gutes, zuhause
/pferde/139851/verkaufe-hannov...Text duplicate [...]
A-TITLE Zur Detailansicht
/pferdezubehoer/153708/verkauf...Text duplicate IMG-ALT Kein Bild
/pferdezubehoer/153708/verkauf...Verkaufe Bandagen in 4er Set
/pferdezubehoer/153707/verkauf...Text duplicate IMG-ALT Kein Bild
/pferdezubehoer/153707/verkauf...Verkaufe Halfter in Braun
/pferdezubehoer/153706/verkauf...IMG-ALT Verkaufe Halfter aus einer Colection
/pferdezubehoer/153706/verkauf...Text duplicate Verkaufe Halfter aus einer Colection
/pferdezubehoer/153706/verkauf...Text duplicate [...]
A-TITLE Zur Detailansicht
/pferde/153703/7-jaehrige-hann...IMG-ALT 7 Jährige Hannoveraner Stute sucht erfahrenen Reiter
/pferde/153703/7-jaehrige-hann...Text duplicate 7 Jährige Hannoveraner Stute sucht erfahrenen Reiter
/pferde/153703/7-jaehrige-hann...Text duplicate [...]
A-TITLE Zur Detailansicht
/saettel/143776/euroriding-jad...IMG-ALT Euroriding Jade GP, 32 Kammer
/saettel/143776/euroriding-jad...Text duplicate Euroriding Jade GP, 32 Kammer
/saettel/143776/euroriding-jad...Text duplicate [...]
A-TITLE Zur Detailansicht
/saettel/141948/top-tom-winter...IMG-ALT Top Tom Winter 56 Sattel
/saettel/141948/top-tom-winter...Text duplicate Top Tom Winter 56 Sattel
/saettel/141948/top-tom-winter...Text duplicate [...]
A-TITLE Zur Detailansicht
/anhaenger-und-kutschen/139846...IMG-ALT Marathonwagen 1 und 2 spännig
/anhaenger-und-kutschen/139846...Text duplicate Marathonwagen 1 und 2 spännig
/saettel/137944/dressursattel-...IMG-ALT Dressursattel Spirig St. Gallen für Großpferd und KleinpferdPony
/saettel/137944/dressursattel-...Text duplicate Dressursattel Spirig St. Gallen für Großpferd und KleinpferdPony
/saettel/137944/dressursattel-...Text duplicate [...]
A-TITLE Zur Detailansicht
/saettel/133518/dressursattel-...IMG-ALT Dressursattel Kieffer Wien DL Professionell
/saettel/133518/dressursattel-...Text duplicate Dressursattel Kieffer Wien DL Professionell
/saettel/133518/dressursattel-...Text duplicate [...]
A-TITLE Zur Detailansicht
/anhaenger-und-kutschen/124694...IMG-ALT Victoria kutschen für Hochzeiten und Feiern
/anhaenger-und-kutschen/124694...Text duplicate Victoria kutschen für Hochzeiten und Feiern
/pferde/155464/wunderschoene-n...IMG-ALT Wunderschöne Nachwuchs-Dressurstute aus eigener Zucht
A-TITLE Stute, Hannoveraner, Fiorissimo, Feel Free
/pferde/155464/wunderschoene-n...Text duplicate Wunderschöne Nachwuchs-Dressurstute aus eigener Zucht
A-TITLE Stute, Hannoveraner, Fiorissimo, Feel Free
/pferde/155464/wunderschoene-n...Text duplicate [...]
A-TITLE Zur Detailansicht
/saettel/143777/sommer-lancelo...IMG-ALT Sommer Lancelot DS, 32 Kammerweite
/saettel/143777/sommer-lancelo...Text duplicate Sommer Lancelot DS, 32 Kammerweite
/saettel/143777/sommer-lancelo...Text duplicate [...]
A-TITLE Zur Detailansicht
/pferde/124676/zukuenftiger-st...IMG-ALT Zukünftiger Stern am Dressurhimmel
A-TITLE Paradiesvogel, Quaterback, Paradiesfalter, Sportpferd, Championatsfohlen, Ausstrahlung, Charmeur
/pferde/124676/zukuenftiger-st...Text duplicate Zukünftiger Stern am Dressurhimmel
A-TITLE Paradiesvogel, Quaterback, Paradiesfalter, Sportpferd, Championatsfohlen, Ausstrahlung, Charmeur
/pferde/124676/zukuenftiger-st...Text duplicate [...]
A-TITLE Zur Detailansicht
/pferdezubehoer/124667/diverse...IMG-ALT diverse Reitartikel NEUWERTIG zu verkaufen
/pferdezubehoer/124667/diverse...Text duplicate diverse Reitartikel NEUWERTIG zu verkaufen
/pferdezubehoer/124667/diverse...Text duplicate [...]
A-TITLE Zur Detailansicht
/pferde/117137/wunderschoenes-...IMG-ALT wunderschönes Reitpferd und Zuchtstute zu verkaufen
A-TITLE Freizeitpferd
/pferde/117137/wunderschoenes-...Text duplicate wunderschönes Reitpferd und Zuchtstute zu verkaufen
A-TITLE Freizeitpferd
/pferde/117137/wunderschoenes-...Text duplicate [...]
A-TITLE Zur Detailansicht
/pferde/117136/erfolgreiche-zu...IMG-ALT erfolgreiche Zuchtstute macht sehr schöne Fohlen
/pferde/117136/erfolgreiche-zu...Text duplicate erfolgreiche Zuchtstute macht sehr schöne Fohlen
https://www.horsebase.de/Anchor No Text
/agb-nutzungsbedingungen.htmlNutzungsbedingungen
/datenschutz.htmlDatenschutz
/regeln.htmlRegeln
/achtung-betrueger.htmlBetrugswarnung
https://www.horsebase.de/faq.htmlFAQ
/kontakt.htmlImpressum
/partner.htmlPartner
/abos.htmlAbos
/datenschutz.htmlMehr erfahren

Server configuration

HTTP redirects
(Critically important)
This page redirects to "https://www.horsebase.de/"
HTTP header
(Important)
No X-Powered HTTP header is sent.
This page uses GZip for compressed data transmission.
Performance
(Somewhat important)
The page response time of 0.65 seconds is longer than the recommended limit of 0.4 seconds. A high response time unnecessarily slows down search engine crawling and results in bad user experience as well.
This page loads 4 CSS files. This may affect the page load time negatively.
This page loads 6 JavaScript files. This may affect the load time negatively.
The file size of the HTML document is fine (60 kB).

HTTP Response Header

NameValue
servernginx
dateThu, 30 Mar 2023 08:03:58 GMT
content-typetext/html; charset=utf-8
content-length12164
connectionkeep-alive
p3pCP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
link</media/system/js/jquery.imageloader.js>; rel=preload; as=script, </media/system/css/jquery-ui-1.7.2.custom.pack.css>; rel=preload; as=style, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/141947/130x100/0000423039.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/kein_bild.png>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/153706/130x100/0000470964.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/153703/130x100/0000470954.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/143776/130x100/0000429857.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/141948/130x100/0000423042.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/139846/130x100/0000414674.jpg>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/137944/130x100/0000406891.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/133518/130x100/0000390605.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/124694/130x100/0000360440.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/155464/130x100/0000478577.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/143777/130x100/0000429861.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/124676/130x100/0000360669.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/124667/130x100/0000360365.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/117137/130x100/0000335382.jpg>; rel=preload; as=image, <https://img1.horsebase.com/images/cads/117136/130x100/0000335380.jpg>; rel=preload; as=image, </templates/waybeyond/css/cads.pack.css>; rel=preload; as=style, <https://www.google.com/jsapi?key=ABQIAAAAxGowbxTXCepwj4r7re0VAhThmdyYcW5uTnKhqD3LqtG8qxFv3hQqDhnd3pIvuLw8uAECbAnA_lmquQ>; rel=preload; as=script, </media/system/js/jquery.min.js>; rel=preload; as=script, </media/system/js/jquery-ui.min.js>; rel=preload; as=script, </media/system/css/colorbox.pack.css>; rel=preload; as=style, </media/system/js/jquery.colorbox.pack.js>; rel=preload; as=script, <https://img4.horsebase.com/images/cads/147271/130x100/0000443831.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/107088/130x100/0000304258.jpg>; rel=preload; as=image, <https://img4.horsebase.com/images/cads/146483/130x100/0000440439.jpg>; rel=preload; as=image, <https://img2.horsebase.com/images/cads/156898/130x100/0000484717.jpg>; rel=preload; as=image, <https://img3.horsebase.com/images/cads/145574/130x100/0000436770.jpg>; rel=preload; as=image, </modules/mod_yoo_login/styles/style.pack.css>; rel=preload; as=style
expiresMon, 1 Jan 2001 00:00:00 GMT
cache-controlpost-check=0, pre-check=0
pragmano-cache
set-cookie67 Characters
last-modifiedThu, 30 Mar 2023 08:03:58 GMT
varyAccept-Encoding
content-encodinggzip
x-frame-optionsSAMEORIGIN
x-xss-protection1; mode=block
x-content-type-optionsnosniff
referrer-policyno-referrer-when-downgrade
content-security-policydefault-src * data: 'unsafe-eval' 'unsafe-inline'
strict-transport-securitymax-age=31536000; includeSubDomains; preload
statuscode200
http_versionHTTP/1.1

External factors

Blacklists
(Nice to have)
This website is not classified "for adult only".
This website has excellent links from other websites.
This page has backlinks from 258 referring domains.
This page has 1,607 backlinks.
This page has backlinks from 84 different ip addresses.
Facebook popularity
(Somewhat important)
The page has 0 shares and comments on Facebook.
Listed on Webwiki
(Nice to have)
This website is listed on Webwiki.

Links from Wikipedia

No links from Wikipedia were found.

Robots.txt

User-agent: WebReaper
User-agent: AlphaSeoBot
User-agent: AlphaSeoBot-SA
User-agent: WebCopier
User-agent: Offline Explorer
User-agent: HTTrack
User-agent: Microsoft.URL.Control
User-agent: EmailCollector
User-agent: penthesilea
User-agent: MJ12bot
User-Agent: Topsy
User-agent: spbot
User-agent: dotbot
User-agent: SeznamBot
Disallow: /

User-agent: OrangeBot
Disallow: /


User-agent: Slurp
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sogou Orion spider
Disallow: /
User-agent: Sogou News Spider
Disallow: /
User-agent: Sogou blog
Disallow: /
User-agent: Sogou spider2
Disallow: /
User-agent: Sogou inst spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-Agent: Linkdexbot/2.0
Disallow: /
User-Agent: Linkdexbot/2.1
Disallow: /
User-Agent: Linkdexbot
Disallow: /
User-agent: Sogou web spider Sogou web spider/4.0
Disallow: /
User-agent: sogou web spider
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: Opoint
Disallow: /
User-agent: 008
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 5.0 Robot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot)
Disallow: /
User-agent: ClarityDailyBot
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; ClarityDailyBot)
Disallow: /

User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot)
Disallow: /
User-agent: PAD-bot/9.0
Disallow: /
User-agent: PAD-bot
Disallow: /
User-agent: Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/)
Disallow: /
User-agent: SoftListBot/2.2
Disallow: /
User-agent: SoftListBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: NativeHost
Disallow: /
User-agent: newsme/1.0; [email protected]
Disallow: /
User-agent: newsme/1.0
Disallow: /
User-agent: newsme
Disallow: /
User-agent: Niki-Bot
Disallow: /
User-agent: NING/1.0
Disallow: /
User-agent: NING
Disallow: /

User-agent: peerindex
Disallow: /
User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
Disallow: /
User-agent: Diffbot/0.1
Disallow: /
User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot)
Disallow: /
User-agent: elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net)
Disallow: /
User-agent: elefent
Disallow: /
User-agent: elefent/1.2
Disallow: /
User-agent: crawler.elefent.net
Disallow: /
User-agent: MS Search 4.0 Robot
Disallow: /
User-agent: MS Search 4.0
Disallow: /
User-agent: MS Search 5.0
Disallow: /
User-agent: MS Search 6.0
Disallow: /
User-agent: MS Search 7.0
Disallow: /
User-agent: MS Search
Disallow: /


User-agent: CCBot/2.0 (http://commoncrawl.org/faq/)
Disallow: /
User-agent: CCBot/2.0
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
Disallow: /
User-agent: rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])
Disallow: /
User-agent: Mail.RU_Bot/Robots/2.0
Disallow: /
User-agent: AOL_Daniel_Clark_Spider/Nutch-0.9 (AOL Search; [email protected])
Disallow: /
User-agent: AolOnScraper/1.0
Disallow: /


User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html; gzip)
Disallow: /

User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html)
Disallow: /

User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /



User-agent: Apache Sites Search Facet
Disallow: /
User-agent: bigdata
Disallow: /

User-agent: Fast corporate crawler
Disallow: /
User-agent: 008
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: magpie-crawler/1.1
Disallow: /
User-agent: MagpieRSS
Disallow: /
User-agent: psnsearch
Disallow: /
User-agent: SLCC2
Disallow: /
User-agent: Aqua_Products
Disallow: /
User-agent: asterias
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow: /
User-agent: Black Hole
Disallow: /
User-agent: BlowFish/1.0
Disallow: /
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bullseye/1.0
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: GuzzleHttp
Disallow: /
User-agent: Go 1.1 package http
Disallow: /
User-agent: Go 1.1
Disallow: /
User-agent: Go 1.2
Disallow: /
User-agent: Go 1.2 package http
Disallow: /
User-agent: gocrawl v0.4
Disallow: /
User-agent: gocrawl
Disallow: /
User-agent: Google-HTTP-Java-Client
Disallow: /
User-agent: Cegbfeieh
Disallow: /
User-agent: Hatena Star
Disallow: /
User-agent: PagePeeker.com
Disallow: /
User-agent: PagePeeker
Disallow: /
User-agent: _agentname
Disallow: /
User-agent: agentname
Disallow: /
User-agent: 1 in Business Networking, www.MarcusWalker.ws
Disallow: /
User-agent: Business Networking
Disallow: /
User-agent: AntiOpensourceBot/1.1
Disallow: /
User-agent: AntiOpensourceBot
Disallow: /
User-agent: antibot-V1.1.5/i586-linux-2.2
Disallow: /
User-agent: Anti-XBOCT/2007 [xxx] (OS; X; JU)
Disallow: /
User-agent: AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected])
Disallow: /
User-agent: Anti-XBOCT
Disallow: /
User-agent: AntBot
Disallow: /
User-agent: Ant-Nutch
Disallow: /
User-agent: AntBot/1.0
Disallow: /
User-agent: AnzwersCrawl/2.0
Disallow: /

User-agent: Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Disallow: /

User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /

User-agent: Wotbox/2.01 (+http://www.wotbox.com/bot/)
Disallow: /

User-agent: Wotbox
Disallow: /

User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: CherryPicker /1.0
Disallow: /
User-agent: CherryPickerElite/1.0
Disallow: /
User-Agent: litefinder
Disallow: /
User-agent: Friendica
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Flamingo
Disallow: /
User-agent: Faraday v0.8.8
Disallow: /
User-agent: Faraday
Disallow: /
User-agent: Barkrowler 0.1.6
Disallow: /
User-agent: Barkrowler
Disallow: /
User-Agent: ExperiBot_v1
Disallow: /
User-Agent: ExperiBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-Agent: omgilibot
Disallow: /
User-Agent: omgili
Disallow: /
User-agent: CherryPickerSE/1.0
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: Cliqzbot
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: FairAd Client
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow: /
User-agent: Gaisbot
Disallow: /
User-agent: GetRight/4.2
Disallow: /
User-agent: grub
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: Harvest/1.5
Disallow: /
User-agent: hloader
Disallow: /
User-agent: ConveraCrawler/0.9e
Disallow: /
User-agent: ConveraCrawler
Disallow: /
User-agent: YahooFeedSeeker/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://publisher.yahoo.com/rssguide)
Disallow: /
User-agent: YahooFeedSeeker/2.0+
Disallow: /
User-agent: YahooFeedSeeker
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: ia_archiver/1.6
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: Iron33/1.0.2
Disallow: /
User-agent: JennyBot
Disallow: /


User-agent: Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)
Disallow: /

User-agent: ExtLinksBot/1.5
Disallow: /

User-agent: ExtLinksBot
Disallow: /

User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /

User-agent: GrapeshotCrawler/2.0
Disallow: /

User-agent: GrapeshotCrawler/2.0
Disallow: /
User-agent: Kenjin Spider
Disallow: /
User-agent: Keyword Density/0.9
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb/clsHTTP
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: LinkScan/8.1a Unix
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LNSpiderguy
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp-trivial/1.34
Disallow: /
User-agent: Mata Hari
Disallow: /
User-agent: Microsoft URL Control
Disallow: /
User-agent: Microsoft URL Control - 5.01.4511
Disallow: /
User-agent: Microsoft URL Control - 6.00.8169
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: MIIxpc/4.2
Disallow: /
User-agent: Mister PiX
Disallow: /
User-agent: moget
Disallow: /
User-agent: moget/2.1
Disallow: /

User-agent: NCBot
Disallow: /
User-agent: Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html)
Disallow: /
User-agent: Testbot/1.0
Disallow: /
User-agent: Testbot
Disallow: /
User-agent: Bot !Susie
Disallow: /
User-agent: Slurp Disallow: /
User-agent: !Susie
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /


User-agent: Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/)
Disallow: /

User-agent: GroupHigh/1.0
Disallow: /


User-agent: G-i-g-a-b-o-t
Disallow: /

User-agent: SeznamBot/3.2
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
Disallow: /

User-agent: Helloo/1.0.4
Disallow: /

User-agent: Mozilla/5.0 (compatible; Helloo/1.0.4; +http://www.helloo.io/;)
Disallow: /

User-agent: SiteExplorer/1.1b
Disallow: /

User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Disallow: /



User-agent: Mail.RU_Bot/Robots
Disallow: /


User-agent: attach
Disallow: /
User-agent: ASPSeek
Disallow: /
User-agent: appie
Disallow: /
User-agent: AbachoBOT
Disallow: /
User-agent: autoemailspider
Disallow: /
User-agent: anarchie
Disallow: /
User-agent: antibot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: B2w
Disallow: /
User-agent: BackWeb
Disallow: /
User-agent: BackDoorBot
Disallow: /
User-agent: Bandit
Disallow: /
User-agent: BatchFTP
Disallow: /
User-agent: Black\ Hole
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: BlowFish
Disallow: /
User-agent: BuiltBotTough
Disallow: /
User-agent: Bot\ mailto
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: Buddy
Disallow: /
User-agent: Bullseye
Disallow: /
User-agent: bumblebee
Disallow: /
User-agent: BunnySlippers
Disallow: /
User-agent: ClariaBot
Disallow: /
User-agent: curl
Disallow: /
User-agent: clsHTTP
Disallow: /
User-agent: ChinaClaw
Disallow: /
User-agent: CheeseBot
Disallow: /
User-agent: CherryPicker
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: CherryPickerSE
Disallow: /
User-agent: CherryPickerElite
Disallow: /
User-agent: Collector
Disallow: /
User-agent: COAST\ WebMaster
Disallow: /
User-agent: cosmos
Disallow: /
User-agent: CopyRightCheck
Disallow: /
User-agent: ColdFusion
Disallow: /
User-agent: Copier
Disallow: /
User-agent: Crescent
Disallow: /
User-agent: DA
Disallow: /
User-agent: DTS\ Agent
Disallow: /
User-agent: DISCo\ Pump
Disallow: /
User-agent: DittoSpyder
Disallow: /
User-agent: Diamond
Disallow: /
User-agent: Download\ Demon
Disallow: /
User-agent: Download\ Wonder
Disallow: /
User-agent: Downloader
Disallow: /
User-agent: dloader
Disallow: /
User-agent: Drip
Disallow: /
User-agent: eCatch
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: Express\ WebPictures
Disallow: /
User-agent: Extreme\ Picture\ Finder
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EasyDL
Disallow: /
User-agent: EirGrabber
Disallow: /
User-agent: EroCrawler
Disallow: /
User-agent: ExtractorPro
Disallow: /
User-agent: EyeNetIE
Disallow: /
User-agent: FAST\ WebCrawler
Disallow: /


User-agent: masscan/1.0 (https://github.com/robertdavidgraham/masscan)
Disallow: /

User-agent: masscan/1.0
Disallow: /

User-agent: masscan
Disallow: /

User-agent: semantic-visions.com crawler
Disallow: /





User-agent: MJ12bot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: ADmantX Platform Semantic Analyzer
Disallow: /

User-agent: proximic
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: DeuSu
Disallow: /

User-agent: SMTBot
Disallow: /

User-agent: GetintentCrawler
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: XoviBot
Disallow: /

User-agent: zgrab
Disallow: /

User-agent: Pigafetta
Disallow: /

User-agent: plukkie
Disallow: /

User-agent: postano
Disallow: /

User-agent: Prlog
Disallow: /

User-agent: PulseCrawler/1.1
Disallow: /

User-agent: QuerySeekerSpider
Disallow: /

User-agent: quipu
Disallow: /

User-agent: Raven
Disallow: /

User-agent: RavenCrawler
Disallow: /

User-agent: RBot
Disallow: /

User-agent: Recorded Future
Disallow: /

User-agent: Riddler
Disallow: /

User-agent: Ruby
Disallow: /

User-agent: RukiCrawler
Disallow: /

User-agent: SafeDNSBot
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: Searchie
Disallow: /

User-agent: semantic-visions.com crawler
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: seocharger-robot
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SEOlyticsCrawler
Disallow: /

User-agent: seoscanners
Disallow: /

User-agent: seoscanners.net
Disallow: /

User-agent: ShowyouBot (http://showyou.com/crawler)
Disallow: /

User-agent: simplecrawler
Disallow: /

User-agent: SISTRIX Crawler
Disallow: /

User-agent: SiteExplorer
Disallow: /

User-Agent: siteluxbot
Disallow: /

User-agent: SmeshBot
Disallow: /

User-agent: SMTBot
Disallow: /

User-agent: srmse/Nutch
Disallow: /

User-agent: SoftListBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: squirrly
Disallow: /

User-agent: ssearch_bot
Disallow: /

User-agent: Stratagems Kumo
Disallow: /

User-agent: tbot-nutch
Disallow: /

User-agent: TWMBot
Disallow: /

User-agent: Typhoeus - https://github.com/typhoeus/typhoeus
Disallow: /

User-agent: uipbot
Disallow: /

User-agent: uMBot-LN
Disallow: /

User-agent: UptimeRobot/2.0
Disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: WebIndex
Disallow: /

User-agent: VeBot
Disallow: /

User-agent: VeriCiteCrawler
Disallow: /

User-agent: WeSEE_Bot
Disallow: /

User-agent: WiseGuys Robot
Disallow: /

User-agent: voltron
Disallow: /

User-agent: woobot
Disallow: /

User-agent: woriobot
Disallow: /

User-agent: wotbox
Disallow: /

User-agent: X-CAD-SE
Disallow: /

User-agent: XoviBot
Disallow: /

User-agent: yacybot
Disallow: /

User-agent: Yahoo Pipes 1.0
Disallow: /

User-agent: YioopBot
Disallow: /

User-agent: crawler.vs.uni-duisburg-essen.de
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: FileHound
Disallow: /
User-agent: Fetch\ API\ Request
Disallow: /
User-agent: FlashGet
Disallow: /
User-agent: FlickBot
Disallow: /
User-agent: FrontPage
Disallow: /
User-agent: FreeFind.com
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: GetSmart
Disallow: /
User-agent: Generic
Disallow: /
User-agent: Go!Zilla
Disallow: /
User-agent: Go-Ahead-Got-It
Disallow: /
User-agent: gotit
Disallow: /
User-agent: Grabber
Disallow: /
User-agent: GrabNet
Disallow: /
User-agent: Grafula
Disallow: /
User-agent: Gulliver
Disallow: /
User-agent: Harvest
Disallow: /
User-agent: HMView
Disallow: /
User-agent: Heretrix
Disallow: /
User-agent: HitboxDoctor
Disallow: /
User-agent: HTTPapp
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: HTTPTrack
Disallow: /
User-agent: HTTPviewer
Disallow: /
User-agent: httplib
Disallow: /
User-agent: httpfetcher
Disallow: /
User-agent: httpscraper
Disallow: /
User-agent: hloader
Disallow: /
User-agent: humanlinks
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: InterGET
Disallow: /
User-agent: Internet\ Ninja
Disallow: /
User-agent: InfoNaviRobot
Disallow: /
User-agent: InternetSeer.com
Disallow: /
User-agent: Iria
Disallow: /
User-agent: IRLbot
Disallow: /
User-agent: JetCar
Disallow: /
User-agent: JOC
Disallow: /
User-agent: JOC\ Web\ Spider
Disallow: /
User-agent: JoBo
Disallow: /
User-agent: Java
Disallow: /
User-agent: JustView
Disallow: /
User-agent: Jonzilla
Disallow: /
User-agent: JennyBot
Disallow: /
User-agent: Kenjin\ Spider
Disallow: /
User-agent: Keyword\ Density
Disallow: /
User-agent: larbin
Disallow: /
User-agent: LeechFTP
Disallow: /
User-agent: Lachesis
Disallow: /
User-agent: LexiBot
Disallow: /
User-agent: libWeb
Disallow: /
User-agent: Libby_
Disallow: /
User-agent: LinkScan
Disallow: /
User-agent: LinkWalker
Disallow: /
User-agent: LinkextractorPro
Disallow: /
User-agent: lftp
Disallow: /
User-agent: likse
Disallow: /
User-agent: Link
Disallow: /
User-agent: lwp-trivial
Disallow: /
User-agent: lwp\ request
Disallow: /
User-agent: Magnet
Disallow: /
User-agent: Mag-Net
Disallow: /
User-agent: Mass\ Downloader
Disallow: /
User-agent: MIIxpc
Disallow: /

User-agent: Missigua
Disallow: /
User-agent: Mewsoft\ Search\ Engine
Disallow: /
User-agent: moget
Disallow: /
User-agent: Mata\ Hari
Disallow: /
User-agent: Memo
Disallow: /
User-agent: Metacarta
Disallow: /
User-agent: Mercator
Disallow: /
User-agent: MIDown\ tool
Disallow: /
User-agent: MFC_Tear_Sample
Disallow: /
User-agent: Mirror
Disallow: /
User-agent: MIIxpc
Disallow: /
User-agent: Mister\ PiX
Disallow: /
User-agent: NationalDirectory\ WebSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: Nikto
Disallow: /
User-agent: Navroad
Disallow: /
User-agent: NearSite
Disallow: /
User-agent: NetAnts
Disallow: /
User-agent: NetSpider
Disallow: /
User-agent: NICErsPRO
Disallow: /
User-agent: NetResearchServer
Disallow: /
User-agent: NetMechanic
Disallow: /
User-agent: Net\ Vampire
Disallow: /
User-agent: Net\ Probe
Disallow: /
User-agent: NetZip
Disallow: /
User-agent: nexuscache
Disallow: /
User-agent: Ninja
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: our\ agent
Disallow: /
User-agent: onestop
Disallow: /
User-agent: oBot
Disallow: /
User-agent: Octopus
Disallow: /
User-agent: Offline\ Explorer
Disallow: /
User-agent: Openfind
Disallow: /
User-agent: Openfind\ data\ gatherer
Disallow: /
User-agent: PageGrabber
Disallow: /
User-agent: Papa\ Foto
Disallow: /

User-agent: Xaldon\ WebSpider
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: Xara
Disallow: /
User-agent: Y!TunnelPro
Disallow: /
User-agent: YahooYSMcm
Disallow: /
User-agent: Zade
Disallow: /
User-agent: ZBot
Disallow: /
User-agent: Zeus
Disallow: /
User-agent: Rogerbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Dotbot
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: AhrefsBot
Disallow: /

User-agent: Xaldon\ WebSpider
Disallow: /

User-agent: Screaming Frog SEO Spider/8.1
Disallow: /

User-agent: DoBot
Disallow: /

User-agent: Laserlikebot
Disallow: /
User-agent: oBot
Disallow: /

User-agent: Neliveto
Disallow: /
User-Agent: ahrefs
Disallow: /
User-Agent: AHC/1.0
Disallow: /
User-Agent: AHC
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Disallow: /

User-agent: evc-batch/2.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Disallow: /

User-agent: SemrushBot/1.2~bl
Disallow: /

User-agent: evc-batch
Disallow: /

User-agent: 007ac9 Crawler
 disallow: /

User-agent: 1 2 3 Submit PRO
 disallow: /

User-agent: 200PleaseBot
 disallow: /

User-agent: 2ADAMbot
 disallow: /

User-agent: 2ADAMbot/1.0
 disallow: /

User-agent: 360Spider
 disallow: /

user-agent: Abonti
 disallow: /

user-agent: Abonti/0.92
 disallow: /

user-agent: abot v1.0
 disallow: /

user-agent: aboutthedomain
 disallow: /

user-agent: Add Catalog
 disallow: /

user-agent: Add Catalog/2.1
 disallow: /

user-agent: AdvBot
 disallow: /

user-agent: AdvBot/2.0
 disallow: /

user-agent: AhrefsBot
 disallow: /

user-agent: Ahrefs-Bot
 disallow: /

user-agent: AhrefsBot/1.0
 disallow: /

user-agent: Ahrefs-Bot/1.0
 disallow: /

user-agent: Ahrefs-Bot/2.0
 disallow: /

user-agent: Ahrefs-Bot/3.0
 disallow: /

user-agent: Ahrefs-Bot/4.0
 disallow: /

user-agent: Ahrefs-Bot/5.0
 disallow: /

user-agent: aiHitBot
 disallow: /

user-agent: aiHitBot/2.9
 disallow: /

user-agent: Anonymous/0.0
 disallow: /

user-agent: Arachnida
 disallow: /

user-agent: Associative Spider
 disallow: /

User-agent: Baiduspider
 disallow: /

User-agent: Baidu Spider
 disallow: /

User-agent: Battleztar Bazinga
 disallow: /

User-agent: Battleztar Bazinga/0.01
 disallow: /

User-agent: BDFetch
 disallow: /

User-agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com
Disallow: /

User-agent: ZmEu
Disallow: /

User-agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)
Disallow: /


User-agent: betaBot
 disallow: /

User-agent: bieshu
 disallow: /

User-agent: Bigli SEO
 disallow: /

User-agent: Blackboard Safeassign
 disallow: /

User-agent: Blazer 1.0
 disallow: /

User-agent: BLEXBot
 disallow: /

User-agent: BLEXBot/1.0
 disallow: /

User-agent: BLP_bbot
 disallow: /

User-agent: BLP_bbot/0.1
 disallow: /

User-agent: BOIA-Accessibility-Agent/PR 1.0
 disallow: /

User-agent: BOT for JCE
 disallow: /

User-agent: BOT/0.1 (BOT for JCE)
 disallow: /

User-agent: BPImageWalker
 disallow: /

User-agent: BPImageWalker/2.0
 disallow: /

User-agent: BUbiNG
 disallow: /

User-agent: BuiBui-Bot
 disallow: /

User-agent: BuiBui-Bot/1.0
 disallow: /

User-agent: ca-crawler
 disallow: /

User-agent: ca-crawler/1.0
 disallow: /

User-agent: CakePHP
 disallow: /

User-agent: Calypso v/0.01
 disallow: /

User-agent: Calypso
 disallow: /

User-agent: CB/Nutch-1.7
 disallow: /

User-agent: CCBot
 disallow: /

User-agent: CCBot/2.0
 disallow: /

User-agent: Checkbot
 disallow: /

User-agent: checkgzipcompression.com
 disallow: /

User-agent: chushou
 disallow: /

User-agent: CloudServerMarketSpider
 disallow: /

User-agent: CloudServerMarketSpider/1.0
 disallow: /

User-agent: Clushbot/3.x-BinaryFury
 disallow: /

User-agent: CMS Crawler
 disallow: /

User-agent: CMS Crawler: http://www.cmscrawler.com
 disallow: /

User-agent: coccoc
 disallow: /

User-agent: CoinCornerBot
 disallow: /

User-agent: CoinCornerBot/1.1
 disallow: /

User-agent: Copyscape
 disallow: /

User-agent: crawler4j
 disallow: /

User-agent: CRAZYWEBCRAWLER 0.9.0
 disallow: /

User-agent: CRAZYWEBCRAWLER 0.9.1
 disallow: /

User-agent: CRAZYWEBCRAWLER 0.9.7
 disallow: /

User-agent: CrazyWebCrawler
 disallow: /

User-agent: CrazyWebCrawler-Spider
 disallow: /

User-agent: Crowsnest
 disallow: /

User-agent: Crowsnest/0.5
 disallow: /

User-agent: Curious George - www.analyticsseo.com/crawler
 disallow: /

User-agent: Curious George
 disallow: /

User-agent: cuwhois
 disallow: /

User-agent: cuwhois/1.0
 disallow: /

User-agent: dahoms
 disallow: /

User-agent: datagnionbot
 disallow: /

User-agent: DeuSu/5.0.2
 disallow: /

User-agent: Digincore
 disallow: /

User-agent: Digincore bot
 disallow: /

User-agent: Dispatch/0.11.0
 disallow: /

User-agent: Domain Re-Animator Bot
 disallow: /

User-agent: DomainAppender /1.0
 disallow: /

User-agent: DomainAppender
 disallow: /

 User-agent: DomainCrawler/3.0
 disallow: /

User-agent: DomainSigmaCrawler
 disallow: /

User-agent: DomainSigmaCrawler/0.1
 disallow: /

User-agent: Domnutch
 disallow: /

User-agent: Domnutch-Bot
 disallow: /

User-agent: Domnutch-Bot/Nutch
 disallow: /

User-agent: Domnutch-Bot/Nutch-1.0
 disallow: /

User-agent: dotbot
 disallow: /

User-agent: ECCP/1.2.1
 disallow: /

User-agent: eCommerceBot
 disallow: /

User-agent: enlle punto com/Nutch-1.9
 disallow: /

User-agent: EPiServer Link Checker
 disallow: /

User-agent: EuripBot
 disallow: /

User-agent: EuripBot/2.0
 disallow: /

User-agent: evc/2.0
 disallow: /

User-agent: evc-batch
 disallow: /

User-agent: evc-batch/2.0
 disallow: /

User-agent: Express WebPictures
 disallow: /

User-agent: Faraday v0.8.8
 disallow: /

User-agent: Faraday
 disallow: /

User-agent: Findxbot
 disallow: /

User-agent: Findxbot/1.0
 disallow: /

User-agent: Flamingo_SearchEngine
 disallow: /

User-agent: Flipboard Robot
 disallow: /

User-agent: GetProxi.es-bot
 disallow: /

User-agent: GetProxi.es-bot/1.1
 disallow: /

User-agent: GigablastOpenSource
 disallow: /

User-agent: GigablastOpenSource/1.0
 disallow: /

User-agent: Girafabot
 disallow: /

User-agent: Gluten Free Crawler
 disallow: /

User-agent: Gluten Free Crawler/1.0
 disallow: /

User-agent: GriffinBot
 disallow: /

User-agent: GrifinBot/0.01
 disallow: /

User-agent: GWPImages
 disallow: /

User-agent: GWPImages/1.0
 disallow: /

User-agent: Haiula
 disallow: /

User-agent: Haiula/1.4
 disallow: /

User-agent: HaosouSpider
 disallow: /

User-agent: Hivemind
 disallow: /

User-agent: HostHarvest
 disallow: /

User-agent: HostHarvest/0.4.28
 disallow: /

User-agent: HRCrawler
 disallow: /

User-agent: HRCrawler/2.0
 disallow: /

User-agent: http://git.io/tl_S2w
 disallow: /

User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp
 disallow: /

User-agent: HubSpot Links Crawler 1.0
 disallow: /

User-agent: HubSpot Webcrawler
 disallow: /

User-agent: HubSpot
 disallow: /

User-agent: hunchan
 disallow: /

User-agent: HyperCrawl
 disallow: /

User-agent: HyperCrawl/0.2
 disallow: /

User-agent: ICAP-IOD
 disallow: /

User-agent: ICC-Crawler
 disallow: /

User-agent: ICC-Crawler/2.0
 disallow: /

User-agent: Ichiro Robot
 disallow: /

User-agent: image.coccoc/1.0
 disallow: /

User-agent: Image2play
 disallow: /

User-agent: Image2play/0.1
 disallow: /

User-agent: Indy Library
 disallow: /

User-agent: InsightsCollector
 disallow: /

User-agent: InsightsCollector/0.1
 disallow: /

User-agent: InsightsCollector/0.1beta
 disallow: /

User-agent: integrity/5
 disallow: /

User-agent: InterNaetBoten
 disallow: /

User-agent: InterNaetBoten/0.99
 disallow: /

User-agent: Barkrowler/0.7 (+http://www.exensa.com/crawl)
 disallow: /

User-agent: Barkrowler
 disallow: /

User-agent: IRL Crawler
 disallow: /

User-agent: James BOT - WebCrawler
 disallow: /

User-agent: James BOT
 disallow: /

User-agent: JamesBOT
 disallow: /

User-agent: JetBrains 5.0
 disallow: /

User-agent: JetBrains
 disallow: /

User-agent: Kraken
 disallow: /

User-agent: Kraken/0.1
 disallow: /

User-agent: Kyoto-Tohoku-Crawler/v1
 disallow: /

User-agent: larbin
 disallow: /

User-agent: lechenie
 disallow: /

User-agent: libwww-perl
 disallow: /

User-agent: link checker
 disallow: /

User-agent: Link/1.0
 disallow: /

User-agent: linkCheck
 disallow: /

User-agent: linkCheckV3.0
 disallow: /

User-agent: Linkdex
 disallow: /

User-agent: linkdex.com/v2.0
 disallow: /

User-agent: linkdex.com/v2.1
 disallow: /

User-agent: LinkdexBot
 disallow: /

User-agent: linkdexbot/2.0
 disallow: /

User-agent: linkdexbot/2.1
 disallow: /

User-agent: linkdexbot-mobile/2.1
 disallow: /

User-agent: LinkpadBot
 disallow: /

User-agent: LinkpadBot/1.06
 disallow: /

User-agent: LinqiaScrapeBot
 disallow: /

User-agent: LinqiaScrapeBot/1.0
 disallow: /

User-agent: Lipperhey SEO Service
 disallow: /

User-agent: Lipperhey
 disallow: /

User-agent: Lipperhey-Kaus-Australis
 disallow: /

User-agent: Lipperhey-Kaus-Australis/5.0
 disallow: /

User-agent: listicka
 disallow: /

User-agent: LSSRocketCrawler
 disallow: /

User-agent: LSSRocketCrawler/1.0 LightspeedSystems
 disallow: /

User-agent: LSSRocketCrawler/1.0
 disallow: /

User-agent: ltx71
 disallow: /

User-agent: LWNutch/Nutch-1.4
 disallow: /

User-agent: Mail.RU
 disallow: /

User-agent: Mail.RU_Bot
 disallow: /

User-agent: Mail.RU_Bot/2.0
 disallow: /

User-agent: Mail.RU_Bot/Fast/2.0
 disallow: /

User-agent: md5sum
 disallow: /

User-agent: md5sum\x22
 disallow: /

User-agent: meanpathbot
 disallow: /

User-agent: MegaIndex.ru
 disallow: /

User-agent: MegaIndex.ru/2.0
 disallow: /

User-agent: mezhpozvonochnoi
 disallow: /

User-agent: Mike-Crawler
 disallow: /

User-agent: MixBot
 disallow: /

User-agent: MixrankBot
 disallow: /

User-agent: MJ12bot
 disallow: /

User-agent: Monkeybot/0.1
 disallow: /

User-agent: my crawler
 disallow: /

User-agent: My Nutch Spider/Nutch-1.9
 disallow: /

User-agent: mycrowl/Nutch-1.9
 disallow: /

User-agent: MyGreatUA/2.0
 disallow: /

User-agent: MyIPTest
 disallow: /

User-agent: NameProtect Robot
 disallow: /

User-agent: NerdyBot
 disallow: /

User-agent: Netcraft Spider
 disallow: /

User-agent: netEstate NE Crawler
 disallow: /

User-agent: NetLyzer FastProbe
 disallow: /

User-agent: NetResearchServer
 disallow: /

User-agent: NetResearchServer/4.0
 disallow: /

User-agent: Nmap Scripting Engine
 disallow: /

User-agent: node.io
 disallow: /

User-agent: node.js
 disallow: /

User-agent: Node/simplecrawler 0.5.2
 disallow: /

User-agent: Node/simplecrawler
 disallow: /

User-agent: oBot/2.3.1
 disallow: /

User-agent: omgilibot
 disallow: /

User-agent: omgilibot/0.4
 disallow: /

User-agent: Online Domain Tools - Online Website Link Checker
 disallow: /

User-agent: Online Domain Tools - Online Website Link Checker/1.2
 disallow: /

User-agent: Openfind Robot
 disallow: /

User-agent: OpenHoseBot
 disallow: /

User-agent: OpenHoseBot/2.1
 disallow: /

User-agent: Openstat
 disallow: /

User-agent: Openstat/0.1
 disallow: /

User-agent: OptimizationCrawler
 disallow: /

User-agent: OptimizationCrawler/0.2
 disallow: /

User-agent: Page Analyzer v4.0
 disallow: /

User-agent: Page Analyzer
 disallow: /

User-agent: PageAnalyzer
 disallow: /

User-agent: PageAnalyzer/1.1
 disallow: /

User-agent: PageAnalyzer/1.5
 disallow: /

User-agent: PagesInventory
 disallow: /

User-agent: Pagespeed/1.1 Fetcher
 disallow: /

User-agent: Pagespeed/1.1
 disallow: /

User-agent: Pagespeedbot
 disallow: /

User-agent: Perl LWP
 disallow: /

User-agent: PHPCrawl
 disallow: /

User-agent: phpSiteCheck 1.0
 disallow: /

User-agent: phpSiteCheck
 disallow: /

User-agent: Plukkie
 disallow: /

User-agent: POGS/2.0
 disallow: /

User-agent: Powermarks
 disallow: /

User-agent: PowerPivot
 disallow: /

User-agent: PRIVACY_ENFAQ.jsp
 disallow: /

User-agent: Prlog
 disallow: /

User-agent: Prlog/1.0
 disallow: /

User-agent: publiclibraryarchive.org
 disallow: /

User-agent: publiclibraryarchive.org/1.0
 disallow: /

User-agent: Pu_iN Crawler
 disallow: /

User-agent: Putin
 disallow: /

User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /

User-agent: Putin spider
 disallow: /

User-agent: qingdao
 disallow: /

User-agent: QlikView
 disallow: /

User-agent: quipu
 disallow: /

User-agent: quipu/1.0
 disallow: /

User-agent: quipu/2.0
 disallow: /

User-agent: R6_CommentReader
 disallow: /

User-agent: R6_FeedFetcher
 disallow: /

User-agent: Riddler
 disallow: /

User-agent: RivalSeek.com-Bot
 disallow: /

User-agent: rogerbot
 disallow: /

User-agent: rogerbot/1.0
 disallow: /

User-agent: rootlink
 disallow: /

User-agent: RU_Bot/2.0
 disallow: /

User-agent: Scopia
 disallow: /

User-agent: Scopia crawler
 disallow: /

User-agent: Scopia crawler 1.0
 disallow: /

User-agent: Scopia crawler 1.1
 disallow: /

User-agent: Scopia crawler 1.2
 disallow: /

User-agent: Scrapy
 disallow: /

User-agent: Scrapy/0.16.5
 disallow: /

User-agent: Scrapy/0.24.4
 disallow: /

User-agent: Scrapy/0.24.5
 disallow: /

User-agent: Scrapy/0.24.6
 disallow: /

User-agent: Scrapy/1.0.1
 disallow: /

User-agent: Screaming Frog SEO Spider
 disallow: /

User-agent: Screaming Frog SEO Spider/2,55
 disallow: /

User-agent: Screaming Frog SEO Spider/2.55
 disallow: /

User-agent: Screaming Frog SEO Spider/3.1
 disallow: /

User-agent: Screaming Frog SEO Spider/3.3
 disallow: /

User-agent: Screaming Frog SEO Spider/4.1
 disallow: /

User-agent: Screaming Frog SEO Spider/5.0
 disallow: /

User-agent: Screaming Frog SEO Spider/5.1
 disallow: /

User-agent: Screaming Frog SEO Spider/5.1 Beta 2
 disallow: /

User-agent: scrutiny/4
 disallow: /

User-agent: SemrushBot
 disallow: /

User-agent: SemrushBot-SA
 disallow: /

User-agent: SEOdiver/1.0
 disallow: /

User-agent: SEOkicks
 disallow: /

User-agent: SEOkicks-Robot
 disallow: /

User-agent: SEOlyticsCrawler
 disallow: /

User-agent: SEOlyticsCrawler/3.0
 disallow: /

User-agent: seoscanners
 disallow: /

User-agent: seoscanners.net/1
 disallow: /

User-agent: SEOstats 2.1.0
 disallow: /

User-agent: Seosys/Nutch-2.3
 disallow: /

User-agent: SetCronJob/1.0
 disallow: /

User-agent: R6_FeedFetcher_(www.radian6.com/crawler)
disallow: /

User-agent: SeznamBot
 disallow: /

User-agent: SheerBoredom.Experimental.Robot
 disallow: /

User-agent: SheerBoredom.Experimental.Robot/0.2
 disallow: /

User-agent: ShowyouBot
 disallow: /

User-agent: Simplecrawler
 disallow: /

User-agent: SISTRIX Crawler
 disallow: /

User-agent: sistrix
 disallow: /

User-agent: SiteBot
 disallow: /

User-agent: SiteBot/0.1
 disallow: /

User-agent: SiteExplorer
 disallow: /

User-agent: SiteExplorer/1.0
 disallow: /

User-agent: SiteExplorer/1.0b
 disallow: /

User-agent: Siteluxbot
 disallow: /

User-agent: Siteluxbot/1.0
 disallow: /

User-agent: SkimBot
 disallow: /

User-agent: RSSingBot (http://www.rssing.com)
disallow: /

User-agent: SkimBot/1.0
 disallow: /

User-agent: sky nutch crawler/Nutch-1.9
 disallow: /

User-agent: SMTBot
 disallow: /

User-agent: SMTBot/1.0
 disallow: /

User-agent: SNK Screenshot Bot
 disallow: /

User-agent: SNK Screenshot Bot/0.20
 disallow: /

User-agent: Sogou Spider
 disallow: /

User-agent: Sogou web spider
 disallow: /

User-agent: SpamBayes
 disallow: /

User-agent: SpamBayes/1.1a3+
 disallow: /

User-agent: spbot
 disallow: /

User-agent: spbot/4.4.2
 disallow: /

User-agent: spiderbot
 disallow: /

User-agent: SpiderLing
 disallow: /

User-agent: Spiderbot/Nutch-1.7
 disallow: /

User-agent: spray-can
 disallow: /

User-agent: spray-can/1.2.1
 disallow: /

User-agent: SSG/3.0
 disallow: /

User-agent: Statastico
 disallow: /

User-agent: Statastico/4.0
 disallow: /

User-agent: Steeler
 disallow: /

User-agent: Steeler/3.5
 disallow: /

User-agent: Stratagems Kumo
 disallow: /

User-agent: Stratagems
 disallow: /

User-agent: StudioFACA Search
 disallow: /

User-agent: StudioFACA
 disallow: /

User-agent: sukibot
 disallow: /

User-agent: sukibot_heritrix
 disallow: /

User-agent: sukibot_heritrix/3.1.1
 disallow: /

User-agent: SurveyBot
 disallow: /

User-agent: Synapse
 disallow: /

User-agent: Synthesio Crawler release MonaLisa
 disallow: /

User-Agent: tbot-nutch/Nutch-1.10
 disallow: /

User-Agent: Traackr.com bot
 disallow: /

User-Agent: trendictionbot
 disallow: /

User-Agent: Trendiction-Bot
 disallow: /

User-Agent: TrueBot
 disallow: /

User-Agent: TrueBot/1.0
 disallow: /

User-Agent: TulipChain/5.xx
 disallow: /

User-Agent: TWMBot/0.1
 disallow: /

User-Agent: Typhoeus
 disallow: /

User-Agent: UCMore Crawler App
 disallow: /

User-Agent: uMBot-LN
 disallow: /

User-Agent: uMBot-LN/1.0
 disallow: /

User-Agent: updown_tester
 disallow: /

User-Agent: URLChecker
 disallow: /

User-agent: V1.0/1.2
 disallow: /

User-agent: w3af.org
 disallow: /

User-agent: WASALive
 disallow: /

User-agent: WASALive-Bot
 disallow: /

User-agent: vBSEO
 disallow: /

User-agent: WBSearchBot
 disallow: /

User-agent: WBSearchBot/1.1
 disallow: /

User-agent: WeAreNotEvil
 disallow: /

User-agent: WebAlta
 disallow: /

User-agent: WebAlta Crawler
 disallow: /

User-agent: Web corpus crawler
 disallow: /

User-agent: WebCookies
 disallow: /

User-agent: WebCookies/1.0
 disallow: /

User-agent: WebCopier vx.xa
 disallow: /

User-agent: Webnest 0.9
 disallow: /

User-agent: WebQL
 disallow: /

User-agent: Webscout
 disallow: /

User-agent: Webscout/1.0
 disallow: /

User-agent: Web-sniffer
 disallow: /

User-agent: Web-sniffer/1.1.0
 disallow: /

User-agent: Webster Pro V3.4
 disallow: /

User-agent: WebTarantula.com Crawler
 disallow: /

User-agent: WeCrawlForThePeace
 disallow: /

User-agent: VegeBot
 disallow: /

 User-agent: Vegi bot
 disallow: /

User-agent: WeLikeLinks
 disallow: /

User-agent: VeriCiteCrawler
 disallow: /

User-agent: VeriCiteCrawler/Nutch-1.9
 disallow: /

User-agent: WhatWeb
 disallow: /

User-agent: WhatWeb/0.4.8-dev
 disallow: /

User-agent: Visited by http://tools.geek-tools.org
 disallow: /


User-agent: voltron
 disallow: /

User-agent: woobot
 disallow: /

User-agent: woobot/1.1
 disallow: /

User-agent: woobot/2.0
 disallow: /

User-agent: Vorboss Web Crawler
 disallow: /

User-agent: Vorboss Web Crawler/Nutch-2.3
 disallow: /

User-agent: WorldBrewBot
 disallow: /

User-agent: WorldBrewBot/2.1
 disallow: /

User-agent: worldwebheritage.org
 disallow: /

User-agent: worldwebheritage.org/1.0
 disallow: /

User-agent: wscheck.com
 disallow: /

User-agent: wscheck.com/1.0.0
 disallow: /

User-agent: www.deadlinkchecker.com
 disallow: /

User-agent: www.petitsage.fr site detector 0.4
 disallow: /

User-agent: WWW-Mechanize
 disallow: /

User-agent: WWW-Mechanize/1.74
 disallow: /

User-agent: Xenu Link Sleuth
 disallow: /

User-agent: Xenu's Link Sleuth
 disallow: /

User-agent: XoviBot
 disallow: /

User-agent: XoviBot/2.0
 disallow: /

User-agent: XSpider
 disallow: /

User-agent: Yandex Robot
 disallow: /

User-agent: Yandex
 disallow: /

User-agent: Yetibot
 disallow: /

User-agent: YisouSpider
 disallow: /

User-agent: yoozBot
 disallow: /

User-agent: yoozBot-2.2
 disallow: /

User-agent: zgrab/0.x
 disallow: /

User-agent: zzabmbot
 disallow: /

User-agent: zzabmbot/1.0
 disallow: /



User-agent: Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-2.3)
Disallow: /
User-agent: Mozilla/3.0 (Slurp/si; [email protected]; http://www.inktomi.com/slurp.html)
Disallow: /




User-agent: Gigabot/2.0
Disallow: /
User-agent: GameSpyHTTP/1.0
Disallow: /
User-agent: Baiduspider ( http://www.baidu.com/search/spider.htm)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Disallow: /

User-Agent: istellabot/t.1.13
Disallow: /
User-Agent: MagpieRSS/0.72 (+http://magpierss.sf.net)
Disallow: /
User-Agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
Disallow: /
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Disallow: /


User-Agent: Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
Disallow: /
User-Agent: Liferea
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-Agent: ahrefsbot
Disallow: /
User-Agent: Ahrefs
Disallow: /
User-Agent: Ahrefsbot
Disallow: /
User-Agent: BoardReader Blog Indexer(http://boardreader.com)
Disallow: /
User-Agent: BoardReader
Disallow: /
User-Agent: YOUBot
Disallow: /
User-Agent: AhrefsBot.com
Disallow: /
User-Agent: yandexBot
Disallow: /
User-Agent: Dow Jones Searchbot
Disallow: /
User-Agent: TurnitinBot
Disallow: /
User-Agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /
User-Agent: Almaden
Disallow: /
User-Agent: eCairn-Grabber/1.0
Disallow: /
User-Agent: eCairn-Grabber
Disallow: /


User-Agent: Amazon
Disallow: /

User-Agent: Gluten Free Crawler/1.0
Disallow: /

User-Agent: DotBot/1.1
Disallow: /

User-Agent: LinqiaCrawlerBot/1.0
Disallow: /

User-Agent: istellabot/t.1.13
Disallow: /

User-Agent: nutch-1.4/Nutch-1.4
Disallow: /

User-Agent: nutch-1.4
Disallow: /

User-Agent: DoBot
Disallow: /

User-Agent: AsapBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-Agent: amazonbot
Disallow: /


User-Agent: Amazon-bot
Disallow:/

User-Agent: Meditoolkitbot
Disallow: /


User-agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Disallow: /

User-agent: Yeti/1.1
Disallow: /

User-agent: Yeti
Disallow: /


User-agent: Yahoo! Slurp
Disallow: /

User-agent: HaoSouSpider
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: Applebot/0.1; +http://www.apple.com/go/applebot
Disallow: /


User-agent: Applebot/0.1
Disallow: /

User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
Disallow: /


User-agent: ia_archiver
Disallow: /

User-agent: Naverbot
Disallow: /

User-agent: seznambot
Disallow: /

User-agent: teoma
Disallow: /



User-agent: Yeti
Disallow: /

User-agent: EtaoSpider
Disallow: /

User-Agent: swebot
Disallow: /

User-Agent: Newsbrain
Disallow: /

User-Agent: Feedreader 3.14
Disallow: /

User-Agent: Feedreader
Disallow: /

User-Agent: Feedreader 3.14 (Powered by Newsbrain)
Disallow: /

User-agent: aihitbot
Disallow: /

User-agent: Exabot
Disallow: /


User-agent: Exabot-Thumbnails
Disallow: /

User-agent: sitebot
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: TwengaBot-Discover
Disallow: /

User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: Bender
Disallow: /

User-agent: discobot
Disallow: /

User-Agent: panscient.com
Disallow: /

User-Agent: searchwebengine.net
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: NerdByNature.Bot
disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: sistrix
Disallow: /

User-agent: SindiceBot
Disallow: /

User-agent: plukkie
Disallow: /

User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html)
Disallow: /

User-agent: SBIder
Disallow: /

User-agent: IWAgent/1.0
Disallow: /

User-agent: IWAgent
Disallow: /

User-agent: BotALot
Disallow:/

User-agent: findfiles.net
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Goodzer
Disallow: /

User-agent: DJEcoBot
Disallow: /

User-agent: DJEcoBot/1.0
Disallow: /

User-agent: YandexBot/3.0
Disallow: /

User-agent: AhrefsBot/5.2
Disallow: /

User-agent: Sogou inst spider
Disallow: /

User-agent: Sogou spider2
Disallow: /

User-agent: Sogou blog
Disallow: /

User-agent: Sogou News Spider
Disallow: /

User-agent: Sogou Orion spider
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: spray-can/1.3.4
Disallow: /

User-agent: spray-can
Disallow: /

User-agent: spray can
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: 360spider
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: BacklinkCrawler
Disallow: /

User-agent: lemurwebcrawler
Disallow: /

User-agent: gigabot
Disallow: /

User-agent: discobot Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /

User-agent: Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot)
Disallow: /

User-agent: Cliqzbot
Disallow: /


User-agent: FAST Enterprise Crawler 6
Disallow: /

User-agent: Sensis.com.au Web Crawler
Disallow: /

User-agent: Baiduspider
Disallow: /



User-agent: Teoma
Disallow: /

User-agent: XilopixBot/0.1
Disallow: /

User-agent: XilopixBot
Disallow: /

User-agent: twiceler
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Arachmo
Disallow: /

User-agent: Acoon
Disallow: /

User-agent: adbeat_bot
Disallow: /

User-agent: AgentLinkSpammer
Disallow: /

User-agent: aiHitBot
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: Scrubby
Disallow: /

User-agent: Robozilla
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: worio bot heritrix
Disallow: /

User-agent: trovitBot
Disallow: /

User-agent: MojeekBot/0.6
Disallow: /

User-agent: rogerbot/1.0
Disallow: /

User-agent: istellabot/t.1.13
Disallow: /

User-agent: SeznamBot/3.2
Disallow: /



User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: careerbot
Disallow: /

User-agent: SogouBot
Disallow: /

User-agent: Sogou spider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: Sogouspider
Disallow: /

User-agent: Qihoo360Bot
Disallow: /

User-agent: EasouSpider
Disallow: /

User-agent: COMODOSpider
Disallow: /

User-agent: findlinks
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: KaloogaBot
Disallow: /

User-agent: Mail.RU_Bot
Disallow: /

User-agent: Mail.RU
Disallow: /

User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Disallow: /

User-agent: SurdotlyBot/1.0
Disallow: /

User-agent: SurdotlyBot
Disallow: /

User-agent: Apexoo Spider (http://www.apexoo.com/spider/)
Disallow: /


User-agent: Apexoo Spider 1.0
Disallow: /

User-agent: Apexoo Spider 1.1
Disallow: /

User-agent: AportCatalogRobot/2.0
Disallow: /

User-agent: Apostrophe/0.2.1.93 (compatible) filecache
Disallow: /

User-agent: Apostrophe
Disallow: /

User-agent: Aplix HTTP/1.0.1
Disallow: /

User-agent: moget
Disallow: /
User-agent: Ichiro
Disallow: /
User-agent: Ichiro 3.0
Disallow: /
User-agent: NaverBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: NetcraftSurveyAgent/1.0
Disallow: /
User-agent: OpenWebIndex/Nutch-1.6
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: panoptaStudyBot
Disallow: /
User-agent: checks.panopta.com
Disallow: /
User-agent: psbot
Disallow: /
User-agent: plukkie
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: SeznamBot/1.0
Disallow: /
User-agent: SeznamBot/1.1
Disallow: /
#SeznamBot/3.0
User-agent: SeznamBot/3.0
Disallow: /
User-agent: SistrixCrawler
Disallow: /
User-agent: Sistrix
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: SISTRIX
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: Sogou web spider
Disallow: /
User-agent: Sosospider+
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: Sosospider/2.0
Disallow: /
User-agent: 360Spider
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: Wada.vn
Disallow: /
User-agent: Wada.vn Vietnamese Search
Disallow: /
User-agent: Wada.vn Vietnamese Search/2.1
Disallow: /
User-agent: Yandex
Disallow: /
User-agent: YandexBot/3.0-MirrorDetector
Disallow: /
User-agent: YandexImages
Disallow: /
User-agent: YandexSomething
Disallow: /
User-agent: Yandex.com
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: YoudaoBot/1.0
Disallow: /
User-agent: Zao
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs
Disallow: /
User-agent: aipbot
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: Alexibot
Disallow: /
User-agent: asterias
Disallow:/
User-agent: Aqua_Products
Disallow: /
User-agent: b2w/0.1
Disallow: /
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: becomebot
Disallow: /
User-agent: Black Hole
Disallow:/
User-agent: BlowFish
Disallow:/
User-agent: Bookmark search tool
Disallow: /
User-agent: BotALot
Disallow: /
User-agent: BotRightHere
Disallow: /
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite
Disallow:/
User-agent: CherryPickerSE
Disallow:/
User-agent: Copernic
Disallow: /
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: FairAd Client
Disallow: /
User-agent: Fasterfox
Disallow: /
User-agent: Flaming AttackBot
Disallow: /
User-agent: Foobot
Disallow:/
User-agent: Gaisbot
Disallow: /
User-agent: GetRight
Disallow: /
User-agent: Harvest
Disallow:/
User-agent: Go!Zilla
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: HTTrack
Disallow: /
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: Iron33
Disallow: /
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density
Disallow:/
User-agent: larbin
Disallow: /
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: Mata Hari
Disallow: /
User-agent: MIIxpc
Disallow:/
User-agent: Mister PiX
Disallow:/

User-agent: moget
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NetMechanic
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: Oracle Ultra Search
Disallow: /
User-agent: PerMan
Disallow: /
User-agent: ProPowerBot
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: psbot
Disallow: /

User-agent: QueryN Metasearch
Disallow:/
User-agent: Radiation Retriever
Disallow: /
User-agent: RepoMonkey
Disallow:/
User-agent: Bait
Disallow:/
User-agent: RMA
Disallow:/
User-agent: searchpreview
Disallow: /
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz
Disallow:/
User-agent: SurveyBot
Disallow: /
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: turingos
Disallow:/
User-agent: TurnitinBot
Disallow: /
User-agent: URL Control
Disallow: /
User-agent: URLy Warning
Disallow:/
User-agent: URL_Spider_Pro
Disallow: /
User-agent: VCI
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebCapture
Disallow: /
User-agent: scalaj-http/1.0
Disallow: /
User-agent: WebCollage
Disallow: /
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: Wget
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu's
Disallow:/
User-agent: Xenu's Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
User-agent: Meltawer
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Cision
Disallow: /

User-agent: Talkwater
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /

User-agent: Zite
Disallow: /

User-agent: flipboard
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: trendybuzz
Disallow: /

User-agent: spotter
Disallow: /

User-agent: scoop.it
Disallow: /

User-agent: linkfluence
Disallow: /

User-agent: 5emeRue
Disallow: /

User-agent: Augure
Disallow: /

User-agent: Corporama
Disallow: /

User-agent: readability.com
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: libwww
Disallow: /

User-agent: wget
Disallow: /

User-agent: 5erue
Disallow: /

User-agent: Sysomos/1.0
Disallow: /

User-agent: adequat
Disallow: /

User-agent: adequat-systems
Disallow: /

User-agent: auramundi
Disallow: /

User-agent: coexel
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: leadbox
Disallow: /

User-agent: mention
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: mytwip
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: mixnode.com
Disallow: /

User-agent: opinion-tracker
Disallow: /

User-agent: proxem
Disallow: /

User-agent: score3
Disallow: /

User-agent: trendeo
Disallow: /

User-agent: vecteurplus
Disallow: /

User-agent: verticalsearch
Disallow: /

User-agent: vsw
Disallow: /

User-agent: winello
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: infoseek
Disallow: /


User-agent: Offline Explorer
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: asknread.com
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: spotter
Disallow: /

User-agent: LexxeBot/1.0
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: Mozilla/5.0 (compatible; spbot/2.0; http://www.seoprofiler.com/bot/ )
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: Nuzzel
Disallow: /

User-agent: Nuzzel Bot
Disallow: /

User-agent: Sosospider+(+http://help.soso.com/webspider.htm)
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: SiteBot/0.1
Disallow: /

User-agent: SiteBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: crystalsemanticsbot
Disallow: /

User-agent: CrystalSemanticsBot
Disallow: /

User-agent: NetSeer crawler
Disallow: /

User-agent: trovitBot
Disallow: /

User-agent: LexxeBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: discobot
Disallow: /

User-agent: Jyxobot
Disallow: /

User-agent: sogou
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: Sogoubot
Disallow: /

User-agent: Sogou.com
Disallow: /

User-agent: sogouspider-106-120-173-138.crawl.sogou.com
Disallow: /

User-agent: crawl13.exabot.com
Disallow: /

User-agent: sogouspider-106-120-173-138
Disallow: /

User-agent: Sogou web spider/4.0
Disallow: /


User-agent: Sogouwebspider
Disallow: /

User-agent: sogouspider
Disallow: /

User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Disallow: /

User-agent: Exabot/3.0
Disallow: /


User-agent: Sogou web spider
Disallow: /

User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: sistrix
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected])
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: NerdByNature.Bot
Disallow: /


User-agent: psbot
Disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: AddThis.com robot [email protected]
Disallow: /

User-agent: AddThis.com
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: proximic
Disallow: /

User-agent: discoverybot
Disallow: /

User-agent: bl.uk_lddc_bot
Disallow: /

User-agent: IstellaBot
Disallow: /

User-agent: seokicks
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: UnisterBot
Disallow: /

User-agent: Bender
Disallow: /

User-agent: wotbox
Disallow: /

User-agent: Yasni
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: netEstate NE Crawler
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: Pixray-Seeker
Disallow: /

User-agent: Linguee
Disallow: /

User-agent: integromedb
Disallow: /

User-agent: SearchmetricsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: BDCbot
Disallow: /

User-agent: GrapeshotCrawler
Disallow: /

User-agent: WeSEE:Search
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: admantx
Disallow: /

User-agent: spbot
Disallow: /

User-agent: BUbiNG
Disallow: /

User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: Harvest/1.5
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Nutch
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: Spiderbot
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: Xenu Link Sleuth/1.3.8
User-agent: yacy
User-agent: yandex
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Spiderbot/Nutch-1.7
User-agent: g2reader-bot/1.0
User-agent: SocialShare/1.0
User-agent: Owlin Bot
User-agent: trendictionbot
Disallow: /

# Disallow: Sistrix
User-agent: sistrix
Disallow: /

# Disallow: Sistrix
User-agent: SISTRIX Crawler
Disallow: /

# Disallow: Sistrix
User-agent: SISTRIX
Disallow: /

# Disallow: SEOkicks-Robot
User-agent: SEOkicks-Robot
Disallow: /

# Disallow: jobs.de-Robot
User-agent: jobs.de-Robot
Disallow: /

# Backlink Analysis
user-agent: AhrefsBot
disallow: /

# Bot der Leipziger Unister Holding GmbH
user-agent: UnisterBot
disallow: /

# http://moz.com/products
User-agent: dotbot
Disallow: /

# http://www.searchmetrics.com
User-agent: SearchmetricsBot
Disallow: /

# http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
User-agent: MJ12bot
Disallow: /

# http://www.domaintools.com/webmasters/surveybot.php
User-agent: SurveyBot
Disallow: /

# http://www.seodiver.com/bot
user-agent: SEOdiver
disallow: /

# http://openlinkprofiler.org/bot
User-agent: spbot
Disallow: /

# http://www.wotbox.com/bot/
User-agent: wotbox
Disallow: /

# http://www.opensiteexplorer.org/dotbot
# http://moz.com/researchtools/ose/dotbot
User-agent: dotbot
Disallow: /

# http://www.meanpath.com/meanpathbot.html
User-agent: meanpathbot
Disallow: /

# http://www.backlinktest.com/crawler.html
User-agent: BacklinkCrawler
Disallow: /

# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler
Disallow: /

# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Disallow: /

# http://www.brandwatch.com/magpie-crawler/
User-agent: magpie-crawler/1.1
Disallow: /

# http://filterdb.iss.net/crawler/
User-agent: oBot
Disallow: /

User-agent: fr-crawler
Disallow: /

# http://webmeup-crawler.com
User-agent: BLEXBot
Disallow: /

# https://megaindex.com/crawler
User-agent: MegaIndex.ru
Disallow: /

User-agent: megaindex.com
Disallow: /

# http://www.cloudservermarket.com
User-Agent: CloudServerMarketSpider
Disallow: /

# http://www.trendiction.de/de/publisher/bot
User-Agent: trendictionbot
Disallow: /

# http://www.exalead.com
User-agent: Exabot
Disallow: /

# http://www.career-x.de/bot.html
User-agent: careerbot
Disallow: /

# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /

User-agent: seoscanners.net/1
Disallow: /

User-agent: MetaJobBot
Disallow: /

User-agent: Spiderbot
Disallow: /

User-agent: LinkStats
Disallow: /

User-agent: JobboerseBot
Disallow: /

User-agent: ICCrawler
Disallow: /

User-agent: Plista
Disallow: /

User-agent: Domain Re-Animator Bot
Disallow: /

User-agent: Aboundexbot
Disallow: /

User-agent: acapbot
Disallow: /

User-agent: AdnormCrawler
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: AOLbot/4.0
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: Attribot
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Bot.AraTurka.com
Disallow: /

User-agent: Butterfly/1.0
Disallow: /

User-agent: BuzzSumo
Disallow: /

User-agent: cis455crawler
Disallow: /

User-agent: ClarityBot
Disallow: /

User-agent: ClarityDailyBot
Disallow: /

User-agent: CMS Crawler
Disallow: /

User-agent: CommonCrawler
Disallow: /

User-agent: crawler4j (http://code.google.com/p/crawler4j/)
Disallow: /

User-agent: CrazyWebCrawler
Disallow: /

User-agent: CrazyWebCrawler-Spider
Disallow: /

User-agent: DataparkSearch
Disallow: /

User-agent: deepcrawl
Disallow: /

User-agent: demandbase-bot
Disallow: /

User-agent: DeuSu
Disallow: /

User-agent: dialogsearch.com
Disallow: /

User-agent: diffbot
Disallow: /

User-agent: dj-research
Disallow: /

User-agent: DomainAppender
Disallow: /

User-agent: DomainMacroCrawler
Disallow: /

User-agent: DomainSigmaCrawler
Disallow: /

User-agent: Domain Re-Animator Bot
Disallow: /

User-agent: Ezooms Robot
Disallow: /

User-agent: Experibot
Disallow: /

User-agent: FavOrg
Disallow: /

# https://www.lipperhey.com/en/about/
User-agent: Lipperhey-Kaus-Australis
Disallow: /

# https://turnitin.com/robot/crawlerinfo.html
User-agent: turnitinbot
Disallow: /

# http://help.coccoc.com/
User-agent: coccoc
Disallow: /

# ubermetrics-technologies.com
User-agent: um-IC
Disallow: /

User-agent: scanbot
Disallow: /

# datenbutler.de
User-agent: mindUpBot
Disallow: /

# http://searchgears.de/uber-uns/crawling-faq.html
User-agent: sg-Orbiter
Disallow: /

# http://commoncrawl.org/faq/
User-agent: CCBot
Disallow: /


# http://linkfluence.net/
User-agent: Kraken
Disallow: /

# http://www.botje.com/plukkie.htm
User-agent: plukkie
Disallow: /

# https://www.safedns.com/searchbot
User-agent: SafeDNSBot
Disallow: /

# http://www.haosou.com/help/help_3_2.html
User-agent: 360Spider
Disallow: /

# http://www.haosou.com/help/help_3_2.html
User-agent: HaosouSpider
Disallow: /

# http://www.moz.com/dp/rogerbot
User-agent: rogerbot
Disallow: /

# http://www.openhose.org/bot.html
User-agent: OpenHoseBot
Disallow: /

# http://www.screamingfrog.co.uk/seo-spider/
User-agent: Screaming Frog SEO Spider
Disallow: /

# http://thumbsniper.com
User-agent: ThumbSniper
Disallow: /

# http://www.radian6.com/crawler
User-agent: R6_CommentReader
Disallow: /

User-agent: ImplisenseBot
Disallow: /

# http://cliqz.com/company/cliqzbot
User-agent: Cliqzbot
Disallow: /

# https://www.aihitdata.com/about
User-agent: aiHitBot
Disallow: /

# http://www.trendiction.com/en/publisher/bot
User-Agent: trendictionbot
Disallow: /

# Riddler
#	Riddler (http://riddler.io/about)
#	http://riddler.io/about
User-agent: Riddler
Disallow: /

# SMTBot
#	http://www.similartech.com/smtbot
#	Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
User-agent: SMTBot
Disallow: /

# ScoutJet
#	http://www.scoutjet.com
#	Mozilla/5.0+(compatible;+ScoutJet;++http://www.scoutjet.com/)
#	Mozilla/5.0+(compatible;+Blekkobot;+ScoutJet;++http://blekko.com/about/blekkobot)
User-agent: ScoutJet
Disallow: /

# SeeBot
#	http://www.seegnify.com/bot
#	seebot/2.0 (+http://www.seegnify.com/bot)
User-agent: SeeBot
Disallow: /

# Archive.org Bot
#	http://archive.org/details/archive.org_bot
#	Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot)
#	Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot)
User-agent: archive.org_bot
Disallow: /

# AdBeat Bot
#	https://www.adbeat.com/operation_policy
#	adbeat_bot
User-agent: adbeat_bot
Disallow: /

# MeanPathBot
#	http://www.meanpath.com/meanpathbot.html
#	Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
User-agent: meanpathbot
Disallow: /

# Mojeek Bot
#	https://www.mojeek.com/bot.html
#	Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
User-agent: MojeekBot
Disallow: /

# Speedy Spider / EntireWeb Search
#	http://www.entireweb.com
#	Speedy+Spider+(http://www.entireweb.com)
User-agent: Speedy Spider
Disallow: /

# GenieO
#	http://www.genieo.com/webfilter.html
#	Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
User-agent: genieo
Disallow: /

# scrapy.org
# by default ROBOTSTXT_OBEY is set to false
#	http://doc.scrapy.org/en/latest/topics/settings.html
User-agent: Scrapy
Disallow: /

# Gigablast Open Source Search Engine/Crawler/Spider
# You can ignore robots.txt
#	http://gigablast.com/admin.html
#	GigablastOpenSource/1.0
User-agent: GigablastOpenSource
Disallow: /

# Semrush Bot
#	http://www.semrush.com/bot.html
#	Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html)
User-agent:	SemrushBot
Disallow: /

# Proximic
#	http://www.proximic.com/info/spider.php
#	Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
User-agent: proximic
Disallow: /

# AppleBot
#	Used by Siri/Spotlight suggestions (commented out for now)
#	http://www.apple.com/go/applebot
#	Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot)
#	Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot)
#	Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot)
User-agent: applebot
Disallow: /

# CCBot
#	http://commoncrawl.org/faq/
#	CCBot/2.0+(http://commoncrawl.org/faq/)
User-agent: CCBot
Disallow: /

# Plukkie
#	http://www.botje.com/plukkie.htm
#	Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
User-agent: Plukkie
Disallow: /

# Lipperhey Kaus Australis
#	https://www.lipperhey.com/en/about/
#	Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
User-agent: lipperhey
Disallow: /

# oBot
#	http://filterdb.iss.net/crawler/
#	Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
#	Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
User-agent: oBot
Disallow: /

# Screaming Frog SEO Spider
#	Downloaded software, robots.txt can be ignored
#	http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
#	Screaming+Frog+SEO+Spider/2.55
#	Screaming+Frog+SEO+Spider/3.3
User-agent: Screaming Frog SEO Spider
Disallow: /

# VegeBot
#	http://www.exactbot.com/vegebot/index.html
User-agent: VegeBot
Disallow: /

# ScreenerBot
#	ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
User-agent: ScreenerBot
Disallow: /

# Garlik from Experian
#	GarlikCrawler/1.2 (http://garlik.com/, [email protected])
User-agent: Garlik
Disallow: /

#	BusinessBot: [email protected]
User-agent: BusinessBot
Disallow: /


#	FAST Enterprise Crawler/5.3.4 ([email protected])
User-agent: FAST Enterprise
Disallow: /

# Nutch
#	http://nutch.apache.org/bot.html
#	Nutch+Spider/Nutch-1.5
#	mycrowl/Nutch-1.9
#	nutch+crawler/Nutch-1.9
#	sky+nutch+crawler/Nutch-1.9
#	Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
User-agent: Nutch
Disallow: /





# Open Link Profiler
#	Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
#	http://OpenLinkProfiler.org/bot
User-agent: spbot
Disallow: /


# James Bot
#	Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
#	http://cognitiveseo.com/bot.html
User-agent: JamesBOT
Disallow: /



# Kraken
#	Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;[email protected])
#	http://linkfluence.net/
User-agent: Kraken
Disallow: /



#	Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
#	http://www.linkdex.com/bots/
User-agent: linkdex
Disallow: /


#	Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
#	Siteimprove.com
User-agent: SiteCheck-sitecrawl by Siteimprove.com
Disallow: /


#	Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
#	http://megaindex.com/crawler
User-agent: MegaIndex
Disallow: /


#	Prlog
#	Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
#	http://prlog.ru/
User-agent: Prlog
Disallow: /

#	Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
#	https://www.aihitdata.com/about
User-agent: AiHitBot
Disallow: /


#	KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
#	http://www.kingland.com
User-agent: KinglandSystemsCorp
Disallow: /


#	Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;[email protected])
#	http://www.sitemapbot.com
User-agent: SitemapBot
Disallow: /


#	Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
#	https://www.deepcrawl.com/bot
User-agent: deepcrawl
Disallow: /


#	Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21
#	http://www.alexa.com/toolbar
#User-agent: AlexaToolbar
#Disallow: /


#	Mozilla/5.0+(compatible;+heritrix/3.3.0-SNAPSHOT-20150302-2206++http://127.0.0.1)
#	https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
User-agent: heritrix
Disallow: /


#	Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
#	http://help.naver.com/robots/
User-agent: Yeti
Disallow: /


#	Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,[email protected]+for+information)
User-agent: NetShelter
Disallow: /

#	VSE/1.0+([email protected])

#	LSSRocketCrawler/1.0 LightspeedSystems
#	http://lightspeedsystems.com
User-agent: LSSRocketCrawler
Disallow: /


#	Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
#	http://www.idmarch.org/bot.html
User-agent: idmarch
Disallow: /


#	Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0
#	Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0+(Following+Mediapartners-Google)
User-agent: Daumoa
Disallow: /




#	Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
#	https://www.companiesintheuk.co.uk/bot.html
User-agent: CukBot
Disallow: /


#	Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
#	http://www.expertsearch.nl/spider
User-agent: ExpertSearchSpider
Disallow: /


#	Mozilla/5.0+(compatible;+Goodzer/2.0;[email protected])
User-agent: Goodzer
Disallow: /


#	WeSEE
#	http://www.wesee.com/
User-agent: WeSEE
Disallow: /


#	http://ltx71.com/
#	ltx71 - (http://ltx71.com/)
User-agent: ltx71
Disallow: /

#	rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,[email protected])
#	http://moz.com/help/pro/what-is-rogerbot-
User-agent: rogerbot
Disallow: /


#	bitlybot
#	http://bit.ly ??
User-agent: bitlybot
Disallow: /


#	Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
#	http://www.trendiction.de/bot
User-agent: trendictionbot
Disallow: /


#	SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
User-agent: SearchMassive
Disallow: /


#	spiderbot
User-agent: spiderbot
Disallow: /


#	yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
#	yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
User-agent: yacybot
Disallow: /


#	AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;[email protected])
#	http://www.aboutus.org/Aboutus:Bot
User-agent: AboutUsBot
Disallow: /




#	Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,[email protected])
#	http://www.opensiteexplorer.org/dotbot
User-agent: DotBot
Disallow: /


#	Mozilla/5.0+(compatible;+BLEXBot/1.0;++http://webmeup-crawler.com/)
#	http://webmeup-crawler.com/
User-agent: BLEXBot
Disallow: /





#	Xenu Link Sleuth 1.3.4
#	Xenu Link Sleuth/1.3.8
User-agent: Xenu Link Sleuth
Disallow: /


# Go language
#	Go+1.1+package+http
User-agent: Go 1.1
Disallow: /


# wotbox.com
#	http://www.wotbox.com/bot/
#	Wotbox/2.01+(+http://www.wotbox.com/bot/)
User-agent: Wotbox
Disallow: /

# yandex
#	http://yandex.com/bots
#	Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
User-agent: YandexBot
Disallow: /


#	Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
#	http://help.yahoo.com/help/us/ysearch/slurp
# User-agent: Yahoo! Slurp
# Disallow: /


# Yahoo Japan
#	http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
#	Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
User-agent: Y!J-ASR
Disallow: /

# domainreanimator.com
User-agent: Domain Re-Animator Bot
Disallow: /

# crazywebcrawler.com
User-agent: CrazyWebCrawler-Spider
Disallow: /

# crazywebcrawler.com
User-agent: CrazyWebCrawler
Disallow: /

# Dow Jones Searchbot
#	Mozilla/5.0+(compatible;+Dow+Jones+Searchbot)
User-agent: Dow Jones SearchBot
Disallow: /

# OpenVAS.org
#User-agent: OpenVAS
#Disallow: /

# WeCrawlForThePeace - We are not Evil
User-agent: WeCrawForThePeace
Disallow: /

# WeLikeLinks - WeAreNotEvil
User-agent: WeLikeLinks
Disallow: /

# icevikatam/1.0
#User-agent: icevikatam
#Disallow: /

User-agent: UbiCrawler
Disallow: /
User-agent: DOC
Disallow: /
User-agent: Zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: SiteSnagger
Disallow: /
User-agent: WebStripper
Disallow: /
User-agent: WebCopier
Disallow: /
User-agent: Fetch
Disallow: /
User-agent: Offline Explorer
Disallow: /
User-agent: Teleport
Disallow: /
User-agent: TeleportPro
Disallow: /
User-agent: WebZIP
Disallow: /
User-agent: Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
Disallow: /
User-agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)
Disallow: /
User-agent: SocialRankIOBot; http://socialrank.io/about
Disallow: /
User-agent: SocialRankIOBot
Disallow: /
User-agent: linko
Disallow: /
User-agent: HTTrack
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: Xenu
Disallow: /
User-agent: larbin
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: Download Ninja
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /
User-agent: WebReaper
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: Curious George
Disallow: /
User-agent: CybEye.com
Disallow: /
User-agent: DoCoMo
Disallow: /
User-agent: ExB Language Crawler
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: Flamingo_SearchEngine
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Genio
Disallow: /
User-agent: LWNutch
Disallow: /
User-agent: LexxeBot
Disallow: /
User-agent: OpenWebIndex
Disallow: /
User-agent: RediffNewsBot
Disallow: /
User-agent: SEOENGWorldBot
Disallow: /
User-agent: Scanmine
Disallow: /
User-agent: ShopWiki
Disallow: /
User-agent: ShowyouBot
Disallow: /
User-agent: Sosospider
Disallow: /
User-agent: WocBot
Disallow: /
User-agent: Yeti
Disallow: /
User-agent: YoudaoBot
Disallow: /
User-agent: daumoa
Disallow: /
User-agent: gsa-crawler
Disallow: /
User-agent: libcrawl
Disallow: /
User-agent: linkdex
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: repparser
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: sindice-site-manager
Disallow: /
User-agent: sogou spider
Disallow: /
User-agent: sogou
Disallow: /
User-agent: woriobot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: yolinkBot
Disallow: /
User-agent: UnisterBot
Disallow: /
User-agent: JikeSpider
Disallow: /
User-agent: Orthogaffe
Disallow: /
User-agent: wget
Disallow: /

# Block Sogou
User-agent: sogou spider
Disallow: /

# Block SEOkicks
User-agent: SEOkicks-Robot
Disallow: /

# Block BlexBot
User-agent: BLEXBot
Disallow: /

# Block SISTRIX
User-agent: SISTRIX Crawler
Disallow: /

# Block Uptime robot
User-agent: UptimeRobot/2.0
Disallow: /

User-agent: 008
Disallow: /

# Block Ezooms Robot
User-agent: Ezooms Robot
Disallow: /

# Block Perl LWP
User-agent: Perl LWP
Disallow: /

# Block BlexBot
User-agent: BLEXBot
Disallow: /

# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /

# Block WiseGuys Robot
User-agent: WiseGuys Robot
Disallow: /

# Block Turnitin Robot
User-agent: Turnitin Robot
Disallow: /


User-agent: HuaweiSymantecSpider
Disallow: /

User-agent: Purebot
Disallow: /

# Block Heritrix
User-agent: Heritrix
Disallow: /

User-agent: eContext/1.0 (eContext Classification Engine)
Disallow: /

User-agent: eContext/1.0
Disallow: /

User-agent: eContext
Disallow: /

# Block pricepi
User-agent: pimonster
Disallow: /
User-agent: Pimonster
Disallow: /

# Block Searchmetrics Bot
User-agent: SearchmetricsBot
Disallow: /

# Block Eniro
User-agent: ECCP/1.0 ([email protected])
Disallow: /

# Block YandexBot
User-agent: Yandex
Disallow: /

# Block Baidu
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /

# Block SoGou
User-agent: Sogou Spider
Disallow: /

# Block Youdao
User-agent: YoudaoBot
Disallow: /

# Block  MegaIndex.ru
User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: Yandex
Disallow: /
user-agent: AhrefsBot
disallow: /
User-Agent: trendictionbot
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: search:robot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: xovi
Disallow: /
User-agent: Cutbot
Disallow: /



User-agent: DotBot
Disallow: /

User-agent: LivelapBot
Disallow: /

User-agent: LS Session
Disallow: /

User-agent: MagicBot
Disallow: /

User-agent: AliveAdvisorCrawler/1.7.1
Disallow: /

User-agent: AliveAdvisorCrawler/1.7.1 (whole web crawler; http://search.aliveadvisor.com; [email protected])
Disallow: /

User-agent: madaali.de
Disallow: /

User-agent: memoryBot
Disallow: /

User-agent: whole web crawler
Disallow: /




User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: MegaIndex.ru/
Disallow: /

User-agent: SputnikBot
Disallow: /

User-agent: UbiCrawler
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: wget
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: berlin-fu-cow
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: semager
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: InAGist URL Resolver
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Insitesbot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: IndeedBot
Disallow: /

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: MegaIndex.ru/2.0
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: megaIndex.ru
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: ltx71
Disallow: /

User-agent: SputnikBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: seoscanners.net
Disallow: /

User-agent: DeuSu
Disallow: /

User-agent: meds-online24.com
Disallow: /

User-agent: seoscanners
Disallow: /

User-agent: GuzzleHttp
Disallow: /

User-agent: IDBot
Disallow: /

User-agent: PRTGCloudBot
Disallow: /

User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /

User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: WebLyzard Crawler
Disallow: /

User-agent: WebCopier v4.3
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: Shim-Crawler
Disallow: /

User-agent: OmniExplorer_Bot
Disallow: /

User-agent: NimbleCrawler
Disallow: /

User-agent: hl_ftien_spider_v1.1
Disallow: /

User-agent: EMPAS_ROBOT
Disallow: /

User-agent: EDI/1.4.1 (Edacious & Intelligent Web Crawler)
Disallow: /

User-agent: (compatible; hetrix/1.8.0+http://www.accelobot.com)
Disallow: /

User-agent: WebReaper [[email protected]]
Disallow: /

User-agent: WebReaper [[email protected]]
Disallow: /

User-agent: Mozilla/5.0 (compatible; Vagabondo/Kliksafe; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Disallow: /

User-agent: Vagabondo/4.0 (webagent at wise-guys dot nl)
Disallow: /

User-agent: Mozilla/4.0 (compatible;  Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)
Disallow: /

# http://www.wise-guys.nl/
User-agent: webcrawler
Disallow: /

User-agent: ilse
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: Bilbo
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: lwp-trivial/5.810
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: discoverybot
Disallow: /

User-agent: SimplePie
Disallow: /

User-agent: SimplePie/1.1.1
Disallow: /

User-agent: SimplePie/1.0.1
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Ocelli/1.4
Disallow: /

User-agent: Ocelli
Disallow: /

User-agent: [email protected]
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: Ezooms/1.0
Disallow: /

User-agent: Ezooms/1.0; [email protected]
Disallow: /

#ad Crawler: http://www.a6corp.com/a6-web-scraping-policy/
User-agent: A6-Indexer/1.0
Disallow: /

#ad Crawler: http://www.grapeshot.co.uk/crawler.php
User-agent: grapeshot
Disallow: /

# http://www.proximic.com/info/spider.php
User-agent: proximic
Disallow: /

# http://www.jobdigger.nl/spider/
User-agent: JobdiggerSpider
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: rogerbot
Disallow:/

User-agent: SEOkicks-Robot
Disallow:/

User-agent: SMTBot
Disallow:/

User-agent: SemrushBot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: EnaBot/1.2 (http://www.enaball.com/crawler.html)
Disallow: /

User-agent: EnaBot
Disallow: /

User-agent: Go-http-client/1.1
Disallow: /


User-agent: SMTBot/1.0
Disallow: /

User-agent: e-SocietyRobot
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: Synthesio Crawler release MonaLisa (contact at synthesio dot fr)
Disallow: /

User-agent: FreeFind
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Caliperbot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: SimplePie
Disallow: /

User-agent: SimplePie/1.1.3 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20081219
Disallow: /

User-Agent: trendictionbot
Disallow: /

# http://sentibot.eu/
User-agent: SentiBot
Disallow: /

User-agent: SentiBot
Disallow: /

User-agent: omgili/0.5 +http://omgili.com
Disallow: /

User-agent: Mediatoolkitbot ([email protected])
Disallow: /

User-agent: Mediatoolkitbot
Disallow: /

User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Disallow: /

User-agent: GrapeshotCrawler/2.0
Disallow: /

User-agent: GrapeshotCrawler
Disallow:/

User-agent: Grapeshot Crawler
Disallow:/

User-agent: Grapeshot
Disallow:/

User-agent: omgili/0.5 +http://omgili.com
Disallow: /

User-agent: omgili
Disallow: /


User-agent: musobot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: SEOkicks-Robot
Disallow: /

User-agent: SimplePie
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: Mozilla/5.0 ResearchBot/0.6.2 (project: flattr-id-meta-tokens) This/1
Disallow: /

User-agent: ResearchBot/0.6.2
Disallow: /

User-agent: ResearchBot
Disallow: /

User-agent: jobs.de-Robot
Disallow: /

User-agent: FreeFind
Disallow: /

User-agent: xingbot
Disallow: /

User-agent: xing bot
Disallow: /

User-agent: xing
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: ChinasoSpider
Disallow: /

User-agent: yisouspider
Disallow: /

User-agent: EasouSpider
Disallow: /


User-agent: SeznamBot
Disallow: /

User-agent: Caliperbot
Disallow: /

User-agent: spbot
Disallow: /

user-agent: EveryoneSocialBot
Disallow: /
user-agent: something
Disallow: /

user-agent: ABCdatos BotLink
user-agent: Acme.Spider
user-agent: Ahoy! The Homepage Finder
user-agent: Alkaline
user-agent: Anthill
user-agent: Walhello appie
user-agent: Arachnophilia
user-agent: Arale
user-agent: Araneo
user-agent: AraybOt
user-agent: ArchitextSpider
user-agent: Aretha
user-agent: ARIADNE
user-agent: arks
user-agent: AskJeeves
user-agent: ASpider (Associative Spider)
user-agent: ATN Worldwide
user-agent: Atomz.com Search Robot
user-agent: AURESYS
user-agent: BackRub
user-agent: Bay Spider
user-agent: Big Brother
user-agent: Bjaaland
user-agent: BlackWidow
user-agent: Die Blinde Kuh
user-agent: Bloodhound
user-agent: Borg-Bot
user-agent: BoxSeaBot
user-agent: bright.net caching robot
user-agent: BSpider
user-agent: CACTVS Chemistry Spider
user-agent: Calif
user-agent: Cassandra
user-agent: Digimarc Marcspider/CGI
user-agent: Checkbot
user-agent: ChristCrawler.com
user-agent: churl
user-agent: cIeNcIaFiCcIoN.nEt
user-agent: CMC/0.01
user-agent: Collective
user-agent: Combine System
user-agent: Conceptbot
user-agent: ConfuzzledBot
user-agent: CoolBot
user-agent: Web Core / Roots
user-agent: XYLEME Robot
user-agent: Internet Cruiser Robot
user-agent: Cusco
user-agent: CyberSpyder Link Test
user-agent: CydralSpider
user-agent: Desert Realm Spider
user-agent: DeWeb(c) Katalog/Index
user-agent: DienstSpider
user-agent: Digger
user-agent: Digital Integrity Robot
user-agent: Direct Hit Grabber
user-agent: DNAbot
user-agent: DownLoad Express
user-agent: DragonBot
user-agent: DWCP (Dridus' Web Cataloging Project)
user-agent: e-collector
user-agent: EbiNess
user-agent: EIT Link Verifier Robot
user-agent: ELFINBOT
user-agent: Emacs-w3 Search Engine
user-agent: ananzi
user-agent: esculapio
user-agent: Esther
user-agent: Evliya Celebi
user-agent: FastCrawler
user-agent: Fluid Dynamics Search Engine robot
user-agent: Felix IDE
user-agent: Wild Ferret Web Hopper #1, #2, #3
user-agent: FetchRover
user-agent: fido
user-agent: Hämähäkki
user-agent: KIT-Fireball
user-agent: Fish search
user-agent: Fouineur
user-agent: Robot Francoroute
user-agent: Freecrawl
user-agent: FunnelWeb
user-agent: gammaSpider, FocusedCrawler
user-agent: gazz
user-agent: GCreep
user-agent: GetBot
user-agent: GetURL
user-agent: Golem
user-agent: Grapnel/0.01 Experiment
user-agent: Griffon
user-agent: Gromit
user-agent: Northern Light Gulliver
user-agent: Gulper Bot
user-agent: HamBot
user-agent: Harvest
user-agent: havIndex
user-agent: HI (HTML Index) Search
user-agent: Hometown Spider Pro
user-agent: ht://Dig
user-agent: HTMLgobble
user-agent: Hyper-Decontextualizer
user-agent: iajaBot
user-agent: IBM_Planetwide
user-agent: Popular Iconoclast
user-agent: Ingrid
user-agent: Imagelock
user-agent: IncyWincy
user-agent: Informant
user-agent: InfoSeek Robot 1.0
user-agent: Infoseek Sidewinder
user-agent: InfoSpiders
user-agent: Inspector Web
user-agent: IntelliAgent
user-agent: I, Robot
user-agent: Iron33
user-agent: Israeli-search
user-agent: JavaBee
user-agent: JBot Java Web Robot
user-agent: JCrawler
user-agent: Jeeves
user-agent: JoBo Java Web Robot
user-agent: Jobot
user-agent: JoeBot
user-agent: The Jubii Indexing Robot
user-agent: JumpStation
user-agent: image.kapsi.net
user-agent: Katipo
user-agent: KDD-Explorer
user-agent: Kilroy
user-agent: KO_Yappo_Robot
user-agent: LabelGrabber
user-agent: larbin
user-agent: legs
user-agent: Link Validator
user-agent: LinkScan
user-agent: LinkWalker
user-agent: Lockon
user-agent: logo.gif Crawler
user-agent: Lycos
user-agent: Mac WWWWorm
user-agent: Magpie
user-agent: marvin/infoseek
user-agent: Mattie
user-agent: MediaFox
user-agent: MerzScope
user-agent: NEC-MeshExplorer
user-agent: MindCrawler
user-agent: mnoGoSearch search engine software
user-agent: moget
user-agent: MOMspider
user-agent: Monster
user-agent: Motor
user-agent: Muncher
user-agent: Muninn
user-agent: Muscat Ferret
user-agent: Mwd.Search
user-agent: Internet Shinchakubin
user-agent: NDSpider
user-agent: Nederland.zoek
user-agent: NetCarta WebMap Engine
user-agent: NetMechanic
user-agent: NetScoop
user-agent: newscan-online
user-agent: NHSE Web Forager
user-agent: Nomad
user-agent: The NorthStar Robot
user-agent: nzexplorer
user-agent: ObjectsSearch
user-agent: Occam
user-agent: HKU WWW Octopus
user-agent: OntoSpider
user-agent: Openfind data gatherer
user-agent: Orb Search
user-agent: Pack Rat
user-agent: PageBoy
user-agent: ParaSite
user-agent: Patric
user-agent: pegasus
user-agent: The Peregrinator
user-agent: PerlCrawler 1.0
user-agent: Phantom
user-agent: PhpDig
user-agent: PiltdownMan
user-agent: Pimptrain.com's robot
user-agent: Pioneer
user-agent: html_analyzer
user-agent: Portal Juice Spider
user-agent: PGP Key Agent
user-agent: PlumtreeWebAccessor
user-agent: Poppi
User-agent: iskanie
user-agent: PortalB Spider
user-agent: psbot
user-agent: GetterroboPlus Puu
user-agent: The Python Robot
user-agent: Raven Search
user-agent: RBSE Spider
user-agent: Resume Robot
user-agent: RoadHouse Crawling System
user-agent: RixBot
user-agent: Road Runner: The ImageScape Robot
user-agent: Robbie the Robot
user-agent: ComputingSite Robi/1.0
user-agent: RoboCrawl Spider
user-agent: RoboFox
user-agent: Robozilla
user-agent: Roverbot
user-agent: RuLeS
user-agent: SafetyNet Robot
user-agent: Scooter
user-agent: Sleek
user-agent: Search.Aus-AU.COM
user-agent: SearchProcess
user-agent: Senrigan
user-agent: SG-Scout
user-agent: ShagSeeker
user-agent: Shai'Hulud
user-agent: Sift
user-agent: Simmany Robot Ver1.0
user-agent: Site Valet
user-agent: Open Text Index Robot
user-agent: SiteTech-Rover
user-agent: Skymob.com
user-agent: SLCrawler
user-agent: Inktomi Slurp
user-agent: Smart Spider
user-agent: Snooper
user-agent: Solbot
user-agent: Spanner
user-agent: Speedy Spider
user-agent: spider_monkey
user-agent: SpiderBot
user-agent: Spiderline Crawler
user-agent: SpiderMan
user-agent: SpiderView(tm)
user-agent: Spry Wizard Robot
user-agent: Site Searcher
user-agent: Suke
user-agent: suntek search engine
user-agent: Sven
user-agent: Sygol
user-agent: TACH Black Widow
user-agent: Tarantula
user-agent: tarspider
user-agent: Tcl W3 Robot
user-agent: TechBOT
user-agent: Templeton
user-agent: TeomaTechnologies
user-agent: TITAN
user-agent: TitIn
user-agent: The TkWWW Robot
user-agent: TLSpider
user-agent: UCSD Crawl
user-agent: UdmSearch
user-agent: UptimeBot
user-agent: URL Check
user-agent: URL Spider Pro
user-agent: Valkyrie
user-agent: Verticrawl
user-agent: Victoria
user-agent: vision-search
user-agent: void-bot
user-agent: Voyager
user-agent: VWbot
user-agent: The NWI Robot
user-agent: W3M2
user-agent: WallPaper (alias crawlpaper)
user-agent: the World Wide Web Wanderer
user-agent: [email protected] by wap4.com
user-agent: WebBandit Web Spider
user-agent: WebCatcher
user-agent: WebCopy
user-agent: webfetcher
user-agent: The Webfoot Robot
user-agent: Webinator
user-agent: weblayers
user-agent: WebLinker
user-agent: WebMirror
user-agent: The Web Moose
user-agent: WebQuest
user-agent: Digimarc MarcSpider
user-agent: WebReaper
user-agent: webs
user-agent: Websnarf
user-agent: WebSpider
user-agent: WebVac
user-agent: webwalk
user-agent: WebWalker
user-agent: WebWatch
user-agent: Wget
user-agent: whatUseek Winona
user-agent: WhoWhere Robot
user-agent: Wired Digital
user-agent: Weblog Monitor
user-agent: w3mir
user-agent: WebStolperer
user-agent: The Web Wombat
user-agent: The World Wide Web Worm
user-agent: WWWC Ver 0.2.5
user-agent: WebZinger
user-agent: XGET
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /

User-agent: Zite
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: Synthesio
Disallow: /

User-agent: trendybuzz
Disallow: /

User-agent: spotter
Disallow: /

User-agent: linkfluence
Disallow: /

User-agent: 5emeRue
Disallow: /

User-agent: Augure
Disallow: /

User-agent: Sindup
Disallow: /

User-agent: Corporama
Disallow: /

User-agent: readability.com
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: k2spider
Disallow: /

User-agent: libwww
Disallow: /

User-agent: 5erue
Disallow: /

User-agent: adequat
Disallow: /

User-agent: adequat-systems
Disallow: /

User-agent: auramundi
Disallow: /

User-agent: coexel
Disallow: /

User-agent: leadbox
Disallow: /

User-agent: Moreover
Disallow: /

User-agent: mytwip
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: opinion-tracker
Disallow: /

User-agent: proxem
Disallow: /

User-agent: score3
Disallow: /

User-agent: trendeo
Disallow: /

User-agent: vecteurplus
Disallow: /

User-agent: verticalsearch
Disallow: /

User-agent: vsw
Disallow: /

User-agent: winello
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: infoseek
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Zealbot
Disallow: /

User-agent: asknread.com
Disallow: /

User-agent: ellisphere
Disallow: /

User-agent: spotter
Disallow: /

User-agent: 008
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: AlvinetSpider
Disallow: /

User-agent: Antenne Hatena
Disallow: /

User-agent: ApocalXExplorerBot
Disallow: /

User-agent: asterias
Disallow: /

User-agent: BackDoorBot/1.0
Disallow: /

User-agent: BizInformation
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: DISCo Pump 3.1
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: DotBot/1.1
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: Flamingo_SearchEngine
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: hloader
Disallow: /

User-agent: httplib
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: HTTrack 3.0
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: Igentia
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: libWeb/clsHTTP
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: lwp-trivial
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: InfoPath
Disallow: /

User-agent: InfoPath.3
Disallow: /

User-agent: Cyberwatcher
Disallow: /

User-agent: SafeDNSBot (https://www.safedns.com/searchbot)
Disallow: /



User-agent: Mata Hari
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: moget
Disallow: /

User-agent: moget/2.1
Disallow: /

User-agent: MS Search 4.0 Robot
Disallow: /

User-agent: MS Search 5.0 Robot
Disallow: /

User-agent: Naverbot
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NetAttache
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: OpenindexSpider
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: psbot
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RMA
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: SightupBot
Disallow: /

User-agent: SiteBot
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SiteSucker
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: sosospider
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: spanner
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: suggybot
Disallow: /

User-agent: SuperBot
Disallow: /

User-agent: SuperBot/2.6
Disallow: /

User-agent: suzuran
Disallow: /

User-agent: Szukacz/1.4
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: TightTwatBot
Disallow: /

User-agent: Titan
Disallow: /

User-agent: toCrawl/UrlDispatcher
Disallow: /

User-agent: TosCrawler
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: True_Robot/1.0
Disallow: /

User-agent: turingos
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: UrlPouls
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: VCI
Disallow: /

User-agent: Web Image Collector
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: WebBandit/3.50
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: webcopy
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: webmirror
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: website extractor
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebStripper/2.02
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: wget
Disallow: /

User-agent: WikioFeedBot
Disallow: /

User-agent: WinHTTrack
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: yacy
Disallow: /

User-agent: YRSPider
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zookabot
Disallow: /

## EXCLUSION BOTS
User-agent: ConveraCrawler
User-agent: Jetbot
User-agent: Meltwater
User-agent: Digimind
User-agent: Knowings
User-agent: kbcrawl
User-agent: AmiSoftware
User-agent: Ask n read
User-agent: Qwam content intelligence
User-agent: Zite
User-agent: FlipboardProxy
User-agent: Youmag
User-agent: Synthesio
User-agent: trendybuzz
User-agent: spotter
User-agent: linkfluence
User-agent: 5emeRue
User-agent: Augure
User-agent: Sindup
User-agent: Corporama
User-agent: readability.com
User-agent: grub-client
User-agent: ia_archiver
User-agent: ia_archiver-web.archive.org
User-agent: k2spider
User-agent: libwww
User-agent: 5erue
User-agent: adequat
User-agent: adequat-systems
User-agent: auramundi
User-agent: coexel
User-agent: leadbox
User-agent: Moreover
User-agent: mytwip
User-agent: NewsNow
User-agent: Newzbin
User-agent: opinion-tracker
User-agent: proxem
User-agent: score3
User-agent: trendeo
User-agent: vecteurplus
User-agent: verticalsearch
User-agent: vsw
User-agent: winello
User-agent: Fetch
User-agent: infoseek
User-agent: MSIECrawler
User-agent: sitecheck.internetseer.com
User-agent: TeleportPro
User-agent: Zealbot
User-agent: asknread.com
User-agent: ellisphere
User-agent: spotter
User-agent: 008
User-agent: Alexibot
User-agent: AlvinetSpider
User-agent: Antenne Hatena
User-agent: ApocalXExplorerBot
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: BizInformation
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DISCo Pump 3.1
User-agent: DittoSpyder
User-agent: dotbot
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Flamingo_SearchEngine
User-agent: Foobot
User-agent: hloader
User-agent: httplib
User-agent: HTTrack
User-agent: HTTrack 3.0
User-agent: humanlinks
User-agent: Igentia
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: JikeSpider
User-agent: Kenjin Spider
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control - 5.01.4511
User-agent: Microsoft URL Control - 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: MLBot
User-agent: moget
User-agent: moget/2.1
User-agent: MS Search 4.0 Robot
User-agent: MS Search 5.0 Robot
User-agent: Naverbot
User-agent: NetAnts
User-agent: NetAttache
User-agent: NetMechanic
User-agent: NICErsPRO
User-agent: Offline Explorer
User-agent: Openfind
User-agent: OpenindexSpider
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: psbot
User-agent: QuepasaCreep
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RMA
User-agent: SemrushBot
User-agent: SightupBot
User-agent: SiteBot
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Sogou web spider
User-agent: sosospider
User-agent: SpankBot
User-agent: spanner
User-agent: Speedy
User-agent: suggybot
User-agent: SuperBot
User-agent: SuperBot/2.6
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: TosCrawler
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: TurnitinBot
User-agent: UrlPouls
User-agent: URLy Warning
User-agent: VCI
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: webcopy
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: webmirror
User-agent: WebReaper
User-agent: WebSauger
User-agent: website extractor
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebStripper/2.02
User-agent: WebZip
User-agent: wget
User-agent: WikioFeedBot
User-agent: WinHTTrack
User-agent: WWW-Collector-E
User-agent: yacy
User-agent: YRSPider
User-agent: Zeus
User-agent: Zookabot
User-agent: Linguee Bot
Disallow: /


User-agent: BLP_bbot/0.1
Disallow: /




User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+)
Disallow: /

User-agent: YandexSomething/1.0
Disallow: /

User-agent: Java/1.6.0_04
Disallow: /

User-agent: 8484 Boston Project v 1.0
Disallow: /

User-agent: Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; m)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; H)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; P)
Disallow: /

User-agent: Yandex/1.01.001 (compatible; Win16; I)
Disallow: /

User-agent: +BecomeBot/3.0
Disallow: /

User-agent: AISearchBot
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: AmazonCloudServices
Disallow: /

User-agent: Apexoo Spider 1.0
Disallow: /

User-agent: Aqua_Products
Disallow: /

User-agent: Arachnoidea
Disallow: /

User-agent: ArchitextSpider
Disallow: /



User-agent: BackDoorBot/1.0
Disallow: /

User-agent: BackDoorBot
Disallow: /



User-agent: DotBot/1.1
Disallow: /




User-agent: YandexBot
Disallow: /

User-agent: YandexBot/3.0
Disallow: /





User-agent: SemrushBot-SA/0.97
Disallow: /




User-agent: archive.org_bot
Disallow: /


User-agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)
Disallow: /


User-agent: AsapBot
Disallow: /


User-agent: semantic-visions.com crawler
Disallow: /

User-agent: MetaURI API/2.0 +metauri.com
Disallow: /

User-agent: MetaURI
Disallow: /


User-agent: citeseerxbot
Disallow: /

User-agent: autokrawl
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /

User-agent: becomebot
Disallow: /

User-agent: BecomeBot/3.0
Disallow: /

User-agent: Bitacle
Disallow: /

User-agent: Bitacle bot
Disallow: /

User-agent: Bitacle bot/1.1
Disallow: /

User-agent: Bitacle*
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: BlowFish/1.0
Disallow: /

User-agent: BobCrawl
Disallow: /

User-agent: boitho
Disallow: /

User-agent: boitho.com
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: bot/1.0
Disallow: /

User-agent: bot/1.0+(bot;+http://;[email protected])
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: botmobi
Disallow: /

User-agent: bsalsa
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: Bullseye/1.0
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Butterfly
Disallow: /

User-agent: Cazoodle-Bot
Disallow: /

User-agent: CazoodleBot
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: Charlotte
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: CherryPickerElite/1.0
Disallow: /

User-agent: CherryPickerSE/1.0
Disallow: /

User-agent: classroombot
Disallow: /

User-agent: ConveraCrawler/0.9d
Disallow: /

User-agent: ConveraCrawler/0.9e
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: cosmos
Disallow: /

User-agent: crawl13.exabot.com
Disallow: /

User-agent: crawl6.exabot.com
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow: /

User-agent: cyberpatrol
Disallow: /

User-agent: cyberpatrolcrawler
Disallow: /

User-agent: datacha0s/2.0
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Disallow: /


User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
Disallow: /

User-agent: dcbot
Disallow: /

User-agent: del.icio.us
Disallow: /

User-agent: del.icio.us-thumbnails/1.0
Disallow: /

User-agent: DepSpid
Disallow: /

User-agent: discobot
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: DoCoMo
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: DTAAgent
Disallow: /

User-agent: eagle
Disallow: /

User-agent: ee:
Disallow: /

User-agent: EmailCollector
Disallow: /

User-agent: EmailSiphon
Disallow: /

User-agent: EmailWolf
Disallow: /

User-agent: envolk
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: EUDORA
Disallow: /

User-agent: Everest-Vulcan
Disallow: /

User-agent: exabot
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: Exabot-Images/1.0
Disallow: /

User-agent: Exabot-Images/3.0
Disallow: /

User-agent: Exabot/2.0
Disallow: /

User-agent: Exabot/3.0
Disallow: /

User-agent: exooba
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: FairAd Client
Disallow: /

User-agent: FairShare
Disallow: /

User-agent: fast
Disallow: /

User-agent: fast Enterprise crawler
Disallow: /

User-agent: fast metaweb crawler
Disallow: /

User-agent: FAST-WebCrawler
Disallow: /

User-agent: fast-Webcrawler
Disallow: /

User-agent: findlinks/1.1.1-a5
Disallow: /

User-agent: findlinks/1.1.3-beta9
Disallow: /

User-agent: Flaming AttackBot
Disallow: /

User-agent: flatlandbot
Disallow: /

User-agent: jack
Disallow: /

User-agent: JS-Kit URL Resolver
Disallow: /

User-agent: KomodiaBot
Disallow: /

User-agent: FollowSiteBot
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: FunWebProducts
Disallow: /

User-agent: FunWebProducts f39-2350
Disallow: /

User-agent: FyberSpider
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: genieBot
Disallow: /

User-agent: genieBot ([email protected])
Disallow: /

User-agent: genieBot+([email protected])
Disallow: /

User-agent: [email protected]
Disallow: /

User-agent: GetRight/4.2
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: Gigabot*
Disallow: /

User-agent: Gigabot/2.0
Disallow: /

User-agent: GingerCrawler
Disallow: /

User-agent: grub
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: gsa-crawler
Disallow: /

User-agent: Gulliver
Disallow: /

User-agent: Harvest/1.5
Disallow: /

User-agent: HenryTheMiragoRobot
Disallow: /

User-agent: heritrix
Disallow: /

User-agent: heritrix/1.12.1 +http://www.page-store.com)
Disallow: /

User-agent: hloader
Disallow: /

User-agent: HMSE_Robot
Disallow: /

User-agent: HornySexSearch
Disallow: /

User-agent: http://www.almaden.ibm.com/cs/crawler
Disallow: /

User-agent: http://www.cuill.com/twiceler/robot.html
Disallow: /

User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/)
Disallow: /

User-agent: httplib
Disallow: /

User-agent: humanlinks
Disallow: /

User-agent: ia_archive
Disallow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

User-agent: ia_archiver/1.6
Disallow: /

User-agent: IBM
Disallow: /

User-agent: IBM_Planetwide
Disallow: /

User-agent: ichiro/2.0
Disallow: /

User-agent: ichiro/2.0 ([email protected])
Disallow: /

User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html)
Disallow: /

User-agent: IconSurf
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: InfoPath
Disallow: /

User-agent: InfoPath.1
Disallow: /

User-agent: InfoPath.2
Disallow: /

User-agent: IntegraTelecom
Disallow: /

User-agent: internetseer
Disallow: /

User-agent: InternetSeer.com
Disallow: /

User-agent: ips-agent
Disallow: /

User-agent: IRLbot
Disallow: /

User-agent: IRLbot/3.0
Disallow: /

User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)
Disallow: /

User-agent: Iron33/1.0.2
Disallow: /

User-agent: Isidorus
Disallow: /

User-agent: Java Bot 2.0
Disallow: /

User-agent: Java/1.5.0_05
Disallow: /

User-agent: Java/1.5.0_10
Disallow: /

User-agent: JavaBot
Disallow: /

User-agent: JavaBot 2.0
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: Jyxobot
Disallow: /

User-agent: kalooga
Disallow: /

User-agent: KaloogaBot
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Keyword Density/0.9
Disallow: /

User-agent: Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)
Disallow: /


User-agent: kyluka
Disallow: /

User-agent: LaBot
Disallow: /

User-agent: larbin
Disallow: /

User-agent: larbin_2.6.1
Disallow: /

User-agent: larbin_2.6.1+([email protected])
Disallow: /

User-agent: laycat
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: LexiBotWeb Image Collector
Disallow: /

User-agent: libcurl-agent
Disallow: /



User-agent: LinkAider
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: LinkScan/8.1a Unix
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LinkWalker/2.0
Disallow: /

User-agent: LinkWalker/2.0 - www.seventwentyfour.com
Disallow: /

User-agent: LMQueueBot/0.2
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: LocalBot
Disallow: /

User-agent: LocalcomBot
Disallow: /

User-agent: looksmart
Disallow: /

User-agent: lwp-trivial/1.34
Disallow: /

User-agent: Mail.Ru
Disallow: /

User-agent: MantraAgent
Disallow: /

User-agent: Mata Hari
Disallow: /


User-agent: Microsoft URL Control
Disallow: /

User-agent: Microsoft URL Control - 5.01.4511
Disallow: /

User-agent: Microsoft URL Control - 6.00.8169
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: MIIxpc/4.2
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: MJ12bot/v1.2.0
Disallow: /

User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? )
Disallow: /

User-agent: MLBot
Disallow: /

User-agent: Modified GMMBotRightHere
Disallow: /

User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Disallow: /

User-agent: Mozilla/5.0+(compatible;+BecomeBot/3.0;++http://www.become.com/site_owners.html)
Disallow: /


User-agent: MSR-ISRCCrawler
Disallow: /

User-agent: nambu
Disallow: /

User-agent: NationalDirectory-SuperSpider
Disallow: /

User-agent: NaverBot
Disallow: /

User-agent: naverbot
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: nexen
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: NG/2.0
Disallow: /

User-agent: nicebot
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: noxtrumbot/1.0
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: obot
Disallow: /

User-agent: Ocelli
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Offline Explorer/1.5
Disallow: /

User-agent: Offline Explorer/1.9
Disallow: /

User-agent: Offline Explorer/2.5
Disallow: /

User-agent: OmniExplorer
Disallow: /

User-agent: OmniExplorer_Bot
Disallow: /

User-agent: OmniExplorer_Bot/1.07
Disallow: /

User-agent: OmniExplorer_Bot/1.09
Disallow: /

User-agent: OmniExplorer_Bot/1.10
Disallow: /

User-agent: OnTownsBot
Disallow: /

User-agent: OOZBOT
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openfind data gathere
Disallow: /

User-agent: OpenX
Disallow: /

User-agent: Oracle Ultra Search
Disallow: /

User-agent: page_prefetcher
Disallow: /


User-agent: PerMan
Disallow: /

User-agent: Piffany_Web_Cacher_v0.91
Disallow: /

User-agent: ProPowerBot/2.14
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: psbot
Disallow: /

User-agent: psbot/0.1
Disallow: /

User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html)
Disallow: /

User-agent: PSIBots
Disallow: /


User-agent: QEAVis
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: Radiation Retriever 1.1
Disallow: /

User-agent: Rankivabot
Disallow: /

User-agent: REAP-Crawler
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow: /

User-agent: RiceComputerArchitecture
Disallow: /

User-agent: RMA
Disallow: /

User-agent: robotgenius
Disallow: /

User-agent: Robozilla
Disallow: /

User-agent: ru_com_viewer
Disallow: /

User-agent: RufusBot
Disallow: /

User-agent: SapphireWebCrawler
Disallow: /

User-agent: Twiceler www.cuill.com/robots.html
Disallow: /


User-agent: k2spider
Disallow: /

User-agent: grub-client
Disallow: /

User-agent: wget
Disallow: /

User-agent: NPBot
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: DOC
Disallow: /

User-agent: Zao
Disallow: /


User-agent: sitecheck.internetseer.com
Disallow: /

User-agent: IsraBot
Disallow: /

User-agent: Orthogaffe
Disallow: /

User-agent: Zealbot
Disallow: /


User-agent: SiteSnagger
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Fetch
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: linko
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Microsoft.URL.Control
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: larbin
Disallow: /

User-agent: libwww
Disallow: /

User-agent: ZyBORG
Disallow: /

User-agent: Download Ninja
Disallow: /

User-agent: fast
Disallow: /

User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Disallow: /

User-agent: Twiceler www.cuill.com/twiceler/robot.html
Disallow: /

User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Disallow: /



User-agent: sbider
Disallow: /

User-agent: SBIder
Disallow: /

User-agent: SBIder/SBIder-0.8.2-dev
Disallow: /

User-agent: scooter
Disallow: /

User-agent: Scooter/2.0
Disallow: /

User-agent: scoutjet
Disallow: /

User-agent: Scrubby
Disallow: /

User-agent: searchme
Disallow: /

User-agent: searchpreview
Disallow: /

User-agent: Sensis Web Crawler
Disallow: /

User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au)
Disallow: /

User-agent: Sensis+Web+Crawler+
Disallow: /

User-agent: Servage
Disallow: /

User-agent: ServageRobot
Disallow: /

User-agent: setooz
Disallow: /

User-agent: ShopWiki
Disallow: /

User-agent: Sidewinder
Disallow: /

User-agent: Sika
Disallow: /

User-agent: sitecat
Disallow: /

User-agent: sitecheck.internetseer.com
Disallow: /


User-agent: SiteSnagger
Disallow: /

User-agent: Slicehost
Disallow: /

User-agent: Slurp.so/1.0
Disallow: /

User-agent: [email protected]
Disallow: /

User-agent: [email protected]
Disallow: /

User-agent: Slurp/2.0-KiteHourly
Disallow: /

User-agent: Slurp/2.0-OwlWeekly
Disallow: /

User-agent: Slurp/2.0j
Disallow: /

User-agent: Slurp/3.0-AU
Disallow: /

User-agent: SlySearch
Disallow: /

User-agent: SlySearch/1.0
Disallow: /

User-agent: SlySearch/1.x
Disallow: /

User-agent: Snapbot
Disallow: /

User-agent: Snapbot/1.0
Disallow: /

User-agent: SnapPreviewBot
Disallow: /

User-agent: Snappy
Disallow: /

User-agent: Snappybot
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: sogou+spider
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: spanner
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: speedyspider
Disallow: /

User-agent: sphsearch
Disallow: /

User-agent: sproose
Disallow: /

User-agent: StackRambler/2.0
Disallow: /

User-agent: StackRambler/2.0+(MSIE+incompatible)
Disallow: /

User-agent: SuperBot/4.4.0.60 (Windows XP)
Disallow: /

User-agent: SuperPages
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: SurveyBot/2.3
Disallow: /

User-agent: SurveyBot/2.3 (Whois Source)
Disallow: /

User-agent: uberbot
Disallow: /

User-agent: UltraSeek
Disallow: /

User-agent: URL_Spider_Pro
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T)
Disallow: /



User-agent: ViolaBot
Disallow: /



User-agent: WebCopier
Disallow: /

User-agent: WebCopier v3.2a
Disallow: /

User-agent: WebCopier v3.3
Disallow: /

User-agent: WebCopier v3.3.0
Disallow: /

User-agent: WebCopier v3.3.2
Disallow: /

User-agent: WebCrawler
Disallow: /

User-agent: WebDataCentreBot
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: WebmasterWorld Extractor
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: Websense
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Webspider
Disallow: /

User-agent: Webster Pro
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebViewer
Disallow: /

User-agent: WebZip
Disallow: /

User-agent: WebZIP/3.5 (http://www.spidersoft.com)
Disallow: /


User-agent: whois.de
Disallow: /

User-agent: whoisde.de
Disallow: /

User-agent: WillyBot
Disallow: /

User-agent: Win32
Disallow: /

User-agent: worio
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: Xenu Link Sleuth
Disallow: /

User-agent: Xenu Link Sleuth 1.2i
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Xenu Link Sleuth/1.3.8
Disallow: /

User-agent: Y!J
Disallow: /

User-agent: y!j-bsc
Disallow: /

User-agent: Y!J-BSC
Disallow: /

User-agent: Y!J-BSC/1.0
Disallow: /

User-agent: TunitinBot
Disallow: /

User-agent: ConveraCrawler
Disallow: /

User-agent: QuepasaCreep
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Digimind
Disallow: /

User-agent: Knowings D
Disallow: /

User-agent: kbcrawl
Disallow: /

User-agent: Wget
Disallow: /

User-agent: AmiSoftware
Disallow: /

User-agent: Newzbin
Disallow: /

User-agent: Ask n read
Disallow: /

User-agent: Qwam content intelligence
Disallow: /



User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Disallow: /

User-agent: y!j/1.0
Disallow: /

User-agent: Yahoo-Blogs
Disallow: /

User-agent: Yahoo-Blogs/v3.9
Disallow: /

User-agent: Yahoo-MMCrawler/3.x
Disallow: /

User-agent: YahooSeeker
Disallow: /

User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
Disallow: /

User-agent: YahooYSMcm
Disallow: /

User-agent: Zite
Disallow: /

User-agent: flipboard
Disallow: /

User-agent: FlipboardProxy
Disallow: /

User-agent: Youmag
Disallow: /

User-agent: YahooYSMcm/2.0.0
Disallow: /

User-agent: yandex
Disallow: /

User-agent: YandexSomething
Disallow: /

User-agent: Yanga
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: yoogliFetchAgent/0.1
Disallow: /

User-agent: Zao
Disallow: /

User-agent: zermelo
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow: /

User-agent: Zeus Link Scout
Disallow: /

User-agent: zibber
Disallow: /

User-agent: zibber v0.1
Disallow: /

User-agent: zibber-v0.1(www.zibb.com/crawler/)
Disallow: /

User-agent: ZupeeCrawler
Disallow: /

User-agent: zyborg
Disallow: /

User-agent: ZyBorg/1.0
Disallow: /

User-agent: BLEXBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: ScoutJet
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Snoothbot
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Daum
Disallow: /
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /
User-agent: Xenu's
Disallow: /
User-agent: WeSEE
Disallow: /
User-agent: ShopWiki
Disallow: /
user-agent: crystalsemantics
disallow: /
user-agent: f-six
disallow: /
user-agent: sitebot
disallow: /
User-agent: trendictionbot
Disallow: /
User-agent: scrapybot
Disallow: /
User-agent: VocusBot
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: ezooms.bot
Disallow: /
User-agent: skimlinks.com
Disallow: /
User-agent: SeznamBot
Disallow: /
User-agent: wikiwix-bot
Disallow: /
User-agent: semantissimo
Disallow: /
User-agent: MLBot
Disallow: /
User-agent: LemonSources
Disallow: /
User-agent: discobot
Disallow: /
User-agent: linkfluence
Disallow: /
User-agent: ltbot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: language-tools.com
Disallow: /
User-agent: TREC-KBA-Bot
Disallow: /
User-agent: yacybot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: Socialradarbot
Disallow: /
User-agent: Havij
Disallow: /
User-agent: proximic
Disallow: /
User-agent: MireoBot
Disallow: /
User-agent: brandwatch
Disallow: /
User-agent: seoprofiler
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: Bot clicker
Disallow: /
User-agent: WBSearchBot
Disallow: /
User-agent: EC2LinkFinder
Disallow: /
User-agent: ExaleadCloudview
Disallow: /
User-agent: AdMedia
Disallow: /
User-agent: grepnetstat.com
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: NerdByNature
Disallow: /
User-agent: Screaming Frog SEO Spider
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: Sindup
Disallow: /
User-agent: netseer
Disallow: /
User-agent: ichiro
Disallow: /
User-agent: TwengaBot
Disallow: /
User-agent: netseer.com
Disallow: /
User-agent: nagios
Disallow: /
User-agent: linkdex.com
Disallow: /
User-agent: FeedRetriever
Disallow: /
User-agent: CrystalSemanticsBot
Disallow: /
User-agent: Argus Presse
Disallow: /
User-agent: IBM ICA Crawler
Disallow: /
User-agent: nutch agent
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: Genieo
Disallow: /
User-agent: SISTRIX Crawler
Disallow: /
User-agent: PaperLiBot
Disallow: /
User-agent: SearchmetricsBot
Disallow: /
User-agent: Linguee
Disallow: /
User-agent: ip-web-crawler.com
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-agent: Mail.RU_Bot
Disallow: /
User-agent: LinksCrawler
Disallow: /
User-agent: archive.org
Disallow: /
User-agent: CompSpyBot
Disallow: /
User-agent: netEstate
Disallow: /
User-agent: MyNutchTest
Disallow: /
User-agent: admantx
Disallow: /
User-agent: genieo.com
Disallow: /
User-agent: mixdata.com
Disallow: /
User-agent: ubermetrics-technologies
Disallow: /
User-agent: grapeshot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: WASALive
Disallow: /
User-agent: URLAppendBot
Disallow: /
User-agent: MaxPointCrawler
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: Aboundexbot
Disallow: /
User-agent: XoviBot
Disallow: /
User-agent: psbot
Disallow: /


User-Agent: Slurp
User-agent: Yahoo! Slurp
User-agent: Baiduspider
User-agent: TweetedTimes Bot
User-agent: TweetmemeBot
User-agent: PaperLiBot
User-agent: BomboraBot
User-agent: WF search
User-agent: heritrix
User-agent: expo9
User-agent: NetSeer
User-agent: AddThis.com
User-agent: Feed
User-agent: Sogou
User-agent: ZumBot
User-agent: Feedspot
User-agent: acebookexternalhit
User-agent: StorygizeBot
User-agent: Scrapy
User-agent: GermCrawler
User-agent: Netvibes
User-agent: Vagabondo
User-agent: Genieo
User-agent: um-IC
User-agent: Typhoeus
User-agent: newspaper
User-agent: weborama-fetcher
User-agent: ScooperBot
User-agent: BYO-2
User-agent: SimplePie
User-agent: GroupHigh
User-agent: feedparser
User-agent: UniversalFeedParser
User-agent: Pulsepoint
User-agent: Steeler
User-agent: GarlikCrawler
User-agent: Flipboard
User-agent: FlipboardRSS
User-agent: Moreover
User-agent: Mercury
User-agent: FeedBurner
User-agent: Yahoo
User-agent: omgili
User-agent: Bloglovin
User-agent: YahooCacheSystem
User-agent: mTalk
User-agent: EveryoneSocialBot
User-agent: EzineArticlesLinkScanner
User-agent: dlvr.it
User-agent: pfblogs
User-agent: Aol
User-agent: ToutiaoSpider
User-agent: Photon
User-agent: PicoFeed
User-agent: um-FC
User-agent: Digg
User-agent: Synaptive
User-agent: YisouSpider
User-agent: BYO-4
User-agent: Feedbin
User-agent: Twingly
User-agent: Tiny
User-agent: WWW-Mechanize
User-agent: CommaFeed
User-agent: TkBot
User-agent: TinEye-bot
User-agent: Exabot
User-agent: coccoc
User-agent: BLEXBot
User-agent: uMBot-LN
User-agent: QuerySeekerSpider
User-agent: Yandex
User-agent: YandexBot
User-agent: YandexMobileBot
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: sogou spider
User-agent: Sogou web spider
User-agent: Baiduspider
User-agent: YoudaoBot
User-agent: MJ12bot
User-agent: spbot
User-agent: 360Spider
User-agent: SeznamBot
User-Agent: AhrefsBot
User-Agent: Linkdex
User-agent: linkdexbot
User-agent: SemrushBot
User-agent: SemrushBot-SA
User-agent: CFNetwork
User-agent: SputnikBot
User-agent: TurnitinBot
User-agent: proximic
User-agent: grapeshot
User-agent: VigLink
User-agent: Clickagy
User-agent: YaK
User-agent: DeuSu
User-agent: Taboolabot
User-agent: Pinterestbot
User-agent: psbot
User-agent: Plukkie
User-agent: GetIntent
User-agent: Facebot
User-agent: linkdexbot
User-agent: Vegi
User-agent: istellabot
User-agent: Leikibot
User-agent: Claritybot
User-agent: PiplBot
User-agent: Feedspotbot
User-agent: moatbot
User-agent: Yeti
User-agent: contxbot
User-agent: SEOkicks-Robot
User-agent: rogerbot
User-agent: Discordbot
User-agent: Tapatalk
User-agent: Mail.RU_Bot
User-agent: coccocbot
User-agent: HybridBot
User-agent: Slack-ImgProxy
User-agent: adbeat_bot
User-agent: Kraken
User-agent: MojeekBot
User-agent: calculon
User-agent: mcsbot
User-agent: avkzarabotok
User-agent: SearchmetricsBot
User-agent: Uptimebot
User-agent: yacybot
User-agent: Applebot
User-agent: Gigabot
User-agent: g2reader-bot
User-agent: Seoterritory
User-agent: AhrefsBot
User-agent: feedbot
User-agent: Findxbot
User-agent: Wotbox
User-agent: SMTBot
User-agent: Ocarinabot
User-agent: Dragonbot
User-agent: Zombiebot
User-agent: metabot.ru
User-agent: RSSMicro
User-agent: _zbot
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: UptimeRobot
User-agent: uipbot
User-agent: ZoominfoBot
User-agent: Flamingo_SearchEngine
User-agent: bot-pge.chlooe.com
User-agent: Girafabot
User-agent: Multiviewbot
User-agent: SafeDNSBot
User-agent: GnowitNewsbot
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Treato-Bot
User-agent: Pvblcbot
User-agent: Laserlikebot
User-agent: trendiction
User-agent: Mediatoolkitbot
User-agent: CharlieBot
User-agent: BUbiNG
User-agent: Superfeedr
User-agent: focusbot
User-agent: Sottopop
User-agent: MaxPointCrawler
User-agent: Nutch-1.10
User-agent: wonderbot
User-agent: zarabotok--doma
User-agent: Cliqzbot
User-agent: DotBot
User-agent: Exabot
User-agent: moatbot
User-agent: Tapatalk
Disallow: /


User-agent: GingerCrawler
Disallow: /

User-agent: nutch
Disallow: /

User-agent: Urlbot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: ScoutJet
Disallow: /

User-agent: 008
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Speedy Spider
Disallow: /

User-agent: Accelobot
Disallow: /

User-agent: dotbot
Disallow: /

User-agent: discobot
Disallow: /

User-agent: AboutUsBot
Disallow: /

User-agent: Vocus
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: plukkie
Disallow: /

User-agent: AhrefsBot
Disallow: /


User-agent: http://ahrefs.com/robot
Disallow: /

User-agent: http://www.genieo.com/webfilter.html
Disallow: /

User-agent: ia_archiver
Disallow: /

User-agent: linkdex.com
Disallow: /

User-agent: linkdex.com/v2.0
Disallow: /

User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])
Disallow: /

User-agent: NetcraftSurveyAgent/1.0; [email protected]
Disallow: /

User-agent: NetcraftSurveyAgent/1.0
Disallow: /

User-agent: NetcraftSurveyAgent
Disallow: /

User-agent: NetcraftSurveyAgent/2.0
Disallow: /

User-agent: SemrushBot/0.95
Disallow: /

User-agent: MJ12bot/v1.4.3
Disallow: /
User-agent: Java/1.6.0_04
Disallow: /
User-agent: JS-Kit URL Resolver, http://js-kit.com/
Disallow: /
User-agent: JS-Kit
Disallow: /
User-agent: NetShelter ContentScan
Disallow: /

User-agent: Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm)
Disallow: /
User-agent: Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1)
Disallow: /
User-agent: Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html)
Disallow: /
User-agent: YYSpider
Disallow: /
User-agent: YisouSpider
Disallow: /

User-agent: PagePeeker2
Disallow: /

User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Disallow: /

User-agent: Ruby
Disallow: /
User-agent: Abonti
Disallow: /
User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Disallow: /
User-agent: SEOkicks
Disallow: /
User-agent: SEOkicks-Robot
Disallow: /
User-agent: Image\ Sucker
Disallow: /


User-agent: SuperBot
Disallow: /

User-agent: Web\ Image\ Collector
Disallow: /
User-agent: Web\ Sucker
Disallow: /

User-agent: UbiCrawler
Disallow: /
User-agent: zao
Disallow: /
User-agent: sitecheck.internetseer.com
Disallow: /
User-agent: Zealbot
Disallow: /
User-agent: MSIECrawler
Disallow: /
User-agent: linko
Disallow: /
User-agent: Microsoft.URL.Control
Disallow: /
User-agent: libwww
Disallow: /
User-agent: ZyBORG
Disallow: /
User-agent: grub-client
Disallow: /
User-agent: k2spider
Disallow: /
User-agent: NPBot
Disallow: /


Crawl-delay: 10
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Disallow: /xmlrpc/

Search preview

www.horsebase.de
Pferde kaufen, Pferde verkaufen und Pferde suchen - Pferdezubehör -...
Pferde kaufen und Pferde verkaufen kann so einfach sein. Präsentieren Sie Ihr Pferd in unserem Pferdemarkt über 200.000 Besuchern pro Monat. Pferdekauf und Pferdeverkauf einfach, gratis und professionell mit einem internationalen Publikum und hohem Bekanntheitsgrad.

Most important keywords

Following keywords were found. You can check the keyword optimization of this page for each keyword.

KeywordResultRecheck
Pferdemarkt76%Check
HorseBase Pferdemarkt72%Check
Pferde69%Check
verkaufen68%Check
Kauf und Verkauf67%Check
Pferde kaufen65%Check
Pferde verkaufen65%Check
verkaufen und Pferde64%Check
HorseBase62%Check
zu verkaufen61%Check

Test up to 1.000 webpages of horsebase.de with our free plan!

Try for free
No trial. It's just free!

Cookie Policy

We use cookies to make our site work and also for analytics and advertising purposes. You can enable or disable optional cookies as desired. See the following links for more information.

We need these so the site can function properly

So we can better understand how visitors use our website

So we can serve you tailored ads and promotions