Muestra a tus visitantes lo bien optimizado que está tu sitio web con el widget de Seobility. La herramienta presenta la puntuación SEO actual de tu Homepage.
User-agent: WebReaper User-agent: AlphaSeoBot User-agent: AlphaSeoBot-SA User-agent: WebCopier User-agent: Offline Explorer User-agent: HTTrack User-agent: Microsoft.URL.Control User-agent: EmailCollector User-agent: penthesilea User-agent: MJ12bot User-Agent: Topsy User-agent: spbot User-agent: dotbot User-agent: SeznamBot Disallow: / User-agent: OrangeBot Disallow: / User-agent: Slurp Disallow: / User-agent: JikeSpider Disallow: / User-agent: Sosospider Disallow: / User-agent: Sogou Orion spider Disallow: / User-agent: Sogou News Spider Disallow: / User-agent: Sogou blog Disallow: / User-agent: Sogou spider2 Disallow: / User-agent: Sogou inst spider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-Agent: Linkdexbot Disallow: / User-Agent: Linkdexbot/2.0 Disallow: / User-Agent: Linkdexbot/2.1 Disallow: / User-Agent: Linkdexbot Disallow: / User-agent: Sogou web spider Sogou web spider/4.0 Disallow: / User-agent: sogou web spider Disallow: / User-Agent: MJ12bot Disallow: / User-Agent: Opoint Disallow: / User-agent: 008 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: MS Search 4.0 Robot Disallow: / User-agent: MS Search 5.0 Robot Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 5.0 Robot) Disallow: / User-agent: ClarityDailyBot Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; ClarityDailyBot) Disallow: / User-agent: Mozilla/5.0 (compatible; 200PleaseBot/1.0; +http://www.200please.com/bot) Disallow: / User-agent: PAD-bot/9.0 Disallow: / User-agent: PAD-bot Disallow: / User-agent: Mozilla/5.0 (compatible; SoftListBot/2.2; +http://www.softlist.us/) Disallow: / User-agent: SoftListBot/2.2 Disallow: / User-agent: SoftListBot Disallow: / User-agent: Diffbot Disallow: / User-agent: NativeHost Disallow: / User-agent: newsme/1.0; [email protected] Disallow: / User-agent: newsme/1.0 Disallow: / User-agent: newsme Disallow: / User-agent: Niki-Bot Disallow: / User-agent: NING/1.0 Disallow: / User-agent: NING Disallow: / User-agent: peerindex Disallow: / User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) Disallow: / User-agent: Diffbot/0.1 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: Mozilla/5.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: Mozilla/5.0 (compatible; MSIE 7.0; Windows NT; MS Search 4.0 Robot) Disallow: / User-agent: elefent/1.2 (web crawler; crawler.elefent.net; webmaster at elefent dot net) Disallow: / User-agent: elefent Disallow: / User-agent: elefent/1.2 Disallow: / User-agent: crawler.elefent.net Disallow: / User-agent: MS Search 4.0 Robot Disallow: / User-agent: MS Search 4.0 Disallow: / User-agent: MS Search 5.0 Disallow: / User-agent: MS Search 6.0 Disallow: / User-agent: MS Search 7.0 Disallow: / User-agent: MS Search Disallow: / User-agent: CCBot/2.0 (http://commoncrawl.org/faq/) Disallow: / User-agent: CCBot/2.0 Disallow: / User-agent: CCBot Disallow: / User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots) Disallow: / User-agent: rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected]) Disallow: / User-agent: Mail.RU_Bot/Robots/2.0 Disallow: / User-agent: AOL_Daniel_Clark_Spider/Nutch-0.9 (AOL Search; [email protected]) Disallow: / User-agent: AolOnScraper/1.0 Disallow: / User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html; gzip) Disallow: / User-agent: Mozilla/5.0 (compatible; XilopixBot/0.1 +http://www.xilopix.com/bot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/) Disallow: / User-agent: Apache Sites Search Facet Disallow: / User-agent: bigdata Disallow: / User-agent: Fast corporate crawler Disallow: / User-agent: 008 Disallow: / User-agent: magpie-crawler Disallow: / User-agent: magpie-crawler/1.1 Disallow: / User-agent: MagpieRSS Disallow: / User-agent: psnsearch Disallow: / User-agent: SLCC2 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: asterias Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: Black Hole Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: BotALot Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: GuzzleHttp Disallow: / User-agent: Go 1.1 package http Disallow: / User-agent: Go 1.1 Disallow: / User-agent: Go 1.2 Disallow: / User-agent: Go 1.2 package http Disallow: / User-agent: gocrawl v0.4 Disallow: / User-agent: gocrawl Disallow: / User-agent: Google-HTTP-Java-Client Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: Hatena Star Disallow: / User-agent: PagePeeker.com Disallow: / User-agent: PagePeeker Disallow: / User-agent: _agentname Disallow: / User-agent: agentname Disallow: / User-agent: 1 in Business Networking, www.MarcusWalker.ws Disallow: / User-agent: Business Networking Disallow: / User-agent: AntiOpensourceBot/1.1 Disallow: / User-agent: AntiOpensourceBot Disallow: / User-agent: antibot-V1.1.5/i586-linux-2.2 Disallow: / User-agent: Anti-XBOCT/2007 [xxx] (OS; X; JU) Disallow: / User-agent: AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; [email protected]) Disallow: / User-agent: Anti-XBOCT Disallow: / User-agent: AntBot Disallow: / User-agent: Ant-Nutch Disallow: / User-agent: AntBot/1.0 Disallow: / User-agent: AnzwersCrawl/2.0 Disallow: / User-agent: Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966) Disallow: / User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) Disallow: / User-agent: Wotbox/2.01 (+http://www.wotbox.com/bot/) Disallow: / User-agent: Wotbox Disallow: / User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html) Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SEOkicks Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPicker /1.0 Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-Agent: litefinder Disallow: / User-agent: Friendica Disallow: / User-agent: Flamingo_SearchEngine Disallow: / User-agent: Flamingo Disallow: / User-agent: Faraday v0.8.8 Disallow: / User-agent: Faraday Disallow: / User-agent: Barkrowler 0.1.6 Disallow: / User-agent: Barkrowler Disallow: / User-Agent: ExperiBot_v1 Disallow: / User-Agent: ExperiBot Disallow: / User-agent: MJ12bot Disallow: / User-Agent: omgilibot Disallow: / User-Agent: omgili Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: EroCrawler Disallow: / User-agent: Cliqzbot Disallow: / User-agent: ExtractorPro Disallow: / User-agent: FairAd Client Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: Foobot Disallow: / User-agent: Gaisbot Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: hloader Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: YahooFeedSeeker/2.0+(compatible;+Mozilla+4.0;+MSIE+5.5;+http://publisher.yahoo.com/rssguide) Disallow: / User-agent: YahooFeedSeeker/2.0+ Disallow: / User-agent: YahooFeedSeeker Disallow: / User-agent: humanlinks Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: JennyBot Disallow: / User-agent: Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html) Disallow: / User-agent: ExtLinksBot/1.5 Disallow: / User-agent: ExtLinksBot Disallow: / User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) Disallow: / User-agent: GrapeshotCrawler/2.0 Disallow: / User-agent: GrapeshotCrawler/2.0 Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: larbin Disallow: / User-agent: LexiBot Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: Mata Hari Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Mister PiX Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: NCBot Disallow: / User-agent: Testbot/1.0 (compatible; Testbot/1.0; +http://www.testbot.com/bot.html) Disallow: / User-agent: Testbot/1.0 Disallow: / User-agent: Testbot Disallow: / User-agent: Bot !Susie Disallow: / User-agent: Slurp Disallow: / User-agent: !Susie Disallow: / User-agent: Slurp Disallow: / User-agent: Teoma Disallow: / User-agent: twiceler Disallow: / User-agent: Gigabot Disallow: / User-agent: Scrubby Disallow: / User-agent: Robozilla Disallow: / User-agent: Nutch Disallow: / User-agent: naverbot Disallow: / User-agent: yeti Disallow: / User-agent: yahoo-mmcrawler Disallow: / User-agent: psbot Disallow: / User-agent: yahoo-blogs/v3.9 Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/) Disallow: / User-agent: GroupHigh/1.0 Disallow: / User-agent: G-i-g-a-b-o-t Disallow: / User-agent: SeznamBot/3.2 Disallow: / User-agent: SeznamBot Disallow: / User-agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/) Disallow: / User-agent: Helloo/1.0.4 Disallow: / User-agent: Mozilla/5.0 (compatible; Helloo/1.0.4; +http://www.helloo.io/;) Disallow: / User-agent: SiteExplorer/1.1b Disallow: / User-agent: Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/) Disallow: / User-agent: Mail.RU_Bot/Robots Disallow: / User-agent: attach Disallow: / User-agent: ASPSeek Disallow: / User-agent: appie Disallow: / User-agent: AbachoBOT Disallow: / User-agent: autoemailspider Disallow: / User-agent: anarchie Disallow: / User-agent: antibot Disallow: / User-agent: asterias Disallow: / User-agent: B2w Disallow: / User-agent: BackWeb Disallow: / User-agent: BackDoorBot Disallow: / User-agent: Bandit Disallow: / User-agent: BatchFTP Disallow: / User-agent: Black\ Hole Disallow: / User-agent: Baidu Disallow: / User-agent: BlowFish Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bot\ mailto Disallow: / User-agent: BotALot Disallow: / User-agent: Buddy Disallow: / User-agent: Bullseye Disallow: / User-agent: bumblebee Disallow: / User-agent: BunnySlippers Disallow: / User-agent: ClariaBot Disallow: / User-agent: curl Disallow: / User-agent: clsHTTP Disallow: / User-agent: ChinaClaw Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: Crescent Disallow: / User-agent: CherryPickerSE Disallow: / User-agent: CherryPickerElite Disallow: / User-agent: Collector Disallow: / User-agent: COAST\ WebMaster Disallow: / User-agent: cosmos Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: ColdFusion Disallow: / User-agent: Copier Disallow: / User-agent: Crescent Disallow: / User-agent: DA Disallow: / User-agent: DTS\ Agent Disallow: / User-agent: DISCo\ Pump Disallow: / User-agent: DittoSpyder Disallow: / User-agent: Diamond Disallow: / User-agent: Download\ Demon Disallow: / User-agent: Download\ Wonder Disallow: / User-agent: Downloader Disallow: / User-agent: dloader Disallow: / User-agent: Drip Disallow: / User-agent: eCatch Disallow: / User-agent: EirGrabber Disallow: / User-agent: Express\ WebPictures Disallow: / User-agent: Extreme\ Picture\ Finder Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: EasyDL Disallow: / User-agent: EirGrabber Disallow: / User-agent: EroCrawler Disallow: / User-agent: ExtractorPro Disallow: / User-agent: EyeNetIE Disallow: / User-agent: FAST\ WebCrawler Disallow: / User-agent: masscan/1.0 (https://github.com/robertdavidgraham/masscan) Disallow: / User-agent: masscan/1.0 Disallow: / User-agent: masscan Disallow: / User-agent: semantic-visions.com crawler Disallow: / User-agent: MJ12bot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Yandex Disallow: / User-agent: ADmantX Platform Semantic Analyzer Disallow: / User-agent: proximic Disallow: / User-agent: CCBot Disallow: / User-agent: DeuSu Disallow: / User-agent: SMTBot Disallow: / User-agent: GetintentCrawler Disallow: / User-agent: Exabot Disallow: / User-agent: XoviBot Disallow: / User-agent: zgrab Disallow: / User-agent: Pigafetta Disallow: / User-agent: plukkie Disallow: / User-agent: postano Disallow: / User-agent: Prlog Disallow: / User-agent: PulseCrawler/1.1 Disallow: / User-agent: QuerySeekerSpider Disallow: / User-agent: quipu Disallow: / User-agent: Raven Disallow: / User-agent: RavenCrawler Disallow: / User-agent: RBot Disallow: / User-agent: Recorded Future Disallow: / User-agent: Riddler Disallow: / User-agent: Ruby Disallow: / User-agent: RukiCrawler Disallow: / User-agent: SafeDNSBot Disallow: / User-agent: Scrapy Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Searchie Disallow: / User-agent: semantic-visions.com crawler Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: seocharger-robot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SEOlyticsCrawler Disallow: / User-agent: seoscanners Disallow: / User-agent: seoscanners.net Disallow: / User-agent: ShowyouBot (http://showyou.com/crawler) Disallow: / User-agent: simplecrawler Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: SiteExplorer Disallow: / User-Agent: siteluxbot Disallow: / User-agent: SmeshBot Disallow: / User-agent: SMTBot Disallow: / User-agent: srmse/Nutch Disallow: / User-agent: SoftListBot Disallow: / User-agent: spbot Disallow: / User-agent: squirrly Disallow: / User-agent: ssearch_bot Disallow: / User-agent: Stratagems Kumo Disallow: / User-agent: tbot-nutch Disallow: / User-agent: TWMBot Disallow: / User-agent: Typhoeus - https://github.com/typhoeus/typhoeus Disallow: / User-agent: uipbot Disallow: / User-agent: uMBot-LN Disallow: / User-agent: UptimeRobot/2.0 Disallow: / User-agent: WBSearchBot Disallow: / User-agent: WebIndex Disallow: / User-agent: VeBot Disallow: / User-agent: VeriCiteCrawler Disallow: / User-agent: WeSEE_Bot Disallow: / User-agent: WiseGuys Robot Disallow: / User-agent: voltron Disallow: / User-agent: woobot Disallow: / User-agent: woriobot Disallow: / User-agent: wotbox Disallow: / User-agent: X-CAD-SE Disallow: / User-agent: XoviBot Disallow: / User-agent: yacybot Disallow: / User-agent: Yahoo Pipes 1.0 Disallow: / User-agent: YioopBot Disallow: / User-agent: crawler.vs.uni-duisburg-essen.de Disallow: / User-agent: BLEXBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: FileHound Disallow: / User-agent: Fetch\ API\ Request Disallow: / User-agent: FlashGet Disallow: / User-agent: FlickBot Disallow: / User-agent: FrontPage Disallow: / User-agent: FreeFind.com Disallow: / User-agent: GetRight Disallow: / User-agent: GetSmart Disallow: / User-agent: Generic Disallow: / User-agent: Go!Zilla Disallow: / User-agent: Go-Ahead-Got-It Disallow: / User-agent: gotit Disallow: / User-agent: Grabber Disallow: / User-agent: GrabNet Disallow: / User-agent: Grafula Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest Disallow: / User-agent: HMView Disallow: / User-agent: Heretrix Disallow: / User-agent: HitboxDoctor Disallow: / User-agent: HTTPapp Disallow: / User-agent: HTTrack Disallow: / User-agent: HTTPTrack Disallow: / User-agent: HTTPviewer Disallow: / User-agent: httplib Disallow: / User-agent: httpfetcher Disallow: / User-agent: httpscraper Disallow: / User-agent: hloader Disallow: / User-agent: humanlinks Disallow: / User-agent: ia_archiver Disallow: / User-agent: InterGET Disallow: / User-agent: Internet\ Ninja Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InternetSeer.com Disallow: / User-agent: Iria Disallow: / User-agent: IRLbot Disallow: / User-agent: JetCar Disallow: / User-agent: JOC Disallow: / User-agent: JOC\ Web\ Spider Disallow: / User-agent: JoBo Disallow: / User-agent: Java Disallow: / User-agent: JustView Disallow: / User-agent: Jonzilla Disallow: / User-agent: JennyBot Disallow: / User-agent: Kenjin\ Spider Disallow: / User-agent: Keyword\ Density Disallow: / User-agent: larbin Disallow: / User-agent: LeechFTP Disallow: / User-agent: Lachesis Disallow: / User-agent: LexiBot Disallow: / User-agent: libWeb Disallow: / User-agent: Libby_ Disallow: / User-agent: LinkScan Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: lftp Disallow: / User-agent: likse Disallow: / User-agent: Link Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp\ request Disallow: / User-agent: Magnet Disallow: / User-agent: Mag-Net Disallow: / User-agent: Mass\ Downloader Disallow: / User-agent: MIIxpc Disallow: / User-agent: Missigua Disallow: / User-agent: Mewsoft\ Search\ Engine Disallow: / User-agent: moget Disallow: / User-agent: Mata\ Hari Disallow: / User-agent: Memo Disallow: / User-agent: Metacarta Disallow: / User-agent: Mercator Disallow: / User-agent: MIDown\ tool Disallow: / User-agent: MFC_Tear_Sample Disallow: / User-agent: Mirror Disallow: / User-agent: MIIxpc Disallow: / User-agent: Mister\ PiX Disallow: / User-agent: NationalDirectory\ WebSpider Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Nikto Disallow: / User-agent: Navroad Disallow: / User-agent: NearSite Disallow: / User-agent: NetAnts Disallow: / User-agent: NetSpider Disallow: / User-agent: NICErsPRO Disallow: / User-agent: NetResearchServer Disallow: / User-agent: NetMechanic Disallow: / User-agent: Net\ Vampire Disallow: / User-agent: Net\ Probe Disallow: / User-agent: NetZip Disallow: / User-agent: nexuscache Disallow: / User-agent: Ninja Disallow: / User-agent: NPBot Disallow: / User-agent: our\ agent Disallow: / User-agent: onestop Disallow: / User-agent: oBot Disallow: / User-agent: Octopus Disallow: / User-agent: Offline\ Explorer Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind\ data\ gatherer Disallow: / User-agent: PageGrabber Disallow: / User-agent: Papa\ Foto Disallow: / User-agent: Xaldon\ WebSpider Disallow: / User-agent: Xenu Disallow: / User-agent: Xara Disallow: / User-agent: Y!TunnelPro Disallow: / User-agent: YahooYSMcm Disallow: / User-agent: Zade Disallow: / User-agent: ZBot Disallow: / User-agent: Zeus Disallow: / User-agent: Rogerbot Disallow: / User-agent: Exabot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Dotbot Disallow: / User-agent: Gigabot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Xaldon\ WebSpider Disallow: / User-agent: Screaming Frog SEO Spider/8.1 Disallow: / User-agent: DoBot Disallow: / User-agent: Laserlikebot Disallow: / User-agent: oBot Disallow: / User-agent: Neliveto Disallow: / User-Agent: ahrefs Disallow: / User-Agent: AHC/1.0 Disallow: / User-Agent: AHC Disallow: / User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Disallow: / User-Agent: Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot ) Disallow: / User-agent: evc-batch/2.0 Disallow: / User-agent: Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html) Disallow: / User-agent: SemrushBot/1.2~bl Disallow: / User-agent: evc-batch Disallow: / User-agent: 007ac9 Crawler disallow: / User-agent: 1 2 3 Submit PRO disallow: / User-agent: 200PleaseBot disallow: / User-agent: 2ADAMbot disallow: / User-agent: 2ADAMbot/1.0 disallow: / User-agent: 360Spider disallow: / user-agent: Abonti disallow: / user-agent: Abonti/0.92 disallow: / user-agent: abot v1.0 disallow: / user-agent: aboutthedomain disallow: / user-agent: Add Catalog disallow: / user-agent: Add Catalog/2.1 disallow: / user-agent: AdvBot disallow: / user-agent: AdvBot/2.0 disallow: / user-agent: AhrefsBot disallow: / user-agent: Ahrefs-Bot disallow: / user-agent: AhrefsBot/1.0 disallow: / user-agent: Ahrefs-Bot/1.0 disallow: / user-agent: Ahrefs-Bot/2.0 disallow: / user-agent: Ahrefs-Bot/3.0 disallow: / user-agent: Ahrefs-Bot/4.0 disallow: / user-agent: Ahrefs-Bot/5.0 disallow: / user-agent: aiHitBot disallow: / user-agent: aiHitBot/2.9 disallow: / user-agent: Anonymous/0.0 disallow: / user-agent: Arachnida disallow: / user-agent: Associative Spider disallow: / User-agent: Baiduspider disallow: / User-agent: Baidu Spider disallow: / User-agent: Battleztar Bazinga disallow: / User-agent: Battleztar Bazinga/0.01 disallow: / User-agent: BDFetch disallow: / User-agent: Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com Disallow: / User-agent: ZmEu Disallow: / User-agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html) Disallow: / User-agent: betaBot disallow: / User-agent: bieshu disallow: / User-agent: Bigli SEO disallow: / User-agent: Blackboard Safeassign disallow: / User-agent: Blazer 1.0 disallow: / User-agent: BLEXBot disallow: / User-agent: BLEXBot/1.0 disallow: / User-agent: BLP_bbot disallow: / User-agent: BLP_bbot/0.1 disallow: / User-agent: BOIA-Accessibility-Agent/PR 1.0 disallow: / User-agent: BOT for JCE disallow: / User-agent: BOT/0.1 (BOT for JCE) disallow: / User-agent: BPImageWalker disallow: / User-agent: BPImageWalker/2.0 disallow: / User-agent: BUbiNG disallow: / User-agent: BuiBui-Bot disallow: / User-agent: BuiBui-Bot/1.0 disallow: / User-agent: ca-crawler disallow: / User-agent: ca-crawler/1.0 disallow: / User-agent: CakePHP disallow: / User-agent: Calypso v/0.01 disallow: / User-agent: Calypso disallow: / User-agent: CB/Nutch-1.7 disallow: / User-agent: CCBot disallow: / User-agent: CCBot/2.0 disallow: / User-agent: Checkbot disallow: / User-agent: checkgzipcompression.com disallow: / User-agent: chushou disallow: / User-agent: CloudServerMarketSpider disallow: / User-agent: CloudServerMarketSpider/1.0 disallow: / User-agent: Clushbot/3.x-BinaryFury disallow: / User-agent: CMS Crawler disallow: / User-agent: CMS Crawler: http://www.cmscrawler.com disallow: / User-agent: coccoc disallow: / User-agent: CoinCornerBot disallow: / User-agent: CoinCornerBot/1.1 disallow: / User-agent: Copyscape disallow: / User-agent: crawler4j disallow: / User-agent: CRAZYWEBCRAWLER 0.9.0 disallow: / User-agent: CRAZYWEBCRAWLER 0.9.1 disallow: / User-agent: CRAZYWEBCRAWLER 0.9.7 disallow: / User-agent: CrazyWebCrawler disallow: / User-agent: CrazyWebCrawler-Spider disallow: / User-agent: Crowsnest disallow: / User-agent: Crowsnest/0.5 disallow: / User-agent: Curious George - www.analyticsseo.com/crawler disallow: / User-agent: Curious George disallow: / User-agent: cuwhois disallow: / User-agent: cuwhois/1.0 disallow: / User-agent: dahoms disallow: / User-agent: datagnionbot disallow: / User-agent: DeuSu/5.0.2 disallow: / User-agent: Digincore disallow: / User-agent: Digincore bot disallow: / User-agent: Dispatch/0.11.0 disallow: / User-agent: Domain Re-Animator Bot disallow: / User-agent: DomainAppender /1.0 disallow: / User-agent: DomainAppender disallow: / User-agent: DomainCrawler/3.0 disallow: / User-agent: DomainSigmaCrawler disallow: / User-agent: DomainSigmaCrawler/0.1 disallow: / User-agent: Domnutch disallow: / User-agent: Domnutch-Bot disallow: / User-agent: Domnutch-Bot/Nutch disallow: / User-agent: Domnutch-Bot/Nutch-1.0 disallow: / User-agent: dotbot disallow: / User-agent: ECCP/1.2.1 disallow: / User-agent: eCommerceBot disallow: / User-agent: enlle punto com/Nutch-1.9 disallow: / User-agent: EPiServer Link Checker disallow: / User-agent: EuripBot disallow: / User-agent: EuripBot/2.0 disallow: / User-agent: evc/2.0 disallow: / User-agent: evc-batch disallow: / User-agent: evc-batch/2.0 disallow: / User-agent: Express WebPictures disallow: / User-agent: Faraday v0.8.8 disallow: / User-agent: Faraday disallow: / User-agent: Findxbot disallow: / User-agent: Findxbot/1.0 disallow: / User-agent: Flamingo_SearchEngine disallow: / User-agent: Flipboard Robot disallow: / User-agent: GetProxi.es-bot disallow: / User-agent: GetProxi.es-bot/1.1 disallow: / User-agent: GigablastOpenSource disallow: / User-agent: GigablastOpenSource/1.0 disallow: / User-agent: Girafabot disallow: / User-agent: Gluten Free Crawler disallow: / User-agent: Gluten Free Crawler/1.0 disallow: / User-agent: GriffinBot disallow: / User-agent: GrifinBot/0.01 disallow: / User-agent: GWPImages disallow: / User-agent: GWPImages/1.0 disallow: / User-agent: Haiula disallow: / User-agent: Haiula/1.4 disallow: / User-agent: HaosouSpider disallow: / User-agent: Hivemind disallow: / User-agent: HostHarvest disallow: / User-agent: HostHarvest/0.4.28 disallow: / User-agent: HRCrawler disallow: / User-agent: HRCrawler/2.0 disallow: / User-agent: http://git.io/tl_S2w disallow: / User-agent: http://www.checkprivacy.or.kr:6600/RS/PRIVACY_ENFAQ.jsp disallow: / User-agent: HubSpot Links Crawler 1.0 disallow: / User-agent: HubSpot Webcrawler disallow: / User-agent: HubSpot disallow: / User-agent: hunchan disallow: / User-agent: HyperCrawl disallow: / User-agent: HyperCrawl/0.2 disallow: / User-agent: ICAP-IOD disallow: / User-agent: ICC-Crawler disallow: / User-agent: ICC-Crawler/2.0 disallow: / User-agent: Ichiro Robot disallow: / User-agent: image.coccoc/1.0 disallow: / User-agent: Image2play disallow: / User-agent: Image2play/0.1 disallow: / User-agent: Indy Library disallow: / User-agent: InsightsCollector disallow: / User-agent: InsightsCollector/0.1 disallow: / User-agent: InsightsCollector/0.1beta disallow: / User-agent: integrity/5 disallow: / User-agent: InterNaetBoten disallow: / User-agent: InterNaetBoten/0.99 disallow: / User-agent: Barkrowler/0.7 (+http://www.exensa.com/crawl) disallow: / User-agent: Barkrowler disallow: / User-agent: IRL Crawler disallow: / User-agent: James BOT - WebCrawler disallow: / User-agent: James BOT disallow: / User-agent: JamesBOT disallow: / User-agent: JetBrains 5.0 disallow: / User-agent: JetBrains disallow: / User-agent: Kraken disallow: / User-agent: Kraken/0.1 disallow: / User-agent: Kyoto-Tohoku-Crawler/v1 disallow: / User-agent: larbin disallow: / User-agent: lechenie disallow: / User-agent: libwww-perl disallow: / User-agent: link checker disallow: / User-agent: Link/1.0 disallow: / User-agent: linkCheck disallow: / User-agent: linkCheckV3.0 disallow: / User-agent: Linkdex disallow: / User-agent: linkdex.com/v2.0 disallow: / User-agent: linkdex.com/v2.1 disallow: / User-agent: LinkdexBot disallow: / User-agent: linkdexbot/2.0 disallow: / User-agent: linkdexbot/2.1 disallow: / User-agent: linkdexbot-mobile/2.1 disallow: / User-agent: LinkpadBot disallow: / User-agent: LinkpadBot/1.06 disallow: / User-agent: LinqiaScrapeBot disallow: / User-agent: LinqiaScrapeBot/1.0 disallow: / User-agent: Lipperhey SEO Service disallow: / User-agent: Lipperhey disallow: / User-agent: Lipperhey-Kaus-Australis disallow: / User-agent: Lipperhey-Kaus-Australis/5.0 disallow: / User-agent: listicka disallow: / User-agent: LSSRocketCrawler disallow: / User-agent: LSSRocketCrawler/1.0 LightspeedSystems disallow: / User-agent: LSSRocketCrawler/1.0 disallow: / User-agent: ltx71 disallow: / User-agent: LWNutch/Nutch-1.4 disallow: / User-agent: Mail.RU disallow: / User-agent: Mail.RU_Bot disallow: / User-agent: Mail.RU_Bot/2.0 disallow: / User-agent: Mail.RU_Bot/Fast/2.0 disallow: / User-agent: md5sum disallow: / User-agent: md5sum\x22 disallow: / User-agent: meanpathbot disallow: / User-agent: MegaIndex.ru disallow: / User-agent: MegaIndex.ru/2.0 disallow: / User-agent: mezhpozvonochnoi disallow: / User-agent: Mike-Crawler disallow: / User-agent: MixBot disallow: / User-agent: MixrankBot disallow: / User-agent: MJ12bot disallow: / User-agent: Monkeybot/0.1 disallow: / User-agent: my crawler disallow: / User-agent: My Nutch Spider/Nutch-1.9 disallow: / User-agent: mycrowl/Nutch-1.9 disallow: / User-agent: MyGreatUA/2.0 disallow: / User-agent: MyIPTest disallow: / User-agent: NameProtect Robot disallow: / User-agent: NerdyBot disallow: / User-agent: Netcraft Spider disallow: / User-agent: netEstate NE Crawler disallow: / User-agent: NetLyzer FastProbe disallow: / User-agent: NetResearchServer disallow: / User-agent: NetResearchServer/4.0 disallow: / User-agent: Nmap Scripting Engine disallow: / User-agent: node.io disallow: / User-agent: node.js disallow: / User-agent: Node/simplecrawler 0.5.2 disallow: / User-agent: Node/simplecrawler disallow: / User-agent: oBot/2.3.1 disallow: / User-agent: omgilibot disallow: / User-agent: omgilibot/0.4 disallow: / User-agent: Online Domain Tools - Online Website Link Checker disallow: / User-agent: Online Domain Tools - Online Website Link Checker/1.2 disallow: / User-agent: Openfind Robot disallow: / User-agent: OpenHoseBot disallow: / User-agent: OpenHoseBot/2.1 disallow: / User-agent: Openstat disallow: / User-agent: Openstat/0.1 disallow: / User-agent: OptimizationCrawler disallow: / User-agent: OptimizationCrawler/0.2 disallow: / User-agent: Page Analyzer v4.0 disallow: / User-agent: Page Analyzer disallow: / User-agent: PageAnalyzer disallow: / User-agent: PageAnalyzer/1.1 disallow: / User-agent: PageAnalyzer/1.5 disallow: / User-agent: PagesInventory disallow: / User-agent: Pagespeed/1.1 Fetcher disallow: / User-agent: Pagespeed/1.1 disallow: / User-agent: Pagespeedbot disallow: / User-agent: Perl LWP disallow: / User-agent: PHPCrawl disallow: / User-agent: phpSiteCheck 1.0 disallow: / User-agent: phpSiteCheck disallow: / User-agent: Plukkie disallow: / User-agent: POGS/2.0 disallow: / User-agent: Powermarks disallow: / User-agent: PowerPivot disallow: / User-agent: PRIVACY_ENFAQ.jsp disallow: / User-agent: Prlog disallow: / User-agent: Prlog/1.0 disallow: / User-agent: publiclibraryarchive.org disallow: / User-agent: publiclibraryarchive.org/1.0 disallow: / User-agent: Pu_iN Crawler disallow: / User-agent: Putin disallow: / User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0 Disallow: / User-agent: Putin spider disallow: / User-agent: qingdao disallow: / User-agent: QlikView disallow: / User-agent: quipu disallow: / User-agent: quipu/1.0 disallow: / User-agent: quipu/2.0 disallow: / User-agent: R6_CommentReader disallow: / User-agent: R6_FeedFetcher disallow: / User-agent: Riddler disallow: / User-agent: RivalSeek.com-Bot disallow: / User-agent: rogerbot disallow: / User-agent: rogerbot/1.0 disallow: / User-agent: rootlink disallow: / User-agent: RU_Bot/2.0 disallow: / User-agent: Scopia disallow: / User-agent: Scopia crawler disallow: / User-agent: Scopia crawler 1.0 disallow: / User-agent: Scopia crawler 1.1 disallow: / User-agent: Scopia crawler 1.2 disallow: / User-agent: Scrapy disallow: / User-agent: Scrapy/0.16.5 disallow: / User-agent: Scrapy/0.24.4 disallow: / User-agent: Scrapy/0.24.5 disallow: / User-agent: Scrapy/0.24.6 disallow: / User-agent: Scrapy/1.0.1 disallow: / User-agent: Screaming Frog SEO Spider disallow: / User-agent: Screaming Frog SEO Spider/2,55 disallow: / User-agent: Screaming Frog SEO Spider/2.55 disallow: / User-agent: Screaming Frog SEO Spider/3.1 disallow: / User-agent: Screaming Frog SEO Spider/3.3 disallow: / User-agent: Screaming Frog SEO Spider/4.1 disallow: / User-agent: Screaming Frog SEO Spider/5.0 disallow: / User-agent: Screaming Frog SEO Spider/5.1 disallow: / User-agent: Screaming Frog SEO Spider/5.1 Beta 2 disallow: / User-agent: scrutiny/4 disallow: / User-agent: SemrushBot disallow: / User-agent: SemrushBot-SA disallow: / User-agent: SEOdiver/1.0 disallow: / User-agent: SEOkicks disallow: / User-agent: SEOkicks-Robot disallow: / User-agent: SEOlyticsCrawler disallow: / User-agent: SEOlyticsCrawler/3.0 disallow: / User-agent: seoscanners disallow: / User-agent: seoscanners.net/1 disallow: / User-agent: SEOstats 2.1.0 disallow: / User-agent: Seosys/Nutch-2.3 disallow: / User-agent: SetCronJob/1.0 disallow: / User-agent: R6_FeedFetcher_(www.radian6.com/crawler) disallow: / User-agent: SeznamBot disallow: / User-agent: SheerBoredom.Experimental.Robot disallow: / User-agent: SheerBoredom.Experimental.Robot/0.2 disallow: / User-agent: ShowyouBot disallow: / User-agent: Simplecrawler disallow: / User-agent: SISTRIX Crawler disallow: / User-agent: sistrix disallow: / User-agent: SiteBot disallow: / User-agent: SiteBot/0.1 disallow: / User-agent: SiteExplorer disallow: / User-agent: SiteExplorer/1.0 disallow: / User-agent: SiteExplorer/1.0b disallow: / User-agent: Siteluxbot disallow: / User-agent: Siteluxbot/1.0 disallow: / User-agent: SkimBot disallow: / User-agent: RSSingBot (http://www.rssing.com) disallow: / User-agent: SkimBot/1.0 disallow: / User-agent: sky nutch crawler/Nutch-1.9 disallow: / User-agent: SMTBot disallow: / User-agent: SMTBot/1.0 disallow: / User-agent: SNK Screenshot Bot disallow: / User-agent: SNK Screenshot Bot/0.20 disallow: / User-agent: Sogou Spider disallow: / User-agent: Sogou web spider disallow: / User-agent: SpamBayes disallow: / User-agent: SpamBayes/1.1a3+ disallow: / User-agent: spbot disallow: / User-agent: spbot/4.4.2 disallow: / User-agent: spiderbot disallow: / User-agent: SpiderLing disallow: / User-agent: Spiderbot/Nutch-1.7 disallow: / User-agent: spray-can disallow: / User-agent: spray-can/1.2.1 disallow: / User-agent: SSG/3.0 disallow: / User-agent: Statastico disallow: / User-agent: Statastico/4.0 disallow: / User-agent: Steeler disallow: / User-agent: Steeler/3.5 disallow: / User-agent: Stratagems Kumo disallow: / User-agent: Stratagems disallow: / User-agent: StudioFACA Search disallow: / User-agent: StudioFACA disallow: / User-agent: sukibot disallow: / User-agent: sukibot_heritrix disallow: / User-agent: sukibot_heritrix/3.1.1 disallow: / User-agent: SurveyBot disallow: / User-agent: Synapse disallow: / User-agent: Synthesio Crawler release MonaLisa disallow: / User-Agent: tbot-nutch/Nutch-1.10 disallow: / User-Agent: Traackr.com bot disallow: / User-Agent: trendictionbot disallow: / User-Agent: Trendiction-Bot disallow: / User-Agent: TrueBot disallow: / User-Agent: TrueBot/1.0 disallow: / User-Agent: TulipChain/5.xx disallow: / User-Agent: TWMBot/0.1 disallow: / User-Agent: Typhoeus disallow: / User-Agent: UCMore Crawler App disallow: / User-Agent: uMBot-LN disallow: / User-Agent: uMBot-LN/1.0 disallow: / User-Agent: updown_tester disallow: / User-Agent: URLChecker disallow: / User-agent: V1.0/1.2 disallow: / User-agent: w3af.org disallow: / User-agent: WASALive disallow: / User-agent: WASALive-Bot disallow: / User-agent: vBSEO disallow: / User-agent: WBSearchBot disallow: / User-agent: WBSearchBot/1.1 disallow: / User-agent: WeAreNotEvil disallow: / User-agent: WebAlta disallow: / User-agent: WebAlta Crawler disallow: / User-agent: Web corpus crawler disallow: / User-agent: WebCookies disallow: / User-agent: WebCookies/1.0 disallow: / User-agent: WebCopier vx.xa disallow: / User-agent: Webnest 0.9 disallow: / User-agent: WebQL disallow: / User-agent: Webscout disallow: / User-agent: Webscout/1.0 disallow: / User-agent: Web-sniffer disallow: / User-agent: Web-sniffer/1.1.0 disallow: / User-agent: Webster Pro V3.4 disallow: / User-agent: WebTarantula.com Crawler disallow: / User-agent: WeCrawlForThePeace disallow: / User-agent: VegeBot disallow: / User-agent: Vegi bot disallow: / User-agent: WeLikeLinks disallow: / User-agent: VeriCiteCrawler disallow: / User-agent: VeriCiteCrawler/Nutch-1.9 disallow: / User-agent: WhatWeb disallow: / User-agent: WhatWeb/0.4.8-dev disallow: / User-agent: Visited by http://tools.geek-tools.org disallow: / User-agent: voltron disallow: / User-agent: woobot disallow: / User-agent: woobot/1.1 disallow: / User-agent: woobot/2.0 disallow: / User-agent: Vorboss Web Crawler disallow: / User-agent: Vorboss Web Crawler/Nutch-2.3 disallow: / User-agent: WorldBrewBot disallow: / User-agent: WorldBrewBot/2.1 disallow: / User-agent: worldwebheritage.org disallow: / User-agent: worldwebheritage.org/1.0 disallow: / User-agent: wscheck.com disallow: / User-agent: wscheck.com/1.0.0 disallow: / User-agent: www.deadlinkchecker.com disallow: / User-agent: www.petitsage.fr site detector 0.4 disallow: / User-agent: WWW-Mechanize disallow: / User-agent: WWW-Mechanize/1.74 disallow: / User-agent: Xenu Link Sleuth disallow: / User-agent: Xenu's Link Sleuth disallow: / User-agent: XoviBot disallow: / User-agent: XoviBot/2.0 disallow: / User-agent: XSpider disallow: / User-agent: Yandex Robot disallow: / User-agent: Yandex disallow: / User-agent: Yetibot disallow: / User-agent: YisouSpider disallow: / User-agent: yoozBot disallow: / User-agent: yoozBot-2.2 disallow: / User-agent: zgrab/0.x disallow: / User-agent: zzabmbot disallow: / User-agent: zzabmbot/1.0 disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org) Mozilla/4.0 (compatible; grub-client-2.3) Disallow: / User-agent: Mozilla/3.0 (Slurp/si; [email protected]; http://www.inktomi.com/slurp.html) Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GameSpyHTTP/1.0 Disallow: / User-agent: Baiduspider ( http://www.baidu.com/search/spider.htm) Disallow: / User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) Disallow: / User-Agent: istellabot/t.1.13 Disallow: / User-Agent: MagpieRSS/0.72 (+http://magpierss.sf.net) Disallow: / User-Agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected]) Disallow: / User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Disallow: / User-Agent: Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/) Disallow: / User-Agent: Liferea Disallow: / User-agent: Sogou web spider Disallow: / User-Agent: ahrefsbot Disallow: / User-Agent: Ahrefs Disallow: / User-Agent: Ahrefsbot Disallow: / User-Agent: BoardReader Blog Indexer(http://boardreader.com) Disallow: / User-Agent: BoardReader Disallow: / User-Agent: YOUBot Disallow: / User-Agent: AhrefsBot.com Disallow: / User-Agent: yandexBot Disallow: / User-Agent: Dow Jones Searchbot Disallow: / User-Agent: TurnitinBot Disallow: / User-Agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-Agent: Almaden Disallow: / User-Agent: eCairn-Grabber/1.0 Disallow: / User-Agent: eCairn-Grabber Disallow: / User-Agent: Amazon Disallow: / User-Agent: Gluten Free Crawler/1.0 Disallow: / User-Agent: DotBot/1.1 Disallow: / User-Agent: LinqiaCrawlerBot/1.0 Disallow: / User-Agent: istellabot/t.1.13 Disallow: / User-Agent: nutch-1.4/Nutch-1.4 Disallow: / User-Agent: nutch-1.4 Disallow: / User-Agent: DoBot Disallow: / User-Agent: AsapBot Disallow: / User-agent: dotbot Disallow: / User-agent: coccocbot-web Disallow: / User-Agent: amazonbot Disallow: / User-Agent: Amazon-bot Disallow:/ User-Agent: Meditoolkitbot Disallow: / User-agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot) Disallow: / User-agent: Yeti/1.1 Disallow: / User-agent: Yeti Disallow: / User-agent: Yahoo! Slurp Disallow: / User-agent: HaoSouSpider Disallow: / User-agent: Baiduspider Disallow: / User-agent: Applebot Disallow: / User-agent: Applebot/0.1; +http://www.apple.com/go/applebot Disallow: / User-agent: Applebot/0.1 Disallow: / User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1) Disallow: / User-agent: ia_archiver Disallow: / User-agent: Naverbot Disallow: / User-agent: seznambot Disallow: / User-agent: teoma Disallow: / User-agent: Yeti Disallow: / User-agent: EtaoSpider Disallow: / User-Agent: swebot Disallow: / User-Agent: Newsbrain Disallow: / User-Agent: Feedreader 3.14 Disallow: / User-Agent: Feedreader Disallow: / User-Agent: Feedreader 3.14 (Powered by Newsbrain) Disallow: / User-agent: aihitbot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Thumbnails Disallow: / User-agent: sitebot Disallow: / User-agent: YandexBot Disallow: / User-agent: Yandex Disallow: / User-agent: TwengaBot-Discover Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Ezooms Disallow: / User-agent: Bender Disallow: / User-agent: discobot Disallow: / User-Agent: panscient.com Disallow: / User-Agent: searchwebengine.net Disallow: / User-agent: MLBot Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: Speedy Disallow: / User-agent: dotbot Disallow: / User-agent: NerdByNature.Bot disallow: / User-agent: WBSearchBot Disallow: / User-agent: sistrix Disallow: / User-agent: SindiceBot Disallow: / User-agent: plukkie Disallow: / User-agent: SBIder/SBIder-0.8.2-dev (http://www.sitesell.com/sbider.html) Disallow: / User-agent: SBIder Disallow: / User-agent: IWAgent/1.0 Disallow: / User-agent: IWAgent Disallow: / User-agent: BotALot Disallow:/ User-agent: findfiles.net Disallow: / User-agent: SeznamBot Disallow: / User-agent: Goodzer Disallow: / User-agent: DJEcoBot Disallow: / User-agent: DJEcoBot/1.0 Disallow: / User-agent: YandexBot/3.0 Disallow: / User-agent: AhrefsBot/5.2 Disallow: / User-agent: Sogou inst spider Disallow: / User-agent: Sogou spider2 Disallow: / User-agent: Sogou blog Disallow: / User-agent: Sogou News Spider Disallow: / User-agent: Sogou Orion spider Disallow: / User-agent: JikeSpider Disallow: / User-agent: spray-can/1.3.4 Disallow: / User-agent: spray-can Disallow: / User-agent: spray can Disallow: / User-agent: Sosospider Disallow: / User-agent: 360spider Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: YoudaoBot Disallow: / User-agent: BacklinkCrawler Disallow: / User-agent: lemurwebcrawler Disallow: / User-agent: gigabot Disallow: / User-agent: discobot Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot) Disallow: / User-agent: Mozilla/5.0 (compatible; Cliqzbot/1.0; +http://cliqz.com/company/cliqzbot) Disallow: / User-agent: Cliqzbot Disallow: / User-agent: FAST Enterprise Crawler 6 Disallow: / User-agent: Sensis.com.au Web Crawler Disallow: / User-agent: Baiduspider Disallow: / User-agent: Teoma Disallow: / User-agent: XilopixBot/0.1 Disallow: / User-agent: XilopixBot Disallow: / User-agent: twiceler Disallow: / User-agent: Baiduspider Disallow: / User-agent: Arachmo Disallow: / User-agent: Acoon Disallow: / User-agent: adbeat_bot Disallow: / User-agent: AgentLinkSpammer Disallow: / User-agent: aiHitBot Disallow: / User-agent: Gigabot Disallow: / User-agent: Scrubby Disallow: / User-agent: Robozilla Disallow: / User-agent: ia_archiver Disallow: / User-agent: worio bot heritrix Disallow: / User-agent: trovitBot Disallow: / User-agent: MojeekBot/0.6 Disallow: / User-agent: rogerbot/1.0 Disallow: / User-agent: istellabot/t.1.13 Disallow: / User-agent: SeznamBot/3.2 Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: careerbot Disallow: / User-agent: SogouBot Disallow: / User-agent: Sogou spider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogouspider Disallow: / User-agent: Qihoo360Bot Disallow: / User-agent: EasouSpider Disallow: / User-agent: COMODOSpider Disallow: / User-agent: findlinks Disallow: / User-agent: Ezooms Disallow: / User-agent: DotBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Mail.RU Disallow: / User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html) Disallow: / User-agent: SurdotlyBot/1.0 Disallow: / User-agent: SurdotlyBot Disallow: / User-agent: Apexoo Spider (http://www.apexoo.com/spider/) Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Apexoo Spider 1.1 Disallow: / User-agent: AportCatalogRobot/2.0 Disallow: / User-agent: Apostrophe/0.2.1.93 (compatible) filecache Disallow: / User-agent: Apostrophe Disallow: / User-agent: Aplix HTTP/1.0.1 Disallow: / User-agent: moget Disallow: / User-agent: Ichiro Disallow: / User-agent: Ichiro 3.0 Disallow: / User-agent: NaverBot Disallow: / User-agent: Yeti Disallow: / User-agent: NetcraftSurveyAgent/1.0 Disallow: / User-agent: OpenWebIndex/Nutch-1.6 Disallow: / User-agent: OpenWebIndex Disallow: / User-agent: panoptaStudyBot Disallow: / User-agent: checks.panopta.com Disallow: / User-agent: psbot Disallow: / User-agent: plukkie Disallow: / User-agent: SeznamBot Disallow: / User-agent: SeznamBot/1.0 Disallow: / User-agent: SeznamBot/1.1 Disallow: / #SeznamBot/3.0 User-agent: SeznamBot/3.0 Disallow: / User-agent: SistrixCrawler Disallow: / User-agent: Sistrix Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: SISTRIX Disallow: / User-agent: sogou spider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sosospider+ Disallow: / User-agent: Sosospider Disallow: / User-agent: Sosospider/2.0 Disallow: / User-agent: 360Spider Disallow: / User-agent: SurveyBot Disallow: / User-agent: Wada.vn Disallow: / User-agent: Wada.vn Vietnamese Search Disallow: / User-agent: Wada.vn Vietnamese Search/2.1 Disallow: / User-agent: Yandex Disallow: / User-agent: YandexBot/3.0-MirrorDetector Disallow: / User-agent: YandexImages Disallow: / User-agent: YandexSomething Disallow: / User-agent: Yandex.com Disallow: / User-agent: YandexBot Disallow: / User-agent: YisouSpider Disallow: / User-agent: YoudaoBot/1.0 Disallow: / User-agent: Zao Disallow: / User-agent: naverbot Disallow: / User-agent: yeti Disallow: / User-agent: yahoo-mmcrawler Disallow: / User-agent: psbot Disallow: / User-agent: asterias Disallow: / User-agent: yahoo-blogs Disallow: / User-agent: aipbot Disallow: / User-agent: ia_archiver Disallow: / User-agent: Alexibot Disallow: / User-agent: asterias Disallow:/ User-agent: Aqua_Products Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow:/ User-agent: becomebot Disallow: / User-agent: Black Hole Disallow:/ User-agent: BlowFish Disallow:/ User-agent: Bookmark search tool Disallow: / User-agent: BotALot Disallow: / User-agent: BotRightHere Disallow: / User-agent: BuiltBotTough Disallow:/ User-agent: Bullseye/1.0 Disallow:/ User-agent: BunnySlippers Disallow:/ User-agent: Cegbfeieh Disallow:/ User-agent: CheeseBot Disallow:/ User-agent: CherryPicker Disallow:/ User-agent: CherryPickerElite Disallow:/ User-agent: CherryPickerSE Disallow:/ User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow:/ User-agent: cosmos Disallow:/ User-agent: Crescent Disallow:/ User-agent: DittoSpyder Disallow:/ User-agent: EmailCollector Disallow:/ User-agent: EmailSiphon Disallow:/ User-agent: EmailWolf Disallow:/ User-agent: EroCrawler Disallow:/ User-agent: ExtractorPro Disallow:/ User-agent: FairAd Client Disallow: / User-agent: Fasterfox Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: Foobot Disallow:/ User-agent: Gaisbot Disallow: / User-agent: GetRight Disallow: / User-agent: Harvest Disallow:/ User-agent: Go!Zilla Disallow:/ User-agent: hloader Disallow:/ User-agent: httplib Disallow:/ User-agent: HTTrack Disallow: / User-agent: humanlinks Disallow:/ User-agent: InfoNaviRobot Disallow:/ User-agent: Iron33 Disallow: / User-agent: JennyBot Disallow:/ User-agent: Kenjin Spider Disallow:/ User-agent: Keyword Density Disallow:/ User-agent: larbin Disallow: / User-agent: LexiBot Disallow:/ User-agent: libWeb/clsHTTP Disallow:/ User-agent: LinkextractorPro Disallow:/ User-agent: LinkScan/8.1a Unix Disallow:/ User-agent: LinkWalker Disallow:/ User-agent: LNSpiderguy Disallow:/ User-agent: lwp-trivial Disallow:/ User-agent: Mata Hari Disallow: / User-agent: MIIxpc Disallow:/ User-agent: Mister PiX Disallow:/ User-agent: moget Disallow:/ User-agent: NetAnts Disallow:/ User-agent: NetMechanic Disallow:/ User-agent: NICErsPRO Disallow:/ User-agent: Offline Explorer Disallow:/ User-agent: Openfind Disallow:/ User-agent: Openfind data gathere Disallow:/ User-agent: Oracle Ultra Search Disallow: / User-agent: PerMan Disallow: / User-agent: ProPowerBot Disallow:/ User-agent: ProWebWalker Disallow:/ User-agent: psbot Disallow: / User-agent: QueryN Metasearch Disallow:/ User-agent: Radiation Retriever Disallow: / User-agent: RepoMonkey Disallow:/ User-agent: Bait Disallow:/ User-agent: RMA Disallow:/ User-agent: searchpreview Disallow: / User-agent: SiteSnagger Disallow:/ User-agent: SpankBot Disallow:/ User-agent: spanner Disallow:/ User-agent: suzuran Disallow:/ User-agent: Szukacz Disallow:/ User-agent: SurveyBot Disallow: / User-agent: Teleport Disallow:/ User-agent: TeleportPro Disallow:/ User-agent: Telesoft Disallow:/ User-agent: The Intraformant Disallow:/ User-agent: TheNomad Disallow:/ User-agent: TightTwatBot Disallow:/ User-agent: Titan Disallow:/ User-agent: toCrawl Disallow:/ User-agent: True_Robot Disallow:/ User-agent: turingos Disallow:/ User-agent: TurnitinBot Disallow: / User-agent: URL Control Disallow: / User-agent: URLy Warning Disallow:/ User-agent: URL_Spider_Pro Disallow: / User-agent: VCI Disallow:/ User-agent: Web Image Collector Disallow:/ User-agent: WebAuto Disallow:/ User-agent: WebBandit Disallow:/ User-agent: WebCapture Disallow: / User-agent: scalaj-http/1.0 Disallow: / User-agent: WebCollage Disallow: / User-agent: WebCopier Disallow:/ User-agent: WebEnhancer Disallow:/ User-agent: WebmasterWorldForumBot Disallow:/ User-agent: WebSauger Disallow:/ User-agent: Website Quester Disallow:/ User-agent: Webster Pro Disallow:/ User-agent: WebStripper Disallow:/ User-agent: WebZip Disallow:/ User-agent: Wget Disallow:/ User-agent: WWW-Collector-E Disallow:/ User-agent: Xenu's Disallow:/ User-agent: Xenu's Link Sleuth 1.1c Disallow:/ User-agent: Zeus Disallow:/ User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow:/ User-agent: Meltawer Disallow: / User-agent: Digimind Disallow: / User-agent: Knowings Disallow: / User-agent: Sindup Disallow: / User-agent: Cision Disallow: / User-agent: Talkwater Disallow: / User-agent: TurnitinBot Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: QuepasaCreep Disallow: / User-agent: Jetbot Disallow: / User-agent: NewsNow Disallow: / User-agent: kbcrawl Disallow: / User-agent: AmiSoftware Disallow: / User-agent: Newzbin Disallow: / User-agent: Ask n read Disallow: / User-agent: Qwam content intelligence Disallow: / User-agent: Zite Disallow: / User-agent: flipboard Disallow: / User-agent: FlipboardProxy Disallow: / User-agent: Youmag Disallow: / User-agent: Synthesio Disallow: / User-agent: trendybuzz Disallow: / User-agent: spotter Disallow: / User-agent: scoop.it Disallow: / User-agent: linkfluence Disallow: / User-agent: 5emeRue Disallow: / User-agent: Augure Disallow: / User-agent: Corporama Disallow: / User-agent: readability.com Disallow: / User-agent: grub-client Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver-web.archive.org Disallow: / User-agent: k2spider Disallow: / User-agent: libwww Disallow: / User-agent: wget Disallow: / User-agent: 5erue Disallow: / User-agent: Sysomos/1.0 Disallow: / User-agent: adequat Disallow: / User-agent: adequat-systems Disallow: / User-agent: auramundi Disallow: / User-agent: coexel Disallow: / User-agent: ellisphere Disallow: / User-agent: leadbox Disallow: / User-agent: mention Disallow: / User-agent: Moreover Disallow: / User-agent: mytwip Disallow: / User-agent: NewsNow Disallow: / User-agent: Newzbin Disallow: / User-agent: mixnode.com Disallow: / User-agent: opinion-tracker Disallow: / User-agent: proxem Disallow: / User-agent: score3 Disallow: / User-agent: trendeo Disallow: / User-agent: vecteurplus Disallow: / User-agent: verticalsearch Disallow: / User-agent: vsw Disallow: / User-agent: winello Disallow: / User-agent: Fetch Disallow: / User-agent: infoseek Disallow: / User-agent: Offline Explorer Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebCopier Disallow: / User-agent: WebStripper Disallow: / User-agent: Zealbot Disallow: / User-agent: asknread.com Disallow: / User-agent: ellisphere Disallow: / User-agent: spotter Disallow: / User-agent: LexxeBot/1.0 Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: Mozilla/5.0 (compatible; spbot/2.0; http://www.seoprofiler.com/bot/ ) Disallow: / User-agent: Sosospider Disallow: / User-agent: Nuzzel Disallow: / User-agent: Nuzzel Bot Disallow: / User-agent: Sosospider+(+http://help.soso.com/webspider.htm) Disallow: / User-agent: rogerbot Disallow: / User-agent: SiteBot/0.1 Disallow: / User-agent: SiteBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Baiduspider Disallow: / User-agent: crystalsemanticsbot Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: NetSeer crawler Disallow: / User-agent: trovitBot Disallow: / User-agent: LexxeBot Disallow: / User-agent: DotBot Disallow: / User-agent: Ezooms Disallow: / User-agent: discobot Disallow: / User-agent: Jyxobot Disallow: / User-agent: sogou Disallow: / User-agent: sogou spider Disallow: / User-agent: Sogoubot Disallow: / User-agent: Sogou.com Disallow: / User-agent: sogouspider-106-120-173-138.crawl.sogou.com Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: sogouspider-106-120-173-138 Disallow: / User-agent: Sogou web spider/4.0 Disallow: / User-agent: Sogouwebspider Disallow: / User-agent: sogouspider Disallow: / User-agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: Sogou web spider Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: sistrix Disallow: / User-agent: heritrix Disallow: / User-agent: GarlikCrawler/1.1 (http://garlik.com/, [email protected]) Disallow: / User-agent: AhrefsBot Disallow: / User-agent: NerdByNature.Bot Disallow: / User-agent: psbot Disallow: / User-agent: WBSearchBot Disallow: / User-agent: AddThis.com robot [email protected] Disallow: / User-agent: AddThis.com Disallow: / User-agent: ia_archiver Disallow: / User-agent: proximic Disallow: / User-agent: discoverybot Disallow: / User-agent: bl.uk_lddc_bot Disallow: / User-agent: IstellaBot Disallow: / User-agent: seokicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: UnisterBot Disallow: / User-agent: Bender Disallow: / User-agent: wotbox Disallow: / User-agent: Yasni Disallow: / User-agent: JikeSpider Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: netEstate NE Crawler Disallow: / User-agent: Exabot Disallow: / User-agent: Pixray-Seeker Disallow: / User-agent: Linguee Disallow: / User-agent: integromedb Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: BDCbot Disallow: / User-agent: GrapeshotCrawler Disallow: / User-agent: WeSEE:Search Disallow: / User-agent: TurnitinBot Disallow: / User-agent: admantx Disallow: / User-agent: spbot Disallow: / User-agent: BUbiNG Disallow: / User-agent: 008 User-agent: Alexibot User-agent: AlvinetSpider User-agent: Antenne Hatena User-agent: ApocalXExplorerBot User-agent: asterias User-agent: BackDoorBot/1.0 User-agent: BizInformation User-agent: Black Hole User-agent: BlowFish/1.0 User-agent: BotALot User-agent: BuiltBotTough User-agent: Bullseye/1.0 User-agent: BunnySlippers User-agent: Cegbfeieh User-agent: CheeseBot User-agent: CherryPicker User-agent: CherryPickerElite/1.0 User-agent: CherryPickerSE/1.0 User-agent: CopyRightCheck User-agent: cosmos User-agent: Crescent User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: DISCo Pump 3.1 User-agent: DittoSpyder User-agent: dotbot User-agent: EmailCollector User-agent: EmailSiphon User-agent: EmailWolf User-agent: EroCrawler User-agent: ExtractorPro User-agent: Flamingo_SearchEngine User-agent: Foobot User-agent: Harvest/1.5 User-agent: hloader User-agent: httplib User-agent: HTTrack User-agent: HTTrack 3.0 User-agent: humanlinks User-agent: Igentia User-agent: InfoNaviRobot User-agent: JennyBot User-agent: JikeSpider User-agent: Kenjin Spider User-agent: LexiBot User-agent: libWeb/clsHTTP User-agent: LinkextractorPro User-agent: LinkScan/8.1a Unix User-agent: LinkWalker User-agent: lwp-trivial User-agent: lwp-trivial/1.34 User-agent: Mata Hari User-agent: Microsoft URL Control - 5.01.4511 User-agent: Microsoft URL Control - 6.00.8169 User-agent: MIIxpc User-agent: MIIxpc/4.2 User-agent: Mister PiX User-agent: MLBot User-agent: moget User-agent: moget/2.1 User-agent: MS Search 4.0 Robot User-agent: MS Search 5.0 Robot User-agent: Naverbot User-agent: NetAnts User-agent: NetAttache User-agent: NetMechanic User-agent: NICErsPRO User-agent: Nutch User-agent: Offline Explorer User-agent: Openfind User-agent: OpenindexSpider User-agent: ProPowerBot/2.14 User-agent: ProWebWalker User-agent: psbot User-agent: QuepasaCreep User-agent: QueryN Metasearch User-agent: RepoMonkey User-agent: RMA User-agent: SemrushBot User-agent: SightupBot User-agent: SiteBot User-agent: SiteSnagger User-agent: SiteSucker User-agent: Sogou web spider User-agent: sosospider User-agent: SpankBot User-agent: spanner User-agent: Speedy User-agent: Spiderbot User-agent: suggybot User-agent: SuperBot User-agent: SuperBot/2.6 User-agent: suzuran User-agent: Szukacz/1.4 User-agent: Teleport User-agent: Telesoft User-agent: The Intraformant User-agent: TheNomad User-agent: TightTwatBot User-agent: Titan User-agent: toCrawl/UrlDispatcher User-agent: TosCrawler User-agent: True_Robot User-agent: True_Robot/1.0 User-agent: turingos User-agent: TurnitinBot User-agent: UrlPouls User-agent: URLy Warning User-agent: VCI User-agent: Web Image Collector User-agent: WebAuto User-agent: WebBandit User-agent: WebBandit/3.50 User-agent: WebCopier User-agent: webcopy User-agent: WebEnhancer User-agent: WebmasterWorldForumBot User-agent: webmirror User-agent: WebReaper User-agent: WebSauger User-agent: website extractor User-agent: Website Quester User-agent: Webster Pro User-agent: WebStripper User-agent: WebStripper/2.02 User-agent: WebZip User-agent: wget User-agent: WikioFeedBot User-agent: WinHTTrack User-agent: WWW-Collector-E User-agent: Xenu Link Sleuth/1.3.8 User-agent: yacy User-agent: yandex User-agent: YRSPider User-agent: Zeus User-agent: Zookabot User-agent: Spiderbot/Nutch-1.7 User-agent: g2reader-bot/1.0 User-agent: SocialShare/1.0 User-agent: Owlin Bot User-agent: trendictionbot Disallow: / # Disallow: Sistrix User-agent: sistrix Disallow: / # Disallow: Sistrix User-agent: SISTRIX Crawler Disallow: / # Disallow: Sistrix User-agent: SISTRIX Disallow: / # Disallow: SEOkicks-Robot User-agent: SEOkicks-Robot Disallow: / # Disallow: jobs.de-Robot User-agent: jobs.de-Robot Disallow: / # Backlink Analysis user-agent: AhrefsBot disallow: / # Bot der Leipziger Unister Holding GmbH user-agent: UnisterBot disallow: / # http://moz.com/products User-agent: dotbot Disallow: / # http://www.searchmetrics.com User-agent: SearchmetricsBot Disallow: / # http://www.majestic12.co.uk/projects/dsearch/mj12bot.php User-agent: MJ12bot Disallow: / # http://www.domaintools.com/webmasters/surveybot.php User-agent: SurveyBot Disallow: / # http://www.seodiver.com/bot user-agent: SEOdiver disallow: / # http://openlinkprofiler.org/bot User-agent: spbot Disallow: / # http://www.wotbox.com/bot/ User-agent: wotbox Disallow: / # http://www.opensiteexplorer.org/dotbot # http://moz.com/researchtools/ose/dotbot User-agent: dotbot Disallow: / # http://www.meanpath.com/meanpathbot.html User-agent: meanpathbot Disallow: / # http://www.backlinktest.com/crawler.html User-agent: BacklinkCrawler Disallow: / # http://www.brandwatch.com/magpie-crawler/ User-agent: magpie-crawler Disallow: / # http://www.brandwatch.com/magpie-crawler/ User-agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) Disallow: / # http://www.brandwatch.com/magpie-crawler/ User-agent: magpie-crawler/1.1 Disallow: / # http://filterdb.iss.net/crawler/ User-agent: oBot Disallow: / User-agent: fr-crawler Disallow: / # http://webmeup-crawler.com User-agent: BLEXBot Disallow: / # https://megaindex.com/crawler User-agent: MegaIndex.ru Disallow: / User-agent: megaindex.com Disallow: / # http://www.cloudservermarket.com User-Agent: CloudServerMarketSpider Disallow: / # http://www.trendiction.de/de/publisher/bot User-Agent: trendictionbot Disallow: / # http://www.exalead.com User-agent: Exabot Disallow: / # http://www.career-x.de/bot.html User-agent: careerbot Disallow: / # https://www.lipperhey.com/en/about/ User-agent: Lipperhey-Kaus-Australis Disallow: / User-agent: seoscanners.net/1 Disallow: / User-agent: MetaJobBot Disallow: / User-agent: Spiderbot Disallow: / User-agent: LinkStats Disallow: / User-agent: JobboerseBot Disallow: / User-agent: ICCrawler Disallow: / User-agent: Plista Disallow: / User-agent: Domain Re-Animator Bot Disallow: / User-agent: Aboundexbot Disallow: / User-agent: acapbot Disallow: / User-agent: AdnormCrawler Disallow: / User-agent: AhrefsBot Disallow: / User-agent: AOLbot/4.0 Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Attribot Disallow: / User-agent: Baiduspider-image Disallow: / User-agent: Bot.AraTurka.com Disallow: / User-agent: Butterfly/1.0 Disallow: / User-agent: BuzzSumo Disallow: / User-agent: cis455crawler Disallow: / User-agent: ClarityBot Disallow: / User-agent: ClarityDailyBot Disallow: / User-agent: CMS Crawler Disallow: / User-agent: CommonCrawler Disallow: / User-agent: crawler4j (http://code.google.com/p/crawler4j/) Disallow: / User-agent: CrazyWebCrawler Disallow: / User-agent: CrazyWebCrawler-Spider Disallow: / User-agent: DataparkSearch Disallow: / User-agent: deepcrawl Disallow: / User-agent: demandbase-bot Disallow: / User-agent: DeuSu Disallow: / User-agent: dialogsearch.com Disallow: / User-agent: diffbot Disallow: / User-agent: dj-research Disallow: / User-agent: DomainAppender Disallow: / User-agent: DomainMacroCrawler Disallow: / User-agent: DomainSigmaCrawler Disallow: / User-agent: Domain Re-Animator Bot Disallow: / User-agent: Ezooms Robot Disallow: / User-agent: Experibot Disallow: / User-agent: FavOrg Disallow: / # https://www.lipperhey.com/en/about/ User-agent: Lipperhey-Kaus-Australis Disallow: / # https://turnitin.com/robot/crawlerinfo.html User-agent: turnitinbot Disallow: / # http://help.coccoc.com/ User-agent: coccoc Disallow: / # ubermetrics-technologies.com User-agent: um-IC Disallow: / User-agent: scanbot Disallow: / # datenbutler.de User-agent: mindUpBot Disallow: / # http://searchgears.de/uber-uns/crawling-faq.html User-agent: sg-Orbiter Disallow: / # http://commoncrawl.org/faq/ User-agent: CCBot Disallow: / # http://linkfluence.net/ User-agent: Kraken Disallow: / # http://www.botje.com/plukkie.htm User-agent: plukkie Disallow: / # https://www.safedns.com/searchbot User-agent: SafeDNSBot Disallow: / # http://www.haosou.com/help/help_3_2.html User-agent: 360Spider Disallow: / # http://www.haosou.com/help/help_3_2.html User-agent: HaosouSpider Disallow: / # http://www.moz.com/dp/rogerbot User-agent: rogerbot Disallow: / # http://www.openhose.org/bot.html User-agent: OpenHoseBot Disallow: / # http://www.screamingfrog.co.uk/seo-spider/ User-agent: Screaming Frog SEO Spider Disallow: / # http://thumbsniper.com User-agent: ThumbSniper Disallow: / # http://www.radian6.com/crawler User-agent: R6_CommentReader Disallow: / User-agent: ImplisenseBot Disallow: / # http://cliqz.com/company/cliqzbot User-agent: Cliqzbot Disallow: / # https://www.aihitdata.com/about User-agent: aiHitBot Disallow: / # http://www.trendiction.com/en/publisher/bot User-Agent: trendictionbot Disallow: / # Riddler # Riddler (http://riddler.io/about) # http://riddler.io/about User-agent: Riddler Disallow: / # SMTBot # http://www.similartech.com/smtbot # Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot) User-agent: SMTBot Disallow: / # ScoutJet # http://www.scoutjet.com # Mozilla/5.0+(compatible;+ScoutJet;++http://www.scoutjet.com/) # Mozilla/5.0+(compatible;+Blekkobot;+ScoutJet;++http://blekko.com/about/blekkobot) User-agent: ScoutJet Disallow: / # SeeBot # http://www.seegnify.com/bot # seebot/2.0 (+http://www.seegnify.com/bot) User-agent: SeeBot Disallow: / # Archive.org Bot # http://archive.org/details/archive.org_bot # Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot) # Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot) User-agent: archive.org_bot Disallow: / # AdBeat Bot # https://www.adbeat.com/operation_policy # adbeat_bot User-agent: adbeat_bot Disallow: / # MeanPathBot # http://www.meanpath.com/meanpathbot.html # Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html) User-agent: meanpathbot Disallow: / # Mojeek Bot # https://www.mojeek.com/bot.html # Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html) User-agent: MojeekBot Disallow: / # Speedy Spider / EntireWeb Search # http://www.entireweb.com # Speedy+Spider+(http://www.entireweb.com) User-agent: Speedy Spider Disallow: / # GenieO # http://www.genieo.com/webfilter.html # Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html) User-agent: genieo Disallow: / # scrapy.org # by default ROBOTSTXT_OBEY is set to false # http://doc.scrapy.org/en/latest/topics/settings.html User-agent: Scrapy Disallow: / # Gigablast Open Source Search Engine/Crawler/Spider # You can ignore robots.txt # http://gigablast.com/admin.html # GigablastOpenSource/1.0 User-agent: GigablastOpenSource Disallow: / # Semrush Bot # http://www.semrush.com/bot.html # Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html) User-agent: SemrushBot Disallow: / # Proximic # http://www.proximic.com/info/spider.php # Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php) User-agent: proximic Disallow: / # AppleBot # Used by Siri/Spotlight suggestions (commented out for now) # http://www.apple.com/go/applebot # Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot) # Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot) # Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot) User-agent: applebot Disallow: / # CCBot # http://commoncrawl.org/faq/ # CCBot/2.0+(http://commoncrawl.org/faq/) User-agent: CCBot Disallow: / # Plukkie # http://www.botje.com/plukkie.htm # Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm) User-agent: Plukkie Disallow: / # Lipperhey Kaus Australis # https://www.lipperhey.com/en/about/ # Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/) User-agent: lipperhey Disallow: / # oBot # http://filterdb.iss.net/crawler/ # Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/) # Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/) User-agent: oBot Disallow: / # Screaming Frog SEO Spider # Downloaded software, robots.txt can be ignored # http://www.screamingfrog.co.uk/seo-spider/user-guide/general/ # Screaming+Frog+SEO+Spider/2.55 # Screaming+Frog+SEO+Spider/3.3 User-agent: Screaming Frog SEO Spider Disallow: / # VegeBot # http://www.exactbot.com/vegebot/index.html User-agent: VegeBot Disallow: / # ScreenerBot # ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com) User-agent: ScreenerBot Disallow: / # Garlik from Experian # GarlikCrawler/1.2 (http://garlik.com/, [email protected]) User-agent: Garlik Disallow: / # BusinessBot: [email protected] User-agent: BusinessBot Disallow: / # FAST Enterprise Crawler/5.3.4 ([email protected]) User-agent: FAST Enterprise Disallow: / # Nutch # http://nutch.apache.org/bot.html # Nutch+Spider/Nutch-1.5 # mycrowl/Nutch-1.9 # nutch+crawler/Nutch-1.9 # sky+nutch+crawler/Nutch-1.9 # Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD) User-agent: Nutch Disallow: / # Open Link Profiler # Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+) # http://OpenLinkProfiler.org/bot User-agent: spbot Disallow: / # James Bot # Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html # http://cognitiveseo.com/bot.html User-agent: JamesBOT Disallow: / # Kraken # Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;[email protected]) # http://linkfluence.net/ User-agent: Kraken Disallow: / # Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/) # http://www.linkdex.com/bots/ User-agent: linkdex Disallow: / # Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com # Siteimprove.com User-agent: SiteCheck-sitecrawl by Siteimprove.com Disallow: / # Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler) # http://megaindex.com/crawler User-agent: MegaIndex Disallow: / # Prlog # Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/) # http://prlog.ru/ User-agent: Prlog Disallow: / # Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about) # https://www.aihitdata.com/about User-agent: AiHitBot Disallow: / # KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com) # http://www.kingland.com User-agent: KinglandSystemsCorp Disallow: / # Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;[email protected]) # http://www.sitemapbot.com User-agent: SitemapBot Disallow: / # Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot) # https://www.deepcrawl.com/bot User-agent: deepcrawl Disallow: / # Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21 # http://www.alexa.com/toolbar #User-agent: AlexaToolbar #Disallow: / # Mozilla/5.0+(compatible;+heritrix/3.3.0-SNAPSHOT-20150302-2206++http://127.0.0.1) # https://webarchive.jira.com/wiki/display/Heritrix/Heritrix User-agent: heritrix Disallow: / # Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/) # http://help.naver.com/robots/ User-agent: Yeti Disallow: / # Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,[email protected]+for+information) User-agent: NetShelter Disallow: / # VSE/1.0+([email protected]) # LSSRocketCrawler/1.0 LightspeedSystems # http://lightspeedsystems.com User-agent: LSSRocketCrawler Disallow: / # Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html) # http://www.idmarch.org/bot.html User-agent: idmarch Disallow: / # Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0 # Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0+(Following+Mediapartners-Google) User-agent: Daumoa Disallow: / # Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html) # https://www.companiesintheuk.co.uk/bot.html User-agent: CukBot Disallow: / # Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider) # http://www.expertsearch.nl/spider User-agent: ExpertSearchSpider Disallow: / # Mozilla/5.0+(compatible;+Goodzer/2.0;[email protected]) User-agent: Goodzer Disallow: / # WeSEE # http://www.wesee.com/ User-agent: WeSEE Disallow: / # http://ltx71.com/ # ltx71 - (http://ltx71.com/) User-agent: ltx71 Disallow: / # rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,[email protected]) # http://moz.com/help/pro/what-is-rogerbot- User-agent: rogerbot Disallow: / # bitlybot # http://bit.ly ?? User-agent: bitlybot Disallow: / # Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11 # http://www.trendiction.de/bot User-agent: trendictionbot Disallow: / # SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites) User-agent: SearchMassive Disallow: / # spiderbot User-agent: spiderbot Disallow: / # yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html # yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html User-agent: yacybot Disallow: / # AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;[email protected]) # http://www.aboutus.org/Aboutus:Bot User-agent: AboutUsBot Disallow: / # Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,[email protected]) # http://www.opensiteexplorer.org/dotbot User-agent: DotBot Disallow: / # Mozilla/5.0+(compatible;+BLEXBot/1.0;++http://webmeup-crawler.com/) # http://webmeup-crawler.com/ User-agent: BLEXBot Disallow: / # Xenu Link Sleuth 1.3.4 # Xenu Link Sleuth/1.3.8 User-agent: Xenu Link Sleuth Disallow: / # Go language # Go+1.1+package+http User-agent: Go 1.1 Disallow: / # wotbox.com # http://www.wotbox.com/bot/ # Wotbox/2.01+(+http://www.wotbox.com/bot/) User-agent: Wotbox Disallow: / # yandex # http://yandex.com/bots # Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots) User-agent: YandexBot Disallow: / # Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) # http://help.yahoo.com/help/us/ysearch/slurp # User-agent: Yahoo! Slurp # Disallow: / # Yahoo Japan # http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/ # Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) User-agent: Y!J-ASR Disallow: / # domainreanimator.com User-agent: Domain Re-Animator Bot Disallow: / # crazywebcrawler.com User-agent: CrazyWebCrawler-Spider Disallow: / # crazywebcrawler.com User-agent: CrazyWebCrawler Disallow: / # Dow Jones Searchbot # Mozilla/5.0+(compatible;+Dow+Jones+Searchbot) User-agent: Dow Jones SearchBot Disallow: / # OpenVAS.org #User-agent: OpenVAS #Disallow: / # WeCrawlForThePeace - We are not Evil User-agent: WeCrawForThePeace Disallow: / # WeLikeLinks - WeAreNotEvil User-agent: WeLikeLinks Disallow: / # icevikatam/1.0 #User-agent: icevikatam #Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html) Disallow: / User-agent: Apache-HttpClient/4.5.3 (Java/1.8.0_112) Disallow: / User-agent: Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html) Disallow: / User-agent: SocialRankIOBot; http://socialrank.io/about Disallow: / User-agent: SocialRankIOBot Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: CompSpyBot Disallow: / User-agent: Curious George Disallow: / User-agent: CybEye.com Disallow: / User-agent: DoCoMo Disallow: / User-agent: ExB Language Crawler Disallow: / User-agent: Ezooms Disallow: / User-agent: Flamingo_SearchEngine Disallow: / User-agent: Genieo Disallow: / User-agent: Genio Disallow: / User-agent: LWNutch Disallow: / User-agent: LexxeBot Disallow: / User-agent: OpenWebIndex Disallow: / User-agent: RediffNewsBot Disallow: / User-agent: SEOENGWorldBot Disallow: / User-agent: Scanmine Disallow: / User-agent: ShopWiki Disallow: / User-agent: ShowyouBot Disallow: / User-agent: Sosospider Disallow: / User-agent: WocBot Disallow: / User-agent: Yeti Disallow: / User-agent: YoudaoBot Disallow: / User-agent: daumoa Disallow: / User-agent: gsa-crawler Disallow: / User-agent: libcrawl Disallow: / User-agent: linkdex Disallow: / User-agent: magpie-crawler Disallow: / User-agent: repparser Disallow: / User-agent: rogerbot Disallow: / User-agent: sindice-site-manager Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou Disallow: / User-agent: woriobot Disallow: / User-agent: yacybot Disallow: / User-agent: yolinkBot Disallow: / User-agent: UnisterBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Orthogaffe Disallow: / User-agent: wget Disallow: / # Block Sogou User-agent: sogou spider Disallow: / # Block SEOkicks User-agent: SEOkicks-Robot Disallow: / # Block BlexBot User-agent: BLEXBot Disallow: / # Block SISTRIX User-agent: SISTRIX Crawler Disallow: / # Block Uptime robot User-agent: UptimeRobot/2.0 Disallow: / User-agent: 008 Disallow: / # Block Ezooms Robot User-agent: Ezooms Robot Disallow: / # Block Perl LWP User-agent: Perl LWP Disallow: / # Block BlexBot User-agent: BLEXBot Disallow: / # Block netEstate NE Crawler (+http://www.website-datenbank.de/) User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/) Disallow: / # Block WiseGuys Robot User-agent: WiseGuys Robot Disallow: / # Block Turnitin Robot User-agent: Turnitin Robot Disallow: / User-agent: HuaweiSymantecSpider Disallow: / User-agent: Purebot Disallow: / # Block Heritrix User-agent: Heritrix Disallow: / User-agent: eContext/1.0 (eContext Classification Engine) Disallow: / User-agent: eContext/1.0 Disallow: / User-agent: eContext Disallow: / # Block pricepi User-agent: pimonster Disallow: / User-agent: Pimonster Disallow: / # Block Searchmetrics Bot User-agent: SearchmetricsBot Disallow: / # Block Eniro User-agent: ECCP/1.0 ([email protected]) Disallow: / # Block YandexBot User-agent: Yandex Disallow: / # Block Baidu User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image Disallow: / # Block SoGou User-agent: Sogou Spider Disallow: / # Block Youdao User-agent: YoudaoBot Disallow: / # Block MegaIndex.ru User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: Yandex Disallow: / user-agent: AhrefsBot disallow: / User-Agent: trendictionbot Disallow: / User-agent: SeznamBot Disallow: / User-agent: search:robot Disallow: / User-agent: XoviBot Disallow: / User-agent: xovi Disallow: / User-agent: Cutbot Disallow: / User-agent: DotBot Disallow: / User-agent: LivelapBot Disallow: / User-agent: LS Session Disallow: / User-agent: MagicBot Disallow: / User-agent: AliveAdvisorCrawler/1.7.1 Disallow: / User-agent: AliveAdvisorCrawler/1.7.1 (whole web crawler; http://search.aliveadvisor.com; [email protected]) Disallow: / User-agent: madaali.de Disallow: / User-agent: memoryBot Disallow: / User-agent: whole web crawler Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: MegaIndex.ru/ Disallow: / User-agent: SputnikBot Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Slurp Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: wget Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: berlin-fu-cow Disallow: / User-agent: JikeSpider Disallow: / User-agent: semager Disallow: / User-agent: BLEXBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: dotbot Disallow: / User-agent: InAGist URL Resolver Disallow: / User-agent: Yandex Disallow: / User-agent: Insitesbot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: IndeedBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: megaIndex.ru Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: ltx71 Disallow: / User-agent: SputnikBot Disallow: / User-agent: spbot Disallow: / User-agent: seoscanners.net Disallow: / User-agent: DeuSu Disallow: / User-agent: meds-online24.com Disallow: / User-agent: seoscanners Disallow: / User-agent: GuzzleHttp Disallow: / User-agent: IDBot Disallow: / User-agent: PRTGCloudBot Disallow: / User-agent: Sogou Pic Spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07) Disallow: / User-agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) Disallow: / User-agent: WebReaper Disallow: / User-agent: WebLyzard Crawler Disallow: / User-agent: WebCopier v4.3 Disallow: / User-agent: Vagabondo Disallow: / User-agent: Shim-Crawler Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: NimbleCrawler Disallow: / User-agent: hl_ftien_spider_v1.1 Disallow: / User-agent: EMPAS_ROBOT Disallow: / User-agent: EDI/1.4.1 (Edacious & Intelligent Web Crawler) Disallow: / User-agent: (compatible; hetrix/1.8.0+http://www.accelobot.com) Disallow: / User-agent: WebReaper [[email protected]] Disallow: / User-agent: WebReaper [[email protected]] Disallow: / User-agent: Mozilla/5.0 (compatible; Vagabondo/Kliksafe; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) Disallow: / User-agent: Vagabondo/4.0 (webagent at wise-guys dot nl) Disallow: / User-agent: Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/) Disallow: / # http://www.wise-guys.nl/ User-agent: webcrawler Disallow: / User-agent: ilse Disallow: / User-agent: Vagabondo Disallow: / User-agent: Bilbo Disallow: / User-agent: PiplBot Disallow: / User-agent: lwp-trivial/5.810 Disallow: / User-agent: dotbot Disallow: / User-agent: discoverybot Disallow: / User-agent: SimplePie Disallow: / User-agent: SimplePie/1.1.1 Disallow: / User-agent: SimplePie/1.0.1 Disallow: / User-agent: Baiduspider Disallow: / User-agent: JikeSpider Disallow: / User-agent: Ocelli/1.4 Disallow: / User-agent: Ocelli Disallow: / User-agent: [email protected] Disallow: / User-agent: Ezooms Disallow: / User-agent: Ezooms/1.0 Disallow: / User-agent: Ezooms/1.0; [email protected] Disallow: / #ad Crawler: http://www.a6corp.com/a6-web-scraping-policy/ User-agent: A6-Indexer/1.0 Disallow: / #ad Crawler: http://www.grapeshot.co.uk/crawler.php User-agent: grapeshot Disallow: / # http://www.proximic.com/info/spider.php User-agent: proximic Disallow: / # http://www.jobdigger.nl/spider/ User-agent: JobdiggerSpider Disallow: / User-agent: BLEXBot Disallow: / User-agent: rogerbot Disallow:/ User-agent: SEOkicks-Robot Disallow:/ User-agent: SMTBot Disallow:/ User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: dotbot Disallow: / User-agent: EnaBot/1.2 (http://www.enaball.com/crawler.html) Disallow: / User-agent: EnaBot Disallow: / User-agent: Go-http-client/1.1 Disallow: / User-agent: SMTBot/1.0 Disallow: / User-agent: e-SocietyRobot Disallow: / User-agent: Synthesio Disallow: / User-agent: Synthesio Crawler release MonaLisa (contact at synthesio dot fr) Disallow: / User-agent: FreeFind Disallow: / User-agent: DBLBot Disallow: / User-agent: SeznamBot Disallow: / User-agent: Caliperbot Disallow: / User-agent: spbot Disallow: / User-agent: SimplePie Disallow: / User-agent: SimplePie/1.1.3 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20081219 Disallow: / User-Agent: trendictionbot Disallow: / # http://sentibot.eu/ User-agent: SentiBot Disallow: / User-agent: SentiBot Disallow: / User-agent: omgili/0.5 +http://omgili.com Disallow: / User-agent: Mediatoolkitbot ([email protected]) Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) Disallow: / User-agent: GrapeshotCrawler/2.0 Disallow: / User-agent: GrapeshotCrawler Disallow:/ User-agent: Grapeshot Crawler Disallow:/ User-agent: Grapeshot Disallow:/ User-agent: omgili/0.5 +http://omgili.com Disallow: / User-agent: omgili Disallow: / User-agent: musobot Disallow: / User-agent: Baiduspider Disallow: / User-agent: TurnitinBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: SimplePie Disallow: / User-agent: Speedy Disallow: / User-agent: Mozilla/5.0 ResearchBot/0.6.2 (project: flattr-id-meta-tokens) This/1 Disallow: / User-agent: ResearchBot/0.6.2 Disallow: / User-agent: ResearchBot Disallow: / User-agent: jobs.de-Robot Disallow: / User-agent: FreeFind Disallow: / User-agent: xingbot Disallow: / User-agent: xing bot Disallow: / User-agent: xing Disallow: / User-agent: DBLBot Disallow: / User-agent: ChinasoSpider Disallow: / User-agent: yisouspider Disallow: / User-agent: EasouSpider Disallow: / User-agent: SeznamBot Disallow: / User-agent: Caliperbot Disallow: / User-agent: spbot Disallow: / user-agent: EveryoneSocialBot Disallow: / user-agent: something Disallow: / user-agent: ABCdatos BotLink user-agent: Acme.Spider user-agent: Ahoy! The Homepage Finder user-agent: Alkaline user-agent: Anthill user-agent: Walhello appie user-agent: Arachnophilia user-agent: Arale user-agent: Araneo user-agent: AraybOt user-agent: ArchitextSpider user-agent: Aretha user-agent: ARIADNE user-agent: arks user-agent: AskJeeves user-agent: ASpider (Associative Spider) user-agent: ATN Worldwide user-agent: Atomz.com Search Robot user-agent: AURESYS user-agent: BackRub user-agent: Bay Spider user-agent: Big Brother user-agent: Bjaaland user-agent: BlackWidow user-agent: Die Blinde Kuh user-agent: Bloodhound user-agent: Borg-Bot user-agent: BoxSeaBot user-agent: bright.net caching robot user-agent: BSpider user-agent: CACTVS Chemistry Spider user-agent: Calif user-agent: Cassandra user-agent: Digimarc Marcspider/CGI user-agent: Checkbot user-agent: ChristCrawler.com user-agent: churl user-agent: cIeNcIaFiCcIoN.nEt user-agent: CMC/0.01 user-agent: Collective user-agent: Combine System user-agent: Conceptbot user-agent: ConfuzzledBot user-agent: CoolBot user-agent: Web Core / Roots user-agent: XYLEME Robot user-agent: Internet Cruiser Robot user-agent: Cusco user-agent: CyberSpyder Link Test user-agent: CydralSpider user-agent: Desert Realm Spider user-agent: DeWeb(c) Katalog/Index user-agent: DienstSpider user-agent: Digger user-agent: Digital Integrity Robot user-agent: Direct Hit Grabber user-agent: DNAbot user-agent: DownLoad Express user-agent: DragonBot user-agent: DWCP (Dridus' Web Cataloging Project) user-agent: e-collector user-agent: EbiNess user-agent: EIT Link Verifier Robot user-agent: ELFINBOT user-agent: Emacs-w3 Search Engine user-agent: ananzi user-agent: esculapio user-agent: Esther user-agent: Evliya Celebi user-agent: FastCrawler user-agent: Fluid Dynamics Search Engine robot user-agent: Felix IDE user-agent: Wild Ferret Web Hopper #1, #2, #3 user-agent: FetchRover user-agent: fido user-agent: Hämähäkki user-agent: KIT-Fireball user-agent: Fish search user-agent: Fouineur user-agent: Robot Francoroute user-agent: Freecrawl user-agent: FunnelWeb user-agent: gammaSpider, FocusedCrawler user-agent: gazz user-agent: GCreep user-agent: GetBot user-agent: GetURL user-agent: Golem user-agent: Grapnel/0.01 Experiment user-agent: Griffon user-agent: Gromit user-agent: Northern Light Gulliver user-agent: Gulper Bot user-agent: HamBot user-agent: Harvest user-agent: havIndex user-agent: HI (HTML Index) Search user-agent: Hometown Spider Pro user-agent: ht://Dig user-agent: HTMLgobble user-agent: Hyper-Decontextualizer user-agent: iajaBot user-agent: IBM_Planetwide user-agent: Popular Iconoclast user-agent: Ingrid user-agent: Imagelock user-agent: IncyWincy user-agent: Informant user-agent: InfoSeek Robot 1.0 user-agent: Infoseek Sidewinder user-agent: InfoSpiders user-agent: Inspector Web user-agent: IntelliAgent user-agent: I, Robot user-agent: Iron33 user-agent: Israeli-search user-agent: JavaBee user-agent: JBot Java Web Robot user-agent: JCrawler user-agent: Jeeves user-agent: JoBo Java Web Robot user-agent: Jobot user-agent: JoeBot user-agent: The Jubii Indexing Robot user-agent: JumpStation user-agent: image.kapsi.net user-agent: Katipo user-agent: KDD-Explorer user-agent: Kilroy user-agent: KO_Yappo_Robot user-agent: LabelGrabber user-agent: larbin user-agent: legs user-agent: Link Validator user-agent: LinkScan user-agent: LinkWalker user-agent: Lockon user-agent: logo.gif Crawler user-agent: Lycos user-agent: Mac WWWWorm user-agent: Magpie user-agent: marvin/infoseek user-agent: Mattie user-agent: MediaFox user-agent: MerzScope user-agent: NEC-MeshExplorer user-agent: MindCrawler user-agent: mnoGoSearch search engine software user-agent: moget user-agent: MOMspider user-agent: Monster user-agent: Motor user-agent: Muncher user-agent: Muninn user-agent: Muscat Ferret user-agent: Mwd.Search user-agent: Internet Shinchakubin user-agent: NDSpider user-agent: Nederland.zoek user-agent: NetCarta WebMap Engine user-agent: NetMechanic user-agent: NetScoop user-agent: newscan-online user-agent: NHSE Web Forager user-agent: Nomad user-agent: The NorthStar Robot user-agent: nzexplorer user-agent: ObjectsSearch user-agent: Occam user-agent: HKU WWW Octopus user-agent: OntoSpider user-agent: Openfind data gatherer user-agent: Orb Search user-agent: Pack Rat user-agent: PageBoy user-agent: ParaSite user-agent: Patric user-agent: pegasus user-agent: The Peregrinator user-agent: PerlCrawler 1.0 user-agent: Phantom user-agent: PhpDig user-agent: PiltdownMan user-agent: Pimptrain.com's robot user-agent: Pioneer user-agent: html_analyzer user-agent: Portal Juice Spider user-agent: PGP Key Agent user-agent: PlumtreeWebAccessor user-agent: Poppi User-agent: iskanie user-agent: PortalB Spider user-agent: psbot user-agent: GetterroboPlus Puu user-agent: The Python Robot user-agent: Raven Search user-agent: RBSE Spider user-agent: Resume Robot user-agent: RoadHouse Crawling System user-agent: RixBot user-agent: Road Runner: The ImageScape Robot user-agent: Robbie the Robot user-agent: ComputingSite Robi/1.0 user-agent: RoboCrawl Spider user-agent: RoboFox user-agent: Robozilla user-agent: Roverbot user-agent: RuLeS user-agent: SafetyNet Robot user-agent: Scooter user-agent: Sleek user-agent: Search.Aus-AU.COM user-agent: SearchProcess user-agent: Senrigan user-agent: SG-Scout user-agent: ShagSeeker user-agent: Shai'Hulud user-agent: Sift user-agent: Simmany Robot Ver1.0 user-agent: Site Valet user-agent: Open Text Index Robot user-agent: SiteTech-Rover user-agent: Skymob.com user-agent: SLCrawler user-agent: Inktomi Slurp user-agent: Smart Spider user-agent: Snooper user-agent: Solbot user-agent: Spanner user-agent: Speedy Spider user-agent: spider_monkey user-agent: SpiderBot user-agent: Spiderline Crawler user-agent: SpiderMan user-agent: SpiderView(tm) user-agent: Spry Wizard Robot user-agent: Site Searcher user-agent: Suke user-agent: suntek search engine user-agent: Sven user-agent: Sygol user-agent: TACH Black Widow user-agent: Tarantula user-agent: tarspider user-agent: Tcl W3 Robot user-agent: TechBOT user-agent: Templeton user-agent: TeomaTechnologies user-agent: TITAN user-agent: TitIn user-agent: The TkWWW Robot user-agent: TLSpider user-agent: UCSD Crawl user-agent: UdmSearch user-agent: UptimeBot user-agent: URL Check user-agent: URL Spider Pro user-agent: Valkyrie user-agent: Verticrawl user-agent: Victoria user-agent: vision-search user-agent: void-bot user-agent: Voyager user-agent: VWbot user-agent: The NWI Robot user-agent: W3M2 user-agent: WallPaper (alias crawlpaper) user-agent: the World Wide Web Wanderer user-agent: w@pSpider by wap4.com user-agent: WebBandit Web Spider user-agent: WebCatcher user-agent: WebCopy user-agent: webfetcher user-agent: The Webfoot Robot user-agent: Webinator user-agent: weblayers user-agent: WebLinker user-agent: WebMirror user-agent: The Web Moose user-agent: WebQuest user-agent: Digimarc MarcSpider user-agent: WebReaper user-agent: webs user-agent: Websnarf user-agent: WebSpider user-agent: WebVac user-agent: webwalk user-agent: WebWalker user-agent: WebWatch user-agent: Wget user-agent: whatUseek Winona user-agent: WhoWhere Robot user-agent: Wired Digital user-agent: Weblog Monitor user-agent: w3mir user-agent: WebStolperer user-agent: The Web Wombat user-agent: The World Wide Web Worm user-agent: WWWC Ver 0.2.5 user-agent: WebZinger user-agent: XGET Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: Jetbot Disallow: / User-agent: Meltwater Disallow: / User-agent: Digimind Disallow: / User-agent: Knowings Disallow: / User-agent: kbcrawl Disallow: / User-agent: AmiSoftware Disallow: / User-agent: Ask n read Disallow: / User-agent: Qwam content intelligence Disallow: / User-agent: Zite Disallow: / User-agent: FlipboardProxy Disallow: / User-agent: Youmag Disallow: / User-agent: Synthesio Disallow: / User-agent: trendybuzz Disallow: / User-agent: spotter Disallow: / User-agent: linkfluence Disallow: / User-agent: 5emeRue Disallow: / User-agent: Augure Disallow: / User-agent: Sindup Disallow: / User-agent: Corporama Disallow: / User-agent: readability.com Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: libwww Disallow: / User-agent: 5erue Disallow: / User-agent: adequat Disallow: / User-agent: adequat-systems Disallow: / User-agent: auramundi Disallow: / User-agent: coexel Disallow: / User-agent: leadbox Disallow: / User-agent: Moreover Disallow: / User-agent: mytwip Disallow: / User-agent: NewsNow Disallow: / User-agent: Newzbin Disallow: / User-agent: opinion-tracker Disallow: / User-agent: proxem Disallow: / User-agent: score3 Disallow: / User-agent: trendeo Disallow: / User-agent: vecteurplus Disallow: / User-agent: verticalsearch Disallow: / User-agent: vsw Disallow: / User-agent: winello Disallow: / User-agent: Fetch Disallow: / User-agent: infoseek Disallow: / User-agent: MSIECrawler Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: TeleportPro Disallow: / User-agent: Zealbot Disallow: / User-agent: asknread.com Disallow: / User-agent: ellisphere Disallow: / User-agent: spotter Disallow: / User-agent: 008 Disallow: / User-agent: Alexibot Disallow: / User-agent: AlvinetSpider Disallow: / User-agent: Antenne Hatena Disallow: / User-agent: ApocalXExplorerBot Disallow: / User-agent: asterias Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: BizInformation Disallow: / User-agent: Black Hole Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BotALot Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: DISCo Pump 3.1 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: dotbot Disallow: / User-agent: DotBot/1.1 Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: EroCrawler Disallow: / User-agent: ExtractorPro Disallow: / User-agent: Flamingo_SearchEngine Disallow: / User-agent: Foobot Disallow: / User-agent: hloader Disallow: / User-agent: httplib Disallow: / User-agent: HTTrack Disallow: / User-agent: HTTrack 3.0 Disallow: / User-agent: humanlinks Disallow: / User-agent: Igentia Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: JennyBot Disallow: / User-agent: JikeSpider Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: LexiBot Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: lwp-trivial Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.3 Disallow: / User-agent: Cyberwatcher Disallow: / User-agent: SafeDNSBot (https://www.safedns.com/searchbot) Disallow: / User-agent: Mata Hari Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Mister PiX Disallow: / User-agent: MLBot Disallow: / User-agent: moget Disallow: / User-agent: moget/2.1 Disallow: / User-agent: MS Search 4.0 Robot Disallow: / User-agent: MS Search 5.0 Robot Disallow: / User-agent: Naverbot Disallow: / User-agent: NetAnts Disallow: / User-agent: NetAttache Disallow: / User-agent: NetMechanic Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Openfind Disallow: / User-agent: OpenindexSpider Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: psbot Disallow: / User-agent: QuepasaCreep Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RMA Disallow: / User-agent: SemrushBot Disallow: / User-agent: SightupBot Disallow: / User-agent: SiteBot Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SiteSucker Disallow: / User-agent: Sogou web spider Disallow: / User-agent: sosospider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: Speedy Disallow: / User-agent: suggybot Disallow: / User-agent: SuperBot Disallow: / User-agent: SuperBot/2.6 Disallow: / User-agent: suzuran Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: Teleport Disallow: / User-agent: Telesoft Disallow: / User-agent: The Intraformant Disallow: / User-agent: TheNomad Disallow: / User-agent: TightTwatBot Disallow: / User-agent: Titan Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: TosCrawler Disallow: / User-agent: True_Robot Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: turingos Disallow: / User-agent: TurnitinBot Disallow: / User-agent: UrlPouls Disallow: / User-agent: URLy Warning Disallow: / User-agent: VCI Disallow: / User-agent: Web Image Collector Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: WebCopier Disallow: / User-agent: webcopy Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: webmirror Disallow: / User-agent: WebReaper Disallow: / User-agent: WebSauger Disallow: / User-agent: website extractor Disallow: / User-agent: Website Quester Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebStripper/2.02 Disallow: / User-agent: WebZip Disallow: / User-agent: wget Disallow: / User-agent: WikioFeedBot Disallow: / User-agent: WinHTTrack Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: yacy Disallow: / User-agent: YRSPider Disallow: / User-agent: Zeus Disallow: / User-agent: Zookabot Disallow: / ## EXCLUSION BOTS User-agent: ConveraCrawler User-agent: Jetbot User-agent: Meltwater User-agent: Digimind User-agent: Knowings User-agent: kbcrawl User-agent: AmiSoftware User-agent: Ask n read User-agent: Qwam content intelligence User-agent: Zite User-agent: FlipboardProxy User-agent: Youmag User-agent: Synthesio User-agent: trendybuzz User-agent: spotter User-agent: linkfluence User-agent: 5emeRue User-agent: Augure User-agent: Sindup User-agent: Corporama User-agent: readability.com User-agent: grub-client User-agent: ia_archiver User-agent: ia_archiver-web.archive.org User-agent: k2spider User-agent: libwww User-agent: 5erue User-agent: adequat User-agent: adequat-systems User-agent: auramundi User-agent: coexel User-agent: leadbox User-agent: Moreover User-agent: mytwip User-agent: NewsNow User-agent: Newzbin User-agent: opinion-tracker User-agent: proxem User-agent: score3 User-agent: trendeo User-agent: vecteurplus User-agent: verticalsearch User-agent: vsw User-agent: winello User-agent: Fetch User-agent: infoseek User-agent: MSIECrawler User-agent: sitecheck.internetseer.com User-agent: TeleportPro User-agent: Zealbot User-agent: asknread.com User-agent: ellisphere User-agent: spotter User-agent: 008 User-agent: Alexibot User-agent: AlvinetSpider User-agent: Antenne Hatena User-agent: ApocalXExplorerBot User-agent: asterias User-agent: BackDoorBot/1.0 User-agent: BizInformation User-agent: Black Hole User-agent: BlowFish/1.0 User-agent: BotALot User-agent: BuiltBotTough User-agent: Bullseye/1.0 User-agent: BunnySlippers User-agent: Cegbfeieh User-agent: CheeseBot User-agent: CherryPicker User-agent: CherryPickerElite/1.0 User-agent: CherryPickerSE/1.0 User-agent: CopyRightCheck User-agent: cosmos User-agent: Crescent User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: DISCo Pump 3.1 User-agent: DittoSpyder User-agent: dotbot User-agent: EmailCollector User-agent: EmailSiphon User-agent: EmailWolf User-agent: EroCrawler User-agent: ExtractorPro User-agent: Flamingo_SearchEngine User-agent: Foobot User-agent: hloader User-agent: httplib User-agent: HTTrack User-agent: HTTrack 3.0 User-agent: humanlinks User-agent: Igentia User-agent: InfoNaviRobot User-agent: JennyBot User-agent: JikeSpider User-agent: Kenjin Spider User-agent: LexiBot User-agent: libWeb/clsHTTP User-agent: LinkextractorPro User-agent: LinkScan/8.1a Unix User-agent: LinkWalker User-agent: lwp-trivial User-agent: lwp-trivial/1.34 User-agent: Mata Hari User-agent: Microsoft URL Control - 5.01.4511 User-agent: Microsoft URL Control - 6.00.8169 User-agent: MIIxpc User-agent: MIIxpc/4.2 User-agent: Mister PiX User-agent: MLBot User-agent: moget User-agent: moget/2.1 User-agent: MS Search 4.0 Robot User-agent: MS Search 5.0 Robot User-agent: Naverbot User-agent: NetAnts User-agent: NetAttache User-agent: NetMechanic User-agent: NICErsPRO User-agent: Offline Explorer User-agent: Openfind User-agent: OpenindexSpider User-agent: ProPowerBot/2.14 User-agent: ProWebWalker User-agent: psbot User-agent: QuepasaCreep User-agent: QueryN Metasearch User-agent: RepoMonkey User-agent: RMA User-agent: SemrushBot User-agent: SightupBot User-agent: SiteBot User-agent: SiteSnagger User-agent: SiteSucker User-agent: Sogou web spider User-agent: sosospider User-agent: SpankBot User-agent: spanner User-agent: Speedy User-agent: suggybot User-agent: SuperBot User-agent: SuperBot/2.6 User-agent: suzuran User-agent: Szukacz/1.4 User-agent: Teleport User-agent: Telesoft User-agent: The Intraformant User-agent: TheNomad User-agent: TightTwatBot User-agent: Titan User-agent: toCrawl/UrlDispatcher User-agent: TosCrawler User-agent: True_Robot User-agent: True_Robot/1.0 User-agent: turingos User-agent: TurnitinBot User-agent: UrlPouls User-agent: URLy Warning User-agent: VCI User-agent: Web Image Collector User-agent: WebAuto User-agent: WebBandit User-agent: WebBandit/3.50 User-agent: WebCopier User-agent: webcopy User-agent: WebEnhancer User-agent: WebmasterWorldForumBot User-agent: webmirror User-agent: WebReaper User-agent: WebSauger User-agent: website extractor User-agent: Website Quester User-agent: Webster Pro User-agent: WebStripper User-agent: WebStripper/2.02 User-agent: WebZip User-agent: wget User-agent: WikioFeedBot User-agent: WinHTTrack User-agent: WWW-Collector-E User-agent: yacy User-agent: YRSPider User-agent: Zeus User-agent: Zookabot User-agent: Linguee Bot Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: MJ12bot/v1.0.8 (http://majestic12.co.uk/bot.php?+) Disallow: / User-agent: YandexSomething/1.0 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; m) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; H) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; P) Disallow: / User-agent: Yandex/1.01.001 (compatible; Win16; I) Disallow: / User-agent: +BecomeBot/3.0 Disallow: / User-agent: AISearchBot Disallow: / User-agent: Alexibot Disallow: / User-agent: AmazonCloudServices Disallow: / User-agent: Apexoo Spider 1.0 Disallow: / User-agent: Aqua_Products Disallow: / User-agent: Arachnoidea Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: BackDoorBot Disallow: / User-agent: DotBot/1.1 Disallow: / User-agent: YandexBot Disallow: / User-agent: YandexBot/3.0 Disallow: / User-agent: SemrushBot-SA/0.97 Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot) Disallow: / User-agent: AsapBot Disallow: / User-agent: semantic-visions.com crawler Disallow: / User-agent: MetaURI API/2.0 +metauri.com Disallow: / User-agent: MetaURI Disallow: / User-agent: citeseerxbot Disallow: / User-agent: autokrawl Disallow: / User-agent: Baiduspider Disallow: / User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html) Disallow: / User-agent: becomebot Disallow: / User-agent: BecomeBot/3.0 Disallow: / User-agent: Bitacle Disallow: / User-agent: Bitacle bot Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: Bitacle* Disallow: / User-agent: Black Hole Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: BobCrawl Disallow: / User-agent: boitho Disallow: / User-agent: boitho.com Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: bot/1.0 Disallow: / User-agent: bot/1.0+(bot;+http://;[email protected]) Disallow: / User-agent: BotALot Disallow: / User-agent: botmobi Disallow: / User-agent: bsalsa Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Butterfly Disallow: / User-agent: Cazoodle-Bot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: Cegbfeieh Disallow: / User-agent: Charlotte Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: classroombot Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraCrawler/0.9e Disallow: / User-agent: Copernic Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: crawl13.exabot.com Disallow: / User-agent: crawl6.exabot.com Disallow: / User-agent: Crescent Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: cyberpatrol Disallow: / User-agent: cyberpatrolcrawler Disallow: / User-agent: datacha0s/2.0 Disallow: / User-agent: DBLBot Disallow: / User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0 Disallow: / User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 Disallow: / User-agent: dcbot Disallow: / User-agent: del.icio.us Disallow: / User-agent: del.icio.us-thumbnails/1.0 Disallow: / User-agent: DepSpid Disallow: / User-agent: discobot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo Disallow: / User-agent: DotBot Disallow: / User-agent: dotbot Disallow: / User-agent: DTAAgent Disallow: / User-agent: eagle Disallow: / User-agent: ee: Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: envolk Disallow: / User-agent: EroCrawler Disallow: / User-agent: EUDORA Disallow: / User-agent: Everest-Vulcan Disallow: / User-agent: exabot Disallow: / User-agent: Exabot Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Images/3.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: exooba Disallow: / User-agent: ExtractorPro Disallow: / User-agent: FairAd Client Disallow: / User-agent: FairShare Disallow: / User-agent: fast Disallow: / User-agent: fast Enterprise crawler Disallow: / User-agent: fast metaweb crawler Disallow: / User-agent: FAST-WebCrawler Disallow: / User-agent: fast-Webcrawler Disallow: / User-agent: findlinks/1.1.1-a5 Disallow: / User-agent: findlinks/1.1.3-beta9 Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: flatlandbot Disallow: / User-agent: jack Disallow: / User-agent: JS-Kit URL Resolver Disallow: / User-agent: KomodiaBot Disallow: / User-agent: FollowSiteBot Disallow: / User-agent: Foobot Disallow: / User-agent: FunWebProducts Disallow: / User-agent: FunWebProducts f39-2350 Disallow: / User-agent: FyberSpider Disallow: / User-agent: Gaisbot Disallow: / User-agent: genieBot Disallow: / User-agent: genieBot ([email protected]) Disallow: / User-agent: genieBot+([email protected]) Disallow: / User-agent: [email protected] Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: Gigabot Disallow: / User-agent: Gigabot* Disallow: / User-agent: Gigabot/2.0 Disallow: / User-agent: GingerCrawler Disallow: / User-agent: grub Disallow: / User-agent: grub-client Disallow: / User-agent: gsa-crawler Disallow: / User-agent: Gulliver Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: HenryTheMiragoRobot Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.12.1 +http://www.page-store.com) Disallow: / User-agent: hloader Disallow: / User-agent: HMSE_Robot Disallow: / User-agent: HornySexSearch Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: HTTP/1.1 zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: httplib Disallow: / User-agent: humanlinks Disallow: / User-agent: ia_archive Disallow: / User-agent: ia_archiver-web.archive.org Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: IBM Disallow: / User-agent: IBM_Planetwide Disallow: / User-agent: ichiro/2.0 Disallow: / User-agent: ichiro/2.0 ([email protected]) Disallow: / User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) Disallow: / User-agent: IconSurf Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: InfoPath Disallow: / User-agent: InfoPath.1 Disallow: / User-agent: InfoPath.2 Disallow: / User-agent: IntegraTelecom Disallow: / User-agent: internetseer Disallow: / User-agent: InternetSeer.com Disallow: / User-agent: ips-agent Disallow: / User-agent: IRLbot Disallow: / User-agent: IRLbot/3.0 Disallow: / User-agent: IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler) Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: Isidorus Disallow: / User-agent: Java Bot 2.0 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: JavaBot Disallow: / User-agent: JavaBot 2.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: Jyxobot Disallow: / User-agent: kalooga Disallow: / User-agent: KaloogaBot Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/) Disallow: / User-agent: kyluka Disallow: / User-agent: LaBot Disallow: / User-agent: larbin Disallow: / User-agent: larbin_2.6.1 Disallow: / User-agent: larbin_2.6.1+([email protected]) Disallow: / User-agent: laycat Disallow: / User-agent: LexiBot Disallow: / User-agent: LexiBotWeb Image Collector Disallow: / User-agent: libcurl-agent Disallow: / User-agent: LinkAider Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: LinkWalker Disallow: / User-agent: LinkWalker/2.0 Disallow: / User-agent: LinkWalker/2.0 - www.seventwentyfour.com Disallow: / User-agent: LMQueueBot/0.2 Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: LocalBot Disallow: / User-agent: LocalcomBot Disallow: / User-agent: looksmart Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: Mail.Ru Disallow: / User-agent: MantraAgent Disallow: / User-agent: Mata Hari Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: MIIxpc Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: Mister PiX Disallow: / User-agent: MJ12bot Disallow: / User-agent: MJ12bot/v1.2.0 Disallow: / User-agent: MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php? ) Disallow: / User-agent: MLBot Disallow: / User-agent: Modified GMMBotRightHere Disallow: / User-agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) Disallow: / User-agent: Mozilla/5.0+(compatible;+BecomeBot/3.0;++http://www.become.com/site_owners.html) Disallow: / User-agent: MSR-ISRCCrawler Disallow: / User-agent: nambu Disallow: / User-agent: NationalDirectory-SuperSpider Disallow: / User-agent: NaverBot Disallow: / User-agent: naverbot Disallow: / User-agent: NetAnts Disallow: / User-agent: NetMechanic Disallow: / User-agent: nexen Disallow: / User-agent: NextGenSearchBot Disallow: / User-agent: NG/2.0 Disallow: / User-agent: nicebot Disallow: / User-agent: NICErsPRO Disallow: / User-agent: noxtrumbot/1.0 Disallow: / User-agent: Nutch Disallow: / User-agent: obot Disallow: / User-agent: Ocelli Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Explorer/1.5 Disallow: / User-agent: Offline Explorer/1.9 Disallow: / User-agent: Offline Explorer/2.5 Disallow: / User-agent: OmniExplorer Disallow: / User-agent: OmniExplorer_Bot Disallow: / User-agent: OmniExplorer_Bot/1.07 Disallow: / User-agent: OmniExplorer_Bot/1.09 Disallow: / User-agent: OmniExplorer_Bot/1.10 Disallow: / User-agent: OnTownsBot Disallow: / User-agent: OOZBOT Disallow: / User-agent: Openbot Disallow: / User-agent: Openfind Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: OpenX Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: page_prefetcher Disallow: / User-agent: PerMan Disallow: / User-agent: Piffany_Web_Cacher_v0.91 Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: ProWebWalker Disallow: / User-agent: psbot Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot/0.1+(+http://www.picsearch.com/bot.html) Disallow: / User-agent: PSIBots Disallow: / User-agent: QEAVis Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: Rankivabot Disallow: / User-agent: REAP-Crawler Disallow: / User-agent: RepoMonkey Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RiceComputerArchitecture Disallow: / User-agent: RMA Disallow: / User-agent: robotgenius Disallow: / User-agent: Robozilla Disallow: / User-agent: ru_com_viewer Disallow: / User-agent: RufusBot Disallow: / User-agent: SapphireWebCrawler Disallow: / User-agent: Twiceler www.cuill.com/robots.html Disallow: / User-agent: k2spider Disallow: / User-agent: grub-client Disallow: / User-agent: wget Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: / User-agent: archive.org_bot Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: IsraBot Disallow: / User-agent: Orthogaffe Disallow: / User-agent: Zealbot Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: fast Disallow: / User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow: / User-agent: Twiceler www.cuill.com/twiceler/robot.html Disallow: / User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: sbider Disallow: / User-agent: SBIder Disallow: / User-agent: SBIder/SBIder-0.8.2-dev Disallow: / User-agent: scooter Disallow: / User-agent: Scooter/2.0 Disallow: / User-agent: scoutjet Disallow: / User-agent: Scrubby Disallow: / User-agent: searchme Disallow: / User-agent: searchpreview Disallow: / User-agent: Sensis Web Crawler Disallow: / User-agent: Sensis Web Crawler (search_comments\\at\\sensis\\dot\\com\\dot\\au) Disallow: / User-agent: Sensis+Web+Crawler+ Disallow: / User-agent: Servage Disallow: / User-agent: ServageRobot Disallow: / User-agent: setooz Disallow: / User-agent: ShopWiki Disallow: / User-agent: Sidewinder Disallow: / User-agent: Sika Disallow: / User-agent: sitecat Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: SiteSnagger Disallow: / User-agent: Slicehost Disallow: / User-agent: Slurp.so/1.0 Disallow: / User-agent: slurp@inktomi Disallow: / User-agent: [email protected] Disallow: / User-agent: Slurp/2.0-KiteHourly Disallow: / User-agent: Slurp/2.0-OwlWeekly Disallow: / User-agent: Slurp/2.0j Disallow: / User-agent: Slurp/3.0-AU Disallow: / User-agent: SlySearch Disallow: / User-agent: SlySearch/1.0 Disallow: / User-agent: SlySearch/1.x Disallow: / User-agent: Snapbot Disallow: / User-agent: Snapbot/1.0 Disallow: / User-agent: SnapPreviewBot Disallow: / User-agent: Snappy Disallow: / User-agent: Snappybot Disallow: / User-agent: sogou spider Disallow: / User-agent: sogou+spider Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: Speedy Disallow: / User-agent: speedyspider Disallow: / User-agent: sphsearch Disallow: / User-agent: sproose Disallow: / User-agent: StackRambler/2.0 Disallow: / User-agent: StackRambler/2.0+(MSIE+incompatible) Disallow: / User-agent: SuperBot/4.4.0.60 (Windows XP) Disallow: / User-agent: SuperPages Disallow: / User-agent: SurveyBot Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: SurveyBot/2.3 (Whois Source) Disallow: / User-agent: uberbot Disallow: / User-agent: UltraSeek Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: URLy Warning Disallow: / User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T) Disallow: / User-agent: ViolaBot Disallow: / User-agent: WebCopier Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.3.0 Disallow: / User-agent: WebCopier v3.3.2 Disallow: / User-agent: WebCrawler Disallow: / User-agent: WebDataCentreBot Disallow: / User-agent: WebEnhancer Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: WebSauger Disallow: / User-agent: Websense Disallow: / User-agent: Website Quester Disallow: / User-agent: Webspider Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebViewer Disallow: / User-agent: WebZip Disallow: / User-agent: WebZIP/3.5 (http://www.spidersoft.com) Disallow: / User-agent: whois.de Disallow: / User-agent: whoisde.de Disallow: / User-agent: WillyBot Disallow: / User-agent: Win32 Disallow: / User-agent: worio Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu Link Sleuth 1.2i Disallow: / User-agent: Xenu's Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Xenu Link Sleuth/1.3.8 Disallow: / User-agent: Y!J Disallow: / User-agent: y!j-bsc Disallow: / User-agent: Y!J-BSC Disallow: / User-agent: Y!J-BSC/1.0 Disallow: / User-agent: TunitinBot Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: QuepasaCreep Disallow: / User-agent: Jetbot Disallow: / User-agent: Meltwater Disallow: / User-agent: NewsNow Disallow: / User-agent: Digimind Disallow: / User-agent: Knowings D Disallow: / User-agent: kbcrawl Disallow: / User-agent: Wget Disallow: / User-agent: AmiSoftware Disallow: / User-agent: Newzbin Disallow: / User-agent: Ask n read Disallow: / User-agent: Qwam content intelligence Disallow: / User-agent: y!j-bsc/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) Disallow: / User-agent: y!j/1.0 Disallow: / User-agent: Yahoo-Blogs Disallow: / User-agent: Yahoo-Blogs/v3.9 Disallow: / User-agent: Yahoo-MMCrawler/3.x Disallow: / User-agent: YahooSeeker Disallow: / User-agent: YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/) Disallow: / User-agent: YahooYSMcm Disallow: / User-agent: Zite Disallow: / User-agent: flipboard Disallow: / User-agent: FlipboardProxy Disallow: / User-agent: Youmag Disallow: / User-agent: YahooYSMcm/2.0.0 Disallow: / User-agent: yandex Disallow: / User-agent: YandexSomething Disallow: / User-agent: Yanga Disallow: / User-agent: Yeti Disallow: / User-agent: yoogliFetchAgent/0.1 Disallow: / User-agent: Zao Disallow: / User-agent: zermelo Disallow: / User-agent: Zeus Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: zibber Disallow: / User-agent: zibber v0.1 Disallow: / User-agent: zibber-v0.1(www.zibb.com/crawler/) Disallow: / User-agent: ZupeeCrawler Disallow: / User-agent: zyborg Disallow: / User-agent: ZyBorg/1.0 Disallow: / User-agent: BLEXBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: dotbot Disallow: / User-agent: ScoutJet Disallow: / User-agent: Exabot Disallow: / User-agent: Snoothbot Disallow: / User-agent: Nutch Disallow: / User-agent: MJ12bot Disallow: / User-agent: Daum Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Xenu's Disallow: / User-agent: WeSEE Disallow: / User-agent: ShopWiki Disallow: / user-agent: crystalsemantics disallow: / user-agent: f-six disallow: / user-agent: sitebot disallow: / User-agent: trendictionbot Disallow: / User-agent: scrapybot Disallow: / User-agent: VocusBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: ezooms.bot Disallow: / User-agent: skimlinks.com Disallow: / User-agent: SeznamBot Disallow: / User-agent: wikiwix-bot Disallow: / User-agent: semantissimo Disallow: / User-agent: MLBot Disallow: / User-agent: LemonSources Disallow: / User-agent: discobot Disallow: / User-agent: linkfluence Disallow: / User-agent: ltbot Disallow: / User-agent: aiHitBot Disallow: / User-agent: language-tools.com Disallow: / User-agent: TREC-KBA-Bot Disallow: / User-agent: yacybot Disallow: / User-agent: SemrushBot Disallow: / User-agent: Socialradarbot Disallow: / User-agent: Havij Disallow: / User-agent: proximic Disallow: / User-agent: MireoBot Disallow: / User-agent: brandwatch Disallow: / User-agent: seoprofiler Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Bot clicker Disallow: / User-agent: WBSearchBot Disallow: / User-agent: EC2LinkFinder Disallow: / User-agent: ExaleadCloudview Disallow: / User-agent: AdMedia Disallow: / User-agent: grepnetstat.com Disallow: / User-agent: Genieo Disallow: / User-agent: NerdByNature Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Genieo Disallow: / User-agent: Sindup Disallow: / User-agent: netseer Disallow: / User-agent: ichiro Disallow: / User-agent: TwengaBot Disallow: / User-agent: netseer.com Disallow: / User-agent: nagios Disallow: / User-agent: linkdex.com Disallow: / User-agent: FeedRetriever Disallow: / User-agent: CrystalSemanticsBot Disallow: / User-agent: Argus Presse Disallow: / User-agent: IBM ICA Crawler Disallow: / User-agent: nutch agent Disallow: / User-agent: omgilibot Disallow: / User-agent: Genieo Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: PaperLiBot Disallow: / User-agent: SearchmetricsBot Disallow: / User-agent: Linguee Disallow: / User-agent: ip-web-crawler.com Disallow: / User-agent: SiteExplorer Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: LinksCrawler Disallow: / User-agent: archive.org Disallow: / User-agent: CompSpyBot Disallow: / User-agent: netEstate Disallow: / User-agent: MyNutchTest Disallow: / User-agent: admantx Disallow: / User-agent: genieo.com Disallow: / User-agent: mixdata.com Disallow: / User-agent: ubermetrics-technologies Disallow: / User-agent: grapeshot Disallow: / User-agent: CCBot Disallow: / User-agent: WASALive Disallow: / User-agent: URLAppendBot Disallow: / User-agent: MaxPointCrawler Disallow: / User-agent: wotbox Disallow: / User-agent: Aboundexbot Disallow: / User-agent: XoviBot Disallow: / User-agent: psbot Disallow: / User-Agent: Slurp User-agent: Yahoo! Slurp User-agent: Baiduspider User-agent: TweetedTimes Bot User-agent: TweetmemeBot User-agent: PaperLiBot User-agent: BomboraBot User-agent: WF search User-agent: heritrix User-agent: expo9 User-agent: NetSeer User-agent: AddThis.com User-agent: Feed User-agent: Sogou User-agent: ZumBot User-agent: Feedspot User-agent: acebookexternalhit User-agent: StorygizeBot User-agent: Scrapy User-agent: GermCrawler User-agent: Netvibes User-agent: Vagabondo User-agent: Genieo User-agent: um-IC User-agent: Typhoeus User-agent: newspaper User-agent: weborama-fetcher User-agent: ScooperBot User-agent: BYO-2 User-agent: SimplePie User-agent: GroupHigh User-agent: feedparser User-agent: UniversalFeedParser User-agent: Pulsepoint User-agent: Steeler User-agent: GarlikCrawler User-agent: Flipboard User-agent: FlipboardRSS User-agent: Moreover User-agent: Mercury User-agent: FeedBurner User-agent: Yahoo User-agent: omgili User-agent: Bloglovin User-agent: YahooCacheSystem User-agent: mTalk User-agent: EveryoneSocialBot User-agent: EzineArticlesLinkScanner User-agent: dlvr.it User-agent: pfblogs User-agent: Aol User-agent: ToutiaoSpider User-agent: Photon User-agent: PicoFeed User-agent: um-FC User-agent: Digg User-agent: Synaptive User-agent: YisouSpider User-agent: BYO-4 User-agent: Feedbin User-agent: Twingly User-agent: Tiny User-agent: WWW-Mechanize User-agent: CommaFeed User-agent: TkBot User-agent: TinEye-bot User-agent: Exabot User-agent: coccoc User-agent: BLEXBot User-agent: uMBot-LN User-agent: QuerySeekerSpider User-agent: Yandex User-agent: YandexBot User-agent: YandexMobileBot User-agent: YandexImageResizer User-agent: YandexImages User-agent: sogou spider User-agent: Sogou web spider User-agent: Baiduspider User-agent: YoudaoBot User-agent: MJ12bot User-agent: spbot User-agent: 360Spider User-agent: SeznamBot User-Agent: AhrefsBot User-Agent: Linkdex User-agent: linkdexbot User-agent: SemrushBot User-agent: SemrushBot-SA User-agent: CFNetwork User-agent: SputnikBot User-agent: TurnitinBot User-agent: proximic User-agent: grapeshot User-agent: VigLink User-agent: Clickagy User-agent: YaK User-agent: DeuSu User-agent: Taboolabot User-agent: Pinterestbot User-agent: psbot User-agent: Plukkie User-agent: GetIntent User-agent: Facebot User-agent: linkdexbot User-agent: Vegi User-agent: istellabot User-agent: Leikibot User-agent: Claritybot User-agent: PiplBot User-agent: Feedspotbot User-agent: moatbot User-agent: Yeti User-agent: contxbot User-agent: SEOkicks-Robot User-agent: rogerbot User-agent: Discordbot User-agent: Tapatalk User-agent: Mail.RU_Bot User-agent: coccocbot User-agent: HybridBot User-agent: Slack-ImgProxy User-agent: adbeat_bot User-agent: Kraken User-agent: MojeekBot User-agent: calculon User-agent: mcsbot User-agent: avkzarabotok User-agent: SearchmetricsBot User-agent: Uptimebot User-agent: yacybot User-agent: Applebot User-agent: Gigabot User-agent: g2reader-bot User-agent: Seoterritory User-agent: AhrefsBot User-agent: feedbot User-agent: Findxbot User-agent: Wotbox User-agent: SMTBot User-agent: Ocarinabot User-agent: Dragonbot User-agent: Zombiebot User-agent: metabot.ru User-agent: RSSMicro User-agent: _zbot User-agent: UptimeRobot User-agent: uipbot User-agent: ZoominfoBot User-agent: Flamingo_SearchEngine User-agent: bot-pge.chlooe.com User-agent: Girafabot User-agent: Multiviewbot User-agent: SafeDNSBot User-agent: GnowitNewsbot User-agent: Treato-Bot User-agent: Pvblcbot User-agent: Laserlikebot User-agent: trendiction User-agent: Mediatoolkitbot User-agent: CharlieBot User-agent: BUbiNG User-agent: Superfeedr User-agent: focusbot User-agent: Sottopop User-agent: MaxPointCrawler User-agent: Nutch-1.10 User-agent: wonderbot User-agent: zarabotok--doma User-agent: Treato-Bot User-agent: Pvblcbot User-agent: Laserlikebot User-agent: trendiction User-agent: Mediatoolkitbot User-agent: CharlieBot User-agent: BUbiNG User-agent: Superfeedr User-agent: focusbot User-agent: Sottopop User-agent: MaxPointCrawler User-agent: Nutch-1.10 User-agent: wonderbot User-agent: zarabotok--doma User-agent: UptimeRobot User-agent: uipbot User-agent: ZoominfoBot User-agent: Flamingo_SearchEngine User-agent: bot-pge.chlooe.com User-agent: Girafabot User-agent: Multiviewbot User-agent: SafeDNSBot User-agent: GnowitNewsbot User-agent: Treato-Bot User-agent: Pvblcbot User-agent: Laserlikebot User-agent: trendiction User-agent: Mediatoolkitbot User-agent: CharlieBot User-agent: BUbiNG User-agent: Superfeedr User-agent: focusbot User-agent: Sottopop User-agent: MaxPointCrawler User-agent: Nutch-1.10 User-agent: wonderbot User-agent: zarabotok--doma User-agent: Treato-Bot User-agent: Pvblcbot User-agent: Laserlikebot User-agent: trendiction User-agent: Mediatoolkitbot User-agent: CharlieBot User-agent: BUbiNG User-agent: Superfeedr User-agent: focusbot User-agent: Sottopop User-agent: MaxPointCrawler User-agent: Nutch-1.10 User-agent: wonderbot User-agent: zarabotok--doma User-agent: Cliqzbot User-agent: DotBot User-agent: Exabot User-agent: moatbot User-agent: Tapatalk Disallow: / User-agent: GingerCrawler Disallow: / User-agent: nutch Disallow: / User-agent: Urlbot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: spbot Disallow: / User-agent: ScoutJet Disallow: / User-agent: 008 Disallow: / User-agent: Yandex Disallow: / User-agent: Speedy Spider Disallow: / User-agent: Accelobot Disallow: / User-agent: dotbot Disallow: / User-agent: discobot Disallow: / User-agent: AboutUsBot Disallow: / User-agent: Vocus Disallow: / User-agent: MJ12bot Disallow: / User-agent: plukkie Disallow: / User-agent: AhrefsBot Disallow: / User-agent: http://ahrefs.com/robot Disallow: / User-agent: http://www.genieo.com/webfilter.html Disallow: / User-agent: ia_archiver Disallow: / User-agent: linkdex.com Disallow: / User-agent: linkdex.com/v2.0 Disallow: / User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected]) Disallow: / User-agent: NetcraftSurveyAgent/1.0; [email protected] Disallow: / User-agent: NetcraftSurveyAgent/1.0 Disallow: / User-agent: NetcraftSurveyAgent Disallow: / User-agent: NetcraftSurveyAgent/2.0 Disallow: / User-agent: SemrushBot/0.95 Disallow: / User-agent: MJ12bot/v1.4.3 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: JS-Kit URL Resolver, http://js-kit.com/ Disallow: / User-agent: JS-Kit Disallow: / User-agent: NetShelter ContentScan Disallow: / User-agent: Mozilla/5.0 (compatible; Sosospider/2.0; +http://help.soso.com/webspider.htm) Disallow: / User-agent: Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 3.1) Disallow: / User-agent: Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html) Disallow: / User-agent: YYSpider Disallow: / User-agent: YisouSpider Disallow: / User-agent: PagePeeker2 Disallow: / User-agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) Disallow: / User-agent: Ruby Disallow: / User-agent: Abonti Disallow: / User-agent: Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html) Disallow: / User-agent: SEOkicks Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: Image\ Sucker Disallow: / User-agent: SuperBot Disallow: / User-agent: Web\ Image\ Collector Disallow: / User-agent: Web\ Sucker Disallow: / User-agent: UbiCrawler Disallow: / User-agent: zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: linko Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / Crawl-delay: 10 User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /xmlrpc/
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
Selecciona una forma de contacto
¿No encuentras lo que buscas? También puedes descargarte la Guía: Primeros pasos, visitar la sección de Preguntas frecuentes, nuestra Wiki SEO: conocimientos básicos o nuestro Blog.
Utilizamos cookies para el buen funcionamiento de nuestra web y con fines analíticos y publicitarios. Puedes activar o desactivar las cookies opcionales. Para más información consulta los siguientes enlaces.
Meta Title
(Deseable)