User-agent: Mozilla/5.0 (compatible; Seznam screenshot-generator 2.1; +http://fulltext.sblog.cz/screenshot/) User-agent: AddThis.com robot [email protected] User-agent: MSRBOT (http://research.microsoft.com/research/sv/msrbot/) User-agent: MSRBOT # Microsoft Research Bot User-agent: SandCrawler # Unknown Microsoft crawler User-agent: Shim-Crawler # Japanese university research User-agent: ScoutJet # Not relevant yet # AhrefsBot User-agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) #Link tracker # User-agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, [email protected]) User-agent: DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, [email protected])DotBot # Link tracker User-agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected]) # Find Links User-agent: findlinks User-agent: findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.4-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.3-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.3-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.3-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.3-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.2-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.1-a1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a9 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a8 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a8 ( http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a7 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a5 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a4 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1-a3 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.1 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.06 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.0.8 (+http://wortschatz.uni-leipzig.de/findlinks/) User-agent: findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/) # User-agent: Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.searchtechnologies.com) User-agent: SputnikBot User-agent: 008 User-agent: Accoona User-agent: AddSugarSpiderBot User-agent: AnyApexBot User-agent: BecomeBot #Consumer shopping User-agent: BeslistBot # Dutch consumer shopping User-agent: BillyBobBot # Dead site User-agent: Bimbot # Hiding identity User-agent: Boitho # Not relevant User-agent: btbot # Torrent search User-agent: CatchBot # Dead Australian search User-agent: cosmos # not relevant User-agent: Covario User-agent: Cerberian # not relevant User-agent: DiamondBot User-agent: Discobot User-agent: EmeraldShield.com User-agent: envolk[ITS]spider User-agent: EsperanzaBot User-agent: Exabot User-agent: FAST Enterprise Crawler User-agent: FAST-WebCrawler User-agent: FDSE robot User-agent: FurlBot User-agent: FyberSpider User-agent: g2crawler User-agent: Gaisbot User-agent: GalaxyBot User-agent: genieBot User-agent: Gigabot User-agent: Girafabot User-agent: HappyFunBot User-agent: Holmes User-agent: htdig User-agent: iaskspider # unknown Chinese spider User-agent: ia_archiver # Alexa crawler User-agent: IRLbot User-agent: IssueCrawler User-agent: Jaxified Bot User-agent: Jyxobot User-agent: KoepaBot User-agent: L.webis User-agent: LapozzBot User-agent: Larbin User-agent: LDSpider User-agent: Linguee Bot User-agent: LinkWalker User-agent: lmspider User-agent: lwp-trivial User-agent: mabontland User-agent: magpie-crawler User-agent: MJ12bot User-agent: mogimogi User-agent: MojeekBot User-agent: Moreoverbot User-agent: Morning Paper User-agent: MVAClient User-agent: mxbot User-agent: NetResearchServer User-agent: NetSeer Crawler User-agent: NewsGator User-agent: NG-Search User-agent: nicebot User-agent: Nusearch User-agent: NutchCVS User-agent: Nymesis User-agent: obot User-agent: oegp User-agent: omgilibot User-agent: OmniExplorer_Bot User-agent: OOZBOT User-agent: Orbiter User-agent: PageBitesHyperBot User-agent: Peew User-agent: polybot User-agent: PostPost User-agent: Psbot - Image search User-agent: PycURL User-agent: Qseero User-agent: Radian6 User-agent: RAMPyBot User-agent: RufusBot User-agent: SBIder User-agent: Scrubby User-agent: SearchSight User-agent: Seekbot User-agent: semanticdiscovery User-agent: Sensis Web Crawler User-agent: SEOChat::Bot User-agent: ShopWiki User-agent: Shoula robot User-agent: silk User-agent: Sitebot User-agent: Snappy User-agent: Speedy Spider User-agent: Sqworm User-agent: suggybot User-agent: SurveyBot User-agent: TerrawizBot User-agent: TheSuBot # TinEye User-agent: TinEye/1.1 (http://tineye.com/crawler.html) User-agent: TinEye # User-agent: truwoGPS User-agent: TurnitinBot User-agent: TweetedTimes Bot User-agent: TwengaBot User-agent: updated User-agent: Urlfilebot User-agent: Vagabondo User-agent: Vortex User-agent: voyager User-agent: VYU2 User-agent: webcollage User-agent: Websquash.com User-agent: wf84 User-agent: WomlpeFactory User-agent: Xaldon_WebSpider User-agent: yacy User-agent: Yasaklibot User-agent: yoogliFetchAgent User-agent: Zao/0.1 (http://www.kototoi.org/zao/) User-agent: Mozilla/4.0 (compatible; Zealbot 1.0) User-agent: zspider/0.9-dev http://feedback.redkolibri.com/ # Zyborg User-agent: Mozilla/4.0 compatible ZyBorg/1.0 DLC ([email protected]; http://www.WISEnutbot.com) User-agent: Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker ([email protected]; http://www.WISEnutbot.com) User-agent: Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker ([email protected]; http://www.WISEnutbot.com) User-agent: Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com) User-agent: Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com) User-agent: Mozilla/4.0 compatible ZyBorg/1.0 ([email protected]; http://www.WISEnutbot.com) # Disallow: / #------------------------------------ # User-agent: Ultraseek Disallow: /*video/ Disallow: /en/video/ Disallow: /ja/video/ Disallow: /fr/video/ Disallow: /zh/video/ Disallow: /de/video/ Disallow: /it/video/ Disallow: /cs/video/ Disallow: /hu/video/ Disallow: /es/video/ Disallow: /ru/video/ Disallow: /pl/video/ Disallow: /pt/video/ Disallow: /ko/video/ Crawl-delay: 30 # ----------------------------------- # Chinese User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html) User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) User-agent: BaiDuSpider User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image User-agent: Sosospider+(+http://help.soso.com/webspider.htm) User-agent: sogou spider User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) User-agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html) User-agent: Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) Disallow: / # Russian # ----------------------------------- # Russian Spiders - Block all languages other than Russian and English User-agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) User-agent: Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) User-agent: Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 User-agent: igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html) User-agent: StackRambler/2.0 (MSIE incompatible User-agent: StackRambler/2.0 Disallow: / # -------------------------------------- # Japanese User-agent: Mozilla/4.0 (compatible; Arachmo) User-agent: ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) User-agent: ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html) User-agent: ichiro/2.0+(http://help.goo.ne.jp/door/crawler.html) User-agent: ichiro/2.0 ([email protected]) User-agent: ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) Disallow: /fr/ Disallow: /zh/ Disallow: /de/ Disallow: /it/ Disallow: /cs/ Disallow: /hu/ Disallow: /es/ Disallow: /ru/ Disallow: /pl/ Disallow: /pt/ Disallow: /ko/ Disallow: /vi/ Disallow: /th/ Crawl-delay: 60 # ---------------------------------------- # German User-agent: ABACHOBot User-agent: Mozilla/4.0 (compatible; B-l-i-t-z-B-O-T) User-agent: iCCrawler (http://www.iccenter.net/bot.htm) # German job listing site User-agent: SynooBot/0.7.1 (SynooBot; http://www.synoo.de/bot.html; [email protected]) User-agent: fastbot.de crawler 2.0 beta (http://www.fastbot.de) Disallow: /ja/ Disallow: /fr/ Disallow: /zh/ Disallow: /it/ Disallow: /cs/ Disallow: /hu/ Disallow: /es/ Disallow: /ru/ Disallow: /pl/ Disallow: /pt/ Disallow: /ko/ Disallow: /vi/ Disallow: /th/ Crawl-delay: 60 # --------------------------------- # French #Block downloads for all user agents User-agent: Pompos/1.3 http://dir.com/pompos.html User-agent: Pompos/1.2 http://pompos.iliad.fr User-agent: Pompos/1.1 http://pompos.iliad.fr User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/) Disallow: /ja/ Disallow: /zh/ Disallow: /de/ Disallow: /it/ Disallow: /cs/ Disallow: /hu/ Disallow: /es/ Disallow: /ru/ Disallow: /pl/ Disallow: /pt/ Disallow: /ko/ Disallow: /vi/ Disallow: /th/ Crawl-delay: 30 # ---------------------------------------- # Czech User-agent: SeznamBot/2.0 (+http://fulltext.seznam.cz/) User-agent: SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/) Disallow: /ja/ Disallow: /fr/ Disallow: /zh/ Disallow: /de/ Disallow: /it/ Disallow: /hu/ Disallow: /es/ Disallow: /ru/ Disallow: /pl/ Disallow: /pt/ Disallow: /ko/ Disallow: /vi/ Disallow: /th/ Crawl-delay: 30 # ---------------------------------------------- # Korean # Block downloads for all user agents User-agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) User-agent: Yeti/1.0 (+http://help.naver.com/robots/) User-agent: Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0 Disallow: /ja/ Disallow: /fr/ Disallow: /zh/ Disallow: /de/ Disallow: /it/ Disallow: /cs/ Disallow: /hu/ Disallow: /es/ Disallow: /ru/ Disallow: /pl/ Disallow: /pt/ Disallow: /vi/ Disallow: /th/ Crawl-delay: 30 # All User Agents # ------------------------------- User-agent: * Disallow: /en/404/ Disallow: /ja/404/ Disallow: /zh/404/ Disallow: /es/404/ Disallow: /de/404/ Disallow: /ru/404/ Disallow: /cs/404/ Disallow: /it/404/ Disallow: /hu/404/ Disallow: /pl/404/ Disallow: /pt/404/ Disallow: /ko/404/ Disallow: */?logout Crawl-delay: 10
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)