# $Id: robots.txt,v 1.9 2024/02/05 23:03:35 karl Exp $ # Robots.txt file # validate at http://www.sxw.org.uk/computing/robots/check.html # always disallow certain big or useless directories, and assorted symlinks. User-agent: * Crawl-Delay: 1 Disallow: /RCS/ Disallow: /cgi-bin Disallow: /devsrc/ Disallow: /ftp/ Disallow: /i-packages/ Disallow: /images/ Disallow: /proTeXt/ Disallow: /svn/ Disallow: /teTeX/ Disallow: /tetex/release Disallow: /tetex/tetex-src Disallow: /tetex/tetex-texmfdist Disallow: /tetex/tetex-texmfmain Disallow: /tetex/texmf Disallow: /tetex/texmf-dist Disallow: /tetex/texmf-local Disallow: /texlive/Contents/ Disallow: /texlive/Images/ Disallow: /texlive/devsrc/ Disallow: /texlive/svn/ Disallow: /texmf/ Disallow: /texmf-dist/ Disallow: /twg/MacTeX/ Disallow: /viewvc-docroot/ Disallow: /~rlevien/scan/ # Mostly from Project Honey Pot User-agent: "ClaudeBot" Disallow: / User-agent: "CFNetwork/129.19" Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.5.0_06 Disallow: / User-agent: Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET Disallow: / User-agent: "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon; .NET CLR 1.1.4322)" Disallow: / User-agent: "Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.8.1.1)" Disallow: / User-agent: "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1)" Disallow: / User-agent: "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us)" Disallow: / User-agent: "Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en)" Disallow: / User-agent: Java/1.5.0_04 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Disallow: / User-agent: Java/1.5.0_02 Disallow: / User-agent: LinkWalker Disallow: / User-agent: Java/1.4.2_03 Disallow: / User-agent: Java/1.5.0_03 Disallow: / User-agent: Java/1.5.0_05 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT) Disallow: / User-agent: Java/1.5.0_07 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Disallow: / User-agent: Java/1.4.2_05 Disallow: / User-agent: Java/1.5.0_01 Disallow: / User-agent: Java/1.5.0_09 Disallow: / User-agent: ISC Systems iRc Search 2.1 Disallow: / User-agent: Java/1.4.2_04 Disallow: / User-agent: Java/1.5.0 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;) Disallow: / User-agent: Java/1.4.2_01 Disallow: / User-agent: Java/1.5.0_10 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) Disallow: / User-agent: Crawler ([email protected]) Disallow: / User-agent: "ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)" Disallow: / ##################################### User-agent: http://www.live.com/ Disallow: / User-agent: http://search.msn.com/msnbot.htm Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: ArribaPacketRat Disallow: / User-agent: autoemailspider Disallow: / User-agent: baiduspider Disallow: / User-agent: Baiduspider Disallow: / User-agent: Bilbo Disallow: / User-agent: DigExt Disallow: / User-agent: dloader(NaverRobot)/1.0 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DTS Agent Disallow: / User-agent: fast Disallow: / User-agent: Getleft 1.1b2 Disallow: / User-agent: girafa Disallow: / User-agent: Gigabot/1.0 Disallow: / User-agent: grub-client Disallow: / User-agent: HTMLAB Disallow: / User-agent: httrack Disallow: / User-agent: ia_archiver Disallow: / User-agent: ImageVampire Disallow: / User-agent: k2spider Disallow: / User-agent: mail sweeper Disallow: / User-agent: MSIE 6.0 Disallow: / User-agent: MSIECrawler Disallow: / User-agent: NetCaptor Disallow: / User-agent: NITLE Blog Spider/0.01 Disallow: / User-agent: NPBot Disallow: / User-agent: nutch Disallow: / User-agent: oBot Disallow: / User-agent: Offline Explorer Disallow: / User-agent: PSBot Disallow: / User-agent: QuepasaCreep v0.9.13 Disallow: / User-agent: searchpreview Disallow:/ User-agent: Scooter/3.3 Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: SpiderKU/0.9 Disallow: / User-agent: Steeler Disallow: / User-agent: SurveyBot/2.3 Disallow: / User-agent: szukacz Disallow: / User-agent: Szukacz/1.9 Disallow: / User-agent: Szukacz/1.10.2 Disallow: / User-agent: Teoma Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Vagabondo/2.1 Disallow: / User-agent: vischeck_spiderBot/0.1libwww-perl/5.48 Disallow: / User-agent: vscooter Disallow: / User-agent: WebCopier v3.3 Disallow: / User-agent: WebCopier v3.2a Disallow: / User-agent: WebCopier Disallow: / User-agent: webcrawler Disallow: / User-agent: Web Downloader/4.9 Disallow: / User-agent: Web Downloader/5.8 Disallow: / User-agent: WebGather 3.0 Disallow: / User-agent: WebStripper/2.56 Disallow: / User-agent: WebZIP/3.65 Disallow: / User-agent: WebZIP Disallow: / #User-agent: Wget #Disallow: / User-agent: YahooSeeker/M1A1-R2D2 Disallow: / User-agent: Yahoo! Slurp Disallow: / User-agent: Yahoo-MMCrawler Disallow: / User-agent: Yeti Disallow: / User-agent: Zao Disallow: / User-agent: Zeus 2.6 Disallow: / User-agent: ConveraCrawler Disallow: / User-agent: msnbot-media/1.0 Disallow: / User-agent: MegaIndex Disallow: / User-agent: SemrushBot Disallow: /
Folgende Keywords wurden erkannt. Überprüfe die Optimierung dieser Keywords für Deine Seite.
(Nice to have)