# ----------------------------------------------------------- # robots.txt for http(s)://tinycms.xyz, last refresh 2021/07/12 # ----------------------------------------------------------- # not all bots below may obey robots.txt in general # or specific rules, respectively # cat /home/wwwlogs/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -nr | head -20 # ----------------------------------------------------------- User-agent: PetalBot Disallow: / User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Seekport Crawler Disallow: / User-agent: barkrowler Disallow: / # semrush bot User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / # ahrefs bot User-agent: AhrefsBot Disallow: / # moz bot User-agent: rogerbot Disallow: / User-agent: dotbot Disallow: / # Wayback Machine User-agent: ia_archiver Disallow: / # https://velen.io/ User-agent: VelenPublicWebCrawler Disallow: / # Baiduspider User-agent: Baiduspider Disallow: / # Block SoGou User-agent: Sogou Spider Disallow: / # Block Youdao User-agent: YoudaoBot Disallow: / # AdsBot User-agent: AdsBot-Google Disallow: /js/ # http://alphaseobot.com/bot.html User-agent: AlphaSeoBot Disallow: / # http://siteexplorer.info/about.html User-agent: SiteExplorer Disallow: / # http://www.sitesucker.us/mac/limitations.html User-agent: SiteSucker Disallow: / # https://www.openindex.io/saas/about-our-spider/ User-agent: OpenindexSpider Disallow: / User-agent: BoogleBot Disallow: / # http://www.backlinktest.com/crawler.html User-agent: BacklinkCrawler Disallow: / User-agent: ZoominfoBot Disallow: / # http://napoveda.seznam.cz/ User-agent: SeznamBot Disallow: / # http://www.website-datenbank.de User-agent: SeznamBot Disallow: / # Block netEstate NE Crawler (+http://www.website-datenbank.de/) User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/) Disallow: / User-agent: ZoominfoBot Disallow: / # Block BlexBot User-agent: BLEXBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: HubSpot Crawler Disallow: / User-agent: SeznamBot Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Mail.Ru Disallow: / User-agent: Serpstatbot Disallow: / User-agent: Baiduspider Disallow: / # https://megaindex.com/crawler User-agent: MegaIndex.ru Disallow: / User-agent: megaindex.com Disallow: / User-agent: SEOkicks Disallow: / User-agent: Keybot translation-search-machine Disallow:/ User-agent: Keywenbot Disallow:/ User-agent: Buck Disallow:/ User-agent: Mediatoolkitbot Disallow: / User-agent: Qwantify Disallow: / # ------------ # not exclude # ------------ User-agent: bingbot Crawl-delay: 5 User-agent: * Allow: / Disallow: /about/ Disallow: /products/ Disallow: /projects/ Disallow: /contact/ Disallow: /about/ Disallow: /contact.html Disallow: /about.html Disallow: /contact-us.html Disallow: /service.html Disallow: /service/
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)