La página no está disponible

Analiza tu página con Seobility
Te quedan 3 de 3 Checks diarios
Esta página no es accesible para el bot de Seobility porque está excluido en el archivo Robots.txt. El bot de Seobility no rastrea nada que excluya a Seobility o *.
URL problemática: http://op.nysed.gov
#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

User-agent: alexa             # Amazon's user agent
Disallow: / # disallow everything

User-agent: Alexabot             # Amazon's user agent
Disallow: / # disallow everything

User-agent: AlexaToolbar             # Amazon's user agent
Disallow: / # disallow everything

User-agent: Aranhabot             # Amazon's user agent
Disallow: / # disallow everything

User-agent: amazon-QBusiness    # Amazon Q Web Crawler
Disallow: / # disallow everything

User-agent: Amazon CloudFront             # Amazon's user agent
Disallow: / # disallow everything

User-agent: Amazonbot             # Amazon's user agent
Disallow: / # disallow everything

User-agent: amzn_assoc             # Amazon's user agent
Disallow: / # disallow everything

User-agent: amazon-kendra-web-crawler-* # all customers of Amazon Kendra's web crawler
Disallow: / # disallow everything

User-agent: amazon-kendra # all customers of Amazon Kendra's web crawler
Disallow: / # disallow everything

User-agent: AmazonAdBot             # Amazon's user agent
Disallow: / # disallow everything

User-agent: AmazonProductDiscoverybot             # Amazon's user agent
Disallow: / # disallow everything

User-agent: AWS*             # Amazon's user agent
Disallow: / # disallow everything

User-agent: aws-internal     # Amazon's user agent
Disallow: / # disallow everything

User-agent: aws*             # Amazon's user agent
Disallow: / # disallow everything

User-agent: echo             # Amazon's user agent
Disallow: / # disallow everything

User-agent: echo*             # Amazon's user agent
Disallow: / # disallow everything

User-agent: *Glue             # Amazon's user agent
Disallow: / # disallow everything

User-agent: Storebot-Google             # Google's user agent
Disallow: / # disallow everything

User-agent: Googlebot-Image             # Google's user agent
Disallow: / # disallow everything

User-agent: Googlebot-News             # Google's user agent
Disallow: / # disallow everything

User-agent: Googlebot-News             # Google's user agent
Disallow: / # disallow everything

User-agent: Googlebot-Video             # Google's user agent
Disallow: / # disallow everything

User-agent: AdsBot-Google-Mobile       # Google's user agent
Disallow: / # disallow everything

User-agent: Mediapartners-Google       # Google's user agent
Disallow: / # disallow everything

User-agent: FeedFetcher-Google        # Google's user agent
Disallow: / # disallow everything

User-agent: GoogleProducer            # Google's user agent
Disallow: / # disallow everything

User-agent: GoogleOther             # Google's user agent
Disallow: / # disallow everything

User-agent: Google-Extended          # Google's user agent
Disallow: / # disallow everything

User-agent: APIs-Google             # Google's user agent
Disallow: / # disallow everything

User-agent: grapeshot               # Oracle's user agent
Disallow: / # disallow everything

User-agent: 173.212soso
Disallow: /

User-agent: 192.comagent
Disallow: /

User-agent: 1noonbot
Disallow: /

User-agent: 1on1searchbot
Disallow: /

User-agent: 360Spider
Disallow: /

User-agent: 3de_search2
Disallow: /

User-agent: 3d_search
Disallow: /

User-agent: 3g bot
Disallow: /

User-agent: 3gse
Disallow: /

User-agent: 404checker
Disallow: /

User-agent: 404enemy
Disallow: /

User-agent: 50.nu
Disallow: /

User-agent: 80legs
Disallow: /

User-agent: A1 sitemap generator
Disallow: /

User-agent: A1 website download
Disallow: /

User-agent: A6-indexer
Disallow: /

User-agent: Aasp
Disallow: /

User-agent: Abachobot
Disallow: /

User-agent: Abonti
Disallow: /

User-agent: Abotemailsearch
Disallow: /

User-agent: Aboundex
Disallow: /

User-agent: Aboutusbot
Disallow: /

User-agent: Accmonitor compliance server
Disallow: /

User-agent: Accoon
Disallow: /

User-agent: Achulkov.net page walker
Disallow: /

User-agent: Acme.spider
Disallow: /

User-agent: Acoonbot
Disallow: /

User-agent: Activetouristbot
Disallow: /

User-agent: Acunetix
Disallow: /

User-agent: Adamm bot
Disallow: /

User-agent: Adbeat_bot
Disallow: /

User-agent: ADmantX
Disallow: /

User-agent: Adminshop.com
Disallow: /

User-agent: Ad muncher
Disallow: /

User-agent: Advanced email extractor
Disallow: /

User-agent: Aesop_com_spiderman
Disallow: /

User-agent: Aespider
Disallow: /

User-agent: AfD-Verbotsverfahren
Disallow: /

User-agent: Af knowledge now verity spider
Disallow: /

User-agent: Aggregator:vocus
Disallow: /

User-agent: Ah-ha.com crawler
Disallow: /

User-agent: Ahrefs
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: AIBOT
Disallow: /

User-agent: Aidu
Disallow: /

User-agent: AiHitBot
Disallow: /

User-agent: Aipbot
Disallow: /

User-agent: Aisiid
Disallow: /

User-agent: Aitcsrobot/1.1
Disallow: /

User-agent: Ajsitemap
Disallow: /

User-agent: Akamai-sitesnapshot
Disallow: /

User-agent: Alexawebsearchplatform
Disallow: /

User-agent: Alexfdownload
Disallow: /

User-agent: Alexibot
Disallow: /

User-agent: Alkalinebot
Disallow: /

User-agent: All acronyms bot
Disallow: /

User-agent: Alligator
Disallow: /

User-agent: AllSubmitter
Disallow: /

User-agent: AlphaBot
Disallow: /

User-agent: Alpha search agent
Disallow: /

User-agent: Amerla search bot
Disallow: /

User-agent: Amfibibot
Disallow: /

User-agent: Ampmppc.com
Disallow: /

User-agent: Amznkassocbot
Disallow: /

User-agent: Anarchie
Disallow: /

User-agent: Anemone
Disallow: /

User-agent: Anonymous
Disallow: /

User-agent: Anotherbot
Disallow: /

User-agent: Answerbot
Disallow: /

User-agent: Answerbus
Disallow: /

User-agent: Answerchase prove
Disallow: /

User-agent: Antbot
Disallow: /

User-agent: Antibot
Disallow: /

User-agent: Antisantyworm
Disallow: /

User-agent: Antro.net
Disallow: /

User-agent: Aonde-spider
Disallow: /

User-agent: Apexoo
Disallow: /

User-agent: Aport
Disallow: /

User-agent: Appengine-google
Disallow: /

User-agent: Appid: s~stremor-crawler-
Disallow: /

User-agent: Aqua_products
Disallow: /

User-agent: Arabot
Disallow: /

User-agent: Arachmo
Disallow: /

User-agent: Arachnophilia
Disallow: /

User-agent: Aria equalizer
Disallow: /

User-agent: Arianna.libero.it
Disallow: /

User-agent: Arikus_spider
Disallow: /

User-agent: Artavisbot
Disallow: /

User-agent: Artera
Disallow: /

User-agent: Art-online.com
Disallow: /

User-agent: Asaha search engine turkey
Disallow: /

User-agent: Ask
Disallow: /

User-agent: Aspider
Disallow: /

User-agent: ASPSeek
Disallow: /

User-agent: Asterias
Disallow: /

User-agent: Astrofind
Disallow: /

User-agent: Athenusbot
Disallow: /

User-agent: Atlocalbot
Disallow: /

User-agent: Atomic_email_hunter
Disallow: /

User-agent: Attach
Disallow: /

User-agent: Attrakt
Disallow: /

User-agent: Attributor
Disallow: /

User-agent: Augurfind
Disallow: /

User-agent: Auresys
Disallow: /

User-agent: Autobaron crawler
Disallow: /

User-agent: Autoemailspider
Disallow: /

User-agent: Autowebdir
Disallow: /

User-agent: Avsearch-
Disallow: /

User-agent: Axfeedsbot
Disallow: /

User-agent: Axonize-bot
Disallow: /

User-agent: Ayna
Disallow: /

User-agent: B2w
Disallow: /

User-agent: BackDoorBot
Disallow: /

User-agent: Backlink-Ceck
Disallow: /

User-agent: Backlink-check
Disallow: /

User-agent: BacklinkCrawler
Disallow: /

User-agent: Backrub
Disallow: /

User-agent: BackStreet
Disallow: /

User-agent: Backstreet browser
Disallow: /

User-agent: BackWeb
Disallow: /

User-agent: Badass
Disallow: /

User-agent: Bandit
Disallow: /

User-agent: Barkrowler
Disallow: /

User-agent: BatchFTP
Disallow: /

User-agent: Battleztar Bazinga
Disallow: /

User-agent: Baypup
Disallow: /

User-agent: BBBike
Disallow: /

User-agent: BDCbot
Disallow: /

User-agent: BDFetch
Disallow: /

User-agent: Becomebot
Disallow: /

User-agent: Becomejpbot
Disallow: /

User-agent: Beetlebot
Disallow: /

User-agent: Bender
Disallow: /

User-agent: Besserscheitern-crawl
Disallow: /

User-agent: BetaBot
Disallow: /

User-agent: Bigado.com
Disallow: /

User-agent: Big brother
Disallow: /

User-agent: Bigcliquebot
Disallow: /

User-agent: Big data
Disallow: /

User-agent: Bigfoot
Disallow: /

User-agent: Biglotron
Disallow: /

User-agent: Bilbo
Disallow: /

User-agent: Bilgibetabot
Disallow: /

User-agent: Bilgibot
Disallow: /

User-agent: Bintellibot
Disallow: /

User-agent: Bitacle
Disallow: /

User-agent: Bitlybot
Disallow: /

User-agent: Bitvouseragent
Disallow: /

User-agent: Bizbot
Disallow: /

User-agent: Bizworks retriever
Disallow: /

User-agent: Blackbird
Disallow: /

User-agent: Blackboard
Disallow: /

User-agent: Black.hole
Disallow: /

User-agent: Black Hole
Disallow: /

User-agent: Blackmask.net search engine
Disallow: /

User-agent: BlackWidow
Disallow: /

User-agent: Bladder fusion
Disallow: /

User-agent: Blaiz-bee
Disallow: /

User-agent: Blexbot
Disallow: /

User-agent: Blinkx
Disallow: /

User-agent: Blitzbot
Disallow: /

User-agent: Blog conversation project
Disallow: /

User-agent: Blogmyway
Disallow: /

User-agent: Blogpulselive
Disallow: /

User-agent: Blogrefsbot
Disallow: /

User-agent: Blogscope
Disallow: /

User-agent: Blogslive
Disallow: /

User-agent: Bloobybot
Disallow: /

User-agent: Blow
Disallow: /

User-agent: BlowFish
Disallow: /

User-agent: Blt
Disallow: /

User-agent: Bnf.fr_bot
Disallow: /

User-agent: Boaconstrictor
Disallow: /

User-agent: Boardreader
Disallow: /

User-agent: Boia.org
Disallow: /

User-agent: Boia-scan-agent
Disallow: /

User-agent: Boi_crawl_00
Disallow: /

User-agent: Boitho
Disallow: /

User-agent: Bolt
Disallow: /

User-agent: Bookmark buddy bookmark checker
Disallow: /

User-agent: Bookmark search tool
Disallow: /

User-agent: Bosug
Disallow: /

User-agent: BotALot
Disallow: /

User-agent: Bot apoena
Disallow: /

User-agent: Botrighthere
Disallow: /

User-agent: Botswana
Disallow: /

User-agent: Bottybot
Disallow: /

User-agent: Bpbot
Disallow: /

User-agent: Braintime_search
Disallow: /

User-agent: Brandprotect
Disallow: /

User-agent: Brandwatch
Disallow: /

User-agent: Brokenlinkcheck.com
Disallow: /

User-agent: Browseremulator
Disallow: /

User-agent: Browsermob
Disallow: /

User-agent: Bruinbot
Disallow: /

User-agent: Bsearchr&d
Disallow: /

User-agent: Bspider
Disallow: /

User-agent: Btbot
Disallow: /

User-agent: Btsearch
Disallow: /

User-agent: Bubing
Disallow: /

User-agent: Buddy
Disallow: /

User-agent: Buibui
Disallow: /

User-agent: Buildcms crawler
Disallow: /

User-agent: BuiltBotTough
Disallow: /

User-agent: BuiltWith
Disallow: /

User-agent: Bullseye
Disallow: /

User-agent: Bumblebee
Disallow: /

User-agent: BunnySlippers
Disallow: /

User-agent: Buscadorclarin
Disallow: /

User-agent: Buscaplus robi
Disallow: /

User-agent: Butterfly
Disallow: /

User-agent: Buyhawaiibot
Disallow: /

User-agent: Buzzbot
Disallow: /

User-agent: BuzzSumo
Disallow: /

User-agent: Byindia
Disallow: /

User-agent: Byspider
Disallow: /

User-agent: Byteserver
Disallow: /

User-agent: Bzbot
Disallow: /

User-agent: Cacheblaster
Disallow: /

User-agent: Caddbot
Disallow: /

User-agent: Cafi
Disallow: /

User-agent: Calculon
Disallow: /

User-agent: Camcrawler
Disallow: /

User-agent: Camelstampede
Disallow: /

User-agent: Canon-webrecord
Disallow: /

User-agent: Careerbot
Disallow: /

User-agent: Cataguru
Disallow: /

User-agent: Catchbot
Disallow: /

User-agent: CATExplorador
Disallow: /

User-agent: Cazoodle
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Ccgcrawl
Disallow: /

User-agent: Ccubee
Disallow: /

User-agent: Cd-preload
Disallow: /

User-agent: Cegbfeieh
Disallow: /

User-agent: Ce-preload
Disallow: /

User-agent: Cerberian drtrs
Disallow: /

User-agent: Cert figleafbot
Disallow: /

User-agent: Cfetch
Disallow: /

User-agent: Cfnetwork
Disallow: /

User-agent: Chameleon
Disallow: /

User-agent: Charlotte
Disallow: /

User-agent: Checkbot
Disallow: /

User-agent: Check&get
Disallow: /

User-agent: Checklinks
Disallow: /

User-agent: CheeseBot
Disallow: /

User-agent: Chemiede-nodebot
Disallow: /

User-agent: CherryPicker
Disallow: /

User-agent: Chilkat
Disallow: /

User-agent: ChinaClaw
Disallow: /

User-agent: Chlooe
Disallow: /

User-agent: Chromeframe
Disallow: /

User-agent: Cipinetbot
Disallow: /

User-agent: Cis455crawler
Disallow: /

User-agent: Citeseerxbot
Disallow: /

User-agent: Clariabot
Disallow: /

User-agent: Claritybot
Disallow: /

User-agent: Climate ark
Disallow: /

User-agent: Climateark spider
Disallow: /

User-agent: Cliqzbot
Disallow: /

User-agent: Cloud mapping
Disallow: /

User-agent: Clshttp
Disallow: /

User-agent: Clushbot
Disallow: /

User-agent: Coast scan engine
Disallow: /

User-agent: Coast webmaster pro
Disallow: /

User-agent: Coccoc
Disallow: /

User-agent: Coccocbot-web
Disallow: /

User-agent: Cogentbot
Disallow: /

User-agent: Cognitiveseo
Disallow: /

User-agent: Collapsarweb
Disallow: /

User-agent: Collector
Disallow: /

User-agent: Colocrossing
Disallow: /

User-agent: Combine
Disallow: /

User-agent: Com.plumanalytics
Disallow: /

User-agent: Connectsearch
Disallow: /

User-agent: Conpilot
Disallow: /

User-agent: Contentsmartz
Disallow: /

User-agent: Contextad bot
Disallow: /

User-agent: Contype
Disallow: /

User-agent: Cookienet
Disallow: /

User-agent: Coolbot
Disallow: /

User-agent: Coolcheck
Disallow: /

User-agent: Copernic
Disallow: /

User-agent: Copier
Disallow: /

User-agent: CopyRightCheck
Disallow: /

User-agent: Copyscape
Disallow: /

User-agent: Core-project
Disallow: /

User-agent: Cosmos
Disallow: /

User-agent: Covario-ids
Disallow: /

User-agent: Cowbot-
Disallow: /

User-agent: Cowdog bot
Disallow: /

User-agent: Crabbybot
Disallow: /

User-agent: Craftbot
Disallow: /

User-agent: C r a w l 3 r
Disallow: /

User-agent: Crawl_application
Disallow: /

User-agent: Crawler@
Disallow: /

User-agent: Crawler43.ejupiter.com
Disallow: /

User-agent: Crawler4j
Disallow: /

User-agent: Crawler.feedback
Disallow: /

User-agent: Crawler_for_infomine
Disallow: /

User-agent: Crawler.kpricorn.org
Disallow: /

User-agent: crawl.sogou.com
Disallow: /

User-agent: Crawly
Disallow: /

User-agent: CrazyWebCrawler
Disallow: /

User-agent: Creativecommons
Disallow: /

User-agent: Crescent
Disallow: /

User-agent: Cs-crawler
Disallow: /

User-agent: Cse html validator
Disallow: /

User-agent: CSHttp
Disallow: /

User-agent: Cshttpclient
Disallow: /

User-agent: Cuasarbot
Disallow: /

User-agent: Culsearch
Disallow: /

User-agent: Curious
Disallow: /

User-agent: Custo
Disallow: /

User-agent: Cvaulev
Disallow: /

User-agent: Cyberdog
Disallow: /

User-agent: Cybernavi_webget
Disallow: /

User-agent: Cyberpatrol sitecat webbot
Disallow: /

User-agent: Cyberspyder
Disallow: /

User-agent: Cydralspider
Disallow: /

User-agent: D1garabicengine
Disallow: /

User-agent: DatabaseDriverMysqli
Disallow: /

User-agent: DataCha0s
Disallow: /

User-agent: Datafountains
Disallow: /

User-agent: Dataparksearch
Disallow: /

User-agent: Dataprovider.com
Disallow: /

User-agent: Datascape robot
Disallow: /

User-agent: Dataspearspiderbot
Disallow: /

User-agent: Dataspider
Disallow: /

User-agent: Dattatec.com
Disallow: /

User-agent: Daumoa
Disallow: /

User-agent: DBLBot
Disallow: /

User-agent: Dcpbot
Disallow: /

User-agent: Declumbot
Disallow: /

User-agent: Deepindex
Disallow: /

User-agent: Deepnet crawler
Disallow: /

User-agent: Deeptrawl
Disallow: /

User-agent: Dejan
Disallow: /

User-agent: Del.icio.us-thumbnails
Disallow: /

User-agent: Deltascan
Disallow: /

User-agent: Delvubot
Disallow: /

User-agent: Demandbase-bot
Disallow: /

User-agent: Demon
Disallow: /

User-agent: Der gro§e bildersauger
Disallow: /

User-agent: Der große bildersauger
Disallow: /

User-agent: Deusu
Disallow: /

User-agent: Devil
Disallow: /

User-agent: Dfs-fetch
Disallow: /

User-agent: Diagem
Disallow: /

User-agent: Diamond
Disallow: /

User-agent: Dibot
Disallow: /

User-agent: Didaxusbot
Disallow: /

User-agent: Digext
Disallow: /

User-agent: Digger
Disallow: /

User-agent: Digincore
Disallow: /

User-agent: Digi-rssbot
Disallow: /

User-agent: Digitalarchivesbot
Disallow: /

User-agent: DigitalPebble
Disallow: /

User-agent: Digout4u
Disallow: /

User-agent: DIIbot
Disallow: /

User-agent: Dillo
Disallow: /

User-agent: Dirbuster
Disallow: /

User-agent: Dir_snatch.exe
Disallow: /

User-agent: Disco
Disallow: /

User-agent: Discobot
Disallow: /

User-agent: Discoverybot
Disallow: /

User-agent: Distilled-reputation-monitor
Disallow: /

User-agent: DittoSpyder
Disallow: /

User-agent: Djangotraineebot
Disallow: /

User-agent: Dkimrepbot
Disallow: /

User-agent: Dmoz downloader
Disallow: /

User-agent: DnyzBot
Disallow: /

User-agent: Docomo
Disallow: /

User-agent: Dof-verify
Disallow: /

User-agent: DomainAppender
Disallow: /

User-agent: DomainCrawler
Disallow: /

User-agent: Domainscan
Disallow: /

User-agent: DomainSigmaCrawler
Disallow: /

User-agent: DomainStatsBot
Disallow: /

User-agent: Domainwatcher bot
Disallow: /

User-agent: Dotbot
Disallow: /

User-agent: Dotspotsbot
Disallow: /

User-agent: Dow jones searchbot
Disallow: /

User-agent: Download
Disallow: /

User-agent: Download Wonder
Disallow: /

User-agent: Doy
Disallow: /

User-agent: Dragonfly
Disallow: /

User-agent: Drip
Disallow: /

User-agent: Drone
Disallow: /

User-agent: Dtaagent
Disallow: /

User-agent: DTS Agent
Disallow: /

User-agent: Dtsearchspider
Disallow: /

User-agent: Dumbot
Disallow: /

User-agent: Dwaar
Disallow: /

User-agent: Dxseeker
Disallow: /

User-agent: Eah
Disallow: /

User-agent: Earth platform indexer
Disallow: /

User-agent: Earth science educator  robot
Disallow: /

User-agent: EasyDL
Disallow: /

User-agent: Ebingbong
Disallow: /

User-agent: Ec2linkfinder
Disallow: /

User-agent: Ecairn-grabber
Disallow: /

User-agent: ECatch
Disallow: /

User-agent: ECCP/1.0
Disallow: /

User-agent: Echoosebot
Disallow: /

User-agent: Ecxi
Disallow: /

User-agent: Edisterbot
Disallow: /

User-agent: Edugovsearch
Disallow: /

User-agent: Egothor
Disallow: /

User-agent: Eidetica.com
Disallow: /

User-agent: EirGrabber
Disallow: /

User-agent: Elblindo the blind bot
Disallow: /

User-agent: Elisabot
Disallow: /

User-agent: Ellerdalebot
Disallow: /

User-agent: Emailcollector
Disallow: /

User-agent: Email extractor
Disallow: /

User-agent: Emailleach
Disallow: /

User-agent: Emailsiphon
Disallow: /

User-agent: EMail Siphon
Disallow: /

User-agent: Emailwolf
Disallow: /

User-agent: EMail Wolf
Disallow: /

User-agent: Emeraldshield
Disallow: /

User-agent: Empas_robot
Disallow: /

User-agent: Enabot
Disallow: /

User-agent: Endeca
Disallow: /

User-agent: Enigmabot
Disallow: /

User-agent: Enswer neuro bot
Disallow: /

User-agent: Enter user-agent
Disallow: /

User-agent: Entitycubebot
Disallow: /

User-agent: EroCrawler
Disallow: /

User-agent: E-societyrobot
Disallow: /

User-agent: Estylesearch
Disallow: /

User-agent: Esyndicat bot
Disallow: /

User-agent: Eurosoft-bot
Disallow: /

User-agent: Evaal
Disallow: /

User-agent: Evc-batch
Disallow: /

User-agent: Eventware
Disallow: /

User-agent: Everest-vulcan inc.
Disallow: /

User-agent: Evil
Disallow: /

User-agent: Exabot
Disallow: /

User-agent: Exactsearch
Disallow: /

User-agent: Exactseek
Disallow: /

User-agent: Exooba
Disallow: /

User-agent: Exploder
Disallow: /

User-agent: Express WebPictures
Disallow: /

User-agent: ExtLinksBot
Disallow: /

User-agent: Extractor
Disallow: /

User-agent: ExtractorPro
Disallow: /

User-agent: Extreme Picture Finder
Disallow: /

User-agent: Eyenetie
Disallow: /

User-agent: EyeNetIE
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: Ez-robot
Disallow: /

User-agent: Factbot
Disallow: /

User-agent: Fairad client
Disallow: /

User-agent: Falcon
Disallow: /

User-agent: Fastbot crawler
Disallow: /

User-agent: Fastbot.de crawler
Disallow: /

User-agent: Fast data search document retriever
Disallow: /

User-agent: Fast esp
Disallow: /

User-agent: Fast-search-engine
Disallow: /

User-agent: Fatbot
Disallow: /

User-agent: Favcollector
Disallow: /

User-agent: Faviconizer
Disallow: /

User-agent: Favorites sweeper
Disallow: /

User-agent: F-bot test pilot
Disallow: /

User-agent: FDM
Disallow: /

User-agent: Fdse robot
Disallow: /

User-agent: Fedcontractorbot
Disallow: /

User-agent: Fembot
Disallow: /

User-agent: FemtosearchBot
Disallow: /

User-agent: Fetch api request
Disallow: /

User-agent: Fetch_ici
Disallow: /

User-agent: Fgcrawler
Disallow: /

User-agent: FHscan
Disallow: /

User-agent: Filangy
Disallow: /

User-agent: Filehound
Disallow: /

User-agent: Fimap
Disallow: /

User-agent: Findanisp.com_isp_finder
Disallow: /

User-agent: Findlinks
Disallow: /

User-agent: Findweb
Disallow: /

User-agent: Firebat
Disallow: /

User-agent: Firefox/7.0
Disallow: /

User-agent: Firstgov.gov search
Disallow: /

User-agent: Flaming attackbot
Disallow: /

User-agent: Flamingo_searchengine
Disallow: /

User-agent: Flashcapture
Disallow: /

User-agent: FlashGet
Disallow: /

User-agent: Flickysearchbot
Disallow: /

User-agent: Fluffy the spider
Disallow: /

User-agent: Flunky
Disallow: /

User-agent: Focused_crawler
Disallow: /

User-agent: Followsite
Disallow: /

User-agent: Foobot
Disallow: /

User-agent: Fooooo_web_video_crawl
Disallow: /

User-agent: Fopper
Disallow: /

User-agent: Formulafinderbot
Disallow: /

User-agent: Forschungsportal
Disallow: /

User-agent: Francis
Disallow: /

User-agent: Fr_crawler
Disallow: /

User-agent: Freeuploader
Disallow: /

User-agent: Freewebmonitoring sitechecker
Disallow: /

User-agent: Freshcrawler
Disallow: /

User-agent: Freshdownload
Disallow: /

User-agent: Freshlinks.exe
Disallow: /

User-agent: Friendfeedbot
Disallow: /

User-agent: Frodo.at
Disallow: /

User-agent: Froggle
Disallow: /

User-agent: FrontPage
Disallow: /

User-agent: Froola bot
Disallow: /

User-agent: Full_breadth_crawler
Disallow: /

User-agent: Fu-nbi
Disallow: /

User-agent: Funnelback
Disallow: /

User-agent: Furlbot
Disallow: /

User-agent: FyberSpider
Disallow: /

User-agent: Fyrebot
Disallow: /

User-agent: G10-bot
Disallow: /

User-agent: Gaisbot
Disallow: /

User-agent: GalaxyBot
Disallow: /

User-agent: Gazz
Disallow: /

User-agent: Gbplugin
Disallow: /

User-agent: Generate_infomine_category_classifiers
Disallow: /

User-agent: Genevabot
Disallow: /

User-agent: Geniebot
Disallow: /

User-agent: Genieo
Disallow: /

User-agent: Geomaxenginebot
Disallow: /

User-agent: Geometabot
Disallow: /

User-agent: Geonabot
Disallow: /

User-agent: Geovisu
Disallow: /

User-agent: GermCrawler
Disallow: /

User-agent: Gethtmlcontents
Disallow: /

User-agent: Getintent
Disallow: /

User-agent: Getleft
Disallow: /

User-agent: GetRight
Disallow: /

User-agent: Getsmart
Disallow: /

User-agent: Geturl.rexx
Disallow: /

User-agent: GetWeb
Disallow: /

User-agent: Giant
Disallow: /

User-agent: Gigablast
Disallow: /

User-agent: Gigablastopensource
Disallow: /

User-agent: Gigabot
Disallow: /

User-agent: G-i-g-a-b-o-t
Disallow: /

User-agent: Girafabot
Disallow: /

User-agent: Gleamebot
Disallow: /

User-agent: Gnome-vfs
Disallow: /

User-agent: Go-Ahead-Got-It
Disallow: /

User-agent: Goforitbot
Disallow: /

User-agent: Goforit.com
Disallow: /

User-agent: Go-http-client
Disallow: /

User-agent: Gold crawler
Disallow: /

User-agent: Goldfire server
Disallow: /

User-agent: Golem
Disallow: /

User-agent: Goodjelly
Disallow: /

User-agent: Gordon-college-google-mini
Disallow: /

User-agent: Goroam
Disallow: /

User-agent: Goseebot
Disallow: /

User-agent: Gotit
Disallow: /

User-agent: Govbot
Disallow: /

User-agent: GoZilla
Disallow: /

User-agent: Go!Zilla
Disallow: /

User-agent: Gpu p2p crawler
Disallow: /

User-agent: Grabber
Disallow: /

User-agent: GrabNet
Disallow: /

User-agent: Grafula
Disallow: /

User-agent: GrapeFX
Disallow: /

User-agent: Grapeshot
Disallow: /

User-agent: Grbot
Disallow: /

User-agent: Greenyogi
Disallow: /

User-agent: GridBot
Disallow: /

User-agent: Gromit
Disallow: /

User-agent: Grub
Disallow: /

User-agent: Gslfbot
Disallow: /

User-agent: GT::WWW
Disallow: /

User-agent: Gulliver
Disallow: /

User-agent: Gulperbot
Disallow: /

User-agent: Gurujibot
Disallow: /

User-agent: Gvcbot.com
Disallow: /

User-agent: Gvc business crawler
Disallow: /

User-agent: Gvc crawler
Disallow: /

User-agent: Gvc search bot
Disallow: /

User-agent: Gvc web crawler
Disallow: /

User-agent: Gvc weblink crawler
Disallow: /

User-agent: Gvc world links
Disallow: /

User-agent: Haansoft
Disallow: /

User-agent: HaosouSpider
Disallow: /

User-agent: Happyfunbot
Disallow: /

User-agent: Harvest
Disallow: /

User-agent: Hatena antenna
Disallow: /

User-agent: Havij
Disallow: /

User-agent: Hawler
Disallow: /

User-agent: Hcat
Disallow: /

User-agent: Hclsreport-crawler
Disallow: /

User-agent: Hd nutch agent
Disallow: /

User-agent: Header_test_client
Disallow: /

User-agent: HEADMasterSEO
Disallow: /

User-agent: Healia
Disallow: /

User-agent: Helix
Disallow: /

User-agent: Here will be link to crawler site
Disallow: /

User-agent: Heritrix
Disallow: /

User-agent: Hiscan
Disallow: /

User-agent: Hisoftware accmonitor server
Disallow: /

User-agent: Hisoftware accverify
Disallow: /

User-agent: Hitcrawler
Disallow: /

User-agent: Hivabot
Disallow: /

User-agent: Hloader
Disallow: /

User-agent: Hmsebot
Disallow: /

User-agent: HMView
Disallow: /

User-agent: Hoge
Disallow: /

User-agent: Holmes
Disallow: /

User-agent: Homepagesearch
Disallow: /

User-agent: Hooblybot-image
Disallow: /

User-agent: Hoowwwer
Disallow: /

User-agent: Hostcrawler
Disallow: /

User-agent: Hsft - link scanner
Disallow: /

User-agent: Hsft - lvu scanner
Disallow: /

User-agent: Hslide
Disallow: /

User-agent: Ht: //check
Disallow: /

User-agent: Htdig
Disallow: /

User-agent: Html link validator
Disallow: /

User-agent: HTMLparser
Disallow: /

User-agent: Httplib
Disallow: /

User-agent: HTTP::Lite
Disallow: /

User-agent: HTTrack
Disallow: /

User-agent: Huaweisymantecspider
Disallow: /

User-agent: Hul-wax
Disallow: /

User-agent: Humanlinks
Disallow: /

User-agent: HybridBot
Disallow: /

User-agent: Hyperestraier
Disallow: /

User-agent: Hyperix
Disallow: /

User-agent: Iblog
Disallow: /

User-agent: Ibuena
Disallow: /

User-agent: Icab
Disallow: /

User-agent: Icds-ingestion
Disallow: /

User-agent: Ichiro
Disallow: /

User-agent: Icopyright conductor
Disallow: /

User-agent: IDBot
Disallow: /

User-agent: IDG/UK
Disallow: /

User-agent: Id-search
Disallow: /

User-agent: Ieautodiscovery
Disallow: /

User-agent: Iecheck
Disallow: /

User-agent: Ihwebchecker
Disallow: /

User-agent: Iiitbot
Disallow: /

User-agent: Iim_405
Disallow: /

User-agent: IlseBot
Disallow: /

User-agent: Iltrovatore
Disallow: /

User-agent: Imagebot
Disallow: /

User-agent: Image Fetch
Disallow: /

User-agent: Image-fetcher
Disallow: /

User-agent: Imagefortress
Disallow: /

User-agent: Imageshereimagesthereimageseverywhere
Disallow: /

User-agent: Image stripper
Disallow: /

User-agent: Image Sucker
Disallow: /

User-agent: Imagevisu
Disallow: /

User-agent: Imds_monitor
Disallow: /

User-agent: Imo-google-robot-intelink
Disallow: /

User-agent: Inagist.com url crawler
Disallow: /

User-agent: IndeedBot
Disallow: /

User-agent: Indexer
Disallow: /

User-agent: Industry cortex webcrawler
Disallow: /

User-agent: Indylabs_marius
Disallow: /

User-agent: Indy Library
Disallow: /

User-agent: Inelabot
Disallow: /

User-agent: Inet32 ctrl
Disallow: /

User-agent: Inetbot
Disallow: /

User-agent: Infolink
Disallow: /

User-agent: Infomine
Disallow: /

User-agent: InfoNaviRobot
Disallow: /

User-agent: Informant
Disallow: /

User-agent: Info seeker
Disallow: /

User-agent: Infoseek sidewinder
Disallow: /

User-agent: InfoTekies
Disallow: /

User-agent: Infousabot
Disallow: /

User-agent: Ingrid
Disallow: /

User-agent: Inktomi
Disallow: /

User-agent: Insightscollector
Disallow: /

User-agent: Insightsworksbot
Disallow: /

User-agent: Inspirebot
Disallow: /

User-agent: Instabid
Disallow: /

User-agent: Insumascout
Disallow: /

User-agent: Intelix
Disallow: /

User-agent: Intelliseek
Disallow: /

User-agent: InterGET
Disallow: /

User-agent: Internetlinkagent
Disallow: /

User-agent: Internet Ninja
Disallow: /

User-agent: Internet radio crawler
Disallow: /

User-agent: InternetSeer
Disallow: /

User-agent: InternetVista monitor
Disallow: /

User-agent: Interseek
Disallow: /

User-agent: Ioi
Disallow: /

User-agent: Ipadd bot
Disallow: /

User-agent: Ips-agent
Disallow: /

User-agent: Ipselonbot
Disallow: /

User-agent: Ip-web-crawler.com
Disallow: /

User-agent: Iria
Disallow: /

User-agent: IRLbot
Disallow: /

User-agent: Iron33
Disallow: /

User-agent: Isara
Disallow: /

User-agent: Isearch
Disallow: /

User-agent: Isilox
Disallow: /

User-agent: Iskanie
Disallow: /

User-agent: IstellaBot
Disallow: /

User-agent: Its-learning crawler
Disallow: /

User-agent: Iu_csci_b659_class_crawler
Disallow: /

User-agent: Ivia
Disallow: /

User-agent: Jadynave
Disallow: /

User-agent: JamesBOT
Disallow: /

User-agent: Jbot
Disallow: /

User-agent: Jbrofuzz
Disallow: /

User-agent: Jemmathetourist
Disallow: /

User-agent: JennyBot
Disallow: /

User-agent: Jetbot
Disallow: /

User-agent: Jetbrains omea pro
Disallow: /

User-agent: JetCar
Disallow: /

User-agent: JikeSpider
Disallow: /

User-agent: Jim
Disallow: /

User-agent: Jobo
Disallow: /

User-agent: Jobspider_ba
Disallow: /

User-agent: Joc
Disallow: /

User-agent: JOC Web Spider
Disallow: /

User-agent: Joedog
Disallow: /

User-agent: Joomla
Disallow: /

User-agent: Jorgee
Disallow: /

User-agent: Joyscapebot
Disallow: /

User-agent: Jspyda
Disallow: /

User-agent: Junut bot
Disallow: /

User-agent: JustView
Disallow: /

User-agent: Jyxobot
Disallow: /

User-agent: Kakclebot
Disallow: /

User-agent: Kalooga
Disallow: /

User-agent: Katatudo-spider
Disallow: /

User-agent: Kbeta1
Disallow: /

User-agent: Keepni web site monitor
Disallow: /

User-agent: Kenjin.spider
Disallow: /

User-agent: Kenjin Spider
Disallow: /

User-agent: Keybot translation-search-machine
Disallow: /

User-agent: Keywenbot
Disallow: /

User-agent: Keyword.density
Disallow: /

User-agent: Keyword Density
Disallow: /

User-agent: Kinjabot
Disallow: /

User-agent: Kitenga-crawler-bot
Disallow: /

User-agent: Kiwistatus
Disallow: /

User-agent: Kmbot-
Disallow: /

User-agent: Kmccrew bot search
Disallow: /

User-agent: Knight
Disallow: /

User-agent: Knowitall
Disallow: /

User-agent: Knowledge.com
Disallow: /

User-agent: Knowledge engine
Disallow: /

User-agent: Koepabot
Disallow: /

User-agent: Koninklijke
Disallow: /

User-agent: Korniki
Disallow: /

User-agent: Kozmosbot
Disallow: /

User-agent: Krowler
Disallow: /

User-agent: Ksbot
Disallow: /

User-agent: K.s.bot
Disallow: /

User-agent: Kuloko-bot
Disallow: /

User-agent: Kulturarw3
Disallow: /

User-agent: Kummhttp
Disallow: /

User-agent: Kurzor
Disallow: /

User-agent: Kyluka crawl
Disallow: /

User-agent: Labhoo
Disallow: /

User-agent: Labourunions411
Disallow: /

User-agent: Lachesis
Disallow: /

User-agent: Lament
Disallow: /

User-agent: Lamerexterminator
Disallow: /

User-agent: Lanshanbot
Disallow: /

User-agent: Lapozzbot
Disallow: /

User-agent: Larbin
Disallow: /

User-agent: Lbot
Disallow: /

User-agent: Leaptag
Disallow: /

User-agent: LeechFTP
Disallow: /

User-agent: LeechGet
Disallow: /

User-agent: Letscrawl.com
Disallow: /

User-agent: LexiBot
Disallow: /

User-agent: Lexxebot
Disallow: /

User-agent: Lftp
Disallow: /

User-agent: Libcrawl
Disallow: /

User-agent: Libiviacore
Disallow: /

User-agent: Libw
Disallow: /

User-agent: LibWeb
Disallow: /

User-agent: Libwhisker
Disallow: /

User-agent: Lightspeedsystems
Disallow: /

User-agent: Likse
Disallow: /

User-agent: Linguee bot
Disallow: /

User-agent: Linkalarm
Disallow: /

User-agent: Linkbot
Disallow: /

User-agent: Linkdexbot
Disallow: /

User-agent: Linkdex.com
Disallow: /

User-agent: LinkextractorPro
Disallow: /

User-agent: Linklint
Disallow: /

User-agent: Linklooker
Disallow: /

User-agent: Linkman
Disallow: /

User-agent: LinkpadBot
Disallow: /

User-agent: LinkScan
Disallow: /

User-agent: LinksManager
Disallow: /

User-agent: Linksmanager.com_bot
Disallow: /

User-agent: Links sql
Disallow: /

User-agent: Linksweeper
Disallow: /

User-agent: Link validator
Disallow: /

User-agent: LinkWalker
Disallow: /

User-agent: LinqiaMetadataDownloaderBot
Disallow: /

User-agent: LinqiaRSSBot
Disallow: /

User-agent: LinqiaScrapeBot
Disallow: /

User-agent: Lipperhey
Disallow: /

User-agent: Litefinder
Disallow: /

User-agent: Litemage_walker
Disallow: /

User-agent: Litlrbot
Disallow: /

User-agent: Little grabber at skanktale.com
Disallow: /

User-agent: Livelapbot
Disallow: /

User-agent: Lm harvester
Disallow: /

User-agent: Lmqueuebot
Disallow: /

User-agent: Lmspider
Disallow: /

User-agent: LNSpiderguy
Disallow: /

User-agent: Loadtimebot
Disallow: /

User-agent: Localcombot
Disallow: /

User-agent: Locust
Disallow: /

User-agent: Lolongbot
Disallow: /

User-agent: Lookbot
Disallow: /

User-agent: Lsearch
Disallow: /

User-agent: Lssbot
Disallow: /

User-agent: Lt scotland checklink
Disallow: /

User-agent: Ltx71
Disallow: /

User-agent: L.webis
Disallow: /

User-agent: LWP
Disallow: /

User-agent: LWP::Simple
Disallow: /

User-agent: Lwp-trivial
Disallow: /

User-agent: Lycos_spider
Disallow: /

User-agent: Lydia entity spider
Disallow: /

User-agent: Lynnbot
Disallow: /

User-agent: Lytranslate
Disallow: /

User-agent: Magnet
Disallow: /

User-agent: Mag-Net
Disallow: /

User-agent: Magpie-crawler
Disallow: /

User-agent: Magus bot
Disallow: /

User-agent: Mail.ru
Disallow: /

User-agent: Mail.RU_Bot
Disallow: /

User-agent: Mainseek_bot
Disallow: /

User-agent: Majestic12
Disallow: /

User-agent: Mammoth
Disallow: /

User-agent: Map robot
Disallow: /

User-agent: MarkMonitor
Disallow: /

User-agent: MarkWatch
Disallow: /

User-agent: Masagool
Disallow: /

User-agent: Masidani_bot_
Disallow: /

User-agent: Masscan
Disallow: /

User-agent: Mass Downloader
Disallow: /

User-agent: Mata.hari
Disallow: /

User-agent: Mata Hari
Disallow: /

User-agent: Matentzn at cs dot man dot ac dot uk
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: Maxamine.com
Disallow: /

User-agent: Maxomobot
Disallow: /

User-agent: Mcbot
Disallow: /

User-agent: Meanpathbot
Disallow: /

User-agent: Mediatoolkitbot
Disallow: /

User-agent: Mediawords
Disallow: /

User-agent: Medrabbit
Disallow: /

User-agent: MegaIndex.ru
Disallow: /

User-agent: Megite
Disallow: /

User-agent: Memacbot
Disallow: /

User-agent: Memo
Disallow: /

User-agent: Mendeleybot
Disallow: /

User-agent: Mercator-
Disallow: /

User-agent: Mercuryboard_user_agent_sql_injection.nasl
Disallow: /

User-agent: Metacarta
Disallow: /

User-agent: Metaeuro web search
Disallow: /

User-agent: Metager2
Disallow: /

User-agent: Metagloss
Disallow: /

User-agent: Metal crawler
Disallow: /

User-agent: Metaquerier
Disallow: /

User-agent: Metaspider
Disallow: /

User-agent: Metaspinner
Disallow: /

User-agent: Metauri
Disallow: /

User-agent: Mfcrawler
Disallow: /

User-agent: MFC_Tear_Sample
Disallow: /

User-agent: Mfhttpscan
Disallow: /

User-agent: Microsoft Data Access
Disallow: /

User-agent: Microsoft URL Control
Disallow: /

User-agent: MIDown tool
Disallow: /

User-agent: MIIxpc
Disallow: /

User-agent: Minibot
Disallow: /

User-agent: Minirank
Disallow: /

User-agent: Mini-robot
Disallow: /

User-agent: Mirror
Disallow: /

User-agent: Missigua locator
Disallow: /

User-agent: Mister.pix
Disallow: /

User-agent: Mister PiX
Disallow: /

User-agent: Miva
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: Mnogosearch
Disallow: /

User-agent: Mod_accessibility
Disallow: /

User-agent: Moduna.com
Disallow: /

User-agent: Moget
Disallow: /

User-agent: Mojeek
Disallow: /

User-agent: Monkeycrawl
Disallow: /

User-agent: Moses
Disallow: /

User-agent: Mowserbot
Disallow: /

User-agent: Mqbot
Disallow: /

User-agent: Mr.4x3
Disallow: /

User-agent: Mse360
Disallow: /

User-agent: MSFrontPage
Disallow: /

User-agent: MSIECrawler
Disallow: /

User-agent: Msindianwebcrawl
Disallow: /

User-agent: Msmobot
Disallow: /

User-agent: Msnptc
Disallow: /

User-agent: Msrabot
Disallow: /

User-agent: Msrbot
Disallow: /

User-agent: MS Web Services Client Protocol
Disallow: /

User-agent: Mt-soft
Disallow: /

User-agent: Muhstik-scan
Disallow: /

User-agent: Multitext
Disallow: /

User-agent: Musobot
Disallow: /

User-agent: Myapp
Disallow: /

User-agent: Mycompanybot
Disallow: /

User-agent: Mycrawler
Disallow: /

User-agent: Myengines-us-bot
Disallow: /

User-agent: Myfamilybot
Disallow: /

User-agent: My-heritrix-crawler
Disallow: /

User-agent: My_little_searchengine_project
Disallow: /

User-agent: Myra
Disallow: /

User-agent: Nabot
Disallow: /

User-agent: Najdi.si
Disallow: /

User-agent: Nambu
Disallow: /

User-agent: Name Intelligence
Disallow: /

User-agent: Nameprotect
Disallow: /

User-agent: Nasa search
Disallow: /

User-agent: Natchcvs
Disallow: /

User-agent: Natweb-bad-link-mailer
Disallow: /

User-agent: Naver
Disallow: /

User-agent: Navroad
Disallow: /

User-agent: NearSite
Disallow: /

User-agent: Nec-meshexplorer
Disallow: /

User-agent: Needle
Disallow: /

User-agent: Neosciocrawler
Disallow: /

User-agent: Nerdbynature.bot
Disallow: /

User-agent: Nerdybot
Disallow: /

User-agent: Nerima-crawl-
Disallow: /

User-agent: Nessus
Disallow: /

User-agent: Nestreader
Disallow: /

User-agent: NetAnts
Disallow: /

User-agent: Netcarta cyberpilot pro
Disallow: /

User-agent: Netcraft
Disallow: /

User-agent: NetEstate NE Crawler
Disallow: /

User-agent: Netexperts
Disallow: /

User-agent: Netid.com bot
Disallow: /

User-agent: NetLyzer
Disallow: /

User-agent: NetMechanic
Disallow: /

User-agent: Netprospector
Disallow: /

User-agent: Netresearchserver
Disallow: /

User-agent: Netseer
Disallow: /

User-agent: Netshift
Disallow: /

User-agent: Netsongbot
Disallow: /

User-agent: Netsparker
Disallow: /

User-agent: NetSpider
Disallow: /

User-agent: Netsrcherp
Disallow: /

User-agent: Nettrack
Disallow: /

User-agent: Net::trackback
Disallow: /

User-agent: Net Vampire
Disallow: /

User-agent: Netvibes
Disallow: /

User-agent: NetZIP
Disallow: /

User-agent: Newmedhunt
Disallow: /

User-agent: News bot
Disallow: /

User-agent: Newsgatherer
Disallow: /

User-agent: Newsgroupreporter
Disallow: /

User-agent: News_search_app
Disallow: /

User-agent: Newstrovebot
Disallow: /

User-agent: NextGenSearchBot
Disallow: /

User-agent: Nextthing.org
Disallow: /

User-agent: Nibbler
Disallow: /

User-agent: Nicebot
Disallow: /

User-agent: NICErsPRO
Disallow: /

User-agent: Niki-bot
Disallow: /

User-agent: Nikto
Disallow: /

User-agent: NimbleCrawler
Disallow: /

User-agent: Nimbus-1
Disallow: /

User-agent: Ninetowns
Disallow: /

User-agent: Ninja
Disallow: /

User-agent: Njuicebot
Disallow: /

User-agent: Nlese
Disallow: /

User-agent: Nmap
Disallow: /

User-agent: Nogate
Disallow: /

User-agent: Norbert the spider
Disallow: /

User-agent: Noteworthybot
Disallow: /

User-agent: NPbot
Disallow: /

User-agent: Nrcan intranet crawler
Disallow: /

User-agent: Nsdl_search_bot
Disallow: /

User-agent: Nuggetize.com bot
Disallow: /

User-agent: Nusearch spider
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: Nu_tch
Disallow: /

User-agent: Nwspider
Disallow: /

User-agent: Nymesis
Disallow: /

User-agent: Nys-crawler
Disallow: /

User-agent: Objectssearch
Disallow: /

User-agent: OBot
Disallow: /

User-agent: Obvius external linkcheck
Disallow: /

User-agent: Ocelli
Disallow: /

User-agent: Octopus
Disallow: /

User-agent: Odp entries t_st
Disallow: /

User-agent: Oegp
Disallow: /

User-agent: Offline.explorer
Disallow: /

User-agent: Offline Explorer
Disallow: /

User-agent: Offline Navigator
Disallow: /

User-agent: Ogspider
Disallow: /

User-agent: Omiexplorer_bot
Disallow: /

User-agent: Omniexplorer
Disallow: /

User-agent: Omnifind
Disallow: /

User-agent: Omniweb
Disallow: /

User-agent: Onetszukaj
Disallow: /

User-agent: Online link validator
Disallow: /

User-agent: Oozbot
Disallow: /

User-agent: Openbot
Disallow: /

User-agent: Openfind
Disallow: /

User-agent: Openintelligencedata
Disallow: /

User-agent: Openisearch
Disallow: /

User-agent: OpenLinkProfiler
Disallow: /

User-agent: Openlink virtuoso rdf crawler
Disallow: /

User-agent: Opensearchserver_bot
Disallow: /

User-agent: OpenVAS
Disallow: /

User-agent: Opidig
Disallow: /

User-agent: Optidiscover
Disallow: /

User-agent: Oracle secure enterprise search
Disallow: /

User-agent: Oracle ultra search
Disallow: /

User-agent: OrangeBot
Disallow: /

User-agent: OrangeSpider
Disallow: /

User-agent: Orisbot
Disallow: /

User-agent: Ornl_crawler
Disallow: /

User-agent: Ornl_mercury
Disallow: /

User-agent: Osis-project.jp
Disallow: /

User-agent: Oso
Disallow: /

User-agent: OutclicksBot
Disallow: /

User-agent: OutfoxBot
Disallow: /

User-agent: Outfoxmelonbot
Disallow: /

User-agent: Owler-bot
Disallow: /

User-agent: Owsbot
Disallow: /

User-agent: Ozelot
Disallow: /

User-agent: P3p client
Disallow: /

User-agent: PageAnalyzer
Disallow: /

User-agent: Page Analyzer
Disallow: /

User-agent: Pagebiteshyperbot
Disallow: /

User-agent: Pagebull
Disallow: /

User-agent: Pagedown
Disallow: /

User-agent: Pagefetcher
Disallow: /

User-agent: PageGrabber
Disallow: /

User-agent: Pagepeeker
Disallow: /

User-agent: Pagerank monitor
Disallow: /

User-agent: Page scorer
Disallow: /

User-agent: PageScorer
Disallow: /

User-agent: Page_verifier
Disallow: /

User-agent: Pamsnbot.htm
Disallow: /

User-agent: Panopy bot
Disallow: /

User-agent: Panscient
Disallow: /

User-agent: Pansophica
Disallow: /

User-agent: Papa Foto
Disallow: /

User-agent: Paperlibot
Disallow: /

User-agent: Parasite
Disallow: /

User-agent: Parsijoo
Disallow: /

User-agent: Pathtraq
Disallow: /

User-agent: Pattern
Disallow: /

User-agent: Patwebbot
Disallow: /

User-agent: Pavuk
Disallow: /

User-agent: Paxleframework
Disallow: /

User-agent: Pbbot
Disallow: /

User-agent: PcBrowser
Disallow: /

User-agent: Pcore-http
Disallow: /

User-agent: Pd-crawler
Disallow: /

User-agent: PECL::HTTP
Disallow: /

User-agent: Penthesila
Disallow: /

User-agent: PeoplePal
Disallow: /

User-agent: Perform_crawl
Disallow: /

User-agent: Perman
Disallow: /

User-agent: Personal ultimate crawler
Disallow: /

User-agent: PHPCrawl
Disallow: /

User-agent: Phpdig
Disallow: /

User-agent: Php version tracker
Disallow: /

User-agent: Picosearch
Disallow: /

User-agent: Picscout
Disallow: /

User-agent: Picsearch
Disallow: /

User-agent: PictureFinder
Disallow: /

User-agent: Pieno robot
Disallow: /

User-agent: Pimonster
Disallow: /

User-agent: Pi-Monster
Disallow: /

User-agent: Pipbot
Disallow: /

User-agent: Pipeliner
Disallow: /

User-agent: Pita
Disallow: /

User-agent: Pixfinder
Disallow: /

User-agent: Pixray
Disallow: /

User-agent: Piyushbot
Disallow: /

User-agent: Planetwork bot search
Disallow: /

User-agent: PleaseCrawl
Disallow: /

User-agent: Plucker
Disallow: /

User-agent: Plukkie
Disallow: /

User-agent: Plumanalytics
Disallow: /

User-agent: Plumtree
Disallow: /

User-agent: Pockey
Disallow: /

User-agent: Pocohttp
Disallow: /

User-agent: POE-Component-Client-HTTP
Disallow: /

User-agent: Pogodak.ba
Disallow: /

User-agent: Pogodak.co.yu
Disallow: /

User-agent: Poirot
Disallow: /

User-agent: Polybot
Disallow: /

User-agent: Pompos
Disallow: /

User-agent: Poodle predictor
Disallow: /

User-agent: Popscreenbot
Disallow: /

User-agent: Postpost
Disallow: /

User-agent: Privacyfinder
Disallow: /

User-agent: Probethenet
Disallow: /

User-agent: Projectwf-java-test-crawler
Disallow: /

User-agent: ProPowerBot
Disallow: /

User-agent: ProWebWalker
Disallow: /

User-agent: Proxem websearch
Disallow: /

User-agent: Proximic
Disallow: /

User-agent: Proxy crawler
Disallow: /

User-agent: Psbot
Disallow: /

User-agent: Pss-bot
Disallow: /

User-agent: Psycheclone
Disallow: /

User-agent: Pub-crawler
Disallow: /

User-agent: Pucl
Disallow: /

User-agent: Pulsebot
Disallow: /

User-agent: Pump
Disallow: /

User-agent: Pwebot
Disallow: /

User-agent: PxBroker
Disallow: /

User-agent: PyCurl
Disallow: /

User-agent: Qeavis agent
Disallow: /

User-agent: Qfkbot
Disallow: /

User-agent: Qualidade
Disallow: /

User-agent: Qualidator.com bot
Disallow: /

User-agent: Quepasacreep
Disallow: /

User-agent: Queryn.metasearch
Disallow: /

User-agent: Queryn metasearch
Disallow: /

User-agent: QueryN Metasearch
Disallow: /

User-agent: Quest.durato
Disallow: /

User-agent: Quick-Crawler
Disallow: /

User-agent: Quintura-crw
Disallow: /

User-agent: Qunarbot
Disallow: /

User-agent: Qwantify
Disallow: /

User-agent: Qweerybot
Disallow: /

User-agent: Qweery_robot.txt_checkbot
Disallow: /

User-agent: R2ibot
Disallow: /

User-agent: R6_commentreader
Disallow: /

User-agent: R6_feedfetcher
Disallow: /

User-agent: R6_votereader
Disallow: /

User-agent: Rabot
Disallow: /

User-agent: Radian6
Disallow: /

User-agent: Radiation retriever
Disallow: /

User-agent: Rampybot
Disallow: /

User-agent: RankActive
Disallow: /

User-agent: RankFlex
Disallow: /

User-agent: RankingBot
Disallow: /

User-agent: RankingBot2
Disallow: /

User-agent: Rankivabot
Disallow: /

User-agent: Rankur
Disallow: /

User-agent: RankurBot
Disallow: /

User-agent: Rational sitecheck
Disallow: /

User-agent: Rcstartbot
Disallow: /

User-agent: RealDownload
Disallow: /

User-agent: Reaper
Disallow: /

User-agent: RebelMouse
Disallow: /

User-agent: Rebi-shoveler
Disallow: /

User-agent: Recorder
Disallow: /

User-agent: Redbot
Disallow: /

User-agent: Redcarpet
Disallow: /

User-agent: RedesScrapy
Disallow: /

User-agent: ReGet
Disallow: /

User-agent: RepoMonkey
Disallow: /

User-agent: Research robot
Disallow: /

User-agent: Riddler
Disallow: /

User-agent: Riight
Disallow: /

User-agent: Ripper
Disallow: /

User-agent: Risenetbot
Disallow: /

User-agent: Riverglassscanner
Disallow: /

User-agent: Robopal
Disallow: /

User-agent: Robosourcer
Disallow: /

User-agent: Robotek
Disallow: /

User-agent: RocketCrawler
Disallow: /

User-agent: Roger
Disallow: /

User-agent: Rogerbot
Disallow: /

User-agent: Rome client
Disallow: /

User-agent: Rondello
Disallow: /

User-agent: Rotondo
Disallow: /

User-agent: Roverbot
Disallow: /

User-agent: Rpt-httpclient
Disallow: /

User-agent: Rtgibot
Disallow: /

User-agent: Rufusbot
Disallow: /

User-agent: Runnk online rss reader
Disallow: /

User-agent: Runnk rss aggregator
Disallow: /

User-agent: S2bot
Disallow: /

User-agent: Safaribookmarkchecker
Disallow: /

User-agent: Safednsbot
Disallow: /

User-agent: Safetynet robot
Disallow: /

User-agent: Saladspoon
Disallow: /

User-agent: SalesIntelligent
Disallow: /

User-agent: Sapienti
Disallow: /

User-agent: Sapphireweb
Disallow: /

User-agent: SBIder
Disallow: /

User-agent: Sbl-bot
Disallow: /

User-agent: ScanAlert
Disallow: /

User-agent: Scanbot
Disallow: /

User-agent: Scan.lol
Disallow: /

User-agent: Scfcrawler
Disallow: /

User-agent: Scich
Disallow: /

User-agent: Scientificcommons.org
Disallow: /

User-agent: Scollspider
Disallow: /

User-agent: Scooperbot
Disallow: /

User-agent: Scooter
Disallow: /

User-agent: Scoutjet
Disallow: /

User-agent: Scrapebox
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Scrawltest
Disallow: /

User-agent: Screaming
Disallow: /

User-agent: ScreenerBot
Disallow: /

User-agent: Scrubby
Disallow: /

User-agent: Scspider
Disallow: /

User-agent: Scumbot
Disallow: /

User-agent: Searcharoo.net
Disallow: /

User-agent: Searchblox
Disallow: /

User-agent: Searchbot
Disallow: /

User-agent: Search-channel
Disallow: /

User-agent: Searchengine
Disallow: /

User-agent: Search-engine-studio
Disallow: /

User-agent: Searchestate
Disallow: /

User-agent: Searchhippo.com
Disallow: /

User-agent: Searchit-bot
Disallow: /

User-agent: Search.kumkie.com
Disallow: /

User-agent: Searchmarking
Disallow: /

User-agent: Searchmarks
Disallow: /

User-agent: Searchmee
Disallow: /

User-agent: SearchmetricsBot
Disallow: /

User-agent: Searchmining
Disallow: /

User-agent: Searchnowbot
Disallow: /

User-agent: Searchpreview
Disallow: /

User-agent: Search publisher
Disallow: /

User-agent: Searchspider.com
Disallow: /

User-agent: Search.updated.com
Disallow: /

User-agent: Search.usgs.gov
Disallow: /

User-agent: Search x-bot
Disallow: /

User-agent: Searqubot
Disallow: /

User-agent: Seb spider
Disallow: /

User-agent: Seekbot
Disallow: /

User-agent: Seeker.lookseek.com
Disallow: /

User-agent: Seeqbot
Disallow: /

User-agent: Seeqpod-vertical-crawler
Disallow: /

User-agent: Selflinkchecker
Disallow: /

User-agent: Semager
Disallow: /

User-agent: Semanticdiscovery
Disallow: /

User-agent: Semantifire
Disallow: /

User-agent: Semisearch
Disallow: /

User-agent: Semrush
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: Seoengworldbot
Disallow: /

User-agent: SEOkicks
Disallow: /

User-agent: SEOlyticsCrawler
Disallow: /

User-agent: Seomoz
Disallow: /

User-agent: SEOprofiler
Disallow: /

User-agent: Seoscanners
Disallow: /

User-agent: SEOstats
Disallow: /

User-agent: Sexsearcher
Disallow: /

User-agent: Shablastbot
Disallow: /

User-agent: Shadowwebanalyzer
Disallow: /

User-agent: Shareaza
Disallow: /

User-agent: Shelob
Disallow: /

User-agent: Sherlock
Disallow: /

User-agent: Shim-crawler
Disallow: /

User-agent: Shodan
Disallow: /

User-agent: Shopsalad
Disallow: /

User-agent: Shopwiki
Disallow: /

User-agent: Showlinks
Disallow: /

User-agent: Showyoubot
Disallow: /

User-agent: Siclab
Disallow: /

User-agent: Silk
Disallow: /

User-agent: Simplepie
Disallow: /

User-agent: Siphon
Disallow: /

User-agent: SISTRIX
Disallow: /

User-agent: Sitebeam
Disallow: /

User-agent: Sitebot
Disallow: /

User-agent: Sitecheck
Disallow: /

User-agent: SiteExplorer
Disallow: /

User-agent: Sitefinder
Disallow: /

User-agent: Siteguardbot
Disallow: /

User-agent: SiteLockSpider
Disallow: /

User-agent: Siteorbiter
Disallow: /

User-agent: SiteSnagger
Disallow: /

User-agent: SiteSucker
Disallow: /

User-agent: Site Sucker
Disallow: /

User-agent: Sitesweeper
Disallow: /

User-agent: Sitevigil
Disallow: /

User-agent: Sitexpert
Disallow: /

User-agent: Skimbot
Disallow: /

User-agent: Skimwordsbot
Disallow: /

User-agent: Skreemrbot
Disallow: /

User-agent: Skywalker
Disallow: /

User-agent: Slackbot-LinkExpanding
Disallow: /

User-agent: Sleipnir
Disallow: /

User-agent: Slow-crawler
Disallow: /

User-agent: SlySearch
Disallow: /

User-agent: Smart-crawler
Disallow: /

User-agent: SmartDownload
Disallow: /

User-agent: Smarte bot
Disallow: /

User-agent: Smartwit.com
Disallow: /

User-agent: SMTBot
Disallow: /

User-agent: Snake
Disallow: /

User-agent: Snapbot
Disallow: /

User-agent: Snap.com beta crawler
Disallow: /

User-agent: Snappreviewbot
Disallow: /

User-agent: Snappy
Disallow: /

User-agent: Snookit
Disallow: /

User-agent: Snooper
Disallow: /

User-agent: Snoopy
Disallow: /

User-agent: SocialRankIOBot
Disallow: /

User-agent: Societyrobot
Disallow: /

User-agent: Socscibot
Disallow: /

User-agent: Soft411 directory
Disallow: /

User-agent: Sogou
Disallow: /

User-agent: sogouspider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: Sohu agent
Disallow: /

User-agent: Sohu-search
Disallow: /

User-agent: Sokitomi crawl
Disallow: /

User-agent: Solbot
Disallow: /

User-agent: Sondeur
Disallow: /

User-agent: Sootle
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: Sottopop
Disallow: /

User-agent: Spacebison
Disallow: /

User-agent: Space bison
Disallow: /

User-agent: SpaceBison
Disallow: /

User-agent: Space fung
Disallow: /

User-agent: Spammen
Disallow: /

User-agent: SpankBot
Disallow: /

User-agent: Spanner
Disallow: /

User-agent: Spatineo monitor controller
Disallow: /

User-agent: Spatineo serval controller
Disallow: /

User-agent: Spatineo serval getmapbot
Disallow: /

User-agent: Spbot
Disallow: /

User-agent: Special_archiver
Disallow: /

User-agent: Speedy
Disallow: /

User-agent: Sphere scout
Disallow: /

User-agent: Sphider
Disallow: /

User-agent: Spiderengine
Disallow: /

User-agent: Spiderku
Disallow: /

User-agent: Spiderman
Disallow: /

User-agent: Spider.terranautic.net
Disallow: /

User-agent: Spinn3r
Disallow: /

User-agent: Spinne
Disallow: /

User-agent: Sportcrew-bot
Disallow: /

User-agent: Sproose
Disallow: /

User-agent: SputnikBot
Disallow: /

User-agent: Spyder3.microsys.com
Disallow: /

User-agent: Sqlmap
Disallow: /

User-agent: Sqlworm
Disallow: /

User-agent: Squidclamav_redirector
Disallow: /

User-agent: Squid-prefetch
Disallow: /

User-agent: Sq webscanner
Disallow: /

User-agent: Sqworm
Disallow: /

User-agent: Srevbot
Disallow: /

User-agent: Sslbot
Disallow: /

User-agent: Ssm agent
Disallow: /

User-agent: S~stremor-crawler
Disallow: /

User-agent: Stackrambler
Disallow: /

User-agent: Stardownloader
Disallow: /

User-agent: Statbot
Disallow: /

User-agent: Statcrawler
Disallow: /

User-agent: Statedept-crawler
Disallow: /

User-agent: Steeler
Disallow: /

User-agent: Stegmann-bot
Disallow: /

User-agent: Stero
Disallow: /

User-agent: Stripper
Disallow: /

User-agent: Stumbler
Disallow: /

User-agent: Suchclip
Disallow: /

User-agent: Sucker
Disallow: /

User-agent: Sucuri
Disallow: /

User-agent: Sumeetbot
Disallow: /

User-agent: Sumitbot
Disallow: /

User-agent: Summizebot
Disallow: /

User-agent: Summizefeedreader
Disallow: /

User-agent: Sunrise xp
Disallow: /

User-agent: SuperBot
Disallow: /

User-agent: SuperHTTP
Disallow: /

User-agent: Superlumin downloader
Disallow: /

User-agent: Superpagesbot
Disallow: /

User-agent: Supremesearch.net
Disallow: /

User-agent: Supybot
Disallow: /

User-agent: Surdotlybot
Disallow: /

User-agent: Surf
Disallow: /

User-agent: Surfbot
Disallow: /

User-agent: SurveyBot
Disallow: /

User-agent: Suzuran
Disallow: /

User-agent: Swebot
Disallow: /

User-agent: Swiftbot
Disallow: /

User-agent: Swish-e
Disallow: /

User-agent: Sygolbot
Disallow: /

User-agent: Synapticwalker
Disallow: /

User-agent: Syntryx ant scout chassis pheromone
Disallow: /

User-agent: Sysscan
Disallow: /

User-agent: Systemsearch-robot
Disallow: /

User-agent: Szukacz
Disallow: /

User-agent: T0PHackTeam
Disallow: /

User-agent: T8Abot
Disallow: /

User-agent: Tailrank
Disallow: /

User-agent: TAkeOut
Disallow: /

User-agent: Talkro web-shot
Disallow: /

User-agent: Tamu_crawler
Disallow: /

User-agent: Tapuzbot
Disallow: /

User-agent: Tarantula
Disallow: /

User-agent: Targetblaster.com
Disallow: /

User-agent: Targetyournews.com bot
Disallow: /

User-agent: Tausdatabot
Disallow: /

User-agent: Taxinomiabot
Disallow: /

User-agent: Teamsoft wininet component
Disallow: /

User-agent: Tecomi bot
Disallow: /

User-agent: Teezirbot
Disallow: /

User-agent: Teleport
Disallow: /

User-agent: TeleportPro
Disallow: /

User-agent: Telesoft
Disallow: /

User-agent: Telesphoreo
Disallow: /

User-agent: Telesphorep
Disallow: /

User-agent: Teradex mapper
Disallow: /

User-agent: Teragram_crawler
Disallow: /

User-agent: Terrawizbot
Disallow: /

User-agent: Testbot
Disallow: /

User-agent: Testing of bot
Disallow: /

User-agent: Textbot
Disallow: /

User-agent: Thatrobotsite.com
Disallow: /

User-agent: The dyslexalizer
Disallow: /

User-agent: The.intraformant
Disallow: /

User-agent: The Intraformant
Disallow: /

User-agent: TheNomad
Disallow: /

User-agent: Theophrastus
Disallow: /

User-agent: Theusefulbot
Disallow: /

User-agent: Thumbbot
Disallow: /

User-agent: Thumbnail.cz robot
Disallow: /

User-agent: Thumbshots-de-bot
Disallow: /

User-agent: T-h-u-n-d-e-r-s-t-o-n-e
Disallow: /

User-agent: Tigerbot
Disallow: /

User-agent: Tineye
Disallow: /

User-agent: Titan
Disallow: /

User-agent: Toata
Disallow: /

User-agent: Tocrawl
Disallow: /

User-agent: To-dress_ru_bot_
Disallow: /

User-agent: To-night-bot
Disallow: /

User-agent: Topicalizer
Disallow: /

User-agent: Topicblogs
Disallow: /

User-agent: Toplistbot
Disallow: /

User-agent: Topserver php
Disallow: /

User-agent: Topyx-crawler
Disallow: /

User-agent: Touche
Disallow: /

User-agent: Tourlentascanner
Disallow: /

User-agent: Toweyabot
Disallow: /

User-agent: Tpsystem
Disallow: /

User-agent: Traazi
Disallow: /

User-agent: Tracemyfile
Disallow: /

User-agent: Transgenikbot
Disallow: /

User-agent: Travelbot
Disallow: /

User-agent: Travellazerbot
Disallow: /

User-agent: Travel-search
Disallow: /

User-agent: Treezy
Disallow: /

User-agent: Trendiction
Disallow: /

User-agent: Trendictionbot
Disallow: /

User-agent: Trendiction.com
Disallow: /

User-agent: Trendiction.de
Disallow: /

User-agent: Trex
Disallow: /

User-agent: Tridentspider
Disallow: /

User-agent: Trovator
Disallow: /

User-agent: True_Robot
Disallow: /

User-agent: Tscholarsbot
Disallow: /

User-agent: Tsm translation-search-machine
Disallow: /

User-agent: Tswebbot
Disallow: /

User-agent: Tulipchain
Disallow: /

User-agent: Turingos
Disallow: /

User-agent: Turnitin
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: Tutorgigbot
Disallow: /

User-agent: Tweetedtimes bot
Disallow: /

User-agent: Tweetmemebot
Disallow: /

User-agent: TwengaBot
Disallow: /

User-agent: Twice
Disallow: /

User-agent: Twikle
Disallow: /

User-agent: Twinuffbot
Disallow: /

User-agent: Twisted pagegetter
Disallow: /

User-agent: Twitturls
Disallow: /

User-agent: Twitturly
Disallow: /

User-agent: Tygobot
Disallow: /

User-agent: Tygoprowler
Disallow: /

User-agent: Typhoeus
Disallow: /

User-agent: Uberbot
Disallow: /

User-agent: Ucb-nutch
Disallow: /

User-agent: Udmsearch
Disallow: /

User-agent: Ufam-crawler-
Disallow: /

User-agent: Unchaos
Disallow: /

User-agent: Unidentified
Disallow: /

User-agent: UnisterBot
Disallow: /

User-agent: Unitek uniengine
Disallow: /

User-agent: Universalsearch
Disallow: /

User-agent: Unwindfetchor
Disallow: /

User-agent: Uoftdb_experiment
Disallow: /

User-agent: Updated
Disallow: /

User-agent: Urlappendbot
Disallow: /

User-agent: Urlblaze
Disallow: /

User-agent: Urlchecker
Disallow: /

User-agent: Url-checker
Disallow: /

User-agent: Urlck
Disallow: /

User-agent: Url control
Disallow: /

User-agent: Urldispatcher
Disallow: /

User-agent: Url_gather
Disallow: /

User-agent: Urlspiderpro
Disallow: /

User-agent: URLy.Warning
Disallow: /

User-agent: URLy Warning
Disallow: /

User-agent: Usaf afkn k2spider
Disallow: /

User-agent: Usasearch
Disallow: /

User-agent: U.s. government printing office
Disallow: /

User-agent: Uss-cosmix
Disallow: /

User-agent: Usyd-nlp-spider
Disallow: /

User-agent: Vacobot
Disallow: /

User-agent: Vacuum
Disallow: /

User-agent: Vadixbot
Disallow: /

User-agent: Vagabondo
Disallow: /

User-agent: Validator
Disallow: /

User-agent: Valkyrie
Disallow: /

User-agent: VB Project
Disallow: /

User-agent: Vbseo
Disallow: /

User-agent: VCI
Disallow: /

User-agent: Vci webviewer vci webviewer win32
Disallow: /

User-agent: Verbstarbot
Disallow: /

User-agent: VeriCiteCrawler
Disallow: /

User-agent: Verifactrola
Disallow: /

User-agent: Verity-url-gateway
Disallow: /

User-agent: Vermut
Disallow: /

User-agent: Versus crawler
Disallow: /

User-agent: Versus.integis.ch
Disallow: /

User-agent: Viasarchivinginformation.html
Disallow: /

User-agent: VidibleScraper
Disallow: /

User-agent: Vipr
Disallow: /

User-agent: Virus_detector
Disallow: /

User-agent: Virus-detector
Disallow: /

User-agent: Virusdie
Disallow: /

User-agent: Visbot
Disallow: /

User-agent: Vishal for clia
Disallow: /

User-agent: Visweb
Disallow: /

User-agent: Vital search'n urchin
Disallow: /

User-agent: Vlad
Disallow: /

User-agent: Vlsearch
Disallow: /

User-agent: Vmbot
Disallow: /

User-agent: Vocusbot
Disallow: /

User-agent: VoidEYE
Disallow: /

User-agent: Voil
Disallow: /

User-agent: Voilabot
Disallow: /

User-agent: Voltron
Disallow: /

User-agent: Vortex
Disallow: /

User-agent: Voyager
Disallow: /

User-agent: Vspider
Disallow: /

User-agent: W3c_unicorn
Disallow: /

User-agent: W3c-webcon
Disallow: /

User-agent: W3search
Disallow: /

User-agent: Wacbot
Disallow: /

User-agent: Wallpapers/3.0
Disallow: /

User-agent: WallpapersHD
Disallow: /

User-agent: Wanadoo
Disallow: /

User-agent: WASALive-Bot
Disallow: /

User-agent: Wastrix
Disallow: /

User-agent: Water conserve portal
Disallow: /

User-agent: Water conserve spider
Disallow: /

User-agent: Watzbot
Disallow: /

User-agent: Wauuu
Disallow: /

User-agent: Wavefire
Disallow: /

User-agent: Waypath
Disallow: /

User-agent: Wazzup
Disallow: /

User-agent: Wbdbot
Disallow: /

User-agent: WBSearchBot
Disallow: /

User-agent: Web-agent
Disallow: /

User-agent: Webalta
Disallow: /

User-agent: Webaltbot
Disallow: /

User-agent: WebAuto
Disallow: /

User-agent: Web Auto
Disallow: /

User-agent: WebBandit
Disallow: /

User-agent: Webbot
Disallow: /

User-agent: Webbul-bot
Disallow: /

User-agent: Webcapture
Disallow: /

User-agent: Web ceo online robot
Disallow: /

User-agent: Webcheck
Disallow: /

User-agent: Webclipping.com
Disallow: /

User-agent: WebCollage
Disallow: /

User-agent: Web Collage
Disallow: /

User-agent: WebCopier
Disallow: /

User-agent: Webcopy
Disallow: /

User-agent: Webcorp
Disallow: /

User-agent: Webcrawler
Disallow: /

User-agent: Web crawler
Disallow: /

User-agent: Webcrawl.net
Disallow: /

User-agent: Webdatacentrebot
Disallow: /

User-agent: WEBDAV
Disallow: /

User-agent: Web downloader
Disallow: /

User-agent: Webdownloader for x
Disallow: /

User-agent: Webdup
Disallow: /

User-agent: Webemailextrac
Disallow: /

User-agent: WebEnhancer
Disallow: /

User-agent: Web Enhancer
Disallow: /

User-agent: WebFetch
Disallow: /

User-agent: Web Fetch
Disallow: /

User-agent: Webgather
Disallow: /

User-agent: Webgobbler
Disallow: /

User-agent: WebGo IS
Disallow: /

User-agent: Web.image.collector
Disallow: /

User-agent: Web image collector
Disallow: /

User-agent: WebImageCollector
Disallow: /

User-agent: Webimages
Disallow: /

User-agent: Webinator-search2
Disallow: /

User-agent: Webinator-wbi
Disallow: /

User-agent: Webindex
Disallow: /

User-agent: Weblayers
Disallow: /

User-agent: WebLeacher
Disallow: /

User-agent: Weblexbot
Disallow: /

User-agent: Weblinker
Disallow: /

User-agent: Web link validator
Disallow: /

User-agent: Weblyzard
Disallow: /

User-agent: Web magnet
Disallow: /

User-agent: Webmastercoffee
Disallow: /

User-agent: Webmasterworld extractor
Disallow: /

User-agent: WebmasterWorldForumBot
Disallow: /

User-agent: Webmeup-crawler
Disallow: /

User-agent: Webminer
Disallow: /

User-agent: Webmoose
Disallow: /

User-agent: Webot
Disallow: /

User-agent: WebPix
Disallow: /

User-agent: Web Pix
Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: Webripper
Disallow: /

User-agent: WebSauger
Disallow: /

User-agent: Web Sauger
Disallow: /

User-agent: Webscan
Disallow: /

User-agent: Websearchbench
Disallow: /

User-agent: Webshag
Disallow: /

User-agent: Website
Disallow: /

User-agent: Web site downloader
Disallow: /

User-agent: WebsiteExtractor
Disallow: /

User-agent: WebsiteQuester
Disallow: /

User-agent: Website Quester
Disallow: /

User-agent: Web-sniffer
Disallow: /

User-agent: Webspear
Disallow: /

User-agent: Websphinx
Disallow: /

User-agent: Webspider
Disallow: /

User-agent: Webster
Disallow: /

User-agent: WebStripper
Disallow: /

User-agent: WebSucker
Disallow: /

User-agent: Web Sucker
Disallow: /

User-agent: Webtrafficexpress
Disallow: /

User-agent: Webtrends link analyzer
Disallow: /

User-agent: Webvac
Disallow: /

User-agent: Webwalk
Disallow: /

User-agent: Webwasher
Disallow: /

User-agent: Webwatch
Disallow: /

User-agent: WebWhacker
Disallow: /

User-agent: Webxm
Disallow: /

User-agent: WebZIP
Disallow: /

User-agent: Weddings.info
Disallow: /

User-agent: Wenbin
Disallow: /

User-agent: Wepa
Disallow: /

User-agent: Wep search
Disallow: /

User-agent: Werelatebot
Disallow: /

User-agent: WeSEE
Disallow: /

User-agent: Whack
Disallow: /

User-agent: Whacker
Disallow: /

User-agent: Whatweb
Disallow: /

User-agent: Whirlpool web engine
Disallow: /

User-agent: Who.is Bot
Disallow: /

User-agent: Whowhere robot
Disallow: /

User-agent: Widow
Disallow: /

User-agent: Wikiabot
Disallow: /

User-agent: Wikio
Disallow: /

User-agent: Wikiwix-bot-
Disallow: /

User-agent: Winhttp
Disallow: /

User-agent: WinHTTrack
Disallow: /

User-agent: Wire
Disallow: /

User-agent: Wisebot
Disallow: /

User-agent: WiseGuys Robot
Disallow: /

User-agent: WISENutbot
Disallow: /

User-agent: Wish-la
Disallow: /

User-agent: Wish-project
Disallow: /

User-agent: Wisponbot
Disallow: /

User-agent: Wmcai-robot
Disallow: /

User-agent: Wminer
Disallow: /

User-agent: Wmsbot
Disallow: /

User-agent: Wonderbot
Disallow: /

User-agent: Woobot
Disallow: /

User-agent: Woriobot
Disallow: /

User-agent: Worldshop
Disallow: /

User-agent: Worqmada
Disallow: /

User-agent: Wotbox
Disallow: /

User-agent: Wprecon
Disallow: /

User-agent: WPScan
Disallow: /

User-agent: Wume_crawler
Disallow: /

User-agent: Www collector
Disallow: /

User-agent: Www-collector-e
Disallow: /

User-agent: WWW-Collector-E
Disallow: /

User-agent: Www-mechanize
Disallow: /

User-agent: WWW-Mechanize
Disallow: /

User-agent: WWW::Mechanize
Disallow: /

User-agent: WWWOFFLE
Disallow: /

User-agent: Wwwrobot
Disallow: /

User-agent: Wwwster
Disallow: /

User-agent: Wwwwanderer
Disallow: /

User-agent: Wwwxref
Disallow: /

User-agent: Wysigot
Disallow: /

User-agent: X09Mozilla
Disallow: /

User-agent: X22Mozilla
Disallow: /

User-agent: Xaldon
Disallow: /

User-agent: Xaldon_WebSpider
Disallow: /

User-agent: Xaldon WebSpider
Disallow: /

User-agent: X-clawler
Disallow: /

User-agent: X-crawler
Disallow: /

User-agent: Xenu
Disallow: /

User-agent: Xerka metabot
Disallow: /

User-agent: Xerka webbot
Disallow: /

User-agent: Xget
Disallow: /

User-agent: Xirq
Disallow: /

User-agent: Xmarksfetch
Disallow: /

User-agent: Xpymep1.exe
Disallow: /

User-agent: Xqrobot
Disallow: /

User-agent: Yacybot
Disallow: /

User-agent: Yacy.net
Disallow: /

User-agent: Yanga worldsearch bot
Disallow: /

User-agent: Yarienavoir.net
Disallow: /

User-agent: Yasaklibot
Disallow: /

User-agent: Yats crawler
Disallow: /

User-agent: Ybot
Disallow: /

User-agent: Yebolbot
Disallow: /

User-agent: Yellowjacket
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: Y!j
Disallow: /

User-agent: Yolinkbot
Disallow: /

User-agent: Yooglifetchagent
Disallow: /

User-agent: Yoono
Disallow: /

User-agent: Yottacars_bot
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: Yourls
Disallow: /

User-agent: Z-add link checker
Disallow: /

User-agent: Zade
Disallow: /

User-agent: Zagrebin
Disallow: /

User-agent: Zao
Disallow: /

User-agent: Zauba
Disallow: /

User-agent: Zauba.io
Disallow: /

User-agent: Zedzo.validate
Disallow: /

User-agent: Zermelo
Disallow: /

User-agent: Zeus
Disallow: /

User-agent: Zgrab
Disallow: /

User-agent: Zibber-v
Disallow: /

User-agent: Zimeno
Disallow: /

User-agent: Zing-bottabo
Disallow: /

User-agent: Zipppbot
Disallow: /

User-agent: Zitebot
Disallow: /

User-agent: ZmEu
Disallow: /

User-agent: Zongbot
Disallow: /

User-agent: Zoomspider
Disallow: /

User-agent: Zotag search
Disallow: /

User-agent: Zsebot
Disallow: /

User-agent: Zuibot
Disallow: /

User-agent: ZumBot
Disallow: /

User-agent: ZyBorg
Disallow: /

User-agent: Zyte
Disallow: /

# ADDED BY MNIT @ DHS 2/28/2019

User-agent: *
Disallow: /imform/*
Disallow: /Insight/*
Disallow: /InsightMetrics/*

User-agent: yandex
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: eContext
Disallow: /

User-agent: archive.org_bot
Disallow: /

User-agent: PiplBot
Disallow: /

User-Agent: Seobility
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: WinistaBot
Disallow: /

User-agent: Flipbot
Disallow: /

User-Agent: The Knowledge AI
Disallow: /

User-Agent: Sogou Web Spider
Disallow: /

User-Agent: Baiduspider
Disallow: /

User-Agent: sSearch Crawler
Disallow: /

User-Agent: Slack-ImgProxy
Disallow: /

User-Agent: Pinterestbot
Disallow: /

User-Agent: Monsidobot
Disallow: /

User-Agent: Screaming Frog SEO Spider
Dissallow: /

User-agent: vebidoobot
Disallow: /

User-Agent: BublupBot
Disallow: /

User-Agent: Thither.Direct Bot/0.0.1
Disallow: /

User-agent: *
# CSS, JS, Images
Allow: /core/*.css$
Allow: /core/*.css?
Allow: /core/*.js$
Allow: /core/*.js?
Allow: /core/*.gif
Allow: /core/*.jpg
Allow: /core/*.jpeg
Allow: /core/*.png
Allow: /core/*.svg
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /profiles/*.svg
# Directories
Disallow: /core/
Disallow: /profiles/
# Files
Disallow: /README.txt
Disallow: /web.config
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register
Disallow: /user/password
Disallow: /user/login
Disallow: /user/logout
# Paths (no clean URLs)
Disallow: /index.php/admin/
Disallow: /index.php/comment/reply/
Disallow: /index.php/filter/tips
Disallow: /index.php/node/add/
Disallow: /index.php/search/
Disallow: /index.php/user/password
Disallow: /index.php/user/register
Disallow: /index.php/user/login
Disallow: /index.php/user/logout
# Disallow search page
Disallow: /search?
La página no está disponible

Política de cookies
