Studentenrabatt.de - SEO Checker

Metaetiquetas

Lista de medios

Estructura de los encabezados

Enlaces detectados en esta página

Cabecera HTTP

Backlinks desde Wikipedia

Robots.txt

#######################################
# ALL BOTS & SPIDERS = NO CRAWL-DELAY #
#######################################

User-agent: *
Allow: /*?page=*
Allow: /*?pagina=*
Allow: *?id=*
Allow: *?group=*
Allow: */stores
Allow: /favicon.ico
Allow: /images/*
Allow: /img/*
Allow: /assets/*
Allow: /.well-known/*
Disallow: *mod_pagespeed_beacon?*
Disallow: *pagespeed*
Disallow: /*sorting*
Disallow: /*disc_from*
Disallow: /*price_from*
Disallow: /*?page=*&*
Disallow: /*?pagina=*&*
Disallow: /?$
Disallow: /shop?*
Disallow: /kortingen?*
Disallow: /lokaal?*
Disallow: /*?$
Disallow: /live/*
Disallow: /shop-goto/specs*
Disallow: */latest/id/*
Disallow: */get-thumbs/*
Disallow: */thumb-up/*
Disallow: */thumb-down/*
Disallow: /account
Disallow: /accounts
Disallow: /promo
Disallow: /verlanglijstje
Disallow: /login
Disallow: /beheer*
Disallow: */hyves.nl
Disallow: */openid
Disallow: *store.php*
Disallow: */store*
Disallow: */afsluiten/*
Disallow: *go.php*
Disallow: */go/*
Disallow: *//*
Disallow: *aanmelden*
Disallow: /goto/*
Disallow: /shop-goto/*
Disallow: /nationaal-goto/*
Disallow: /lokaal-goto/*
Disallow: */go-to*
Disallow: *filter?*
Disallow: *PageSpeed=noscript*
Disallow: *ModPagespeed*
Disallow: *?menu=*
Disallow: *disc_type=*
Disallow: *filtered=*
Disallow: *disc_from=*
Disallow: *disc_to=*
Disallow: *tag%5Bt90%5D=*
Disallow: *?tag%*
Disallow: *[*
Disallow: *]*
Disallow: *?search=*
Disallow: */search/*
Disallow: *?zoek=*
Disallow: */zoek/*
Disallow: */zoeken/*
Disallow: *?suche=*
Disallow: */suche/*
Disallow: /share/*
Disallow: /wp-*
Disallow: /config*
Disallow: *.env*
Disallow: *env.txt*
Disallow: *.git*
Disallow: *.vscode*
Disallow: *.DS_Store
Disallow: *.bak
Disallow: /old/*
Disallow: /backup/*
Disallow: /upload/*
Disallow: /html/*
Disallow: /tel:*

# DUPLICATE CATEGORIES
Allow: */broeken/broeken/*
Allow: */jassen/jassen/*
Allow: */kostuums/kostuums/*
Disallow: */broeken/broeken
Disallow: */jassen/jassen
Disallow: */kostuums/kostuums

##################################################
# SEARCH ENGINE BOTS / SPIDERS = CRAWL-DELAY: 20 #
##################################################

User-agent: Bingbot
User-Agent: msnbot
User-Agent: Slurp
User-Agent: Yahoo! Slurp
User-agent: DuckDuckBot
User-Agent: Ingrid
User-agent: teoma
Crawl-delay: 20
Allow: /*?page=*
Allow: /*?pagina=*
Allow: *?id=*
Allow: *?group=*
Allow: */stores
Allow: /favicon.ico
Allow: /images/*
Allow: /img/*
Allow: /assets/*
Disallow: *mod_pagespeed_beacon?*
Disallow: *pagespeed*
Disallow: /*sorting*
Disallow: /*disc_from*
Disallow: /*price_from*
Disallow: /*?page=*&*
Disallow: /*?pagina=*&*
Disallow: /?$
Disallow: /shop?*
Disallow: /kortingen?*
Disallow: /lokaal?*
Disallow: /*?$
Disallow: /live/*
Disallow: /shop-goto/specs*
Disallow: */latest/id/*
Disallow: */get-thumbs/*
Disallow: */thumb-up/*
Disallow: */thumb-down/*
Disallow: /account
Disallow: /accounts
Disallow: /promo
Disallow: /verlanglijstje
Disallow: /login
Disallow: /beheer*
Disallow: */hyves.nl
Disallow: */openid
Disallow: *store.php*
Disallow: */store*
Disallow: */afsluiten/*
Disallow: *go.php*
Disallow: */go/*
Disallow: *aanmelden*
Disallow: /goto/*
Disallow: /shop-goto/*
Disallow: /nationaal-goto/*
Disallow: /lokaal-goto/*
Disallow: */go-to*
Disallow: *filter?*
Disallow: *PageSpeed=noscript*
Disallow: *ModPagespeed*
Disallow: *?menu=*
Disallow: *disc_type=*
Disallow: *filtered=*
Disallow: *disc_from=*
Disallow: *disc_to=*
Disallow: *tag%5Bt90%5D=*
Disallow: *?tag%*
Disallow: *[*
Disallow: *]*
Disallow: /share/*
Disallow: /wp-*
Disallow: /config*
Disallow: *.env*
Disallow: *env.txt*
Disallow: *.git*
Disallow: *.vscode*
Disallow: *.DS_Store
Disallow: *.bak
Disallow: /old/*
Disallow: /backup/*
Disallow: /upload/*
Disallow: /html/*
Disallow: /tel:*

# DUPLICATE CATEGORIES
Allow: */broeken/broeken/*
Allow: */jassen/jassen/*
Allow: */kostuums/kostuums/*
Disallow: */broeken/broeken
Disallow: */jassen/jassen
Disallow: */kostuums/kostuums

###################################################
# MARKTING & SEO BOTS / SPIDERS = CRAWL-DELAY: 15 #
###################################################

User-agent: rogerbot
User-agent: SEMrushBot
User-agent: Semrush
User-agent: SemrushBot
User-agent: SemrushBot/1.2~bl
User-agent: SemrushBot-SA
User-agent: SemrushBot-BA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SemrushBot-CT
User-agent: SemrushBot-BM
User-agent: MJ12Bot
User-agent: DotBot
User-agent: MauiBot
User-agent: Pinterest
User-agent: Bytespider
crawl-delay: 15
Allow: /*?page=*
Allow: /*?pagina=*
Allow: *?id=*
Allow: *?group=*
Allow: */stores
Allow: /favicon.ico
Allow: /images/*
Allow: /img/*
Allow: /assets/*
Disallow: *mod_pagespeed_beacon?*
Disallow: *pagespeed*
Disallow: /*sorting*
Disallow: /*disc_from*
Disallow: /*price_from*
Disallow: /*?page=*&*
Disallow: /*?pagina=*&*
Disallow: /?$
Disallow: /shop?*
Disallow: /kortingen?*
Disallow: /lokaal?*
Disallow: /*?$
Disallow: /live/*
Disallow: /shop-goto/specs*
Disallow: */latest/id/*
Disallow: */get-thumbs/*
Disallow: */thumb-up/*
Disallow: */thumb-down/*
Disallow: /account
Disallow: /accounts
Disallow: /promo
Disallow: /verlanglijstje
Disallow: /login
Disallow: /beheer*
Disallow: */hyves.nl
Disallow: */openid
Disallow: *store.php*
Disallow: */store*
Disallow: */afsluiten/*
Disallow: *go.php*
Disallow: */go/*
Disallow: *aanmelden*
Disallow: /goto/*
Disallow: /shop-goto/*
Disallow: /nationaal-goto/*
Disallow: /lokaal-goto/*
Disallow: */go-to*
Disallow: *filter?*
Disallow: *PageSpeed=noscript*
Disallow: *ModPagespeed*
Disallow: *?menu=*
Disallow: *disc_type=*
Disallow: *filtered=*
Disallow: *disc_from=*
Disallow: *disc_to=*
Disallow: *tag%5Bt90%5D=*
Disallow: *?tag%*
Disallow: *[*
Disallow: *]*
Disallow: /share/*
Disallow: /wp-*
Disallow: /config*
Disallow: *.env*
Disallow: *env.txt*
Disallow: *.git*
Disallow: *.vscode*
Disallow: *.DS_Store
Disallow: *.bak
Disallow: /old/*
Disallow: /backup/*
Disallow: /upload/*
Disallow: /html/*
Disallow: /tel:*

# DUPLICATE CATEGORIES
Allow: */broeken/broeken/*
Allow: */jassen/jassen/*
Allow: */kostuums/kostuums/*
Disallow: */broeken/broeken
Disallow: */jassen/jassen
Disallow: */kostuums/kostuums

###############################
# BLOCKING BAD BOTS & SPIDERS #
###############################

User-agent: AspiegelBot
User-agent: PetalBot
User-agent: Yandex
User-agent: YandexBot
User-agent: AhrefsBot
User-agent: EtaoSpider
User-agent: RukiCrawler
User-agent: Bilbo
User-agent: Libwww-perl
User-agent: moget
User-agent: ichiro
User-agent: NaverBot
User-agent: Yeti
User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
User-agent: sogouspider
User-agent: sogou spider
User-agent: Sogou web spider
User-agent: Sogou inst spider
User-agent: Sogou spider2
User-agent: Sogou Pic Spider
User-agent: Sogou head spider
User-agent: Sogou Orion spider
User-agent: Sogou-Test-Spider
User-agent: ShopWiki
User-Agent: Fasterfox
User-agent: teracent-feed-processing
User-agent: 360Spider
User-agent: 404checker
User-agent: 404enemy
User-agent: 80legs
User-agent: Abonti
User-agent: Aboundex
User-agent: Aboundexbot
User-agent: Acunetix
User-agent: ADmantX
User-agent: AfD-Verbotsverfahren
User-agent: AIBOT
User-agent: AiHitBot
User-agent: Aipbot
User-agent: Alexibot
User-agent: Alligator
User-agent: AllSubmitter
User-agent: AlphaBot
User-agent: Anarchie
User-agent: Ankit
User-agent: Apexoo
User-agent: archive.org_bot
User-agent: arquivo.pt
User-agent: arquivo-web-crawler
User-agent: Aspeigel
User-agent: ASPSeek
User-agent: Asterias
User-agent: Attach
User-agent: autoemailspider
User-agent: AwarioRssBot
User-agent: AwarioSmartBot
User-agent: BackDoorBot
User-agent: Backlink-Ceck
User-agent: backlink-check
User-agent: BacklinkCrawler
User-agent: BackStreet
User-agent: BackWeb
User-agent: Badass
User-agent: Bandit
User-agent: Barkrowler
User-agent: BatchFTP
User-agent: Battleztar Bazinga
User-agent: BBBike
User-agent: BDCbot
User-agent: BDFetch
User-agent: BetaBot
User-agent: Bigfoot
User-agent: Bitacle
User-agent: Blackboard
User-agent: Black Hole
User-agent: BlackWidow
User-agent: BLEXBot
User-agent: Blow
User-agent: BlowFish
User-agent: Boardreader
User-agent: Bolt
User-agent: BotALot
User-agent: Brandprotect
User-agent: Brandwatch
User-agent: Buddy
User-agent: BuiltBotTough
User-agent: BuiltWith
User-agent: Bullseye
User-agent: BunnySlippers
User-agent: BuzzSumo
User-agent: Calculon
User-agent: CATExplorador
User-agent: CazoodleBot
User-agent: CCBot
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CheTeam
User-agent: ChinaClaw
User-agent: Chlooe
User-agent: Claritybot
User-agent: Cliqzbot
User-agent: Cloud mapping
User-agent: coccocbot-web
User-agent: Cogentbot
User-agent: cognitiveseo
User-agent: Collector
User-agent: com.plumanalytics
User-agent: Copier
User-agent: CopyRightCheck
User-agent: Copyscape
User-agent: Cosmos
User-agent: Craftbot
User-agent: crawler4j
User-agent: crawler.feedback
User-agent: crawl.sogou.com
User-agent: CrazyWebCrawler
User-agent: Crescent
User-agent: CrunchBot
User-agent: CSHttp
User-agent: Curious
User-agent: Custo
User-agent: DatabaseDriverMysqli
User-agent: DataCha0s
User-agent: DBLBot
User-agent: demandbase-bot
User-agent: Demon
User-agent: Deusu
User-agent: Devil
User-agent: Digincore
User-agent: DigitalPebble
User-agent: DIIbot
User-agent: Dirbuster
User-agent: Disco
User-agent: Discobot
User-agent: Discoverybot
User-agent: Dispatch
User-agent: DittoSpyder
User-agent: DnyzBot
User-agent: DomainAppender
User-agent: DomainCrawler
User-agent: DomainSigmaCrawler
User-agent: DomainStatsBot
User-agent: Download Wonder
User-agent: Dragonfly
User-agent: Drip
User-agent: DSearch
User-agent: DTS Agent
User-agent: EasyDL
User-agent: Ebingbong
User-agent: eCatch
User-agent: ECCP/1.0
User-agent: Ecxi
User-agent: EirGrabber
User-agent: EMail Siphon
User-agent: EMail Wolf
User-agent: EroCrawler
User-agent: evc-batch
User-agent: Evil
User-agent: Exabot
User-agent: Express WebPictures
User-agent: ExtLinksBot
User-agent: Extractor
User-agent: ExtractorPro
User-agent: Extreme Picture Finder
User-agent: EyeNetIE
User-agent: Ezooms
User-agent: facebookscraper
User-agent: FDM
User-agent: FemtosearchBot
User-agent: FHscan
User-agent: Fimap
User-agent: Firefox/7.0
User-agent: FlashGet
User-agent: Flunky
User-agent: Foobot
User-agent: Freeuploader
User-agent: FrontPage
User-agent: FyberSpider
User-agent: Fyrebot
User-agent: GalaxyBot
User-agent: Genieo
User-agent: GermCrawler
User-agent: Getintent
User-agent: GetRight
User-agent: GetWeb
User-agent: Gigablast
User-agent: Gigabot
User-agent: G-i-g-a-b-o-t
User-agent: Go-Ahead-Got-It
User-agent: Gotit
User-agent: GoZilla
User-agent: Go!Zilla
User-agent: Grabber
User-agent: GrabNet
User-agent: Grafula
User-agent: GrapeFX
User-agent: GrapeshotCrawler
User-agent: GridBot
User-agent: GT::WWW
User-agent: Haansoft
User-agent: HaosouSpider
User-agent: Harvest
User-agent: Havij
User-agent: HEADMasterSEO
User-agent: heritrix
User-agent: Heritrix
User-agent: Hloader
User-agent: HMView
User-agent: HTMLparser
User-agent: HTTP::Lite
User-agent: HTTrack
User-agent: Humanlinks
User-agent: HybridBot
User-agent: Iblog
User-agent: IDBot
User-agent: Id-search
User-agent: IlseBot
User-agent: Image Fetch
User-agent: Image Sucker
User-agent: IndeedBot
User-agent: Indy Library
User-agent: InfoNaviRobot
User-agent: InfoTekies
User-agent: instabid
User-agent: Intelliseek
User-agent: InterGET
User-agent: Internet Ninja
User-agent: InternetSeer
User-agent: internetVista monitor
User-agent: ips-agent
User-agent: Iria
User-agent: IRLbot
User-agent: Iskanie
User-agent: IstellaBot
User-agent: JamesBOT
User-agent: Jbrofuzz
User-agent: JennyBot
User-agent: JetCar
User-agent: Jetty
User-agent: JikeSpider
User-agent: JOC Web Spider
User-agent: Joomla
User-agent: Jorgee
User-agent: JustView
User-agent: Jyxobot
User-agent: Kenjin Spider
User-agent: Keyword Density
User-agent: Kinza
User-agent: Kozmosbot
User-agent: Lanshanbot
User-agent: Larbin
User-agent: LeechFTP
User-agent: LeechGet
User-agent: LexiBot
User-agent: Lftp
User-agent: LibWeb
User-agent: Libwhisker
User-agent: LieBaoFast
User-agent: Lightspeedsystems
User-agent: Likse
User-agent: Linkdexbot
User-agent: LinkextractorPro
User-agent: LinkpadBot
User-agent: LinkScan
User-agent: LinksManager
User-agent: LinkWalker
User-agent: LinqiaMetadataDownloaderBot
User-agent: LinqiaRSSBot
User-agent: LinqiaScrapeBot
User-agent: Lipperhey
User-agent: Lipperhey Spider
User-agent: Litemage_walker
User-agent: Lmspider
User-agent: LNSpiderguy
User-agent: Ltx71
User-agent: lwp-request
User-agent: LWP::Simple
User-agent: lwp-trivial
User-agent: Magnet
User-agent: Mag-Net
User-agent: magpie-crawler
User-agent: Mail.RU_Bot
User-agent: Majestic12
User-agent: Majestic-SEO
User-agent: Majestic SEO
User-agent: MarkMonitor
User-agent: MarkWatch
User-agent: Masscan
User-agent: Mass Downloader
User-agent: Mata Hari
User-agent: Mb2345Browser
User-agent: meanpathbot
User-agent: Meanpathbot
User-agent: MeanPath Bot
User-agent: Mediatoolkitbot
User-agent: mediawords
User-agent: MegaIndex.ru
User-agent: Metauri
User-agent: MFC_Tear_Sample
User-agent: MicroMessenger
User-agent: Microsoft Data Access
User-agent: Microsoft URL Control
User-agent: MIDown tool
User-agent: MIIxpc
User-agent: Mister PiX
User-agent: Mojeek
User-agent: Mojolicious
User-agent: Morfeus Fucking Scanner
User-agent: MQQBrowser
User-agent: Mr.4x3
User-agent: MSFrontPage
User-agent: MSIECrawler
User-agent: Msrabot
User-agent: muhstik-scan
User-agent: Musobot
User-agent: Name Intelligence
User-agent: Nameprotect
User-agent: Navroad
User-agent: NearSite
User-agent: Needle
User-agent: Nessus
User-agent: NetAnts
User-agent: Netcraft
User-agent: netEstate NE Crawler
User-agent: NetLyzer
User-agent: NetMechanic
User-agent: NetSpider
User-agent: Nettrack
User-agent: Net Vampire
User-agent: Netvibes
User-agent: NetZIP
User-agent: NextGenSearchBot
User-agent: Nibbler
User-agent: NICErsPRO
User-agent: Niki-bot
User-agent: Nikto
User-agent: NimbleCrawler
User-agent: Nimbostratus
User-agent: Ninja
User-agent: Nmap
User-agent: NPbot
User-agent: Nutch
User-agent: oBot
User-agent: Octopus
User-agent: Offline Explorer
User-agent: Offline Navigator
User-agent: OnCrawl
User-agent: Openfind
User-agent: OpenLinkProfiler
User-agent: Openvas
User-agent: OpenVAS
User-agent: OPPO A33
User-agent: OrangeBot
User-agent: OrangeSpider
User-agent: OutclicksBot
User-agent: OutfoxBot
User-agent: PageAnalyzer
User-agent: Page Analyzer
User-agent: PageGrabber
User-agent: page scorer
User-agent: PageScorer
User-agent: Pandalytics
User-agent: Panscient
User-agent: Papa Foto
User-agent: Pavuk
User-agent: pcBrowser
User-agent: PECL::HTTP
User-agent: PeoplePal
User-agent: PHPCrawl
User-agent: Picscout
User-agent: Picsearch
User-agent: PictureFinder
User-agent: Pimonster
User-agent: Pi-Monster
User-agent: Pixray
User-agent: PleaseCrawl
User-agent: plumanalytics
User-agent: Pockey
User-agent: POE-Component-Client-HTTP
User-agent: polaris version
User-agent: Probethenet
User-agent: ProPowerBot
User-agent: ProWebWalker
User-agent: Psbot
User-agent: Pump
User-agent: PxBroker
User-agent: PyCurl
User-agent: QueryN Metasearch
User-agent: Quick-Crawler
User-agent: RankActive
User-agent: RankActiveLinkBot
User-agent: RankFlex
User-agent: RankingBot
User-agent: RankingBot2
User-agent: Rankivabot
User-agent: RankurBot
User-agent: RealDownload
User-agent: Reaper
User-agent: RebelMouse
User-agent: Recorder
User-agent: RedesScrapy
User-agent: ReGet
User-agent: RepoMonkey
User-agent: Ripper
User-agent: RocketCrawler
User-agent: RSSingBot
User-agent: s1z.ru
User-agent: SalesIntelligent
User-agent: SBIder
User-agent: ScanAlert
User-agent: Scanbot
User-agent: scan.lol
User-agent: ScoutJet
User-agent: Scrapy
User-agent: Screaming
User-agent: ScreenerBot
User-agent: Searchestate
User-agent: SearchmetricsBot
User-agent: SEOkicks
User-agent: SEOkicks-Robot
User-agent: SEOlyticsCrawler
User-agent: Seomoz
User-agent: SEOprofiler
User-agent: seoscanners
User-agent: SeoSiteCheckup
User-agent: SEOstats
User-agent: serpstatbot
User-agent: sexsearcher
User-agent: Shodan
User-agent: Siphon
User-agent: SISTRIX
User-agent: Sitebeam
User-agent: SiteExplorer
User-agent: Siteimprove
User-agent: SiteLockSpider
User-agent: SiteSnagger
User-agent: SiteSucker
User-agent: Site Sucker
User-agent: Sitevigil
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SMTBot
User-agent: Snake
User-agent: Snapbot
User-agent: Snoopy
User-agent: SocialRankIOBot
User-agent: Sociscraper
User-agent: Sosospider
User-agent: Sottopop
User-agent: SpaceBison
User-agent: Spammen
User-agent: SpankBot
User-agent: Spanner
User-agent: sp_auditbot
User-agent: Spbot
User-agent: Spinn3r
User-agent: SputnikBot
User-agent: spyfu
User-agent: Sqlmap
User-agent: Sqlworm
User-agent: Sqworm
User-agent: Steeler
User-agent: Stripper
User-agent: Sucker
User-agent: Sucuri
User-agent: SuperBot
User-agent: SuperHTTP
User-agent: Surfbot
User-agent: SurveyBot
User-agent: Suzuran
User-agent: Swiftbot
User-agent: sysscan
User-agent: Szukacz
User-agent: T0PHackTeam
User-agent: T8Abot
User-agent: tAkeOut
User-agent: Teleport
User-agent: TeleportPro
User-agent: Telesoft
User-agent: Telesphoreo
User-agent: Telesphorep
User-agent: The Intraformant
User-agent: TheNomad
User-agent: Thumbor
User-agent: TightTwatBot
User-agent: Titan
User-agent: Toata
User-agent: Toweyabot
User-agent: Tracemyfile
User-agent: Trendiction
User-agent: Trendictionbot
User-agent: trendiction.com
User-agent: trendiction.de
User-agent: True_Robot
User-agent: Turingos
User-agent: Turnitin
User-agent: TurnitinBot
User-agent: TwengaBot
User-agent: Twice
User-agent: Typhoeus
User-agent: UnisterBot
User-agent: Upflow
User-agent: URLy.Warning
User-agent: URLy Warning
User-agent: Vacuum
User-agent: Vagabondo
User-agent: VB Project
User-agent: VCI
User-agent: VeriCiteCrawler
User-agent: VidibleScraper
User-agent: Virusdie
User-agent: VoidEYE
User-agent: Voil
User-agent: Voltron
User-agent: Wallpapers/3.0
User-agent: WallpapersHD
User-agent: WASALive-Bot
User-agent: WBSearchBot
User-agent: Webalta
User-agent: WebAuto
User-agent: Web Auto
User-agent: WebBandit
User-agent: WebCollage
User-agent: Web Collage
User-agent: WebCopier
User-agent: WEBDAV
User-agent: WebEnhancer
User-agent: Web Enhancer
User-agent: WebFetch
User-agent: Web Fetch
User-agent: WebFuck
User-agent: Web Fuck
User-agent: WebGo IS
User-agent: WebImageCollector
User-agent: WebLeacher
User-agent: WebmasterWorldForumBot
User-agent: webmeup-crawler
User-agent: WebPix
User-agent: Web Pix
User-agent: WebReaper
User-agent: WebSauger
User-agent: Web Sauger
User-agent: Webshag
User-agent: WebsiteExtractor
User-agent: WebsiteQuester
User-agent: Website Quester
User-agent: Webster
User-agent: WebStripper
User-agent: WebSucker
User-agent: Web Sucker
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WeSEE
User-agent: Whack
User-agent: Whacker
User-agent: Whatweb
User-agent: Who.is Bot
User-agent: Widow
User-agent: WinHTTrack
User-agent: WiseGuys Robot
User-agent: WISENutbot
User-agent: Wonderbot
User-agent: Woobot
User-agent: Wotbox
User-agent: Wprecon
User-agent: WPScan
User-agent: WWW-Collector-E
User-agent: WWW-Mechanize
User-agent: WWW::Mechanize
User-agent: WWWOFFLE
User-agent: x09Mozilla
User-agent: x22Mozilla
User-agent: Xaldon_WebSpider
User-agent: Xaldon WebSpider
User-agent: Xenu
User-agent: xpymep1.exe
User-agent: YoudaoBot
User-agent: Zade
User-agent: Zauba
User-agent: zauba.io
User-agent: Zermelo
User-agent: Zeus
User-agent: zgrab
User-agent: Zitebot
User-agent: ZmEu
User-agent: ZumBot
User-agent: ZyBorg
Disallow:/
Disallow:/*

Snippet (vista previa de los resultados de búsqueda)