This site can not be crawled by the Seobility bot. The robots.txt file restricts access for our bot. No URLs can be crawled that deny access to the user agent Seobility or *.
Problem URL: http://dot.ny.gov
# $Id: robots.txt # # This is a file retrieved by webwalkers a.k.a. spiders that # conform to a defacto standard. # See <URL:http://www.robotstxt.org/wc/exclusion.html#robotstxt> # # Format is: # User-agent: <name of spider> # Disallow: <nothing> | <path> # ----------------------------------------------------------------------------- User-agent: nys-crawler Disallow: User-agent: WC3-checklink Disallow: User-agent: Googlebot Disallow: User-agent: Inktomi Slurp Disallow: User-agent: MSNBot Disallow: User-agent: AskJeeves Disallow: User-agent: InfoSeek Robot 1.0 Disallow: User-agent: InfoSeek Sidewinder Disallow: User-agent: nys-qa-crawler Disallow: User-agent: * Disallow: / Disallow: // Disallow: /portal/page/portal/ Disallow: /pls/ Disallow: /portalHelp2/ Disallow: /portal/pls/ Disallow: /tmp Disallow: /export/home/appsrv/utilizationReports