This site can not be crawled by the Seobility bot. The robots.txt file restricts access for our bot. No URLs can be crawled that deny access to the user agent Seobility or *.
Problem URL: http://weine.de
## GENERAL SETTINGS ## Enable robots.txt rules for all crawlers User-agent: * Sitemap: https://www.weine.de/sitemap.xml ## DEVELOPMENT RELATED SETTINGS ## Do not crawl development files and folders: CVS, svn directories and dump files Disallow: CVS Disallow: .svn Disallow: .idea Disallow: .sql Disallow: .tgz ## GENERAL MAGENTO SETTINGS User-agent: * Disallow: / Crawl-delay: 60 ## Allow some bots to crawl User-agent: Googlebot User-agent: bingbot User-agent: Slurp User-agent: msnbot User-agent: Mediapartners-Google* User-agent: Yahoo-MMCrawler User-agent: sistrix User-agent: AhrefsBot Disallow: ## Do not crawl Magento admin page User-agent: * Disallow: /admin/ ## Do not crawl common Magento technical folders User-agent: * Disallow: /app/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /lib/ Disallow: /sztools/ Disallow: /pkginfo/ Disallow: /shell/ Disallow: /var/ Disallow: /report/ Disallow: /tmp/ ## Do not crawl common Magento files User-agent: * Disallow: /api.php Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /get.php Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /README.txt Disallow: /RELEASE_NOTES.txt ## MAGENTO SEO IMPROVEMENTS ## Do not crawl sub category pages that are sorted or filtered. User-agent: * Disallow: /*? Disallow: /*?dir* Disallow: /*?dir=desc Disallow: /*?dir=asc Disallow: /*?limit=all Disallow: /*?mode* ## Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs. User-agent: * Disallow: /index.php/ ## Do not crawl links with session IDs User-agent: * Disallow: /*?SID= ## Do not crawl checkout and user account pages User-agent: * Disallow: /checkout/ Disallow: /onestepcheckout/ Disallow: /customer/ Disallow: /customer/account/ Disallow: /customer/account/login/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /review/ ## Do not crawl seach pages and not-SEO optimized catalog links User-agent: * Disallow: /catalogsearch/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ ## Do not crawl availability check ## User-agent: * Disallow: /productalert/add/stock/product_id/ ## SERVER SETTINGS ## Do not crawl common server technical folders and files User-agent: * Disallow: /cgi-bin/ Disallow: /cleanup.php Disallow: /apc.php Disallow: /memcache.php Disallow: /phpinfo.php