User-agent: Scrapy Allow: / # This is to slow down any crawling so as not to put pressure on your server Crawl-Delay: 30 # The following are what we normally don't want to be crawled Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Allow: /wp-content/uploads/ Disallow: /tag/ # uncomment if you are not using tags Disallow: /category/ # uncomment if you are not using categories Disallow: /author/ # uncomment for single user blogs Disallow: /feed/ Disallow: /trackback/ Disallow: /index.php # separate directive for the main script file of WP Disallow: /*? # search results Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: */feed/ Disallow: */trackback/ # Disallow: */print/ Sitemap: http://roemersee.de/sitemap.xml.gz Sitemap: http://roemersee.de/sitemap.xml
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)