Robots.txt
## GENERAL SETTINGS
User-agent: *
Allow:/
## CRAWL BUDGET OPTIMIZATION
User-agent: *
# Disallow: /*?
Disallow: */s/
Disallow: */modal
Disallow: /api/
Disallow: /labels/
## PRIVATE URLS
User-agent: *
Disallow: */user-management
Disallow: */authentication
Disallow: */purchase
Disallow: */error
Disallow: */m/
Disallow: */sm/
## SPECIFIC BOTS SETTINGS
User-agent: noxtrumbot
Crawl-delay: 50
User-agent: msnbot
Crawl-delay: 30
User-agent: Slurp
Crawl-delay: 10
## SITEMAPS
# La url debe ser absoluta https://...
Sitemap: https://www.online.bmsupermercados.es/sitemap.xml
(Nice to have)