Robots.txt
User-agent: *
Disallow: /login/
Disallow: /sign-in/
Disallow: /sign-up/
Disallow: /search/
User-agent: Googlebot
Disallow: /*/arcmap/10.3/
Disallow: /*/arcmap/10.3.1/
Disallow: /*/arcmap/10.4/
Disallow: /*/arcmap/10.5/
Disallow: /*/arcmap/10.6/
User-agent: Bingbot
Disallow: /*/arcmap/10.3/
Disallow: /*/arcmap/10.3.1/
Disallow: /*/arcmap/10.4/
Disallow: /*/arcmap/10.5/
Disallow: /*/arcmap/10.6/
User-agent: Slurp
Disallow: /*/arcmap/10.3/
Disallow: /*/arcmap/10.3.1/
Disallow: /*/arcmap/10.4/
Disallow: /*/arcmap/10.5/
Disallow: /*/arcmap/10.6/
User-agent: DuckDuckBot
Disallow: /*/arcmap/10.3/
Disallow: /*/arcmap/10.3.1/
Disallow: /*/arcmap/10.4/
Disallow: /*/arcmap/10.5/
Disallow: /*/arcmap/10.6/
User-agent: Baiduspider
Disallow: /*/arcmap/10.3/
Disallow: /*/arcmap/10.3.1/
Disallow: /*/arcmap/10.4/
Disallow: /*/arcmap/10.5/
Disallow: /*/arcmap/10.6/
Sitemap: http://desktop.arcgis.com/sitemap_index.xml
(Nice to have)