# disable crawling non-article pages, # but allow loading CSS/JS for site previews; # per https://www.mediawiki.org/w/index.php?title=Manual:Robots.txt&oldid=5844010 User-agent: * Allow: /w/load.php? Disallow: /w/ # block bots from Bytedance (owners of TikTok); # they seem to have been crawling BS01 a lot, # but according to the following links, # the bots are known not to respect robots.txt rules, # so we block them using .htaccess instead # https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/ # https://stackoverflow.com/a/58307944
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)