I am consulting on a website project that is using XML Sitemap 7.x. Their robots.txt is set to:

User-agent: *
Disallow: /

When I mentioned this to the site owner, he checked with the developer who said this was correct because they are submitting the XML sitemap to Google automatically using the module. This seems odd, to say the least.

Would there ever be a legitimate reason to replace the default Drupal 7 robots.txt with this one?

Thanks,
Maria

Comments

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Fixed

No, the modules have every reason to work together. Note that the robots.txt you specified is telling all search engines to ignore all pages, even if they were pinged or have a list of pages that should be updated.

justmagicmaria’s picture

justmagicmaria commented

Thank you, Dave, for your quick response and for confirming what I suspected.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.