Robots.txt has a slight error in it. It blocks robots from crawling mysite.com/search/ , but doesn't block them from crawling mysite.com/search . So they will still get the same page. Here's a patch to add the variations into robots.txt .

I'm not very familiar with patches, but this looks right to me.

CommentFileSizeAuthor
robots.patch3.81 KBAnonymous (not verified)

Comments

Status: Needs review » Needs work

The last submitted patch, robots.patch, failed testing.

dddave’s picture

dddave commented
Status: Needs work » Closed (won't fix)

This seems to be an deliberate decision. I am no expert when it comes to robots.txt but I think disallowing with a trailing slash is sufficient. Can you link to a ressource telling us to disallow both variants?