Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
The default robots.txt does not cooperate very well with multilingual sites - it blocks /admin/ but not /fr/admin , more importantly, /search is covered, but /xx/search isn't, which causes googlebot to crawl our search-results.
Since robots.txt only accepts * as a wildcard and the main issue is the crawling of search, I would propose adding /*/search to the disallow list.
Comment | File | Size | Author |
---|---|---|---|
#3 | drupal-robotstxt-2195283-3.patch | 360 bytes | stefan.r |
Comments
Comment #1
stefan.r CreditAttribution: stefan.r commentedsee #180379: Fix path matching in robots.txt
To be complete, we can do
Comment #2
bart.hanssens CreditAttribution: bart.hanssens commentedadded tag
Comment #3
stefan.r CreditAttribution: stefan.r commentedComment #4
stefan.r CreditAttribution: stefan.r commentedAdded to Openfed.
Comment #7
bart.hanssens CreditAttribution: bart.hanssens commented