By Anonymous (not verified) on
I added Disallow: *?* to my robots.txt in addition to the standard stuff. This to make sure that nothing was being accidentally picked up (multilanguage site and clean url's and all).
When I test this it works. However I noticed that the ?page= is also being blocked.
So I wrote this piece of code: Allow: /*?page=*
I am not sure this would work though. I tried searching for this, but search engines don't pick this up.
Has anyone else tried this?
