By sopia on
Does anyone know about internal wildcards in robot.txt? Basically, I want to put in something like:
User-agent: *
Disallow: /user/*/whatchamacallit
I don't want it to disallow everything after /user/ - just the /whatchamacallit for each user.
I can't find any good documentation on this. Anybody know?