user agents not accepted [#46328]

Crawlers has to be given access by ip and not by the user agent.

These were grabbed from the bad behavior log. But they don´t apply. Only by using the ips access was permitted.

"Bitacle bot/1.1",
"ping.blo.gs/2.0",
"Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.dlc@looksmart.net; h**p://www.WISEnutbot.com)",
"Blogbot 0.9.0 (h**p://blogbot.dk/info/robot.php)",

Comments

Comment #1

dpangier commented 29 January 2006 at 11:27

Assigned:

Unassigned

» dpangier

Reported to the upstream author of Bad Behaviour.

Comment #2

dpangier commented 30 January 2006 at 09:22

From Michael Hampton:

The ones for Blogbot and Bitacle bot should be reported to the software authors so they can fix their software. blo.gs has already been contacted and advised to fix their software.

And the "ZyBorg Dead Link Checker" is blocked intentionally by Bad Behaviour as it is one of a set of apparently malicious bots run by this company (LookSmart). Check to see if it requested /robots.txt -- last time I looked, some of its bots did not at all, and some did request it, but ignored its contents entirely.

The user agent whitelisting seems to be broken, and I'm not exactly sure why. I had thought I fixed it, but perhaps I hadn't. I'll check into it.