I find that my site is being crawled by agents such as:

Mozilla/5.0 (compatible; Google Desktop)
bitlybot
Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)

These are obviously crawlers and not browsers. However, the browscap data seems to treat these as Browsers.

Would it be possible to add a feature wherein the webmaster can override the browscap data and ask the module to treat specific "Browsers" as crawlers and specific "Crawlers" as browsers?

Regards,
KH

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

RobLoach’s picture

Title: Custom Classification for User Agents » Administration to edit the Browscap data
Version: 6.x-1.0 » 6.x-1.x-dev
Component: Miscellaneous » User interface
Status: Active » Needs work

Would be pretty neat to edit this stuff in the interface itself. Currently, all you can do is edit in the MySQL table. The next update, however, you'll loose your changes.

greggles’s picture

Status: Needs work » Active

Yeah, we would need an "overrides" table to store this admin-entered-information that gets merged whenever the browscap.ini file iupdate or is used first in PHP or something.

RobLoach’s picture

Auto Admin might make providing administration around the Browscap data easy.

RobLoach’s picture

Status: Active » Needs review
FileSize
1.26 KB

Auto Admin unfortunately doesn't provide support for editing BLOB data. Maybe if we extend Auto Admin to support external handling of special fields? Hmmm.

_rune’s picture

Auto Admin unfortunately doesn't provide support for editing BLOB data

Hi I am the author of Auto Admin. Adding support for Blobs is a _really_ quick fix. Let me now if you still need it!

Cheers
_rune

RobLoach’s picture

Great idea. I've created an issue here: #656016: Support for BLOBs

fthieme’s picture

Another thought on this: There are two ways to mark them as bots.

1st: Mark them as crawlers locally. This needs some kind of override to the official data and some interface to manage those overrides.

2nd: Tell Gary about them and mark them as crawlers globally. IMHO the better solution, because the everyone, who use browscap will profit from it. So you need a button "tell" and some infrastructure to actually let Gary know about it.

greggles’s picture

In my experience Gary is really quick to figure out stuff himself. Perhaps some documentation to explain that?

ianchan’s picture

subscribe

Devin Carlson’s picture

Version: 6.x-1.x-dev » 7.x-1.x-dev

Marked #909532: iPad recognition as a duplicate.

liquidcms’s picture

i took a different approach to editing bcap data directly.

included here:
- patch for browscap.module which adds a hook to allow modules to alter browscap data pulled from the browscap table
- module called browscap_overrides which adds an admin page where you can set preg match patterns to match against browscap table useragent entries to allow modifying anything in the useragent data entry.

example screenshot shows setting all iPad agents to set ismobiledevice = 0 regardless of what the browscap table has. http://screencast.com/t/OANvATPI

liquidcms’s picture

FileSize
2.49 KB

oops.. i mistakenly packaged the module under Browscap; which doesn't exist. This version removes package (i.e. places it under Other) - although there likely should be a Browscap package.. :)

seanB’s picture

I just tested the patch and the browscap_overrides module. I like the approach. It works perfectly.
Some small remarks:

  • The module caches the browscap_get_browser() response (this confuses me since the clear cache of the devel module didn't always worked)
  • The patterns are case sensitive in overriding settings. Make sure all the properties are lowercase.
  • I get an error after visiting the module page for the first time when there is no data saved yet:
    Notice: Undefined index: browscap_overrides_override_1 in browscap_overrides_admin_settings_form() (line 86 of sites/all/modules/contrib/browscap_overrides/browscap_overrides.module).

Thanks for the fix!

seanB’s picture

Since Windows Phone 8.0 user agents are not yet added to the browscap module I tried to add some settings through the browscap_overrides module. The following improvements could be very helpfull:

  • Add the original user agent in the call to the hooks (see attached patch)
  • user the original user agent to match patterns (since user agents which aren't present in the DB return %
  • use the explode() function in stead of split() since it is deprecated in PHP 5.3.0

New patch and code are attached.

liquidcms’s picture

FileSize
2.6 KB

Sean, yes, using original useragent string makes much more sense. i added one other slight tweak to this - description text with an example on the admin page; since pretty unlikely anyone could guess from what's there as to how to use this.

would be nice to get this included with bcap module as a support module

Hydra’s picture

This sounds great, would love to see that as a supported module in browscap, too.
Little problem when using the example code of
/.*iPad.*/
ismobiledevice=0

This is causing the iPhone to be detected as zero, too. WIthout the override, it works just as expected. Good part, it works for the ipad, but sadly also for the iphone, can someone confirm this or tell me what I did wrong.

appreciate

greggles’s picture

There's now a github issue tracker for collaborating on the data. https://github.com/browscap/browscap/issues and a google group https://groups.google.com/forum/#!forum/browscap

I feel like this really diminishes the need for editing the values inside the module.

AohRveTPV’s picture

Issue summary: View changes
Status: Needs review » Closed (won't fix)

Agree with #17. Also I would prefer if this were implemented, that it be done so in a separate .module. In my humble opinion, Browscap is a lightweight interface module that should be focused on downloading and providing an interface to the Browscap data. Curating and/or using that data is another matter.

AohRveTPV’s picture

Status: Closed (won't fix) » Active

Is there a need to override specific data in another module, if you are able to get the data changed via the Browscap project? Maybe the hook has a use case I am not thinking of.

AohRveTPV’s picture

Version: 7.x-1.x-dev » 7.x-2.x-dev
Status: Active » Postponed (maintainer needs more info)
AohRveTPV’s picture

Status: Postponed (maintainer needs more info) » Closed (won't fix)

No response to #19, so closing.