I find that my site is being crawled by agents such as:
Mozilla/5.0 (compatible; Google Desktop)
bitlybot
Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
These are obviously crawlers and not browsers. However, the browscap data seems to treat these as Browsers.
Would it be possible to add a feature wherein the webmaster can override the browscap data and ask the module to treat specific "Browsers" as crawlers and specific "Crawlers" as browsers?
Regards,
KH
Comment | File | Size | Author |
---|---|---|---|
#15 | browscap_overrides.zip | 2.6 KB | liquidcms |
#14 | browscap_overrides.zip | 2.39 KB | seanB |
#14 | browscap-override_hook.patch | 834 bytes | seanB |
#12 | browscap_overrides.zip | 2.49 KB | liquidcms |
#11 | browscap-override_hook.patch | 758 bytes | liquidcms |
Comments
Comment #1
RobLoachWould be pretty neat to edit this stuff in the interface itself. Currently, all you can do is edit in the MySQL table. The next update, however, you'll loose your changes.
Comment #2
gregglesYeah, we would need an "overrides" table to store this admin-entered-information that gets merged whenever the browscap.ini file iupdate or is used first in PHP or something.
Comment #3
RobLoachAuto Admin might make providing administration around the Browscap data easy.
Comment #4
RobLoachAuto Admin unfortunately doesn't provide support for editing BLOB data. Maybe if we extend Auto Admin to support external handling of special fields? Hmmm.
Comment #5
_rune CreditAttribution: _rune commentedHi I am the author of Auto Admin. Adding support for Blobs is a _really_ quick fix. Let me now if you still need it!
Cheers
_rune
Comment #6
RobLoachGreat idea. I've created an issue here: #656016: Support for BLOBs
Comment #7
fthieme CreditAttribution: fthieme commentedAnother thought on this: There are two ways to mark them as bots.
1st: Mark them as crawlers locally. This needs some kind of override to the official data and some interface to manage those overrides.
2nd: Tell Gary about them and mark them as crawlers globally. IMHO the better solution, because the everyone, who use browscap will profit from it. So you need a button "tell" and some infrastructure to actually let Gary know about it.
Comment #8
gregglesIn my experience Gary is really quick to figure out stuff himself. Perhaps some documentation to explain that?
Comment #9
ianchan CreditAttribution: ianchan commentedsubscribe
Comment #10
Devin Carlson CreditAttribution: Devin Carlson commentedMarked #909532: iPad recognition as a duplicate.
Comment #11
liquidcms CreditAttribution: liquidcms commentedi took a different approach to editing bcap data directly.
included here:
- patch for browscap.module which adds a hook to allow modules to alter browscap data pulled from the browscap table
- module called browscap_overrides which adds an admin page where you can set preg match patterns to match against browscap table useragent entries to allow modifying anything in the useragent data entry.
example screenshot shows setting all iPad agents to set ismobiledevice = 0 regardless of what the browscap table has. http://screencast.com/t/OANvATPI
Comment #12
liquidcms CreditAttribution: liquidcms commentedoops.. i mistakenly packaged the module under Browscap; which doesn't exist. This version removes package (i.e. places it under Other) - although there likely should be a Browscap package.. :)
Comment #13
seanBI just tested the patch and the browscap_overrides module. I like the approach. It works perfectly.
Some small remarks:
Notice: Undefined index: browscap_overrides_override_1 in browscap_overrides_admin_settings_form() (line 86 of sites/all/modules/contrib/browscap_overrides/browscap_overrides.module).
Thanks for the fix!
Comment #14
seanBSince Windows Phone 8.0 user agents are not yet added to the browscap module I tried to add some settings through the browscap_overrides module. The following improvements could be very helpfull:
New patch and code are attached.
Comment #15
liquidcms CreditAttribution: liquidcms commentedSean, yes, using original useragent string makes much more sense. i added one other slight tweak to this - description text with an example on the admin page; since pretty unlikely anyone could guess from what's there as to how to use this.
would be nice to get this included with bcap module as a support module
Comment #16
Hydra CreditAttribution: Hydra commentedThis sounds great, would love to see that as a supported module in browscap, too.
Little problem when using the example code of
/.*iPad.*/
ismobiledevice=0
This is causing the iPhone to be detected as zero, too. WIthout the override, it works just as expected. Good part, it works for the ipad, but sadly also for the iphone, can someone confirm this or tell me what I did wrong.
appreciate
Comment #17
gregglesThere's now a github issue tracker for collaborating on the data. https://github.com/browscap/browscap/issues and a google group https://groups.google.com/forum/#!forum/browscap
I feel like this really diminishes the need for editing the values inside the module.
Comment #18
AohRveTPV CreditAttribution: AohRveTPV commentedAgree with #17. Also I would prefer if this were implemented, that it be done so in a separate .module. In my humble opinion, Browscap is a lightweight interface module that should be focused on downloading and providing an interface to the Browscap data. Curating and/or using that data is another matter.
Comment #19
AohRveTPV CreditAttribution: AohRveTPV commentedIs there a need to override specific data in another module, if you are able to get the data changed via the Browscap project? Maybe the hook has a use case I am not thinking of.
Comment #20
AohRveTPV CreditAttribution: AohRveTPV commentedComment #21
AohRveTPV CreditAttribution: AohRveTPV commentedNo response to #19, so closing.