The attached patch adds the following features to Drupal HEAD:

- Setting Access Permissions by referrer.

The attached patch fixes the following bugs:

- Port numbers on same domain cause Drupal to mark an external referrer

This patch changes the following modules, by function:

- user.module
-- user_admin_access_check Now allows checking based on referrer.
-- _user_admin_access_form Now has options for banning based on referrer.
-- user_admin_access Now lists referrer bans.
-- user_help Show referrer banning in documentation.
- bootstrap.inc
-- _drupal_bootstrap Checks to ensure user is not banned by referrer.
- statistics.module
-- statistics_top_referrers Has "quick-links" for banning referring domain. Fixes port issue.

I have tested this module and it works correctly on Drupal HEAD.

Robin

Comments

dries’s picture

What is the point of referrer banning? To stop people from linking your site? I'd think this is only useful for 0.001% of all Drupal sites.

killes@www.drop.org’s picture

I've asked Robin to implement this because a referrer spammer was spamming drupal.org with porn referrers. This was annoying because it made the "top referrers" page basically useless. I only got rid of the spammer because Kjartan grepped the apache logs for me and told me the IP.

bertboerland’s picture

dries, take a look at http://drupal.org/admin/logs/referrers. mine is even worse, nearly 15% of all referrers are pr0n sites. so i do think it is usefull to ban based on referrer, not to prohibits linkers but referrer spammers

robin monks’s picture

This is also useful for sites that try to pop other sites into IFrames (sometime invisable).

This code does need review. I want to make sure that this will work on all server/OS/php types.

Robin

dries’s picture

I'm still not convinced this feature is needed. I'm tempted to say "won't fix" but I'll let this patch float in the review queue until enough people showed interest in it.

eldarin’s picture

A useful feature for sites plagued by the spammers on the net - another sad example is http://www.xmlrpc.com/stats/referers .

deekayen’s picture

I'd like to see this feature, but I would make a change to the patch.

In statistics.module, + array('data' => t('Ban domain')) isn't exactly clear. We know what it does in the context of this issue, but I can see some of my users thinking that it will do a DNS lookup and ban any users trying to visit from another ISP. If it were my patch, I'd make it 'Ban visitors clicking from this webpage', but that might be wordy for some of you.

robin monks’s picture

If it were my patch, I'd make it 'Ban visitors clicking from this webpage', 
but that might be wordy for some of you.

Don't forget, this is going in a table header. Something like that on my display would take the table halfway to greenland.
"Ban Referring Domain" or "Ban Referrer" may be better choices, but I can't understand how someone could think a domain was an ISP??? I mean, a domain is a domain. Even non-tech-types know a domain is a .com.
I'd like others opinions on this.

I feel the common concensus thus far is that this patch would be a good addition to Drupals banning system. However, the code still needs to have some more testers. This is also a good time to mention any changes you'd like to see.

Robin

robin monks’s picture

StatusFileSize
new7.97 KB

New patch against HEAD, changes the title to "Ban referrer"

Robin

minimism’s picture

As a sufferer of referrer spam (my logs+cache+sessions tables added over 20MB to my database, causing me to go over-quota with my hosting service == cost; as well as the increased traffic making my site go 25% over bandwidth allowance == more cost) would the referrer block generate any output (i.e a 404 message) or would Drupal output a null page (such as cron.php)?

My 'over-bandwidth' problem is not going to be helped if each request gets a 404 message!

Hosting Geek’s picture

I know lighttpd has this feature... maybe you would want to use it instead of apache it seems to have a lot fo features apache is missing plus it uses less load and memory... maybe drupal.org should of swaped to it when it was having load problems... to late to sudgest it now... but if drupal.org ever start getting load problems again it might be the first thing you would want to do.

robin monks’s picture

The module stops the request before it hits the DB or graphics and just ouputs (in plain text).

The referrer ___ has been banned.

Or similar. No "building" take place, the request is effictively killed.

Robin

Uwe Hermann’s picture

StatusFileSize
new7.83 KB

Patch didn't apply anymore (tried to re-add '?>'). Here's an updated patch. From a quick glance the code looks good, but I didn't test thoroughly.

+1 for the feature, though.

robin monks’s picture

StatusFileSize
new7.37 KB

Rerolled the patch again...HEAD is contantly breaking...

Com'on Dries, all +'s :)

Robin

zach harkey’s picture

+1

sangamreddi’s picture

+1

jon@jony.net’s picture

+1 on this feature. but I'd like an option to redirect specific referes to specific path.

nickl’s picture

Status: Needs review » Closed (won't fix)

Patch Bingo:

Sounds viable and it did have lots of support but since Dries was tempted to change the status to 'won't fix' in July 31, 2005 and nothing more has happened here since September I'll do the same now.

This is to give you a chance to reopen this issue by marking it patch (code needs work) and I'll be happy to reroll the work done to HEAD and adding the specific path functionality requested if you can motivate with enough reason to do so.

Let yourself be heard...

Christefano-oldaccount’s picture

Version: x.y.z » 6.x-dev
Status: Closed (won't fix) » Needs work

Not sure what happened here. I'm reopening this to see if anyone wants this added to 6.x. I do.

mcurry’s picture

+1

example... http://www.roadcarvin.com - We're getting hammered by referrer spam - it appears to be a botnet. Can't (efficiently) ban by IP because the IP addresses are spread out all over the place. By the time you notice that an IP address is being used and ban it, the botnet is using another batch of machines.

Need a way to ban referrers by regular expression, not just static text...

catch’s picture

Version: 6.x-dev » 7.x-dev

I also get referrer spam, so +1, bumping.

podarok’s picture

subscribe!
what about user-agent banning? or redirecting

freephile’s picture

+1 This is definitely a feature that Drupal sites would benefit from. Although you could potentially handle this at the infrastructure level (webserver), I think it's most likely that a site administrator who is generating the referrer stats in an organization would like to have this functionality right at their fingertips. Banning on regex, or partial names with wildcards would be useful. Keep in mind that the 'referring site' does not actually have any links from their domain to yours. They simply send out a crawling bot that provides falsified 'referrer' headers.

robin monks’s picture

I'm working on a re-roll for 7.x.

Robin
I ♥ Bugz

robin monks’s picture

Status: Needs work » Needs review

Here it is, all ported to 7.x. I did testing of this locally on LigHTTPd and it worked like a charm.

Robin
I ♥ Bugz

robin monks’s picture

StatusFileSize
new8.58 KB
new8.58 KB

Um..please bear in mind this is my first core patch in some time, and little things like actually attaching the patch get a little fuzzy after awhile ;)

Robin

lilou’s picture

Status: Needs review » Needs work

Patch need to be re-rolled.

robin monks’s picture

Another re-roll coming up!

Robin
I ♥ Bugz

robin monks’s picture

The entire drupal_is_denied function in HEAD had been extremely simplified, so, this is going to take a little longer than I was expecting :)

Robin

catch’s picture

Not just drupal_is_denied() - access rules is gone too, including the table. #228594: UMN Usability: split access rules into an optional module

robin monks’s picture

Yeah, basically everything that was more user-centric was stripped. EG, not allowing some email addresses or usernames. Which is fine. But referrer banning couldn't really live well as a contrib, since it would need to happen early in bootstrap to be effective.

I've about 30% through reroll.

Robin

robin monks’s picture

Status: Needs work » Needs review
StatusFileSize
new9.03 KB

It's here!

I tested locally and it seemed to work fine. It mirrors to IP Blocking functionality.

--Robin

robin monks’s picture

New version.

Robin

robin monks’s picture

A new version ready for testing. Thanks to Morbus and cwgordon7 in IRC for suggestions.

New in this version:
- Both IP blocking and referrer blocking now use the same access perm, 'block traffic'.
- Error message when a request is blocked has been changed to use 'referrer' rather than 'referer'.
- Comment made above HTTP_REFERER to remind everyone that Apache spells this wrong, it's not a typo (at least on our part).
- Unnecessary #submit and #validate formAPI items removed for cleanliness.
- Added doxygen to drupal_blocked_referrer
- Added update function to system.install

Robin

psicomante’s picture

Wonderful patch, thanks Robin; i can't wait until it'll fixed :P

robin monks’s picture

Keeping up with head. Also fixed a bug where you were unable to delete referrers (thanks redndahead in IRC!). Also added back a feature in the original patch but I didn't port to 7.x; you can once again choose to ban referrers directly from your top referrers report in statistics module :)

Robin

stella’s picture

Coder only spotted a few minor problems with the patch:

system.install : @@ -568,6 +568,29 @@
Line 4: There should be no trailing spaces
Line 26: There should be no trailing spaces

system.module : @@ -503,22 +503,44 @@
Line 20: There should be no trailing spaces
Line 41: There should be no trailing spaces
robin monks’s picture

StatusFileSize
new14.33 KB

Thanks stella! This patch should make coder happy ;-)

Robin

BioALIEN’s picture

Robin, wonderful patch! Definite a life saver especially for users on shared hosting.

This should've made it to D5, lets please get this to D7. Now I can regain some meaningful stats back in my Drupal logs!

maartenvg’s picture

Status: Needs review » Needs work

Patch no longer applies.

maarten@VRIEZER:/var/www/home/d7$ patch -p0 < referrer_banning\[1\].patch
patching file includes/bootstrap.inc
Hunk #1 succeeded at 915 (offset 8 lines).
Hunk #2 succeeded at 930 (offset 8 lines).
Hunk #3 FAILED at 1054.
1 out of 3 hunks FAILED -- saving rejects to file includes/bootstrap.inc.rej
patching file modules/statistics/statistics.admin.inc
Hunk #2 FAILED at 123.
1 out of 2 hunks FAILED -- saving rejects to file modules/statistics/statistics.admin.inc.rej
patching file modules/system/system.admin.inc
patching file modules/system/system.install
Hunk #2 succeeded at 3071 (offset -1 lines).
patching file modules/system/system.module
Hunk #1 FAILED at 171.
Hunk #4 succeeded at 2062 (offset 6 lines).
1 out of 4 hunks FAILED -- saving rejects to file modules/system/system.module.rej
aspilicious’s picture

Version: 7.x-dev » 8.x-dev
Status: Needs work » Closed (won't fix)

Is this still as relevant as it was in 2005 o_O? Put status back if you feel this is still needed.