Spam handling usability issues

Hi Gary,

Thanks, I appreciate the feedback. As you can see, this module is fairly young and still has plenty of room for improvement. I'm glad to get your suggestions -- keep them coming. :)

1. Low protection: I can add the anatomical keyphrase into the db manually, as in
INSERT INTO spam_tokens VALUES("Levitra","1","0","99","1097334991"); ... this is naively copied and modified from the example training file, but what I learned by adding rules this way is that one instance is not enough -- the next day, 28 more comments, all of them with the noted key phrase. It seems that without a bulk-training mode on a sample with repeated examples, it could take a long long time before my site sees any useful protection.

It does take time to train a Bayesian filter. This is always true, including with popular tools such as Spamassassin. Generally the filter needs to see a few hundred spam or more before it's able to start automatically catching spam. As for "key phrases", that's not really how this module works. Instead, it breaks content into "words", then finds the 15 most "interesting" words and sees if they appear more often in spam content or non-spam content. Based on that, it decides whether it should mark the new content as spam or non-spam.

I intend to introduce a Markonian tokenizer some time soon, at which time the module will begin to recognize phrases in addition to words. When time permits.

2. Slow defense: Related to (1) and maybe even the same thing, there's no facility for the use-case to escalate a key-phrase; when I get these anatomical enhancement adds flooding in today, I can't ban one and be protected from any more; I have to ban a lot of them and even then only see results statistically. When the spammers fling dozens of crap balls at you, statistical is not good enough.

In theory, if you train the filter with each new spam content, it should eventually "learn" enough spam "words" and start marking a high percentage of them correctly. (I would hope >95%). Proper and consistent training is key, however.

3. Awkward recovery: Ok, let's work through the scenario of 28 enhancement ads ... how do you (a) train the filtre on them and (b) how do you remove them. the answer appears to be: One at a time. This is a big show-stopper for any practical use of spam.module as it is today -- one at a time you must click the comment, scroll down, click Mark as Spam, go back to the comment and Delete, then go back to the list of comments and do the same for the other 27.

True, there is a fair amount of manual effort involved. I'll give this some thought. More comments on this below.

I didn't. What I did was mark the first two as spam, then fire up mysql at the command line and issue a command somewhat like delete from comments where comment like '%anatomic-item enlargement%';... it works, but it's not the sort of solution we can recommend to the average webmaster, plus you have to purge your cache to be completely rid of them.

This is a bad idea. You want to train the filter with _all_ spam that you see. If you only train it with a few of the spam that you see, it will take it longer to learn. Even if all the comments are identical, it's in your best interests to train with each and every one of them that the module gets wrong. (The default is "TOE" mode, meaning "Train On Error". Any time the spam filter incorrectly marks content, you need to train it what it should have done. That's how it learns.)

I repeat: always correct the module when it makes mistakes. If it doesn't recognize a spam comment as spam, you need to correct it each and every time or it will continue to make mistakes.

1. Mark as Spam: this should be one of the options on the admin/comments, and it should be a batch operation via checkboxes -- 28 clicks to mark them, or maybe, as with webmail programs, show one or two dozen at a time with a javascript-hooked button to Select All and Invert Selection, and then action buttons for the selected set to mark the batch as spam and/or delete the works. Unpublish is a first step, but I really don't want to be paying for hosting space for 365x28 viagra adverts.

I did not do this for one simple reason: it requires patches to the Drupal core. I have patched Drupal core many times over the years, but have found that I simply don't have time to keep up with the patches each time a new version is released. Of course, if there was interest from Dries et al in getting the spam filtering functionality into core itself, then many improvements could happen.

That said, I will consider the possibility of introducing some patches for the 4.5 tarball to allow mass-marking messages. In your case, it sounds like this is essential for the module to be usable. And it would not be difficult.

As for deleting spam content, I think this is a bad idea. The reason being that I only started working on this module a little over a month ago, and it will go through some major changes in the coming months. If you delete your spam, then you'll either not be able to upgrade when a new version comes out (in which I change the tokenizer logic to recognize phrases in addition to just "words"), or you'll have to "forget" everything you've taught it so far and start over from the beginning. I firmly believe that at this time deleting spam content is not in your best interests.

2. Two Stage Filtering: Spamassassin uses this approach and it's a very good method -- it is true that some spam is very clever and while obvious to human readers, totally defies pattern-matching, and for this we need the Bayesian analysis. But it is also true that some spam, like the two dozen per day I've collected this week, have obvious keywords and key patterns that could be legitimate, but which my blog can live without. Instant disqualification. Zap. Those that pass the array of perl regex filters, those then go on to the more slippery Bayesian test.

I like this idea quite a bit. It is a simple enough change, though I'm not currently sure on the user interface. In any case, I'll add this functionality at some point in the relatively near future. (I'll have limited spare time for the next couple of weeks to months) Ideally I'd like to make it so you scroll through phrases the module has seen and select the ones that immediately qualify content as being spam. However for now it will be simpler to have the admin manually enter such words/phrases.

I realize that both extensions are non-trivial, one of them may even require new hooks in the comment module -- I didn't say the fix would be easy, only that without them, I don't think Drupal can work out well for those who aren't comfortable with direct SQL access (plus, once I do my SQL delete, I suppose I must also flush my cache too, right?) -- we do need to face this scourge of blog-spam and offer whatever help we can to bring spam.module up to production environment requirements asap.

If you want to help improve this module, something you can do is to save all of your spam. And after you have a lot of it (hundreds to thousands), dump it all to a text file and mail the dumpfile to me. That would greatly help me to improve the tokenizer logic, and to better tune the Bayesian filter.

Comments?

I have some suggestions for things you might try. First, enable 'Advanced configuration' of the module. Review the options there, as it may help to do some tuning. The module is currently tuned based on some papers I read about spam filtering email. But spam comments are quite a bit different, and obviously the tuning will have to change. (That's why I'd like to get a dump of all your spam -- it will help me to get the default tuning right)

Some things you might want to tune: If you have a lot of shorter spam comments, drop the number of words examined from 15 down to 10 or even 5. That could result in a lot of false positives, but it should help the module to be more responsive quicker. You may also want to increase the 'assign unkown token probability' from 40 to something much higher -- this will cause the filter to assume words it hasn't seen before are spam, rather than non spam. Again, this could result in an increase in false positives, but it will also allow the filter to catch more spam. Perhaps try setting it to 60, or even 70?

Finally, you could lower the default threshold from 80, to maybe 60 or 50.

Of course, making any or all of these changes you will want to carefully watch for false positives. And anything the spam filter incorrectly marks as spam, be _sure_ to tell it really wasn't spam. If you tune too far, you'll find instead of always clicking 'mark as spam', instead you'll be always clicking 'mark as not spam'. The key is to test and find that happy medium...

That said, I will consider the possibility of introducing some patches for the 4.5 tarball to allow mass-marking messages. In your case, it sounds like this is essential for the module to be usable. And it would not be difficult.

Okay, patches are available in the new "optional" directory. Apply the patch against the Drupal 4.5.0 comment module and it will give you the ability to mass-mark comments as spam or not spam. (It also adds the ability to mass-delete, mass-publish, and mass-unpublish comments).

(Note: You will also need an updated spam.module, as I had to fix an existing function, and introduce a new one...)

In the future, please file unique feature requests as seperate issues.

the comment.module patch works wonderfully well on the CVS, but I just thought I'd let you know that I had to remove the spam.module in order to see the modules page

I'm unable to duplicate this. I've got a clean 4.5.0 tarball installation with the comment.module patch applied, and the spam.module installed, and everything works fine. You're saying if you move the spam.module out of the modules directory, then 'admin/modules' displays a list of all available modules. But if you move the spam.module into the directory, then you just get a blank page at 'admin/modules'?

I suspect that you've got something funky in your database. Perhaps do a sanity check on your 'system' table?

Anyway, I had a spare hour today and i'm 75% of the way done with implementing regex filtering in addition the Bayesian filtering. I'm hoping to have time to finish tonight before I have to call it a night.

Okay, grab the latest spam.module. You'll need to update your database as well, creating the 'spam_custom' table. Then go to "administer > spam > custom filters" and you can add all the items from your blacklist. Remember to add the beginning and ending /'s, and add an "i" at the end if the matching should be case insensitive. For example, from your blacklist:

     \b(annunci|canzoni|cartoni|donne|scarica|sesso|sonnerie)([\w\d_.-]+)+\.\w{2,3}\b

Should be entered as a custom filter as:

     /\b(annunci|canzoni|cartoni|donne|scarica|sesso|sonnerie)([\w\d_.-]+)+\.\w{2,3}\b/i

At some point I will probably enhance this functionality to validate regex strings. Other suggestions on the usability, etc, are of course very welcome.

Perhaps a default blacklist could be shipped with the module, though I'm not sure if this would be locale specific.

I'm marking this feature request fixed as both of your requests are now implemented. If you have further problems with "admin > modules" noit displaying, please open a seperate support request or bug report.