The /admin/spam page shows an overview of the spam filter's effectiveness. The number of learned non-spam postings is always incorrect. This seems to be because the variable used, learned_notspam, is never set and does not exist in my database.

Comments

jeremy’s picture

jeremy commented
Assigned: Unassigned » jeremy

The counter "learned_notspam" is set in the function spam_save_tokens(). (A grep won't find it because it uses a variable to determine if it's updating "learned_spam" or "learned_notspam".)

I reworded the statistics overview to give useful data. The original overview data was no longer useful once the filter got rebuilt (ie, upgrading to a new tokenizer). It now shows the percentage of false negatives, and the percentage of false positives, all since the last filter rebuild.

junyor’s picture

junyor commented

There's definitely a problem, though it might be an initialization issue. I haven't received any non-spam comments since installing the module. The stats show 60 spams and -60 non-spams (it's always the opposite of the number of spams). It looks like spam_save_tokens() only sets the spam_statistics values when there's a change. Thus, this value will be incorrect until you receive non-spam. I assume the same goes for other values.

jeremy’s picture

jeremy commented

Yes, there was a problem. I was doing some subtraction that I shouldn't have. But I fixed this last night.

Please download the latest module (1.9.2.19) and let me know if you still have a problem.

junyor’s picture

junyor commented

Better, but not quite there. This is what I have now:

This site has had a combined total of 92 spam comment and spam node postings. The last spam posting to this site was at 1:09 pm on Thursday, December 16, 2004.

of the 31 (0%) automatically detected spam postings were incorrectly marked as spam. 61 of the 61 (100%) automatically detected non-spam postings were incorrectly marked as non-spam. This is an overall accuracy of 33.7%.

There seems to be some words missing and the statistics don't make sense.

jeremy’s picture

jeremy commented

Whoops. I neglected to mark some variables as int's, so when they were unset they were showing up as "" instead of 0. Your text is close, but the second paragraph should read:

"0 of the 31 (0%) automatically detected spam postings were incorrectly marked as spam. 61 of the 61 (100%) automatically detected non-spam postings were incorrectly marked as non-spam. This is an overall accuracy of 33.7%."

That is 100 - ((0 + 61) / (31 + 61) * 100) = 33.7. ie, the module has scanned 92 postings, and only 31 of them were correctly marked. (This is due to a very high false-positive rate, something you need to look at. Is this from over-zealous custom filters? URL filters?)

In any case, I realize this morning it is confusing to talk about what was incorrect. I've reworded this so now you'd see:

"31 of the 31 (100%) automatically detected spam postings were correctly marked as spam. 0 of the 61 (0%) automatically detected non-spam postings were correctly marked as non-spam. This is an overall filter accuracy of 33.7%."

Fixed in CVS and 4.5.

junyor’s picture

junyor commented

I haven't had any false positives.

0 of the 61 (0%) automatically detected non-spam postings were correctly marked as non-spam.

The above is correct, but confusing. There were 61 undetected spam comments that I manually marked as spam.

jeremy’s picture

jeremy commented

The fact that I got confused is proof enough for me that it's confusing. ;)

Yes, this text needs to be rewritten. If you (or anyone else) has suggestions for how you'd like the statistics page to read that's not confusing, please post it -- that'd be very helpful. (I still want to include both numbers and percentages) ie, use the data in our previous examples, but arrange it however you like...

I also need to introduce some of the other available counters into the page.

Anonymous’s picture

(not verified) commented