I have a site which has been overrun with spam, but also has a lot of legitimate content posted.

I'd like to run the spam auto-detection on existing forum posts and comments. I'm new to the spam module so please let me know if this is easily done. I imagine doing it via cron jobs, inspecting a little bit of content at a time and marking as spam (but not deleting).

If this has not been done already, I could write the cron hook, but please let me know what functions in the spam api to call.

Thanks.

Comments

Dave Cohen’s picture

Beginning to think I won't get any help with this.

Looking through the code, I see that just about anytime the spam module does anything, it calls drupal_goto(). For example, both spam_scan() and spam_content_update() will call drupal_goto() if they find any spam. This makes them useless for calling during a cron job.

So my suspicion is no one has managed to do what I want to do, which is to automatically scan existing content and mark probable spam as spam.

jeremy’s picture

Category: support » feature

There is code for this in the 5.x version of the module, but sadly it was never implemented in the 6.x version. Patches are welcome, marking as a feature request.

Dave Cohen’s picture

Thanks for the response. This isn't a good patch, but just the code I'm experimenting with right now. I find I could not use any functions like spam_mark_as_spam() or spam_content_is_spam() because those function lead to unsavory recursion and ultimately a drupal_goto, which as I said breaks the cron jobs.

So I put the following sort of hack in my own module (although something like it could be placed in spam.module). Seems to be working OK so far. The code has a lot of print output because I'm still debugging and tweaking.


function sunflower_custom_cron() {
  print(__FUNCTION__);
  print('<pre>');

  if (module_exists('spam')) {
    if (user_access('bypass filters')) {
      print ("Logged in as administrator, skipping spam check.\n\n");
    }
    else {
      // Changed the threshold because I found real content being marked as spam.  Still figuring out the right \
value.                                                                                                           
      $GLOBALS['conf']['spam_threshold'] = 97; // Be very tolerant.                                              

      $start_nid = variable_get('sunflower_custom_spam_scan_start', 0);
      $limit = variable_get('sunflower_custom_spam_scan_limit', 10);
      print("sunflower_custom_cron running with start_nid=$start_nid and limit=$limit.\n\n");
      flush();

      $result = db_query("SELECT nid FROM {node} WHERE nid > %d AND type='forum' ORDER BY nid LIMIT %d",
                         $start_nid, $limit);

      while ($data = db_fetch_object($result)) {
        $node = node_load($data->nid);
        $a = (array) $node;
        $score = spam_content_filter($a, 'node', array());
        if (spam_score_is_spam($score)) {
          // We know it's spam.  We want to both mark as spam and unpublish.                                     
          // Unfortunately every API in spam.module is useless during cron.                                      
          // That module provides no way to mark as spam or unpublish without                                    
          // also calling drupal_goto, which will break cron.                                                    
          $hostname = spam_invoke_module('node', 'hostname', $node->nid);
          db_query("INSERT INTO {spam_tracker} (content_type, content_id, score, hostname, timestamp) VALUES ('%\
s', %d, %d, '%s', %d)", 'node', $node->nid, $score, $hostname, time());
          db_query("UPDATE {node} SET status=0 where nid=%d", $node->nid);
          $is_spam = TRUE;
        }
        else {
          $is_spam = FALSE;
        }
        print(t("Node !link scored %score, is_spam is %is_spam", array(
                  '!link' => l($node->title, 'node/' . $node->nid),
                  '%score' => $score,
                  '%is_spam' => $is_spam ? "true" : "false",
                )));
        print("\n\n");
        flush();



        // Increment variable so next cron does not repeat.                                                      
        variable_set('sunflower_custom_spam_scan_start', $node->nid);
      }


      $start = variable_get('sunflower_custom_spam_scan_comment_start', 0);
      $limit = variable_get('sunflower_custom_spam_scan_limit', 10);

      $result = db_query("SELECT cid FROM {comments} WHERE cid > %d ORDER BY cid LIMIT %d",
                         $start, $limit);

      while ($data = db_fetch_object($result)) {
        $comment = _comment_load($data->cid);
        $a = (array) $comment;
        $score = spam_content_filter($a, 'comment', array());
        if (spam_score_is_spam($score)) {
          // We know it's spam.  We want to both mark as spam and unpublish.                                     
          // Unfortunately every API in spam.module is useless during cron.                                      
          // That module provides no way to mark as spam or unpublish without                                    
          // also calling drupal_goto, which will break cron.                                                    
          $hostname = spam_invoke_module('comment', 'hostname', $comment->cid);
          db_query("INSERT INTO {spam_tracker} (content_type, content_id, score, hostname, timestamp) VALUES ('%\
s', %d, %d, '%s', %d)", 'comment', $comment->cid, $score, $hostname, time());
          db_query("UPDATE {comments} SET status=0 where cid=%d", $comment->cid);
          $is_spam = TRUE;
        }
        else {
          $is_spam = FALSE;
        }
        print(t("comment !link scored %score, is_spam is %is_spam", array(
                  '!link' => l($comment->subject, 'node/' . $comment->nid, array('fragment' => 'comment-' . $com\
ment->cid)),
                  '%score' => $score,
                  '%is_spam' => $is_spam ? "true" : "false",
                )));
        print("\n");
        flush();

        // Increment variable so next cron does not repeat.                                                      
        variable_set('sunflower_custom_spam_scan_comment_start', $comment->cid);
      }
    }
  }
  print("</pre>");
}
chrislabeard’s picture

Any updates on this?

Got a site with a ton of spam on it need to clean it up.

hermann77’s picture

Issue tags: +spam module scan old existing content
StatusFileSize
new9.31 KB
new9.25 KB

Hi,
patch spam.module and spam_filter_url.module and you will get a button "scan" in admin/content/spam/list
It scans only "nodes", but I'm going to do it more generic, may be in the next days.

gnassar’s picture

#3 is correct. The "standalone" functions are kind of a mess and not really standalone at all, and haven't been since the D5 version.

I was actually working on refactoring and re-modularizing all this code as my One Last Project(tm) before a D7 release, so we didn't go into D7 with a mess, but that took on a life of its own. The code I have now is more a rewrite of Spam than a patch. Not sure whether to submit it as a 6.x-2.x-dev or just call it a fork and make a new module.

avpaderno’s picture

Issue summary: View changes
Status: Active » Closed (outdated)
Issue tags: -spam module scan old existing content

I am closing this issue, since Drupal 6 isn't supported anymore.