Hi!

Just so loving your module, thank you so much!

I've changed the time period a little to reflect shorter time deletion of activities (have a lot of users here :)

 $cron_delete_options = array(
    0 => t('Never, my queries are limited manually'),
    600 => t('Older than 10 minutes'),
    3600 => t('Older than an hour'),
    86400 => t('Older than a day'),
    604800 => t('Older than a week'),
    2678400 => t('Older than a month'),
    5270400 => t('Older than two months'),
    7948800 => t('Older than three months'),
  );

and running cron every 15 minutes

Was wondering, is it perhaps possible to say delete everything in a time period, but keep for example last 10, 20 or 30 activities per user for the user profile (personal heartbeat) ? I think the project/activity module has something similar.

Is this perhaps possible please?

Comments

Stalski’s picture

Status: Active » Needs work

I'll look at it. I won't be easy but you certainly have a point there.
- Resetting status to get my attention -

liliplanet’s picture

That is wonderful, thank you Stalski. As mentioned a similar concept exists at http://drupal.org/project/activity :)

Stalski’s picture

First part with the new selections , pushed to git.

liliplanet’s picture

fabulous! looking so forward Staski, thank you!

Stalski’s picture

Really sweating on this. One day to create a query is my current record now. Specialists are helping me.
I checked activity and I don't see a working and scalable solution there? Can you point me the code where you saw this?

Stalski’s picture

Issue tags: +MySQL, +advanced
StatusFileSize
new0 bytes

@liliplanet The patch included works on the dev. Can you test this for me? Don't forget to backup the database.
I worked on this query a whole day and I think I can go bad in the way that too recent messages would be deleted, however the query should make sure that 10 activity records for each user are kept in database.

- EDIT -
If someone comes by, please review this code for me.

  $cron_delete_time = variable_get('heartbeat_activity_log_cron_delete', 2678400);
  $keep_latest_number = variable_get('heartbeat_activity_records_per_user', 10);

  // Delete activity older than the expiration date, while
  // keeping the latest X for each user.
  if ($cron_delete_time) {

    $expire = $_SERVER['REQUEST_TIME'] - $cron_delete_time;

    // Activity Ids that can not be removed (latest activity per user)
    $keep_uaids = array();

    // Calculate the latest activity for each user.
    $result = db_query("SELECT
        t1.uid,
        t1.uaid as 'uaid',
        COUNT(*) as 'rows_per_user',
        t1.timestamp as 'real_date',
        MIN(t2.timestamp) as 'oldest_date',
        count(t2.uid) AS 'count'
      FROM {heartbeat_activity} AS t1
      INNER JOIN {heartbeat_activity} AS t2 ON t1.uid = t2.uid AND t2.timestamp >= t1.timestamp
      WHERE (t1.timestamp, t1.uaid) < (t2.timestamp, t2.uaid)
      GROUP BY t1.uid, t1.uaid HAVING COUNT(t2.uid) <= %d
      ORDER BY t1.uid, t1.uaid, t1.timestamp DESC", $keep_latest_number);
    while ($row = db_fetch_object($result)) {
      $keep_uaids[$row->uaid] = $row->uaid;
    }

    $arguments = array_merge(array($expire), $keep_uaids);
    $delete_result = db_query("SELECT uaid
      FROM {heartbeat_activity}
      WHERE
        timestamp < %d
      AND
        uaid NOT IN (" . db_placeholders($keep_uaids) . ") ", $arguments);
    while ($row = db_fetch_object($delete_result)) {
      _heartbeat_activity_delete($row->uaid);
    }

  }

TODO Get language in the query so it's 10 per language. So in multilingual sites, you can set the configuration I added to double the time or something.

Stalski’s picture

Status: Needs work » Needs review
Stalski’s picture

Status: Needs review » Fixed

It does not break anything, so I pushed it go git to get more feedback (or some feedback or none is even better).

liliplanet’s picture

StatusFileSize
new15.61 KB

Stalski, I will test today but in the meantime you can see the settings at admin/settings/activity/settings

and have attached a screenshot and believe the code is in activity.admin.inc

/**
 * Form builder to dispaly settings for activity module
 */
function activity_settings_form(&$form_state = NULL) {
  $form['activity_expiration'] = array(
    '#type' => 'fieldset',
    '#title' => t('Activity Expiration Settings'),
    '#element_validate' => array('activity_expire_validate'),
  );
  
  $form['activity_expiration']['activity_expire'] = array(
    '#type' => 'select',
    '#title' => t('Activity log purge'),
    '#description' => t("Allows you to set a time limit for storing activity records. Select 0 to keep all activity records."),
    '#options' => drupal_map_assoc(array(0, 3600, 7200, 14400, 21600, 43200, 86400, 604800, 1209600, 2419200, 7257600, 15724800, 31536000), 'format_interval'),
    '#default_value' => variable_get('activity_expire', 0),
  );
  
  $form['activity_expiration']['activity_min_count'] = array(
    '#type' => 'select',
    '#title' => t('Minimum Activities'),
    '#description' => t('This is the minimum number activities that the user must have created before deleting any old activities.'),
    '#options' => drupal_map_assoc(range(0, 200, 5)),
    '#default_value' => variable_get('activity_min_count', 0),
  );
  

not sure if the following is part of it ..

/**
 * Batch deletion step.
 *
 * @param $aid
 *  The actions.aid for this template.
 * @param $batch_context
 *  The context array for this batch operation.
 */
function activity_batch_delete($aid, &$batch_context) {
  if (!isset($batch_context['sandbox']['last_activity_id'])) {
    $batch_context['sandbox']['last_activity_id'] = 0;
    $batch_context['sandbox']['progress'] = 0;
    $batch_context['sandbox']['max'] = db_result(db_query("SELECT COUNT(aid) FROM {activity} WHERE actions_id = '%s'", $aid));
  }
  $limit = 200;
  $activity_to_be_deleted = db_query_range("SELECT aid FROM {activity} WHERE aid > %d AND actions_id = '%s' ORDER BY aid ASC", $batch_context['sandbox']['last_activity_id'], $aid, 0, $limit);
  $activity_ids = array();
  while ($row = db_fetch_object($activity_to_be_deleted)) {
    $activity_ids[] = $row->aid;
    $batch_context['sandbox']['last_activity_id'] = $row->aid;
    $batch_context['sandbox']['progress']++;
  }

  activity_delete($activity_ids);
  // Check if not finished.
  if ($batch_context['sandbox']['progress'] < $batch_context['sandbox']['max']) {
    $batch_context['finished'] = $batch_context['sandbox']['progress'] / $batch_context['sandbox']['max'];
  }
  else {
    // If finished, delete the sandbox.
    unset($batch_context['sandbox']);
  }
  
}

/**
 * Batch regeneration step.
 *
 * @param $aid
 *  The actions.aid for this template.
 * @param $hook
 *  The name of the hook being recorded.
 * @param $op
 *  The op for the hook being recorded.
 * @param $batch_context
 *  An array representing the context for the batch operation.
 */
function activity_batch_regenerate_step($aid, $batch, $hook, $op, &$batch_context) {
  $module = activity_module_name($hook);
  $info  = activity_get_module_info($module);
  $load_callback = $info->context_load_callback;
  $limit = 50;

  // Set up the sandbox for the regeneration.
  if (!isset($batch_context['sandbox']['list'])) {
    $batch_context['sandbox']['list'] = $batch;
    $batch_context['sandbox']['max'] = count($batch_context['sandbox']['list']);
    $batch_context['sandbox']['progress'] = 0;

    $parameters = db_result(db_query("SELECT a.parameters FROM {actions} a WHERE aid = '%s'", $aid));
    $batch_context['sandbox']['parameters'] = unserialize($parameters);
  }

  $count = 0;
  foreach ($batch_context['sandbox']['list'] as $key => $event) {
    $context = array();
    if (++$count > $limit) {
      break;
    }
    else {
      $context = $load_callback($hook, $op, $event['id']);
      if (!empty($context)) {
        $context += array(
          'created' => $event['created'],
          'actor' => $event['actor'],
        );
        activity_record(NULL, $context + $batch_context['sandbox']['parameters']);
      }
      $batch_context['sandbox']['progress']++;
    }
    unset($batch_context['sandbox']['list'][$key]);
  }
  // Check if not finished.
  if ($batch_context['sandbox']['progress'] < $batch_context['sandbox']['max']) {
    $batch_context['finished'] = $batch_context['sandbox']['progress'] / $batch_context['sandbox']['max'];
  }
  else {
    // If finished, delete the sandbox.
    unset($batch_context['sandbox']);
  }
}

Hope this helps some and will revert asap with results :) Thank you so much Stalski!

liliplanet’s picture

also found this is activity.module

/**
 * Implementation of hook_cron().
 */
function activity_cron() {
  // default is 2 weeks 0
  $expire = variable_get('activity_expire', 0);
  $min = variable_get('activity_min_count', 0);
  if (!empty($expire) && empty($min)) {
    db_query("DELETE m, at, aa, a FROM {activity} a
             LEFT JOIN {activity_access} aa ON aa.aid = a.aid
             INNER JOIN {activity_targets} at ON at.aid = a.aid
             INNER JOIN {activity_messages} m ON m.amid = at.amid
             WHERE a.created < %d", $_SERVER['REQUEST_TIME'] - $expire
    );
  }
  elseif (!empty($expire)) {
    // SELECT members with the min number of activies and they have an min(created) older then the expire
    $uid_sql = "SELECT uid, min(created) as min, count(aid) as count FROM {activity} GROUP BY uid HAVING min < %d AND count > %d";
    $uid_result = db_query($uid_sql, $_SERVER['REQUEST_TIME'] - $expire, $min);
    $uids = array();
    while ($uid_obj = db_fetch_object($uid_result)) {
      $uids[] = $uid_obj->uid;
    }
    
    if (!empty($uids)) {
      // DELETE where uid IN () ^^ AND a.created < expire
      $args = array_merge($uids, array($_SERVER['REQUEST_TIME'] - $expire));
      db_query("DELETE m, at, aa, a FROM {activity} a
             LEFT JOIN {activity_access} aa ON aa.aid = a.aid
             INNER JOIN {activity_targets} at ON at.aid = a.aid
             INNER JOIN {activity_messages} m ON m.amid = at.amid
             WHERE a.uid IN(" . db_placeholders($uids) . ")
             AND a.created < %d", $args
      );
    }
  }
}
Stalski’s picture

Yes, I found that code too. This does not what you asked for. It's a solution that is less scalable and does not exactly the same.
Without being to hush, this seems like a fix more than a scalable database solution.

So now, I am even more proud on the result :)

liliplanet’s picture

Wow Stalski, thank you so much, so far running perfectly! You are a Super-Star! Will revert if anything changes, but looking very good :)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

gcassie’s picture

I wonder if you might consider another approach to this query. If the heartbeat_activity table is very large and joined to itself in a situation like this were it can't use indexes effectively due to the comparison operators, the query can take many minutes to execute, if it finishes at all. For example, on a site with ~20K entries in the heartbeat_activity table this query is effectively blocking cron from running.

    $users = db_query("SELECT uid FROM {users} WHERE status = 1");
    $query = array();
    $args = array();
    while ($account = db_fetch_object($users)) {
      $query[] = "(SELECT uid, uaid FROM {heartbeat_activity} WHERE uid = %d ORDER BY uaid DESC LIMIT %d)";
      $args[] = $account->uid;
      $args[] = $keep_latest_number;
    }
    
    $result = db_query(implode("UNION", $query), $args);

It looks more complex but on that same set of data executes in only a couple seconds.

There's a nice writeup of why/how this works here.

Stalski’s picture

Ok, did not know that. This query I built together with one of the main figures in the mysql IRC room and I got the guarantee that it would be scalable. It could be you are correct and therefore I will take some advanced tests for this.
Is it a solution to add indexed to some fields you think?

About the sql, the timestamp still needs to be put into query as well. So the last query will stay the same.

Also, I prefer this to be taken up in another issue. See #1240436: Delete old activity on cron needs to be scalable.