Mailhandler comment date and time stored incorrectly in the latest incarnations of Drupal 6.

Our goal is to get the correct date and time to show up in from archive maillist into forum imported data.

Findings:

A comment on a topic is saved via comment_save, (not node_save).
Created is however never set, thus empty than then code stated

Luckily comment_save does allow to set these values. See below for the code in the current Drupal 6.x and 7.x dev version.


...

      db_update('comment')
        ->fields(array(
          'status' => $comment->status,
          'created' => $comment->created,
          'changed' => $comment->changed,

...

The mailhandler sets the timesamp value of the comment, which is irrelevant with latest Drupal 6. And I've read in comment.install that

* Split {comment}.timestamp into 'created' and 'changed', improve indexing on {comment}.

indicating that using timestamp is obsolete anyhow.

It should have set the $comment->changed value instead.

First what is not correct is the line

...

 $node->date = $node->changed = format_date($header->udate, 'custom', 'Y-m-d H:i:s O');

...

$node->date is indeed a formated date BUT $node->changed should be a node UNIX timestamp value

Also the Timezone is not taken into account, so we need to fix this also see:
maybe for a fix base thi on some code advised lower in this post http://drupal.org/node/187449

change is needed in the code like


-  $node->date = $node->changed = format_date($header->udate, 'custom', 'Y-m-d H:i:s O');

+  $node->changed = $header->udate;
+  // Note that date is NOT stored as UNIX timestamp          
+  $node->date = format_date($header->udate, 'custom', 'Y-m-d H:i:s O');

BUT There are more issues left in the node_save() part
Need to add something to mailhandler like


...

/**
 * Implementation of hook_node_presave(). To make sure can modify node->changed value
 */
function mailhandler_node_presave($node) {
  $node->changed = $node->created;
  //Obsolete not needed anymore: $node->timestamp = $node->created;
}

...

Will make a patch against code of mailinglist-6.x-1.x-dev if it all works and I've tested it wit my test sets.

Comments

cor3huis’s picture

Where Sergey Demidenko ( http://drupal.org/user/573574 ) already nicely wrote:

If you are importing a set of emails from an MBOX and some emails are added into drupal in the form of reply comments, these comments dates will be set to the date of import, not the date the email was sent.

In mailhandler.module mailhandler_comment_submit function. I set $node->created and $node->changed to a timestamp generated from $node->date (which is set to the date of the email).

Insert this code:

...
// Get $node->created from $node->date
$string = $node->date;
$string_explode = explode(' ', $string);
$string_date = $string_explode[0];
$string_time = $string_explode[1];
$string_date_explode = explode('-', $string_date);
$string_time_explode = explode(':', $string_time);
$timestamp = mktime($string_time_explode[0],$string_time_explode[1],$string_time_explode[2],$string_date_explode[1], $string_date_explode[2], $string_date_explode[0]);
$node->created = $timestamp;
$node->changed = $timestamp;
...

A very nice investigation already, however not taken into account the original line that is wrong where $node->created gets a date format in and not a unix timestamp.

Also the full date issue need to look into since sometimes there is no udate in the emailbody but no fallback scenario is created. Als othe timezone of original email is not take into account. This has to be crossreleated to the timezone of the server set datetime.

cor3huis’s picture

There is issue http://drupal.org/node/47170 describing the situation is some detail. However this issue is two separate issues in one ticket :(. I will resolve the second one and then this issue is a duplicate.

cor3huis’s picture

Issue tags: +comments, +date, +time, +import, +email, +archive, +mbox, +backdate, +created, +changed
cor3huis’s picture

StatusFileSize
new1.06 KB

To make sure Sergey's nice effort does not get lost , attached his patch here, (even though the patch is not fully correct)

cor3huis’s picture

Assigned: cor3huis » Unassigned
cor3huis’s picture

Title: Mailhandler comment date and time stored incorrectly » Mailhandler comment created and changed date and time stored incorrectly
sdemi’s picture

@cor3huis:

Using your method, did this also update the timestamps in the node_comment_statistics table? Since this is used to sort by "last post" date on the forums.

cor3huis’s picture

StatusFileSize
new24.47 KB

I cannot confirm(yet), only that I use "Official" Drupal API calls in the preliminary fix. It would be great if you could test that.

BTW there is a handy module called Krumo http://drupal.org/project/krumo where you can look at the LIVE values

cor3huis’s picture

FYI currently walking through all the Mailhandler issue from bottom up to see if I can fix "low hanging fruit" issue types so the issue list is less cluttered and we can focus on having a even better mailhandler module by fixing the real open defects.

cor3huis’s picture

Priority: Normal » Major
cor3huis’s picture

If you already have incorrect node_comment_statistics running this to fix the node comment_http://drupal.org/node/137458#comment-3832938

But testing is first best done on a clean database.

As before we will use only official drupal API calls, thus if we adhere to the API all should be OK. if not OK node_comment_statistics could also have a defect ;)

cor3huis’s picture

Remark

Going through the full mailhandler.module code I came across node_object_prepare. and there if not isset node created the current time is set. So if we would not set the created later on the current server time will be set. sometine to keep in mind

The Drupal API reference

...
<?php
function node_object_prepare(&$node) {
  // Set up default values, if required.
  if (!isset($node->created)) {
    $node->created = time();
  }
  if (!isset($node->date)) {
    $node->date = format_date($node->created, 'custom', 'Y-m-d H:i:s O');
  }
...
cor3huis’s picture

Implementing comment_presave and comment_nodeapi do not seem to work :( , ongoing investigation

cor3huis’s picture

Investigating and debug showed, D6.19 and 6.20 still do not have the comment_presave hook implemented :(

The current Drupal 6.x-dev I've cloned from GIT does!

   module_invoke_all('comment_presave', $comment);
   module_invoke_all('entity_presave', $comment, 'comment');

However I found a way for node_save, and for datetimestamps it does work already. Before a patch can be released the whole issues must be solved, inclusive the TimeZone Timestamp issues. Still working to get all fixed before NewYear 2011

I've have to setup a local server for easier testing and debugging first...

cor3huis’s picture

Have a local working version and also a codechange to solve the Mailhandler to listhandler messages not sorted an threaded correctly if the message ID was not in the order of mail arrival date. Will make patches and submit here for review. Need to cleanup code to Drupal standards and remove some debugging code first.

cor3huis’s picture

Status: Needs work » Patch (to be ported)
StatusFileSize
new14.23 KB

Patch for 1.x included, plz. test and integrate if OK

cor3huis’s picture

Status: Patch (to be ported) » Needs review

Changed status, It would be great if it would be integrated into the main dev version. and make a new patch if needed for the 6.x-1.x-dev version of 14-march-2011 with all improvements of ilo

Status: Needs review » Needs work

The last submitted patch, mailhandler.module.patch, failed testing.

cor3huis’s picture

Thanks for test, will make a new patch but confused against which version of 6.x-v1.x.
It would be great if anyone could guide me there.

Daniel_KM’s picture

Hi,

Yes, me too: I don't understand which release the patch is used with.

Has it been ported to 6.x-2.dev?

Sincerely,

Daniel Berthereau
Knowledge manager

ilo’s picture

Can anyone of you comfirm that this only happens with comments? cor3huis, I'll do the patch if you want.

Daniel_KM’s picture

Hi,

Thanks for publishing dev release.

I test module 6.x-1.x (16th May release) and it works fine (with Drupal 6.20), but I need to correct these lines to avoid the node is marked as edited:

line 732, change from :

  // We want the comment to have the email time, not the current time
  $node->timestamp = $node->created;

to

  // We want the comment to have the email time, not the current time.
  $node->timestamp = $node->changed = $node->created;

and line 776, after the node is saved, add :

        $args = array();
        $sql = 'UPDATE {node} n SET n.changed = %d WHERE n.nid = %d';
        $args[] = $node->created;
        $args[] = $node->nid;
        $result = db_query($sql, $args);

        $args = array();
        $sql = 'UPDATE {node_revisions} n SET n.timestamp = %d WHERE n.nid = %d';
        $args[] = $node->created;
        $args[] = $node->nid;
        $result = db_query($sql, $args);

I would prefer use $node->timestamp = $node->changed = $node->created; because it's more Drupal compliant, but it doesn't work.

Sincerely,

Daniel Berthereau
Knowledge manager

ilo’s picture

Daniel_KM, thanks for the report, but I'm guessing that you are saying something totally different here. The issue is about comments and you are talking about nodes.

What is your issue? mind you if I ask to open another issue if your topic is not related to comments?

thanks in advance

Daniel_KM’s picture

Yes, you are right, the issue appears only with nodes, not with comments. I have no problem with comments times.

Sincerely,

Daniel Berthereau
Knowledge manager

ilo’s picture

Then Daniel, open another issue to discuss it, so we can use this one for issues with time in comments.

danepowell’s picture

Component: Code » Mailhandler
Status: Needs work » Closed (won't fix)

Sorry, 6.x-1.x is no longer a supported release. Please try upgrading to 6.x-2.x and reopen if still an issue.