I have an intermittent problem and I cannot isolate what causes it. So I'm hoping others have some insight.
I have a site using mailhandler with OG, Messaging, Notifications, etc., to run mailinglists. One of the lists is moderated. An email comes in and when mailhandler import runs it is supposed to create a node that is unpubllshed and a send email action is triggered to notify the moderator of the new message. The email to the moderator, like all outgoing emails form the site, is formatted by HTMLMail.

Most emails are retrieved from mailboxes, nodes are created, and the moderator is notified. But some messages are not handled properly. Instead of creating a node, the action is triggered, sending an email that is not formatted by HTMLMail to at least one moderator.

If I use webmail to go to the mailbox I can mark the message as unseen and almost every time, on the next cron cycle the message is imported properly. Occassionally it takes 2 or 3 tries before a node is created by the email.

I have found a couple of common factors, but they are not universally applicable. Initially it appeared that most of the errors occurred when emails were sent from Macs using Apple Mail. Initially I thought it might have to do with the default character encoding of these messages. But that does not appear to be the case. Some messages sent from PCs also fail to create nodes.

Another factor appears to be that some portion of these problem emails are sent from AOL accounts. But I cannot figure out why that would be so.

The fact that I can mark messages as unseen and they then are imported properly throws all of this into question because the message is unchanged and imports properly. I suppose it could be something that happens on our mail server during the import, but I have had instances where multiple messages are retrieved and all but one create nodes.

Any suggestions greatly appreciated.

Comments

danepowell’s picture

danepowell commented

Very peculiar. When you mark a failing message as unread and re-import, are you sure that it's the *only* unread message in the mailbox? And does it really seem random as to whether or not it gets imported each time you try?

Additionally, are there any errors in the dblog, in your PHP errors log, or in the server mail log?

Finally, one thing you could try would be to go to your feeds importer ($base_url/admin/build/feeds/edit/mailhandler_nodes/settings/MailhandlerFetcher if you are using the default mailhandler_nodes importer) and change the filter from 'nodes' to 'all'.

Edit: What I'm trying to determine is whether the problem is really due to the failed message, or to something else such as other messages that fail at the same time, or intermittent errors with the mailbox connection, etc...

rsbecker’s picture

rsbecker commented

Let me try to answer your questions.

There are no error messages in the dblog or in the mail server log (in this case Postfix). I have not checked the PHP log, but I will the next time it happens.

I have changed the fetcher to retrieve all. I cloned the default Mailhandler feed importer, and I thought I had done that when I set it up. But in my setup comments go to a different mailbox, so it is unlikely that this setting is the problem. I'll watch it.

I originally set the mailbox to delete messages after importing. When I discovered this problem I set it to leave the messages in the mailbox so we could recover them. I then figured out about marking them unseen.

The problem isn't entirely random. The reason I suspected macs was that I had a couple of people who could not create nodes. All of there messages failed. They used Macs and tended to have character encryption defaulting to something other than UTF-8. But most Mac users' messages make nodes, and some of them aren't UTF-8 either.

I'll let you know if the logs tell me anything more.

danepowell’s picture

danepowell commented

Let me know if you are able to discern any other sort of pattern. The fact that the same message will sometimes get skipped and sometimes get imported led me to suspect that there are other unread messages that are getting imported at the same time, and gumming up the whole process. But if that's not the case, I don't really know.

It doesn't sound like an intermittent connection problem- if Mailhandler fails to connect to a mailbox, it should throw a watchdog error (or an exception). If you are importing automatically on cron, you might try manually triggering the import to see if that makes a difference- you might see errors that might not show up in the logs.

rsbecker’s picture

rsbecker commented

Today, after I changed the fetcher to All that importer stopped working altogether. I got the following error.

[Wed Jan 04 19:01:02 2012] [error] [client 72.249.82.226] PHP Fatal error: Cannot instantiate abstract class MailhandlerFilters in /var/www/html/drupal/sites/all/modules/mailhandler/mailhandler.module on line 168

If I tried to import manually I got a 500 HTTP error. When I returned the setting to nodes only it worked again. I have no clue what this means.

There were no error messages in the db log or the mail server log. I see the message in the mail server log, and it appears to be delivered normally.

As I said at the start, I'm not sure whether the problem is the messages themselves, the mail server or Mailhandler. I hoped someone else might have seen this behavior and figured out what was happening. I'll keep watching and see if I can identify any patterns.

danepowell’s picture

danepowell commented

That error was fixed almost a month ago... #1369672: Error when using 'All' fetcher filter

Please update to the latest development version or 6.x-2.4 and try again.

danepowell’s picture

danepowell commented
Status: Active » Postponed (maintainer needs more info)
danepowell’s picture

danepowell commented
Status: Postponed (maintainer needs more info) » Closed (cannot reproduce)

No response, going to assume that this was solved by upgrading to the latest release.

rsbecker’s picture

rsbecker commented

I'm afraid it is not fixed, but I still have no clear idea what the problem is. It appears likely that the issue is somehow related to Apple mail because most, if not all of the messages that do not form nodes come from Mac users. But not all emails from Macs have a problem. There seem to be a few users whose emails get stuck, but some emails from those users go through normally.

Don't spe,nd time on this. If I can isolate the problem I'll let you know.

danepowell’s picture

danepowell commented
Status: Closed (cannot reproduce) » Active

No problem, I'm still interested in helping if I can. Just to confirm, you did upgrade to the latest release and it didn't help?

Can you send me a raw copy of a problematic email? Either post it here in a code block or as an attachment, or use my contact form.

danepowell’s picture

danepowell commented

Thanks, I received a copy of your message, but did not see anything peculiar in it.

Just to confirm, you *have* updated to the latest 6.x-2.x-dev release? There have been many bug fixes lately. But to be honest, this sounds more like a connection problem than a problem with Mailhandler.

I would highly recommend making sure that PHP and IMAP are configured to log errors, and that there are no errors in the logs.

danepowell’s picture

danepowell commented
Status: Active » Closed (cannot reproduce)

Make sure you are using the 'all' filter. That's been cropping up a lot lately.

rsbecker’s picture

rsbecker commented
Status: Closed (cannot reproduce) » Active

The fetcher is set to "all." This still happens occasionally. But for some reason I cannot find it is less often now.

I have installed elysia cron and assigned longer intervals to many tasks. So it is less likely that mailhandler runs up against a cron timeout. That's the best explanation I can come up with.

danepowell’s picture

danepowell commented

I've heard reports, especially from people using POP, that the cron job can lock occasionally when trying to connect to the mailbox. This seems to be due to network problems, or a problem with the PHP IMAP library. Usually there's an error in the mail log or apache log about the connection timing out, or about not being able to open the stream.

danepowell’s picture

danepowell commented

This issue might help you once it's finished: #2076309: Better reporting of IMAP errors

danepowell’s picture

danepowell commented
Status: Active » Postponed (maintainer needs more info)

I'm guessing this is related to #1911598: Cron job locks when PHP IMAP can't open stream. Now that #2076309: Better reporting of IMAP errors is finished, if you wait 12 hours for a dev release to roll and try it out, you should start getting errors in your dblog if anything goes wrong with the imap connection. This should greatly help in diagnosing the issue.

On the other hand, you said this isn't much of a problem any more, so feel free to mark is as 'fixed' or 'duplicate'.

rsbecker’s picture

rsbecker commented
Status: Postponed (maintainer needs more info) » Active

Dane

I'm swamped with work until the end of September. But I will test the latest version when that's done.

Failure to retrieve messages is still an intermittent issue on one of my sites.

danepowell’s picture

danepowell commented
Status: Active » Closed (won't fix)