Issues read from mailserver out of order prevent correct threading

cor3huis, I'll take this one and provide a patch later.

After some research, I've found that for IMAP it could be a great improvement, but pratically a blocker for pop accounts, because php's imap extension donwloads full message body when sorting POP accounts. On the other hand, as long as the module retrieves a message by its number, I think it will be easier to sort the array of message numbers instead.

I have also found that it might be quicker to call imap_headers to return all the headers in a single network call, with a great performance boost on larger mailboxes. The dark part of this is that headers might be limited to 25 chars subject according to a comment in php's site.

Cor3huis, do you have a large inbox to do some performance testing? I'll be upload a pair of scripts to do so.

I'm not getting any significant improvement using imap_headers(). Looks like imap_headerinfo() performs call and caches the headers, using that cached information in all subsequent imap_headerinfo() calls.

as_mailhandler.php: imap > Unread messages: 443 time 14.690289974213 secs
fetch_headers.php: imap > Unread messages: 443 time 15.039510011673 secs

The 'not so funny part' is that the same logic as in mailhandler does not return the same results using POP as protocol (against the same email account):

as_mailhandler.php: pop > Unread messages: 276 time 102.59804010391 secs
fetch_headers.php: pop > Unread messages: 0 time 130.51864504814 secs

Comparing sorted and unsorted:
as_mailhandler.php: imap > Unread messages: 443 time 14.128609113693 secs
as_mailhandler_imap_sorted.php: Imap > Unread messages: 443 time 15.430547952652 secs (once it reached: 13.430300951004 secs)

Using apache's benchmark tool (ab) results are more or less the same.

The mailbox information I've used:
stdClass Object
(
[flags] => 31
[messages] => 702 <-- total messages
[recent] => 0
[unseen] => 443 <-- new messages as processed by mailhandler.
[uidnext] => 704
[uidvalidity] => 2
)

So, after this, I come to the conclusion that sorting does not provide such a pain, regardless it performs a little bit slower (as a background process it might not be so important), but I also think that we should reconsider the whole retrieval stuff as long as pop and imap results are not the same.

Just curiosity, cor3huis, what server does not return the email list sorted by arrival date using default imap_open?, I'm asking because gmail does.

Cor3huis, considering your description, and in a setup where there are several mailboxes, do you think it makes sense to parse all headers of all mailboxes, sort the messages by their arrival date (mixing mails from all mailboxes) and then perform a retrieve operation from all these mailboxes, reading the emails according to this arrival date (and not only per mailbox)? This is the only way I can think of to get all emails from all mailboxes in the correct order.

Marking as critical as it may affect the way current retrieving process works.

Sorry, 6.x-1.x is no longer a supported release. Please try upgrading to 6.x-2.x and reopen if still an issue.