Steps to reproduce this bug are simple: Send a new message to yourself. (However stupid this might seem, it's entirely possible to happen either by mistake, or when a new user on the site rushes to explore features by sending "this is my first message" to himself.)

What happens:
- The new thread is listed with incorrect number of replies (1 where it should be 0)
- When viewing the thread, it's not marked as read, so the bold title in messages list persists, as well as number of "new" messages in navigation menu item. (Only marking it as read explicitly through operations on listing page works.)

Internally, the database table {pm_index} gets two entries for the same message/thread/user combination, one as "sender" (not new), another as "recipient" (is new). Now, the listing query gets two messages, while doing COUNT(pmi.thread_id), resulting in one more reply shown in the count. Also, the load query gets two rows, but _privatemsg_load() only fetches one (as it's supposed to load a single message), and since this is the "sender" row without "is new", the message is not considered as unread in privatemsg_view() and therefore not marked as read. So, the other row with "is new" persists, and forces the message to show as unread forever.

My first way of fixing this assumes, that we want to keep the database behavior as is (more on that assumption later). If that's the case, we should fix the queries. The listing query needs COUNT(DISTINCT pmi.mid) to really count messages, not just [possibly duplicate] rows, and the load query needs to group possibly duplicate rows, and do MAX(is_new) (additionally, MAX is needed on all other columns, due to GROUP BY now present).

I've already created a patch for that, and it works fine for me. Attaching that one - so if we decide this is the way to go, then it's there.

But now I feel like we shouldn't insert duplicate rows into {pm_index} in the first place. Going to examine how to fix that later today.

Comments

JirkaRybka’s picture

Status: Needs work » Needs review
StatusFileSize
new1.16 KB

OK, the second way of fixing this is rather simple: In _privatemsg_send() we gather the list of {pm_index} records into an array first, so that duplicates collapse into single records nicely. The author is added first, so whenever he is one of recipients too, he gets the record as unread message (that's what he surely expects, when sending to himself).

Note, that incorrect number of replies on listing also occured, when the recipient was specified twice - although privatemsg UI seems to avoid that, it's still possible to pass a recipients list with duplicates for example to privatemsg_new_thread() [I tested, it really triggered the wrong-number-of-replies problem on recipient's list]. So, this patch also makes _privatemsg_send() more robust regarding duplicates.

Two patches on this thread: Both fix the problem. The one attached to this comment avoids saving duplicates to {pm_index}, while the previous one deals with the consequences, if there were such duplicates already. We might want to commit one or the other, or both for a really fool-proof fix.

naheemsays’s picture

I like the first approach (as the second does not fix the problem for existing users), but I will leave it to others to decide. I think this bug was uncovered by #444686: Only update status if necessary

JirkaRybka’s picture

Version: » 6.x-1.x-dev

On second look, I would vote for committing BOTH these fixes, as one fix the consequences of having duplicates (which is good to have, for existing users, but not only), and the other fix the case where duplicates are created (which is not nice regardless the immediate outcome). Leaving to others, too.

Except that this is rolled against current 1.x-dev (clicked a wrong row first time) ;-)

naheemsays’s picture

If we do change it, there would need to be an update function to fix the past messages index.

Saying that, there must have been a reason for going with the approach we did. Just can't remember it.

JirkaRybka’s picture

I can't see how having two *identical* [after setting as already read] rows is supposed to make sense - I guess it's more of previously untested/unseen edge case. Just thinking out loud, I wasn't there when that code got written ;-)

naheemsays’s picture

its an edgecase that was covered before - the status on all messages being viewed got set to read. But there was an optimisation done recently where it was only set to read when it was previously unread. Obviously this caused problems in some cases where it was being incorrectly detected as read due to duplicate data.

As for reasoning - I was there and I played a large role in the current schema, but I still cannot remember many of the finer details.

I think a large part of it was to be able to tell if you sent the message to yourself or not.

JirkaRybka’s picture

Anyway, from my angle of view, I see no need to discuss this at lengths (feel free to disagree). Both the approaches are basically fine for me, so if there are concerns about the duplicates-removal change, we can just fix the queries per my initial post. I would be happy to see this fixed either way, and I have absolutely no intention to make this a lengthy/depressing thread. :-)

naheemsays’s picture

StatusFileSize
new1.48 KB

reroll of the first patch to remove undesireable changes - the following bit changes the counting from threads to messages and that is not what the modules does so far:

-    $fragments['select'][]      = 'COUNT(pmi.thread_id) as count';
+    $fragments['select'][]      = 'COUNT(DISTINCT pmi.mid) as count';

If this change of behaviour is wanted, it should be discussed and decided on as a group beforehand.

The rest of the changes that fix the actual bug remain.

EDIT - please ignore this one and use the patch in the original post.

JirkaRybka’s picture

That change fixed the second part of the bug, which is: Duplicate records in the index cause more replies to be shown in overview, i.e. a completely new thread (no replies) sent to oneself is shown as having 1 reply. That's similar to "forever unrerad" part, as it's caused by the same problem (duplicate entries), and results in broken display on overview screen.

The query didn't count threads as such - it's a query grouped by threads to retrieve single rows for the overview [threads] list, retrieving message counts inside these groups [inside overview table rows, that is]. Previously, it only just counted [message-oriented] rows returned [grouped] by the query; instead of COUNT(pmi.thread_id), we might use just any of the columns retrieved by that query, with pretty much the same effect. But however, when we have duplicate records for some messages, we need to eliminate these - and that's COUNT(DISTINCT pmi.mid), which only just removes duplicates from message count, without really changing anything else. COUNT(DISTINCT pmi.thread_id) won't work, since the thread ID is the same for all records in each group, so we'll get always a count of 1.

If we can't use the query fix for some reason (which I believe is not the case), we need to go with #1 instead.

naheemsays’s picture

oh yes, sorry about that. Your first patch seems right and it is not changing behaviour in the way I thought it was.

@litwol: please use patch in comment 1 original post.

JirkaRybka’s picture

Or nbz (most probably) meant the patch in the initial post to be used - the one in #1 is the other of my two approaches (although it works for me too, only in a different way).

naheemsays’s picture

yes, thanks. I have corrected my post.

berdir’s picture

I don't really like that we have to GROUP for a single message load but proper solving of the bug (and avoiding duplicate entries in pm_index) needs more thought and time.

Another possibility would be "ORDER BY is_new DESC LIMIT 1" (with http://api.drupal.org/api/function/db_query_range/6). But I'm not sure what is better/faster..

naheemsays’s picture

StatusFileSize
new1.41 KB

Ok, following Berdir's approach - works for me and should potentially be faster.

naheemsays’s picture

StatusFileSize
new1.79 KB

and now using db_query_range instead of just tagging LIMIT 1 onto the end of the "order by".

JirkaRybka’s picture

I tested #15 on my setup:

- Patch is rolled with Windows CR/LF line endings, and so completely fails to apply on my environment (Linux command-line patch utility). But however, I've re-written all the line endings in patch file manually in a text editor, and proceeded to further testing.
- Patch then applied to latest (a few minutes ago downloaded) privatemsg-6.x-1.x-dev.tar.gz with offset -22 lines. No big deal, I guess.
- Code looks good, and works fine for me.

So I would say this is RTBC as far as I'm concerned, except that it was my own patch initially and I had problems to make it apply - so others should decide on that part.

berdir’s picture

- Patch is rolled with Windows CR/LF line endings, and so completely fails to apply on my environment (Linux command-line patch utility). But however, I've re-written all the line endings in patch file manually in a text editor, and proceeded to further testing.

I've seen that too, but my patch utility just issues a warning and removes them themself.

$ patch -v
patch 2.5.9

JirkaRybka’s picture

My end, too:

$ patch -v
patch 2.5.9

I tried --ignore-whitespace switch (which is really the only one shown by --help, that seems even remotely relevant, but no luck). Weird, but not really a problem for the actual code in the patch.

ilo’s picture

Status: Needs review » Reviewed & tested by the community

Applies fine and works as expected. rtbc as commented by irc

litwol’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new1.19 KB

This is my take on this issue. please review.

JirkaRybka’s picture

No time to actually apply&test the patch today (I'll be back on Tuesday), but from quick reading - I guess this will work, but it feels weird to me to have a mixture of both the approaches (fix one part in loading query, and other part at saving time). That's just a feeling, though.

Otherwise, I see two points to discuss:
- As #4 mentions, this won't fix existing data (already existing "forever unread" messages)
- Why insert a row with "new", and then update it - shouldn't we just insert the correct one first time?

litwol’s picture

Status: Needs review » Needs work

You are correct regarding #4, marking this a needs work accordingly.

regarding inserting one record as read and other as new: During one insert the message is inserted as *recipient* hence the new mark, the second time its inserted as *author* hence unread mark. I have no good ideas on how to fix this besides fixing our db schema and that is an oportunity we must catch when d7 branch opens up (very soon).

JirkaRybka’s picture

Status: Needs work » Needs review

Also, if I send a message to myself (as a new user exploring features, or just wanting to create a "reminder to myself" message), I most probably expect it to show as unread, until I really view it. Although I certainly did read the message while typing/submitting it, having it shown as "read" without visiting the thread might be a bit unexpected behavior.

JirkaRybka’s picture

Status: Needs review » Needs work

cross-posted, sorry...

JirkaRybka’s picture

#22: I meant a bit different thing: Inserting all the recipients rows as 'unread', and then immediately update one of these to 'read' is unneeded extra query (although it's not run frequently, I admit). So I thought it might be better to add a condition inside the loop, so that the 'recipient' row for the author is inserted without the new flag initially.

See also #23, though.

litwol’s picture

#25 i have no opinion regarding "perfecness" of either approach.

regarding #23, i agree. This is a rather *magical* behavior to mark them read... would it be better if we mark them as unread instead ?

JirkaRybka’s picture

I would say yes, mark as unread. It would be more of an expected behavior, and thinking of it, having BOTH the rows unread should avoid the problem all the same. It's also simpler code, just changing the status on the extra row for the author.

But however, I smell a risk about the unread messages count - I'm not sure, but having TWO unread rows might show two unread messages in the title? (I didn't test)

naheemsays’s picture

@JirkaRybka - but that would be broken behaviour in the default sense where the author is not also in the recipients.

litwol’s picture

Status: Needs work » Needs review
StatusFileSize
new4.49 KB

Here we go.

1) messages marked UNREAD
2) update hook. i am using a double joint query to get only results that have this problem. meaning only messages that were sent to self but have also been viewed at least once. any messages marked unread will not be affected by this.
3) added few comments to explain what is going on
4) moved hook_update_6001 bellow 6000

naheemsays’s picture

Status: Needs review » Needs work
  // The author has already read this message so set to "read" - should fix bug where the author has been added as a recipient too.
  privatemsg_message_change_status($mid, PRIVATEMSG_UNREAD, $message['author']);

That should be PRIVATEMSG_READ as now if you send a message to anybody, it is marked as unread in your own inbox too even if you are not a recipient.

also, for consistency, should it be "$message['author']->uid"?

litwol’s picture

StatusFileSize
new4.65 KB

fixed typo. status update happens only when author is also a recipient.

also, for consistency, should it be "$message['author']->uid"?

No, status change function requires user object, not UID.

liam mcdermott’s picture

Status: Needs work » Reviewed & tested by the community

go go go!

litwol’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new4.68 KB

Implementing berdir's idea from IRC. this removes the need for the use of privatemsg_message_change_status while sending message.

berdir’s picture

Status: Needs review » Reviewed & tested by the community

Phew... let's commit this :)

litwol’s picture

Status: Reviewed & tested by the community » Fixed

I forever silence this monster now.

JirkaRybka’s picture

Status: Fixed » Needs work

I'm back from my short holidays, so I tested the new dev version. I have a few concerns about this patch:

- The update function 6003 lacks { } around table names (three instances), so it fails horribly on prefixed tables.

- The code comment on adding the extra index record for the author shouldn't mention "set column new to 0" anymore, as it's not true now. BTW, there's also a typo ("tha author") and double-space on that line, and the new comment above should have a dot at the end.

I have no time to roll a patch now (my deadline is too close, so switching to "do only what's absolutely necessary for the site" mode), but still I consider this "needs work" because of the update function failure.

Otherwise, everything works fine on my test install, so I confirm the bug being fixed, except the above.

naheemsays’s picture

Status: Needs work » Needs review
StatusFileSize
new2.21 KB

Can I just add two words: grr argh!

Attached is patch that should fix those problems.

litwol’s picture

Status: Needs review » Fixed

Thanks for catching this.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.