As a followup to #1710688: Activity stream & Email notification system.
Not explicity shown but suggested in the Commons interractive prototype is the "Your activity stream" functionality, a listing of activity that corresponds to the "following" flag for nodes, users, terms and groups.
This issue is for discussion about how to best record and display the on-site activity stream.
All message types have the uid property, which corresponds to the acting user. Messages also entity reference fields for target users and target nodes that the message is about.
"My activity stream" - A single listing of message entities.
A couple of approaches to displaying this listing.
A) A single query generated by a View with many JOINs and OR clauses
. To do this, we'd need to be:
- Joining message entity user property to flag:
The message entity UID property matches a user entity flagged by the current user
- Joining message entity reference field to flag:
One of the message referenced target users is flagged by the current user
The referenced target node entities are flagged by the current user
- Joining taxonomy term or group membership data to message entity reference field to flag:
The referenced target node entities are in a group flagged by the current user
The referenced target node entities contain a term flagged by the current user
Approach A seems pretty ugly because:
- We'd be combining multiple JOINs & OR clauses with an ORDER BY clause, which in preliminary EXPLAINs in MariaDB 5.2.8 will include temp tables, even when add an index on {message}.timestamp.
- This would require adding multiple Views relationships which prevent the View from having any results, even without adding filters, so it's not clear that this approach would even display the desired information.
B) Multiple queries passed into a single Views filter
1) Query for NIDs, UIDs, and TIDs that the user is following based on what they have flagged.
2) Then, find a list of message IDs related to those entities.
(We'd be examining these tables: node, message, target_nodes, target_users, og_membership, field_data_field_topics)
And looking for data such as:
- Where target_nodes in (1 ,2, 3, 4)
- Where target_users in (4,5 6)
- Where message.uid in (4,5,6)
- node joined to field_data_field_topics (or similar) (where in term ids)
- node joined to og_membership where in (group ids)
3) Finally, render the list of messages and apply the order clause.
This could be achieved with Views via a custom filter that separately runs its own queries, and generates a simple IN() query containing list of message IDS. Add an ORDER BY and LIMIT clause and the view is done.
C)Alter the Views query to do a single query without explicit mids and improve indexes that will prevent writing to disk
Below is the output from an EXPLAIN on a simple query generated by a view that connects the referenced target nodes field to the commons_follow_node flag. We could investigate optimizations that would make queries such as this one -- but with many more joins for the various types of flags more scalable & performant.
 acquia_commons_c71_taxonomy_term_data.png)
| Comment | File | Size | Author |
|---|---|---|---|
| #4 | (MySQL 5.2.8-MariaDB-log) acquia_commons_c71_taxonomy_term_data.png | 90.31 KB | ezra-g |
Comments
Comment #1
lightsurge commentedAre there any performance pitfalls with INs that might include tons of message IDs?
I've used the method in B before and it worked quite well but I did sort of wonder if I was hammering the sql engine.
For example, if the stream is paged showing the latest 10 items, the INs in the parent query would still have to include all message ids to cover all the pages, because you can't really page each query and synchronise each with the parent query.
Say for example we have 50 pages in our stream with 10 messages each, that's 500 message IDs and one rather grotesquely large parent query.
Comment #2
lightsurge commentedI remember in Commons 6.x some of the filtering of the stream ended up having to be done in views pre_render, which made for ugly pagination.
Rather than anything like that or the method in B) with bits and pieces here and there tinkering with the list, it would be so nice to just have just the one View that generates the stream, it makes it so much easier to maintain and extend... So my vote would be for A), if it's feasible :/
Comment #3
ezra-g commentedIt sounds like I didn't do a good job of explaining how this would work: We'd still have a single query in a single view, it would just be a single IN() query with a separate initial query to get the IN() values.
Comment #3.0
ezra-g commentedAdding note about schema.
Comment #4
ezra-g commentedAttaching a screenshot to support option C.
Comment #4.0
ezra-g commentedadding option c.
Comment #5
lightsurge commentedWell, from conversation on irc think I did understand you right, but that the problems illustrated in #1 (of ending up with a query like SELECT stream_items WHERE mid IN (1,2,3,4,5,6,7,8 ... 500)) are intended to be offset by only having the most recent content updates in the stream.
Comment #5.0
lightsurge commentedAdd explain query screenshot.
Comment #6
gregglesI'm not the best mysql optmizer, but I have some experience with it. I've had some experience trying to join 5 tables with lots of odd conditions that ended up running for hours (even after adding the indexes that seemed to make sense) when then breaking it apart into 2 queries joining 3 tables each (i.e. Approach B) would work well. So, I vote for B for now, benchmark a bit, and refactor if/when there are problems.
Comment #7
moshe weitzman commentedI agree that B seems sane. Presumably the small "interior" queries return a timestamp and the views filter sorts on that before returning the list of mids. I think this would be quite stable as folks page through the list.
Comment #8
amitaibuUp until now we opted for B on our scenarios. It's also more flexible --allowing each module to return its message IDs.
Comment #9
lightsurge commentedThink I'll definitely acquiesce to those more experienced here! I suppose, if these secondary queries were built on top of Views as well as the primary query, that'll actually give even greater flexibility to site admins wanting to customise the overall outcome.
As a site admin who'd probably want to increase the number of messages appearing in the stream to the maximum possible without causing performance degradation, I suppose in terms of documentation I'd appreciate a suggestion of how many items would be safe in the average server environment, how sql will cope with queries with multiple INs is a total mystery to me.. happy to look at this and do some testing once we have the stream views to play with.
Comment #10
ezra-g commentedWe've been proceeding based on this architecture.
Comment #11.0
(not verified) commentedFix screenshot.