I just recently upgrade my D5 community site to D6 (test setup). The /forum page loads times have increased dramatically (from 5 sec page load to +3 min). The problem only occurs for the main "/forum" page, inside pages load fine. From my research and chatting with fellow Drupalers I think there could be a few reasons why this is occurring.

One reason which people have pointed out is possibly due to n.type = 'forum' condition having been removed from the forum_get_forums() function. I introduced this condition again into the queries and the load time got reduced from 3 minutes for the /forum page to roughly about 25 seconds.

Now if I run the below query (from the function forum_get_forums()) on my D5 installation it would take .2 seconds, but if I run the same query on my fresh D6 upgraded db, it would take about 8 seconds. See below the explain when run for the D5 and D6 databases:

Output from explain run on my D5 database:

mysql> explain SELECT ncs.last_comment_timestamp, IF (ncs.last_comment_uid != 0, u2.name, ncs.last_comment_name) AS last_comment_name, ncs.last_comment_uid FROM node n INNER JOIN users u1 ON n.uid = u1.uid INNER JOIN term_node tn ON n.nid = tn.nid INNER JOIN node_comment_statistics ncs ON n.nid = ncs.nid INNER JOIN users u2 ON ncs.last_comment_uid=u2.uid WHERE n.status = 1 AND n.type = 'forum' AND tn.tid = 64 ORDER BY ncs.last_comment_timestamp DESC LIMIT 0, 1;
+----+-------------+-------+--------+---------------------------------------------------+---------+---------+-------------------------+------+----------------------------------------------+
| id | select_type | table | type   | possible_keys                                     | key     | key_len | ref                     | rows | Extra                                        |
+----+-------------+-------+--------+---------------------------------------------------+---------+---------+-------------------------+------+----------------------------------------------+
|  1 | SIMPLE      | tn    | ref    | PRIMARY,nid,tid                                   | tid     | 4       | const                   |  221 | Using index; Using temporary; Using filesort | 
|  1 | SIMPLE      | n     | ref    | PRIMARY,node_type,status,uid,node_status_type,nid | PRIMARY | 4       | ql.tn.nid               |    1 | Using where                                  | 
|  1 | SIMPLE      | u1    | eq_ref | PRIMARY                                           | PRIMARY | 4       | ql.n.uid                |    1 | Using where; Using index                     | 
|  1 | SIMPLE      | ncs   | eq_ref | PRIMARY                                           | PRIMARY | 4       | ql.tn.nid               |    1 |                                              | 
|  1 | SIMPLE      | u2    | eq_ref | PRIMARY                                           | PRIMARY | 4       | ql.ncs.last_comment_uid |    1 | Using where                                  | 
+----+-------------+-------+--------+---------------------------------------------------+---------+---------+-------------------------+------+----------------------------------------------+

Output from explain run on my D6 database:

mysql> explain SELECT ncs.last_comment_timestamp, IF (ncs.last_comment_uid != 0, u2.name, ncs.last_comment_name) AS last_comment_name, ncs.last_comment_uid FROM node n INNER JOIN users u1 ON n.uid = u1.uid INNER JOIN term_node tn ON n.vid = tn.vid INNER JOIN node_comment_statistics ncs ON n.nid = ncs.nid INNER JOIN users u2 ON ncs.last_comment_uid=u2.uid WHERE n.status = 1 AND n.type = 'forum' AND tn.tid = 64 ORDER BY ncs.last_comment_timestamp DESC LIMIT 0, 1;
+----+-------------+-------+--------+------------------------------------------------+-----------+---------+-------------------------------+-------+----------------------------------------------+
| id | select_type | table | type   | possible_keys                                  | key       | key_len | ref                           | rows  | Extra                                        |
+----+-------------+-------+--------+------------------------------------------------+-----------+---------+-------------------------------+-------+----------------------------------------------+
|  1 | SIMPLE      | n     | ref    | PRIMARY,vid,node_type,uid,node_status_type,nid | node_type | 14      | const                         | 41256 | Using where; Using temporary; Using filesort | 
|  1 | SIMPLE      | u1    | eq_ref | PRIMARY                                        | PRIMARY   | 4       | qltestd5.n.uid                |     1 | Using where; Using index                     | 
|  1 | SIMPLE      | ncs   | eq_ref | PRIMARY                                        | PRIMARY   | 4       | qltestd5.n.nid                |     1 |                                              | 
|  1 | SIMPLE      | u2    | eq_ref | PRIMARY                                        | PRIMARY   | 4       | qltestd5.ncs.last_comment_uid |     1 | Using where                                  | 
|  1 | SIMPLE      | tn    | ref    | vid                                            | vid       | 4       | qltestd5.n.vid                |     1 | Using where                                  | 
+----+-------------+-------+--------+------------------------------------------------+-----------+---------+-------------------------------+-------+----------------------------------------------+

Notice the first row in each of those explains for D5 shows the value 221 rows as apposed to for D6 shows rows values as 41,256.

Anybody has any input as to why such a huge discrepancy in these numbers and why the same query on D6 take so much longer compared to D5.

Note: There is one difference between the 2 queries, the field vid is used in D6 for the table term_node whereas nid is used in D5. I don't have revisions enabled for the most part on my site so I figured this wouldn't make such a difference (perhaps I'm wrong).

CommentFileSizeAuthor
#6 forum.with-stats.tar_.gz18.56 KBrgristroph

Comments

vm’s picture

Status: Active » Fixed

Moheed, I thought we fixed this in IRC?

Though I was unaware that you are using 6.9. Code base should be updated to Drupal 6.13 which is the latest release at this time.

Also of note, this would be better served in the forums.

moeed’s picture

Status: Fixed » Active

I thought it was fixed but actually the problem remained. Strangely, every 3 or 4 clicks, the page starts taking that long. As well I included the query explain in the above post in case that is significant. I had spoken to webchick at Drupalcon Paris last week and she had recommended I post it up here so I figured I might as well.

moeed’s picture

I thought it was fixed but actually the problem remained. Strangely, every 3 or 4 clicks, the page starts taking that long. As well I included the query explain in the above post in case that is significant. I had spoken to webchick at Drupalcon Paris last week and she had recommended I post it up here so I figured I might as well.

I put D6.9 because it doesn't give you an option for the latest 6.13 in the issue creation form. Actually I am running the latest code base 6.13.

Thanks.

vm’s picture

Version: 6.9 » 6.13

moved to 6.13

this is still better served in the forums.

moeed’s picture

Posting in the forum. See you there :o)

rgristroph’s picture

StatusFileSize
new18.56 KB

I have a first cut at something I think should solve this issue. I wrote it with Moeed (the original poster of this issue) for QatarLiving.com, but it is not "live" on that site right now, it is only on a development site; however, working with a copy of the full database with all forums and comments that the live site has, it runs quite quickly.

What this does is:

1) Create a table "forum_statistics" that has a row for each forum, keeping track of the most recent post and user and total number of topics and posts (i.e., all the summary information displayed on the /forums page that takes a while to build with a resource-intensive query).

2) Hooks the comment posting (hook_comment) and creation of a forum node (nodeapi) so that statistics are updated when a post or topic is made, and the statistics are updated cheaply by just incrementing the number of posts / topics. Note that rebuilding the stats for just one forum is not so bad, and users might tolerate a slight delay when posting, so we could just re-calculate the stats for the forum in this case.

3) A new button in the admin menu for forums allows you to re-calculat all forum stats in case they get out of date.

4) A hook_cron also recalculates the forum stats. I am not sure I want to leave this in the final version. It could be configured to turn on and off, and another option is to make a drush command to re-calculate the statistics, so that could be run from the linux cron separately from the normal drupal cron. The statistics recalculation takes as much as 3 minutes, and that seems excessive for a cron running every 5 minutes, for example.

I am attaching a tar file of the whole /modules/forum directory. I will make a patch against core and post it in a followup message, I have to isolate just the changes I made, because I was not working off of Drupal head. Note that I made the changes starting with the Pressflow patched version of Drupal (latest version, 6.15.73) however I don't think this should make any difference.

As I am still cleaning this up, any suggestions for changes or code formatting or etc would be appreciated.

--Rob

kars-t’s picture

Status: Active » Fixed

Hi

I am closing this issue to clean up the issue queue. Feel free to reopen the issue if there is new information and the problem still resides. If not please make sure you close your issues that you don't need any more.

Maybe you can get support from the local user group. Please take a look at this list at groups.drupal.org.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.