Taxi page times out with large Taxonomies
| Project: | Ordered Taxonomy Display - Taxi |
| Version: | 5.x-1.1 |
| Component: | Code |
| Category: | bug report |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Jump to:
Hi,
while the "Taxi" module works fine with small taxonomies, the display/generation of the "Taxonomy tree page" times out with medium to large vocabularies (several thousands of terms). I've been testing this on six different D5 sites of which five are *not* affeced, only the oldest and largest one. I can try to give you some figures about the taxonomy's sizes, if you're interested.
Also, on such large sites, rendering pages of content types with "Ordered Taxonomy Display" enabled, dies completely: Loading the page does result in a timeout after a few minutes; during this time, it causes 100% CPU load on a dual core system which will effect other sites/processes running on the same server (take care if you're hosted in a shared environment!). "Cache Settings" seem to have no noticable effect on such sites. "Taxi" simply does not work in these cases.
Since some other modules seem to have similar problems, this issue seems to be hard to solve programatically; I'd suggest for the "Taxi" module to scan for the size of the taxonomy and to issue at least an warning. However, other modules with similar problems don't do that, either ;)
Greetings, -asb
#1
Thanks for the warning. As soon as I have time, I'll look into it, maybe doing some research in order to attempt to reduce cpu load; if that still does not work, then I'll take your suggestion of warning about large taxonomies.
It is somewhat reasonable, however, since the building of the hierarchical tree takes it's good share of cpu. I'll try to calculate the resources needed for the table to generate.
A question here; is the page crashing in a node which does contain a lot of taxonomies also?
Thanks again!
#2
Hi LisandroEC,
at the affected site, no pages are created, as soon as "Taxi" is enabled for the node's content type. I wouldn't speak of page crashing in this case since it simply times out after some time; it might be possible to to tweak the server's configuration in a way, that theoretically, after some waiting, a page might be produced (e.g. MySQL connection timeouts). Other modules quickly give a WSOD, indicating that something has died.
Since I'm not a programmer, I can only provide jackass advice; maybe the hierarchical tree could be built "in the background", or in segments via cron or some kind of job queueing? MediaWiki (the software behind WIkipedia) utilizes jobs for long running tasks [1]; since MediaWiki needs to scale massively, this seems to be the only possible approach (as far as I know, nobody in Drupal development thinks seriously about sites with several millions of nodes).
And, please remember: In my case, five of six sites work fine with "Taxi", and other modules like "Sitemap" have similar issues. Before breaking your head (does this expression exist in English?) you might want to have a look at this thread regarding performance issues of the XML Sitemap module: http://drupal.org/node/198173 - it took weeks to solve...
Thanks & greetings, -asb
[1] http://meta.wikimedia.org/wiki/Help:Job_queue
#3
Hey, thanks a lot for your patience.
I believe this week I can take a look on this issue, since I have a little more time (and I bought myself a computer for my own use at home :P) as well as implementing some feature requests.
As soon as I can, I will let you know.
Thanks a million, again!
Lisandro.
#4
Hi LisandroEC,
thanks for the recent updates; they're working fine, and relieve of patching the module ;)
Here's some data about the taxonomy of the site where "Taxi" doesn't work for me:
mysql> SELECT count(*) FROM term_data;
+----------+
| count(*) |
+----------+
| 3290 |
+----------+
1 row in set (0.01 sec)
mysql> SELECT count(*) FROM term_hierarchy;
+----------+
| count(*) |
+----------+
| 3696 |
+----------+
1 row in set (0.00 sec)
For comparison: Here's some data about a site where "Taxi" is working fine:
mysql> SELECT count(*) FROM term_data;
+----------+
| count(*) |
+----------+
| 2360 |
+----------+
1 row in set (0.00 sec)
mysql> SELECT count(*) FROM term_hierarchy;
+----------+
| count(*) |
+----------+
| 2401 |
+----------+
1 row in set (0.00 sec)
Either there is (a) a general scalability issue for taxnomies > 3k terms, or (b) a specific problem with the taxonomy on the first mentioned site. If it's (a), it should be addressed by the "Taxi" module somehow, if it's (b), well, then I have a major problem ;-/ I think, with the devel module you can pretty easy teproduce the scenarios. The first site has some 25k nodes, the second about 10k (if it's relevant).
Also, I noticed that enabling "Taxi" on the first site with disabling *all* but one small vocabulary still kills the site. With this "workaround" (displaying just a fragement of the whole taxonomy), I can partially use other problematic tools like "Sitemap" or "Sitemenu" module; for "Taxi" this seems not to work - maybe this helps to reproduce the issue.
Thanks again for this nice menu!
Greetings from Berlin, -asb
#5
I'll take this under consideration, thanks a lot for the info.
I was also thinking - but only thinking, didn't test it though - that
maybe a PHP config setting:
memory_limit = 32M
may be a possible culprit.
The symptoms of timing out when the script consumes memory over this
limit -a couple of times, when testing other projects, something like
this happened to me- are similar, a time out. Well, apache logs
informed this to me, that the memory limit for the execution of the
scripts was exceeded.
May be another thing to check.
Thanks!
#6
Yes, PHP memory_limit is always a critical point when WSODs appear; the problematic site is currently running with memory_limit = 80M, and I incresed this in steps of 96M, 128Mup to 256M (which significantly slows down the site, but does not make the WSOD disappear). Since higher amounts seemed unreasonable and quite insane, I did non continue this test further.
To my current knowledge, Drupal plus PHP memory_limit are moving targets; what memory is really required can be evaluated to some degree with trial and error, but there are no generalizable recommendations; it depends too much on highly variable factors like the number and kind of enabled modules, number of nodes and users, etc. If there is a formula to calculate memory requirements for a given setup, I'd love to hear about.
Greetings, -asb