Reduce memory usage to improve scalability
| Project: | Hierarchical Select |
| Version: | 6.x-3.x-dev |
| Component: | Code - Taxonomy |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | postponed |
This is a feature request to reduce memory usage on page load for large taxonomy. It would improve scalability.
To reproduce this issue use a 20MB taxonomy and monitor memory usage. Taxonomy is 3 levels deep.
On page load Hierarchical Select 6.x-3.x-dev (2009-Jul-29) uses 287MB of memory (RAM). And this is for only one visitor.
Most share hosting providers have 32MB limit. Most populare VPS hosting plans are between 360MB & 720MB limit.
To get above memory usage result I type in the following command in Ubuntu terminal. This will output the memory usage every 1 second for 50 iterations (50 secondes). Display in megabyte (M).
vmstat -S M 1 50
Another option to monitor module memory usage is the Devel module. Under 'Performance Logging' maximum memory usage. http://drupal.org/project/devel
To quickly and easily create a 20MB taxonomy I use the Devel module. And its 'Generate content' feature. Hope this help.
#1
This is because I use use the Taxonomy API, which is unscalable. This can be circumvented by creating our own queries.
Will be for HS 4 probably. Patches are welcome, of course. If you provide a patch, I'll commit it as soon as it's ready instead of waiting for version 4.
#2
@Wim: Thanks I understand now. I'm not a coder but will be happy to contribute patch testing.
@All: Any volunteers for creating a patch with custom queries to workaround the unscalable Taxonomy API?
#3
@all: If someone else wants to contribute memory usage testing here are 2 large taxonomy files. And a small one for quick test. To download right click on below link then select
Save Link Asoption.vocabulary-large-utf8-1.5mb.csv
vocabulary-large-utf8-20mb.csv
vocabulary-small-utf8-2kb.csv
File format is UTF-8
3 levels deep
The easiest way that I know of to import large taxonomy files is using the batch feature of 'taxonomy_csv' module version 4.1 or more recent at http://drupal.org/project/taxonomy_csv
Find attached screenshot for module settings.
Note that those links aren't permanent. So if they don't work that's normal.