too much memory consumption
danithaca - July 3, 2009 - 15:03
| Project: | Recommender API |
| Version: | 6.x-2.0-beta3 |
| Component: | Code |
| Category: | task |
| Priority: | normal |
| Assigned: | danithaca |
| Status: | postponed |
Jump to:
Description
Test w/ 50 users and 500 nodes, replace missing data with zero. 32M PHP max memory limit will cause error. Set to 64M memory is fine.
Running w/ Drush doesn't have such a problem.
#1
exceed memory happens at computeSimilarityMemory() => saveSimilarityMatrix() => Matrix::get().
#2
This might be impossible to fix......
For such a complex computation requirement, it is normal to use lots of memory. In fact, the Java implementation, Mahout, recommends using 1000MB memory. The default PHP max_memory setting is only 8MB, or 32MB. I don't think it's possible to fit.
For now, just use the Drush script.
#3
#4
I'm not sure what to do about this:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 73 bytes) in /sites/all/modules/recommender/Matrix.php on line 263I'm on a VPS with 256 MB of RAM. I used the drush script (needless to say trying to run Recommender through the web interface crashed).
Any ideas?
#5
I upgraded my VPS to 512mb of RAM and no longer have this problem, but it still seems like this should not occur for folks using 256mb.
#6
well, i'm sorry to learn that. can't think of a better idea due to the complexity of the task (which involves large matrix computation).
for future reference, this might be solved by distribute the computation to another cheaper server through REST/WebService/whatever. refere to #503212: add Apache Mahout web services support
#7
Having similar woes. We have a local news site with some 6,500 nodes and ~200 users. After bumping php from 115MB to 200MB we still run out of mem using drush:
Running history_rec at 2009-11-14T13:29:35-05:00 ..........PHP Fatal error: Allowed memory size of 208666624 bytes exhausted (tried to allocate 35 bytes) in /var/.../httpdocs/sites/all/modules/recommender/Recommender.php on line 390
Drush command could not be completed. [error]
Update: I got brave (reckless) and pushed it to 300MB on our 512 server and it ran.
But with us adding some ~700 nodes a month I have to wonder how long we'll be able to use this module. I realize there's not too much that can be done ... it's a memory intensive task.
Are there any work arounds? Some way to do it on a local machine?
#8
Yes. I realized memory consumption is a problem.... But since it involves matrix computing, where everything has to be in the memory, there's no good way to fix the problem ....
Two suggestions:
1. Run the computation in a development mirror of the site (where you can use >512M memory), and then import the database table (recommender_*) to the production server. You should be able to write a script for it if it's repetitive.
2. Wait for #503212: add Apache Mahout web services support
#9
Makes sense. I'd encourage you to add those two points to the readme.txt as I wonder if many would simply give up on the module before researching enough to come to that conclusion.
#10
Passing thought: how do older modules like http://drupal.org/project/cre handle it? I've been using that on a D5 site and its memory requirements seem ok.