Test w/ 50 users and 500 nodes, replace missing data with zero. 32M PHP max memory limit will cause error. Set to 64M memory is fine.

Running w/ Drush doesn't have such a problem.

Comments

danithaca’s picture

danithaca commented

exceed memory happens at computeSimilarityMemory() => saveSimilarityMatrix() => Matrix::get().

danithaca’s picture

danithaca commented
Status: Active » Postponed

This might be impossible to fix......
For such a complex computation requirement, it is normal to use lots of memory. In fact, the Java implementation, Mahout, recommends using 1000MB memory. The default PHP max_memory setting is only 8MB, or 32MB. I don't think it's possible to fit.
For now, just use the Drush script.

danithaca’s picture

danithaca commented
Category: bug » task
kostajh’s picture

kostajh commented
Version: 6.x-2.x-dev » 6.x-2.0-beta3

I'm not sure what to do about this:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 73 bytes) in /sites/all/modules/recommender/Matrix.php on line 263

I'm on a VPS with 256 MB of RAM. I used the drush script (needless to say trying to run Recommender through the web interface crashed).

Any ideas?

kostajh’s picture

kostajh commented

I upgraded my VPS to 512mb of RAM and no longer have this problem, but it still seems like this should not occur for folks using 256mb.

danithaca’s picture

danithaca commented

well, i'm sorry to learn that. can't think of a better idea due to the complexity of the task (which involves large matrix computation).
for future reference, this might be solved by distribute the computation to another cheaper server through REST/WebService/whatever. refere to #503212: add Apache Mahout web services support

ken hawkins’s picture

ken hawkins commented

Having similar woes. We have a local news site with some 6,500 nodes and ~200 users. After bumping php from 115MB to 200MB we still run out of mem using drush:

Running history_rec at 2009-11-14T13:29:35-05:00 ..........
PHP Fatal error:  Allowed memory size of 208666624 bytes exhausted (tried to allocate 35 bytes) in /var/.../httpdocs/sites/all/modules/recommender/Recommender.php on line 390
Drush command could not be completed.                                                                                 [error]

Update: I got brave (reckless) and pushed it to 300MB on our 512 server and it ran.

But with us adding some ~700 nodes a month I have to wonder how long we'll be able to use this module. I realize there's not too much that can be done ... it's a memory intensive task.

Are there any work arounds? Some way to do it on a local machine?

danithaca’s picture

danithaca commented

Yes. I realized memory consumption is a problem.... But since it involves matrix computing, where everything has to be in the memory, there's no good way to fix the problem ....

Two suggestions:
1. Run the computation in a development mirror of the site (where you can use >512M memory), and then import the database table (recommender_*) to the production server. You should be able to write a script for it if it's repetitive.
2. Wait for #503212: add Apache Mahout web services support

ken hawkins’s picture

ken hawkins commented

Makes sense. I'd encourage you to add those two points to the readme.txt as I wonder if many would simply give up on the module before researching enough to come to that conclusion.

joachim’s picture

joachim commented

Passing thought: how do older modules like http://drupal.org/project/cre handle it? I've been using that on a D5 site and its memory requirements seem ok.

angusmccloud’s picture

angusmccloud commented

Has anyone built a script like this?

I currently run my recommendations offline (it's hitting about 400 users, 1500 nodes -- so my host can't do it). Right now I manually download my _votingapi_vote table (from phpmyadmin), overwrite the local copy, run recommendations on my local machine (is it faster to do this via drush than the interface?), and then upload the _recommendation tables.

Has anyone automated this process?

danithaca’s picture

danithaca commented

@angusmccloud: sorry to know you have to go through a process like that. I'll try to write a script and automate this. See #698204: automate data export/import. However, my priority is first #673786: add views support

angusmccloud’s picture

angusmccloud commented

@danithaca: thanks!! Would definitely be a real treat if that was automated and I didn't have to be watching it all the time and RDing in to make new recommendations :)

chawl’s picture

chawl commented

How about ~45,000 nodes and ~22,000 users?

Even 2GB limit is consumed in a minute with drush.

There should be some more optimized C, java, or python library somewhere :/

danithaca’s picture

danithaca commented
Status: Postponed » Closed (won't fix)

This problem is not supported anymore because I plan to switch to other architecture. Please refer to #816112: Evaluate alternatives to boost performance (CPU/RAM)

danithaca’s picture

danithaca commented
Status: Closed (won't fix) » Closed (works as designed)
danithaca’s picture

danithaca commented

Submitted GSoC application to work on this issue: http://groups.drupal.org/node/137054