Hi,

I tried to do a full item-to-item run of the fivestar recommender, and at the end when it was trying to save the prediction data to the database it got an out of memory error. I ran it twice to make sure I got the same error at the same point -- and I did.

The full log from the command prompt is attached.

I'm running the 9/20/11 version of all modules (6.x-3.0-alpha2).
There are 297k ratings in the votingapi table.

Let me know if there's any other info I can provide.

Thanks,
-Connor

CommentFileSizeAuthor
OutOfMemory.txt9.92 KBangusmccloud

Comments

danithaca’s picture

danithaca commented
Status: Active » Needs work

This is indeed a bug. The problem is that all prediction/similarity data are generated and save to memory first before save to database. I need to make it a queue and save periodically to database to save memory.

For now, a quick work around is to increase Java memory. In run.bat, change:
java -cp %CLASSPATH% org.drupal.project.recommender.RecommenderApp
to:
java -Xmx10000M -cp %CLASSPATH% org.drupal.project.recommender.RecommenderApp

If it still doesn't work, increase -Xmx even more.

angusmccloud’s picture

angusmccloud commented

Is there a staging table that it writes back to before going to the final prediction table? Or does it do delete/inserts one user at a time (or set of users)?

I guess what I'm wondering is will it truncate the prediction table before it begins writing in data, and then it'll fill it up in chunks of users at a time? In this scenario, there'd be some users that have no predictions during that time while it's being rebuilt.

(You've probably thought of all this, but I felt the need to say it anyway)

danithaca’s picture

danithaca commented

This problem is fixed in 7.x-4.x release. Basically, I make it multi-threading, so there's a new thread writing data back to the database concurrently with the recommendation computation. That way, it won't require much memory because data are write back immediately after it's computed. Also it increase speed.

I'll see how much work it'll cost to port it to the 6.x-3.x release.

danithaca’s picture

danithaca commented
Status: Needs work » Fixed

This is fixed in 6.x-3.0-alpha3. Also, please update async_command to 6.x-1.0-alpha3.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.