Closed (fixed)
Project:
Recommender API
Version:
6.x-3.0-alpha1
Component:
Code
Priority:
Major
Category:
Feature request
Assigned:
Reporter:
Created:
24 Jun 2011 at 13:28 UTC
Updated:
1 Oct 2011 at 16:51 UTC
Jump to comment: Most recent file
Comments
Comment #1
angusmccloud commentedThanks Dan!
I'm more than happy to use the 6.3 release when it's ready (instead of this code we have hacked together). Particularly if ti can do incremental indexing and recommendation creation.
At this point I'll probably wait for 6.3 to do this -- but the next piece I was going to work on adding was a way to rebuild the similarities table without destroying the table (until the rebuild is complete). That way if it takes 12-24 hours to recalculate those values, you can keep making predictions using the old similarities until it finishes (just have separate threads running).
Right now I use two computers to do this, but ideally I'll keep it all on one box (without having two instances of drupal installed).
Anyway -- very excited to switch to Mahout with incremental builds!
Still on target for the end of August?
Cheers!
Comment #2
danithaca commentedThanks for the feedback Connor. Mahout is super fast, so it won't take 12-24 hours computation anymore :)
I'll make sure not to destroy the old similarity data before the new data is generated in the 3.x releases.
The target date is Aug or Sep.
Cheers~
Comment #3
angusmccloud commentedThat's awesome! Very much looking forward to this one Dan.
Let me know if there's anything I can do to help out (keeping in mind you're WAY better at this than I am)
Comment #4
danithaca commentedWorking on this now. D6 backport of the dependancy async_command is done.
Comment #5
danithaca commentedDone. Most of the helper modules (history_rec, fivestar_rec, uc_rec, similargroups) have been updated to use 6.x-3.0-alpha1.
I haven't done much testing though. Please test it first on dev environment and then production. Let me know if there's any problem.
Comment #6
angusmccloud commentedDan,
I installed the newest version of the recommender (on a fresh d6 install -- so nothing installed except modules related to the recommender).
I'm using the fivestar recommender, running item-to-item. I ran it first for two users worth of ratings (about 500 ratings). It finished really quickly (whew).
I then tried to run it for 10k users (270k ratings), and it's erroring out. I can't tell what the error is: it looks like it's scrolling through ratings for a while, then it throws an error. A screenshot of the error is attached.
Let me know if you have any ideas,
-Connor
Comment #7
angusmccloud commentedI tried with smaller chunks of ratings:
* 5000 = failed (same error)
* 1000 = failed (same error)
* 500 = failed (same error)
* 100 = failed (same error)
* 50 = succeeded (1400 ratings)
Comment #8
danithaca commentedConnor,
The computation ran without problem. The problem was that it had too much data to save back to the database all at once. You can set "db_max_batch_size" in "config.properties" to be 5000 so that it'll save 5000 computed results per batch at a time. Or, you can set "max_allowed_packet" in your MySQL database (http://dev.mysql.com/doc/refman/5.5/en/packet-too-large.html).
I've seen this problem in my dev environment before, so it should fix the problem.
Another thing to note is that the Mahout algorithm implementation is slightly different from RecAPI 2.x, so you might see some differences in the recommendations it generates. Let me know if the recommendations make sense or not. Thanks.
Btw, please submit new issues for other bugs. It'll be easier for me to keep track. Thanks.
--daniel
Comment #9
angusmccloud commentedSorry for using this thread, will use a new one next time and let you merge if they're related.
Neither of those things fixed it -- will open in a new thread instead of continuing here.