Devel_generate is restricted to the php timeout. However if devel_generate used the new batch API, it could be used to an unlimited number of nodes. I wanted to create 100,000 nodes for testing some search changes, which simply can't be done with the current implementation. Yet, thanks to the batch API, this CAN be done relatively easily in 6.x.
| Comment | File | Size | Author |
|---|---|---|---|
| #17 | 177722.patch | 9.66 KB | douggreen |
| #16 | 177722-7x.patch | 9.56 KB | douggreen |
| #14 | 177722-7x.patch | 9.55 KB | douggreen |
| #13 | 177722.patch | 9.66 KB | douggreen |
| #12 | 177722.patch | 9.69 KB | douggreen |
Comments
Comment #1
moshe weitzman commentedmy understanding is that mysql time doesn't count toward the timeout so perhaps this is not needed? did you try it? anyway, this is a probably a good idea.
Comment #2
douggreen commentedThe attached patch adds the batch API for the admin/content/generate form. I've marked the patch as "needs work" only because similar (but easier) functions need to be created for the user and taxonomy generate forms. I've put a @TODO in the comment to indicate where this needs to be done. Also, there's really no reason that most of the work needs to stay in the new devel_generate_batch.inc file. I did most of the work here, mostly so that I could have a clean patch and change as little of the existing code as possible.
Comment #3
moshe weitzman commentedi'd like to hear from chad or dww about any project profile impact.
Comment #4
douggreen commentedI don't understand what you mean by "project profile impact." How would this patch to devel module impact the project module?
Comment #5
hunmonk commentedwe need to keep
devel_generate_content()consistent IMO. there's no need to break any existing programatic implementations of the function when we can make it a wrapper.also, it might be best to change over taxo and user creation in new issues, instead of creating a mega patch.
Comment #6
douggreen commentedI've reimplemented
devel_generate_contentfor backward compatibility. I've also removed the TODO's by restoring the original generate taxonomies and users functions as-is... It makes less sense to change these because you're less likely to want large numbers of those tables.Comment #7
moshe weitzman@drupal.org commentedi committed some pending changes i had to devel_generate_content. this likely needs a reroll. sorry about that. if you can reroll, i will commit before anything else sneaks in. thx.
Comment #8
douggreen commentedLIVE FROM THE MINNESOTA SEARCH SPRINT, I'm reviving this old issue in preparation for search performance testing... I'm working off of a 6.x version for now, but will also post a 7.x version (which I suspect will be almost identical, if not exactly identical). This patch isn't quite right, but it's close.
Comment #9
douggreen commented... and a working version. However, it's not deleting content properly. I'm not sure if the not deleting content properly is a problem with devel or this patch, but because it's not working right, I'm still leaving as CNR.
Comment #10
moshe weitzman commentedHi Doug. If you are already in devel_generate and are motivated to do so, I want to switch the node creation so that taxonomy and comment work on the pre-save operation of nodeapi(). Other modules will plug themselves in there too (I want bulk creation of og nodes and og posts). Just a thought. Thanks for working on this.
Note that project module's install profile uses node creation programmatically so they won't be able to use the batch creation but I think thats fine and we are not likely to break anything for them.
Comment #11
douggreen commentedI fixed the delete problem. The attached patch now works. I used it last night to create 100,000 nodes.
Comment #12
douggreen commentedAnd one more roll, this time with the new devel_generate_batch.inc.
Comment #13
douggreen commentedThis fixes num_comments.
Comment #14
douggreen commentedAnd here's the 7.x patch
Comment #15
yched commentedNo time to actually test, but shouldn't this line
be
?
Comment #16
douggreen commentedThanks yched, Yes it should! Attached patch is for 7.x.
Comment #17
douggreen commentedAnd here's the 6.x patch.
Comment #18
douggreen commentedI'd love to see this committed before things get out-of-date again... Would you like me to commit it, or would you like some additional testing/comments? BTW, the batch api only kicks in if you try to create more than 50 nodes, so if you're testing, try creating 500 nodes.
Comment #19
moshe weitzman commented@Doug - you may commit this to DRUPAL-6--1 and HEAD. I hope you will stay around to fix any bugs that may come up. Also note that I committed to devel_generate yesterday so please be careful to merge before committing.
Comment #20
douggreen commentedCommitted to both DRUPAL-6--1 and HEAD. Thanks!
Comment #21
Anonymous (not verified) commentedAutomatically closed -- issue fixed for two weeks with no activity.