Devel_generate is restricted to the php timeout. However if devel_generate used the new batch API, it could be used to an unlimited number of nodes. I wanted to create 100,000 nodes for testing some search changes, which simply can't be done with the current implementation. Yet, thanks to the batch API, this CAN be done relatively easily in 6.x.

CommentFileSizeAuthor
#17 177722.patch9.66 KBdouggreen
#16 177722-7x.patch9.56 KBdouggreen
#14 177722-7x.patch9.55 KBdouggreen
#13 177722.patch9.66 KBdouggreen
#12 177722.patch9.69 KBdouggreen
#11 177722.patch7.41 KBdouggreen
#9 177722.patch7.38 KBdouggreen
#8 177722.patch7.25 KBdouggreen
#6 177722_0.patch8.19 KBdouggreen
#2 177722.patch8.78 KBdouggreen
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

my understanding is that mysql time doesn't count toward the timeout so perhaps this is not needed? did you try it? anyway, this is a probably a good idea.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
Status: Active » Needs work
FileSize
177722.patch8.78 KB

The attached patch adds the batch API for the admin/content/generate form. I've marked the patch as "needs work" only because similar (but easier) functions need to be created for the user and taxonomy generate forms. I've put a @TODO in the comment to indicate where this needs to be done. Also, there's really no reason that most of the work needs to stay in the new devel_generate_batch.inc file. I did most of the work here, mostly so that I could have a clean patch and change as little of the existing code as possible.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

i'd like to hear from chad or dww about any project profile impact.

douggreen’s picture

douggreen CreditAttribution: douggreen commented

I don't understand what you mean by "project profile impact." How would this patch to devel module impact the project module?

hunmonk’s picture

hunmonk CreditAttribution: hunmonk commented

we need to keep devel_generate_content() consistent IMO. there's no need to break any existing programatic implementations of the function when we can make it a wrapper.

also, it might be best to change over taxo and user creation in new issues, instead of creating a mega patch.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
Status: Needs work » Needs review
FileSize
177722_0.patch8.19 KB

I've reimplemented devel_generate_content for backward compatibility. I've also removed the TODO's by restoring the original generate taxonomies and users functions as-is... It makes less sense to change these because you're less likely to want large numbers of those tables.

moshe weitzman@drupal.org’s picture

moshe weitzman@drupal.org CreditAttribution: moshe weitzman@drupal.org commented
Status: Needs review » Needs work

i committed some pending changes i had to devel_generate_content. this likely needs a reroll. sorry about that. if you can reroll, i will commit before anything else sneaks in. thx.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
Version: 7.x-1.x-dev » 6.x-1.x-dev
Component: Code » devel_generate
FileSize
177722.patch7.25 KB

LIVE FROM THE MINNESOTA SEARCH SPRINT, I'm reviving this old issue in preparation for search performance testing... I'm working off of a 6.x version for now, but will also post a 7.x version (which I suspect will be almost identical, if not exactly identical). This patch isn't quite right, but it's close.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
FileSize
177722.patch7.38 KB

... and a working version. However, it's not deleting content properly. I'm not sure if the not deleting content properly is a problem with devel or this patch, but because it's not working right, I'm still leaving as CNR.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented

Hi Doug. If you are already in devel_generate and are motivated to do so, I want to switch the node creation so that taxonomy and comment work on the pre-save operation of nodeapi(). Other modules will plug themselves in there too (I want bulk creation of og nodes and og posts). Just a thought. Thanks for working on this.

Note that project module's install profile uses node creation programmatically so they won't be able to use the batch creation but I think thats fine and we are not likely to break anything for them.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
Status: Needs work » Needs review
FileSize
177722.patch7.41 KB

I fixed the delete problem. The attached patch now works. I used it last night to create 100,000 nodes.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
FileSize
177722.patch9.69 KB

And one more roll, this time with the new devel_generate_batch.inc.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
FileSize
177722.patch9.66 KB

This fixes num_comments.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
FileSize
177722-7x.patch9.55 KB

And here's the 7.x patch

yched’s picture

yched CreditAttribution: yched commented

No time to actually test, but shouldn't this line

+    $operations = array('devel_batch_generate_content_kill', array());

be 

+    $operations[] = array('devel_batch_generate_content_kill', array());

?

douggreen’s picture

douggreen CreditAttribution: douggreen commented
FileSize
177722-7x.patch9.56 KB

Thanks yched, Yes it should! Attached patch is for 7.x.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
FileSize
177722.patch9.66 KB

And here's the 6.x patch.

douggreen’s picture

douggreen CreditAttribution: douggreen commented

I'd love to see this committed before things get out-of-date again... Would you like me to commit it, or would you like some additional testing/comments? BTW, the batch api only kicks in if you try to create more than 50 nodes, so if you're testing, try creating 500 nodes.

moshe weitzman’s picture

moshe weitzman CreditAttribution: moshe weitzman commented
Status: Needs review » Reviewed & tested by the community

@Doug - you may commit this to DRUPAL-6--1 and HEAD. I hope you will stay around to fix any bugs that may come up. Also note that I committed to devel_generate yesterday so please be careful to merge before committing.

douggreen’s picture

douggreen CreditAttribution: douggreen commented
Status: Reviewed & tested by the community » Fixed

Committed to both DRUPAL-6--1 and HEAD. Thanks!

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.