This is a matching issue for a RTBC D7 core issue: #1040790: _field_info_collate_fields() memory usage.
Right now D7 loads all fields and all instances into memory on each request.
This can get out of control fast, causing memory usage to skyrocket (especially if the site is a Commerce site with many product types and product display types).
Numbers from the core issue:
With free-for-all fieldable entity types, that array can rapidly grow out of hand (measurements are from @catch in the OP):
140k on a fresh install
1M on installing media + media_gallery contrib modules
3M on non unrealistic sites
10M+ on some commerce sites and other sites with many bundles/instances
Once the change lands in core, fields & instances will be cached & retrieved on a per entity type & bundle basis, significantly decreasing memory usage.
However, for that to happen, contrib modules need to stop requesting all the fields all the time.
Commerce is right now pretty bad at examining too many fields & instances, which means that the core patch provides almost no memory savings unless that is fixed.
| Comment | File | Size | Author |
|---|---|---|---|
| #18 | commerce-7283548-18.patch | 3.89 KB | pjcdawkins |
| #7 | 1809916-decrease-memory-usage-7.patch | 12.3 KB | bojanz |
| #6 | 1809916-decrease-memory-usage-temp.patch | 11.35 KB | bojanz |
| #6 | 1809916-decrease-memory-usage-temp2.patch | 11.28 KB | bojanz |
| #4 | 1809916-decrease-memory-usage-3.patch | 12.29 KB | bojanz |
Comments
Comment #1
bojanz commentedFirst stab at this.
7 files changed, 124 insertions(+), 99 deletions(-)
Of course, this doesn't create any problem with the current Drupal core.
Note that because of reference checking, we are still loading all fields when a customer profile / line item gets deleted.
This could be fixed by changing the logic in:
so that it assumes these entity types are always referenced from orders, but it's a logic change so I didn't want to attempt it at this point.
I did kill a few kittens with random cleanups in the functions I was modifying, so sorry about that.
Comment #2
bojanz commentedBetter patch.
Comment #4
bojanz commentedLet's try this one.
The tax field changes were made unnecessary by #1791226: use hook_field_widget_form_alter() instead of hook_field_attach_form() for adding tax to price field widgets.
Also removed the changes in the same style to commerce_order_field_attach_form(), we should actually fix it in another issue in the same way as #1791226: use hook_field_widget_form_alter() instead of hook_field_attach_form() for adding tax to price field widgets (converting to a widget_form_alter).
Comment #6
bojanz commentedLet's see which part fails.
Comment #7
bojanz commentedThis one will pass.
Comment #8
joshmillerJust read through the patch and had two comments. Not sure this counts as a "review" ... but hey some feedback is better than none :D
The original comment here suggested we needed to confirm this addressfield is a current profile. Is this new logic addressing that possibility? Perhaps leaving the comment would not be a bad idea.
Are we losing flexibility or is this always going to be a node?
Comment #9
joshmillerAlso, thought I would look at the memory savings. I don't have a fancy way to look at individual function loads and memory usage, but I do have the trusty devel memory stats at the end of the page... We are saving memory here, folks... I setup a Kickstart 2 RC2 instance and patched it with the following patches (and without). Before each page load, I'd clear cache all using drush.
D7 Patch = http://drupal.org/files/field-info-cache-1040790-144-D7.patch
DC Patch = http://drupal.org/files/1809916-decrease-memory-usage-7.patch
Comment #10
bojanz commentedField injection always worked for nodes only.
EDIT: This is untrue, it also works for line items currently (for some odd reason).
Comment #11
rszrama commentedYou sure about that? I was under the impression it would work for any referencing entity type, though the main use case was node.
Comment #12
rszrama commentedDo you suppose this issue is now deprecated by #1960856: Bring Field API Memory improvements from Drupal 7.22 to Drupal Commerce.?
Comment #13
bojanz commentedOf course not, this is the patch for that issue :)
Comment #14
j0rd commentedThanks for already being pro-active about this.
As for @joshmiller remarks in comment #9, if you're looking to get memory statistics on a function level I strongly recommend looking into installing XHPROF, a wonderful PHP performance debugger created by facebook. It gives you memory usage by function and overall. It also gives you other useful information like how many times a function is called, and how much CPU time they all take up. It's the best tool I've found for debugging PHP performance issues.
For those who are interested Drupal 7.22 included some new patches to help solve these issues, but contrib modules need to be updated to take advantage.
#1915646: The Field API's memory usage has been optimized (includes recommended changes to contributed modules to leverage the changes)
"The Field API's memory usage has been optimized (includes recommended changes to contributed modules to leverage the changes)"
http://drupal.org/node/1915646
I've created issues about improving contrib for the changes in 7.22 to other issues queues, like ApacheSolr, Message and CTools. If you're looking for more gains in KickStart I believe Search API, Rules and Inline Entity Form might also benefit from some tweaks.
Comment #15
rszrama commentedI think I'm going to hold off committing this for 1.6 in the interest of both:
Also, I need to get XHPROF going. : )
Comment #16
damien tournoud commented^ This looks like a really inefficient way of doing
field_info_instances($entity_type):)Comment #17
rszrama commentedWith Drupal 7.24 being a security release, we can now ensure that all users will have the function field_info_field_map(). This was designed as a less memory intensive replacement to field_info_fields(), so if we can use it to avoid calls to field_info_fields() in any given page request, let's do it. : )
https://api.drupal.org/api/drupal/modules%21field%21field.info.inc/funct...
Comment #18
pjcdawkins commentedIn response to #16 and #17 here's my attempt at a simpler patch. I have not touched hook_update_N() functions as the memory usage there is probably less likely to have an impact.
Comment #19
rszrama commentedThanks! Very timely patch - reviewed and committed.
Re: #16, I checked, and the reason I eschewed field_info_instances() in the past was the lack of field type in the return value. : P
Commit: http://drupalcode.org/project/commerce.git/commitdiff/237c28d
Comment #20
rszrama commentedQuick side note: I'm removing the explicit dependence on >= 7.22 in the .info files so sites / tests running from -dev still pass.
Comment #22
berdir