This is a matching issue for a RTBC D7 core issue: #1040790: _field_info_collate_fields() memory usage.

Right now D7 loads all fields and all instances into memory on each request.
This can get out of control fast, causing memory usage to skyrocket (especially if the site is a Commerce site with many product types and product display types).

Numbers from the core issue:

With free-for-all fieldable entity types, that array can rapidly grow out of hand (measurements are from @catch in the OP):

140k on a fresh install
1M on installing media + media_gallery contrib modules
3M on non unrealistic sites
10M+ on some commerce sites and other sites with many bundles/instances

Once the change lands in core, fields & instances will be cached & retrieved on a per entity type & bundle basis, significantly decreasing memory usage.
However, for that to happen, contrib modules need to stop requesting all the fields all the time.

Commerce is right now pretty bad at examining too many fields & instances, which means that the core patch provides almost no memory savings unless that is fixed.

Comments

bojanz’s picture

Priority: Normal » Major
Status: Active » Needs review
StatusFileSize
new18.88 KB

First stab at this.

7 files changed, 124 insertions(+), 99 deletions(-)

Of course, this doesn't create any problem with the current Drupal core.

Note that because of reference checking, we are still loading all fields when a customer profile / line item gets deleted.
This could be fixed by changing the logic in:

commerce_customer_commerce_customer_profile_delete
commerce_line_item_field_attach_delete
commerce_line_item_delete_references
commerce_order_form_commerce_customer_customer_profile_delete_form_alter

so that it assumes these entity types are always referenced from orders, but it's a logic change so I didn't want to attempt it at this point.

I did kill a few kittens with random cleanups in the functions I was modifying, so sorry about that.

bojanz’s picture

StatusFileSize
new18.85 KB

Better patch.

Status: Needs review » Needs work

The last submitted patch, 1809916-decrease-memory-usage-2.patch, failed testing.

bojanz’s picture

Status: Needs work » Needs review
StatusFileSize
new12.29 KB

Let's try this one.

The tax field changes were made unnecessary by #1791226: use hook_field_widget_form_alter() instead of hook_field_attach_form() for adding tax to price field widgets.
Also removed the changes in the same style to commerce_order_field_attach_form(), we should actually fix it in another issue in the same way as #1791226: use hook_field_widget_form_alter() instead of hook_field_attach_form() for adding tax to price field widgets (converting to a widget_form_alter).

Status: Needs review » Needs work

The last submitted patch, 1809916-decrease-memory-usage-3.patch, failed testing.

bojanz’s picture

Status: Needs work » Needs review
StatusFileSize
new11.28 KB
new11.35 KB

Let's see which part fails.

bojanz’s picture

StatusFileSize
new12.3 KB

This one will pass.

joshmiller’s picture

Just read through the patch and had two comments. Not sure this counts as a "review" ... but hey some feedback is better than none :D

+++ b/modules/customer/commerce_customer.module
@@ -997,9 +997,10 @@ function commerce_customer_field_widget_form(&$form, &$form_state, $field, $inst
-        // First make sure this addressfield is part of the current profile.
-        if (!empty($element['profiles'][$key][$field_name]['#language'])) {
+      foreach (field_info_instances('commerce_customer_profile', $profile->type) as $field_name => $field_instance) {

The original comment here suggested we needed to confirm this addressfield is a current profile. Is this new logic addressing that possibility? Perhaps leaving the comment would not be a bad idea.

+++ b/modules/product_reference/commerce_product_reference.module
@@ -25,70 +25,63 @@ function commerce_product_reference_commerce_product_uri($product) {
-            $extra[$entity_type][$bundle_name]['display']['product:' . $product_extra_field_name] = $product_extra_field;
-          }
+          $extra['node'][$bundle_name]['display']['product:' . $product_extra_field_name] = $product_extra_field;

Are we losing flexibility or is this always going to be a node?

joshmiller’s picture

Also, thought I would look at the memory savings. I don't have a fancy way to look at individual function loads and memory usage, but I do have the trusty devel memory stats at the end of the page... We are saving memory here, folks... I setup a Kickstart 2 RC2 instance and patched it with the following patches (and without). Before each page load, I'd clear cache all using drush.

D7 Patch = http://drupal.org/files/field-info-cache-1040790-144-D7.patch
DC Patch = http://drupal.org/files/1809916-decrease-memory-usage-7.patch

Path Shutdown Peak Savings
/collection/wear 104.88 MB 106 MB -
Patched 103.33 MB 104.5 MB 1.5 MB
/tops/commerce-guys-womens-tee 100.82 MB 102 MB -
Patched 100.33 MB 101.25 MB .75 MB
bojanz’s picture

Are we losing flexibility or is this always going to be a node?

Field injection always worked for nodes only.

EDIT: This is untrue, it also works for line items currently (for some odd reason).

rszrama’s picture

You sure about that? I was under the impression it would work for any referencing entity type, though the main use case was node.

rszrama’s picture

bojanz’s picture

Of course not, this is the patch for that issue :)

j0rd’s picture

Thanks for already being pro-active about this.

As for @joshmiller remarks in comment #9, if you're looking to get memory statistics on a function level I strongly recommend looking into installing XHPROF, a wonderful PHP performance debugger created by facebook. It gives you memory usage by function and overall. It also gives you other useful information like how many times a function is called, and how much CPU time they all take up. It's the best tool I've found for debugging PHP performance issues.

For those who are interested Drupal 7.22 included some new patches to help solve these issues, but contrib modules need to be updated to take advantage.

#1915646: The Field API's memory usage has been optimized (includes recommended changes to contributed modules to leverage the changes)
"The Field API's memory usage has been optimized (includes recommended changes to contributed modules to leverage the changes)"
http://drupal.org/node/1915646

I've created issues about improving contrib for the changes in 7.22 to other issues queues, like ApacheSolr, Message and CTools. If you're looking for more gains in KickStart I believe Search API, Rules and Inline Entity Form might also benefit from some tweaks.

rszrama’s picture

I think I'm going to hold off committing this for 1.6 in the interest of both:

  1. Taking a closer look at the change notice and optimization options and
  2. Preventing an update debacle like we had with the update to require Drupal 7.15 (in this case we have a function available only in Drupal 7.22)

Also, I need to get XHPROF going. : )

damien tournoud’s picture

+  if ($entity_type) {
+    $info = entity_get_info($entity_type);
+    $bundles = array_keys($info['bundles']);
+
+    foreach ($bundles as $bundle) {
+      foreach (field_info_instances($entity_type, $bundle) as $field_name => $instance) {
+        $field = field_info_field($field_name);

^ This looks like a really inefficient way of doing field_info_instances($entity_type) :)

rszrama’s picture

With Drupal 7.24 being a security release, we can now ensure that all users will have the function field_info_field_map(). This was designed as a less memory intensive replacement to field_info_fields(), so if we can use it to avoid calls to field_info_fields() in any given page request, let's do it. : )

https://api.drupal.org/api/drupal/modules%21field%21field.info.inc/funct...

pjcdawkins’s picture

Issue summary: View changes
StatusFileSize
new3.89 KB

In response to #16 and #17 here's my attempt at a simpler patch. I have not touched hook_update_N() functions as the memory usage there is probably less likely to have an impact.

rszrama’s picture

Status: Needs review » Fixed

Thanks! Very timely patch - reviewed and committed.

Re: #16, I checked, and the reason I eschewed field_info_instances() in the past was the lack of field type in the return value. : P

Commit: http://drupalcode.org/project/commerce.git/commitdiff/237c28d

rszrama’s picture

Quick side note: I'm removing the explicit dependence on >= 7.22 in the .info files so sites / tests running from -dev still pass.

Status: Fixed » Needs work

The last submitted patch, 18: commerce-7283548-18.patch, failed testing.

berdir’s picture

Status: Needs work » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.