Prevent EntityReference from loading all options when the options will be limited

Looks like an interesting approach, but code would need some comments to explain how it works in order to be maintainable.

Any update on this? my node form takes for every to load with +20000 entities referenced by entity reference field which is limited by another field in the same node form by this module.

It seems that the entity reference field is loading all options although I checked limit options checkbox.

I can't even edit the field from the manage fields page of the content type and I get this error about memory limit

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 102 bytes) in /path/to/mysite/includes/database/query.inc on line 1855 

EDIT: the patch is working great for me except for some php warnings, see the the recent log messages:

Location	http://mysite.com/en/system/ajax
Referrer	http://mysite/en/node/add/type
Message	Warning: Invalid argument supplied for foreach() in _reference_option_limit_get_entityreference_values() (line 642 of /path/to/mysite/sites/all/modules/reference_option_limit/reference_option_limit.module).
Severity	warning

UPDATE: I don't know what exactly happened, but I can't submit the form any more. I get this message:

An illegal choice has been detected. Please contact the site administrator.

after submitting the form, and the entity reference field is surrounded by red, and the only choice avaliable inside it is "None"

Please help!

Just started using this module and it does exactly what I need although I did notice some loading when its going through the options. This is on a test site and I'd imagine if would get worse once I have 200+ enteries.

I can donate some $$ to anyone who can make this patch work.

I too can donate some money to that.

Thanks for the patch!

Not sure if this will help anyone, but I made a slight change to the patched code to address an 'invalid argument supplied foreach' warning I was getting. Will have to manually replace the modified lines to use.

/**
 * Helper function to abstract the values of an entity reference field.
 */
function _reference_option_limit_get_entityreference_values($field_name, $entity, $check_values = array()) {
  $values = array();
  if (isset($check_values[$field_name])) {
    $process = $check_values[$field_name];
  }
  elseif (isset($entity->$field_name)) {
    $process = $entity->$field_name;
  }

  // Changed foreach from foreach($array => $value)
  // to foreach($array as $key => $value)
  if (isset($process) && is_array($process)) {
    if (!empty($process[LANGUAGE_NONE])) {
      foreach($process as $lang => $target) {
        if (!empty($target)) {
          $values[] = $target;
        }
      }
    }
  }

  return $values;
}

+ $form['instance']['options_limit_fieldset']['options_optomize_entityreferences'] = array(

typo: 'optimize'

Could we have some comments here to explain briefly how this functionality works, with a @see to the alter hook?

+ '#description' => t('Prevent entityreference from loading all possible entities. May not work for all configurations; use with care.'),

Which configurations won't it work with?

+ * Implements hook_query_TAG_alter().

Explain briefly how the code flow gets here -- whose query are we altering? A @see to where the query building takes place would be good.

+ if (!empty($instance['options_limit']) && !empty($instance['options_limit_fields']) && !empty($instance['options_optomize_entityreferences']) && empty($current_query_conditions['entity_id'])) {

Argh! Unreadable. Split this up into bail conditions: the first two together, probably.

Also, what's the last condition about? Comments!!!

+ $empty = !empty($field['options_limit_empty_behaviour']);

Variable needs a better name.

$empty = empty() just looks evil!

+ if (_reference_option_limit_get_entityreference_values($field_name_matching, $entity, $_POST) || !empty($instance['options_limit_empty_behaviour'])) {

No idea what's going on here.

+function _reference_option_limit_get_entityreference_values($field_name, $entity, $check_values = array()) {

Needs param docs.

+ $process = $check_values[$field_name];

What does $process mean? Is it a boolean to indicate we should do stuff? Or something else?

+ return $values;

We're returning an array, but the caller only uses us in a boolean context. WTF?

It's not that I'm unwilling to work on this, it's that I don't feel I can. This is a hunk of code that's been contributed by someone else, but it's complicated and woefully lacking in explanations of how it operates. I can't commit it to the module, no matter how many people say it works for them, because that would create a situation where the module that I am responsible for contains code that I *don't understand and have no hope of understanding*.

Perhaps someone can contact @hefox and ask for further clarification? Otherwise, I don't see this issue advancing at all.

How about a different approach? for example you could make the entity reference field hidden and it will show up only when the controller field has any value but "None", I don't mean to hide it by rules or conditional fields, because I have tried this with no luck.

What I mean is to completely unset it from the form while being built, and set it again with the change of the controlling field.

This I could easily do by PHP & JQuery outside Drupal.

Greetings hefox,

I'm trying to figure this out by any way. What do you mean by limiting the selection with view? could you please elaborate!

I got some ideas from this, I'll check it out and bring a patch.

//Josef

Here's a refactored, simplified version of hefox's approach, removing unneccessary conditions and adding default field and value support.

Tested with a sample of 6000 limiting nodes and 10 matching. On my test setup, enabling "Hide all options" and "Optimization" made a significant impact on loading the form and added a ~300% speedup on AJAX calls. I'm not sure how comprehensive this is, but you can always shut it off if it doesn't work :)

//Josef

Over and out.

Greetings Josef,

Thank you for your patch and time! Here is what I did:
I Applied your patch against the latest dev version.
Checked the optimize check box in the field UI page.

Everything just went fine in the node form except when I try to upload an image and hit the upload button, I get the "An illegal choice has been detected. Please contact the site administrator." error message.

And I can't submit the form

And from the logs, I get these:

Location	http://mysite/en/file/ajax/field_my_photo/und/0/form-e_-dvuVqIMcUlHoBROp2ihwXOQme1V2_B7arWjO_bo4
Referrer	http://mysite/en/node/add/my-type
Message	Illegal choice 37596 in my-field element.
Severity	error

Location	http://mysite/en/system/ajax
Referrer	http://mysite/en/node/add/my-type
Message	Notice: Undefined index: #ajax in ajax_form_callback() (line 379 of /path/to/mysite/includes/ajax.inc).
Severity	notice

UPDATE;
I can submit the form if I don't upload photos by clicking the upload button, and by disabling the Entity Reference Autofill module.

However I can't remove or add some photos after that in the node edit form, I get the same Illegal choice message.

Ill test this today when I gets some free time to install the patch.

The patch in #14 is not getting along with other AJAX fields - I suspect it has something to do with the guard conditions in reference_option_limit_form_alter():

  // If we are on an AJAX call, check that it's one that concerns us.
  if (!empty($form_state['triggering_element'])) {
    // ...
  }

If you're not using other AJAX stuff in your module, it should be safe to enable.

I've applied the patch but I guess I wont know how well it'll work untill I get all my content in. I'm not using any file handling fields that might require ajax so can't test for that. If we get that sorted then I can make a donation :)

Turns out options are rebuilt on irrelevant AJAX calls too, which causes validation to fail if we return before the options are built in form_alter(). Fortunately, as the previous options are available in the form state, we can copy the options from there.

From the first test, everything seems to be working as expected now without and with the Entity Reference Autofill module. I will do more tests and report back.

Thank you Josef! you saved my life.

I can confirm now after heavy testing that the patch is working perfectly.
Thank you again Josef!

Is this going to be committed soon?

I'd like to have #2244079: add tests for entityreference fields before I make more changes to the code.

I don't understand! What is required there?

As a maintainer, I have a responsibility to ensure that nothing gets broken by new commits. Automated testing is one way to help with this. The patch here changes a lot of things, and is big and scary, and I'm doing this in my free time on top of a ton of work and family responsibilities. I'd like to have the reassurance that automated tests would give me before I commit this patch.

My problem is I don't understand what automated test are. So if you need me to do tests please tell me how.
Thanks!

Actually, the patch doesn't change much at all. Only actual changes are some added "bail" logic in reference_option_limit_form_alter(). Every other addition is guarded by the new "optimize_entityreferences" setting. Be that as it may, I can see why you want tests before commiting a large patch.

What are tests?

//Josef

Patch #21 work fine with entity reference field, but I doesn't work with taxonomy term reference fields in node form, and views exposed filters.

limiting the option loaded in "limited" field is indeed really important. On a form where a have this kind of field enable...it load all the options doing a node_access query for each of them, resulting in a 30 sec php timeout. I first played a view to limit them myself and it was really fast but it's not working anymore with the new api version. This is a major hangup as timeouts prevents important site functionality to work. Of course, as admin...no problem...will give the patch a test...with entity reference.

Well this module now has a ton of tests that comprehensively cover its behaviour, so that's the testing side of things done. What this patch here needs is a reroll.

On second thoughts, tests that this new feature works would be good too.

Could possibly be done with a test module that implements hook_entity_load(), and tracks what was loaded.

what version of the code this patch have been worked on? It was submitted a year ago, but the code changed a lot since then. I'm having trouble applying the patch...I went and did manually, but I get ajax errors when selecting values in the form...I guess it might be related to the code version. Also, I don't see any new options in the field ui...should there be some?
Another thing...it works when logged as admin....but I still get that 30sec timeout when logged as another user.

Yes, the patch has become out of date and it needs to be rerolled.

I'm afraid this module had a lot of patches at about the same time, so it was inevitable that there would be clashes.

would love to help, but seems like I can't make it to work even on the old file I have....keep getting parse error with just the first few lines..even if it's look right in the file. Trying to understand the code as well so...
Josef would probably be the best to reroll...or at least, someone that knows the new code enough.

does anyone happen to have a working version of the module that include this patch? I'm looking for a working starting point, as I can't seem to get the very basic to work. Showing the new checkbox in the fields setting work...but that...even a line feed in the code breaks it...get some php warning like unexpected character in input: '....something is not right..

I don't have a patched version of this atm, but judging from the date of the patch, it should be written for this version.

Also, I will not have time to re-roll this in the foreseeable future.

What I'm doing right now is to update instance settings:

    [widget] => Array
        (
            [type] => options_select
            [weight] => 29
            [settings] => Array
                (
                    [skip_options_list] => 1
                )

            [module] => options
        )

Then in entityreference.module:

function entityreference_options_list($field, $instance = NULL, $entity_type = NULL, $entity = NULL) {
  if (!empty($instance['widget']['settings']['skip_options_list']) && $instance['widget']['settings']['skip_options_list']) {
    return array();
  }
...

For me it works as I update all fields which I don't want to auto load all options.
Please let me know if that makes sense and maybe it could be an easy solution.

DO NOT USE!

I took a shot at rerolling this. I'm not entirely sure about the incredibly lazy approach I took to rerolling this though - it works at preventing horrible load times on ~10,000 entities (at least on top of the patch stack I'm using? only briefly tested it alone) but if I'm understanding things right, it could be further optimized to avoid redundant queries and entity rebuilds?

Also, I feel like to be complete this could be extended to do something about the field on the manage fields UI too.

DO NOT USE!

My previous patch contains a nasty logic error. It needs more cleanup than this really, but here's a fixed version at least.

Also rerolled against HEAD while I was at it. (#43 is against #1986532-24: Support organic groups OG reference widget and so is the interdiff).

The lazy approach I was trying to use can have some really bad consequences when inside an IEF, and there's nothing to prevent you from trying to set things up like that in that patch. I'll reroll this at some point, but the solution that avoids nasty weird things happening involves code from #1989262-21: Prevent EntityReference from loading all options when the options will be limited and #2589699-5: Doesn't work with inline entity form widget?.

is anyone able to make this work with 1.6 and recent update entity and core?

This should not even be listed as a feature, but more as a bug. If you have just a few items in the list it works, but as soon as you hame a bigger number, it doesn't.

just tried patch #21 and it looks like it might need a re-roll
(tried manually as well with no success)

right now if i enable the module and try to do a "simple" filtering i always end up with :
Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 4096 bytes) in ...\includes\database\database.inc on line 2227

I also tried to apply the patch in #21 but it failed too, it would be great if someone could reroll it!
I allow myself to raise the priority and change the category of this issue because it is pretty important for performance.