#491190: Provide a taxonomy term field type brings up a quite difficult problem. Any reference type field which references a fieldable entity (nodereference, userreference, term reference etc.) needs to load information about the objects it references in order to render information about them. In CCK D6 (and nodereference HEAD) this is done in the formatter - which means a query for every referenced node to get it's title etc.

The proposed answer to this from discussion in #111127: Cache node_load is to load the information in hook_field_load() - where it's cached. However hook_field_load() is called from field_attach_load() which happens during object loading itself (i.e. node_load_multiple()).

This leads to the potential for infinite loops - node/1 references node/2 references node/1 means node_load(1) / node_load(2) / node_load(1) etc. Even without infinite loops, it'd be quite possible to set off loading chains where node/1 is connected to 2000 other nodes indirectly or something.

So... I think the ideal option from a performance perspective would be to be able to load the referenced objects outside of entity loading itself, but before we get to the point of rendering individual entities.

For nodes, we've already got a decent spot to put this in: node_build_multiple() - http://api.drupal.org/api/function/node_build_multiple/7 - we could just add field_attach_pre_render('node', $nodes) (not fussed on the name) right at the beginning of node_build_multiple() and that would allow reference modules to do an $entity_load_multiple() on the objects referenced and add them to the field array - without risking infinite loops - since we'd only do this additional loading when we're about to view something, not when it's being loaded.

The only disadvantage here is we don't cache things in the field cache - but all the _multiple() functions are pretty cheap, and it'll keep the cache entries smaller as well. This was one of my main arguments for caching at the object level rather than the field level, but it'll work well enough with either.

Attached patch isn't much more than pseudo code, just something to look at because when writing the above it looks really abstract...

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

yched’s picture

I'm wondering if this is just making field_attach_sanitize() multiple ?

Other annoying thing is that, really, only the formatter knows what it will need to do its job. Doing a full node_load() at pre-render is pointless if you only end up displaying 'node title as link'.

So maybe what we need in fact is a new *formatter* hook instead ? (a 'multiple' one)

Frando’s picture

Just a quick note: I like the idea, sounds great and +1 to keep loading referenced entities to the formatter, as only the formatter knows what kind of data is needed for the display. So it would be the formatter that implements the new hook. I don't like the name of the hook, though - when I read the issue's title I immediately though of drupal_render()'s #pre_render property. This is confusing. I would propose hook_field_build() or hook_field_build_multiple(). This keeps it in line with the regular node process.

catch’s picture

Title: Add hook_field_pre_render() » Add multiple hook for formatters

Having the formatter implement the hook sounds great. We can just keep it in the same place and rename it. I'll have a look later for suitable places to use this for terms and users.

catch’s picture

Status: Needs work » Needs review
FileSize
3.09 KB

Let's try this. Found spots for node, user and taxonomy terms. Renamed to field_attach_formatter_load() / hook_formatter_load()

yched’s picture

Thanks for tackling this, catch. A couple remarks:

- field_attach_formatter_load() needs a $build_mode param, so we know which formatter to use.

- _field_invoke_multiple('formatter_load') will call the field-type module's hook_field_formatter_load(). Not good if the formatter is implemented by a 3rd party module.
This needs to be _field_invoke_multiple_default('formatter_load'), backed by a field_default_formatter_load() function, that:
finds the right formatter used for that build mode (I think _field_get_formatter() does that - see the 1st lines in field_default_view()),
invokes the right hook in the formatter module.
[edit: Note that _field_invoke_default('form_errors') in field_attach_form_validate() works exactly like that - except the intent is to call a widget hook instead of a formatter hook here]

- detail, but I'm not sure of the _load in the name. This happens way past the object 'load' time, so it might be confusing. Well, we can fine-tune later on if needed.

Status: Needs review » Needs work

The last submitted patch failed testing.

catch’s picture

yched. Thanks for the review. I haven't really looked at formatters in any depth so appreciate the pointers. Won't have much time today for a re-roll though.

On build mode, good point, bit useless without that, what do we put for users and terms at the moment though?

yched’s picture

"what do we put for users and terms at the moment though"
I'd say like field_attach_view(), have the $build_mode arg default to 'full'.

catch’s picture

Status: Needs work » Needs review
FileSize
4.86 KB

OK here's a re-roll.

Has some notices because the $instance passed to field_default_formatter_load() is an array for some reason. Needs a bit of cleanup as well, but on its way.

Status: Needs review » Needs work

The last submitted patch failed testing.

catch’s picture

Status: Needs work » Needs review
FileSize
6.21 KB

OK figured out what's going on - since instances are per-bundle, you need to first work out which formatter is being used for which bundle, then pass objects to the hook per-formatter rather than all at once.

This patch should pass tests, field_default_formatter_load() is looking a bit ropey now though, and docs could use some help. Hoping for some more feedback from yched. The hook currently takes $objects and $formatter - we may need more parameters to make that work.

yched’s picture

Thanks catch.
Fixed a few glitches in field_default_formatter_load() ($instance only contains the name of the formatter type, we need to load the formatter-type info + backup to default formatter for the field type if the original formatter happens to be unavailabe).

The logic implied by 'multiple' is really hairy in there :-(. Attaching what I have so far, will revisit shortly.

Also, we probably need to implement an actual use case to fine tune the list of parameters we need in hook_field_formatter_load().

yched’s picture

Additionally: I'm not too happy that we add a new field_attach_[op]() to the list of functions that entites need to call in order to be 'fieldable'.
So far those functions are bound to common workflow steps that make general sense in a reasonable workflow for any entity, fieldable or not: f_a_[op] where [op] is load, submit, update, view... This new f_a_formatter_load(), OTOH, doesn't really make sense if you don't know the guts of Field API.

Strictly speaking, this step should be hardcoded within field_attach_view(). But: we want a 'multiple' op, and field_attach_view() is single :-(
Not sure what's the cost of making field_attach_view() multiple...
It would make field_default_view() a little more complex, but that's internal. Not sure how it would impact node.module's code for frontpage listing, for instance.

Or maybe it's just a question of finding the right name for that new op: f_a_prepare_view() or something...

yched’s picture

Updated patch.

I figured out the logic in field_default_formatter_load() a little bit better, by comparing with hook_widget() and hook_field_load().

In fact, we need to group objects, instances, items par formatter-module. Then we call hook_field_formatter_load() on each module, with the relevant objects, instances and items.
hook_field_formatter_load() works by altering the $items, just like hook_field_load().

Now we need to find the best way to test this :-), and validate by actual implementations in followup patches.
We also need to solve #13 (at least settle on a name)

Status: Needs review » Needs work

The last submitted patch failed testing.

catch’s picture

Patch is looking good apart from those exceptions.

On #13 - I'm fine with field_attach_prepare_view() as a name, can't think of anything better. I'm also fine with merging this with field_attach_view() if it's possible - but at the moment we're calling these in different functions in node.module - so IMO that's something to look at in a (possibly critical) followup patch.

On testing - we have coverage for most of the code - as evidenced by all the test failures each time we change something. I think we're better off swapping taxonomy_field_load() out to taxonomy_formatter_load() in the terms as fields patch if this gets in (or in this patch if that gets in first), and then this would be included in the tests there in an actual use case - no point doing it in a dummy module if we have a real one. Could also convert nodereference to use it (side note - any discussion on where nodereference and userreference live in D7?).

yched’s picture

On testing - well, strictly speaking, all other field_attach_*() are explicitly tested in field.test (against dummy test entities and field types defined in field_test.module), so this one should be as well.
I'll try to look into the exceptions later today.

yched’s picture

Status: Needs work » Needs review
FileSize
7.81 KB

Attached patch fixes the test exceptions and renames to _prepare_view().
CNR for the bot, but still needs tests.

Status: Needs review » Needs work

The last submitted patch failed testing.

yched’s picture

"Failed to run tests" - what does this mean ?

yched’s picture

Status: Needs work » Needs review
FileSize
17.61 KB

With test for hook_field_formatter_prepare_view().
This should be ready now.

Status: Needs review » Needs work

The last submitted patch failed testing.

catch’s picture

I'm not around much until next week but all of yched's changes look great - only briefly skimmed over the test changes but they seem pretty consistent with what's already there, so should be RTBC once the single test fail is gone.

yched’s picture

Status: Needs work » Needs review
FileSize
17.25 KB

Should fix the failure.

catch’s picture

Had another read through and this looks great to me, not sure if I can mark RTBC since I was involved in the issue quite a bit.

yched’s picture

Status: Needs review » Reviewed & tested by the community

If we both agree on the code, let's be bold and set this to RTBC.

Status: Reviewed & tested by the community » Needs work

The last submitted patch failed testing.

bjaspan’s picture

Doh!

yched’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
14.8 KB

Straight reroll. The failed hunks were only whitespace fixes in user.module, automatically added by my IDE.
I removed them.

[edit: actually they were removed today in http://drupal.org/cvs?commit=240510, hence the fail here]

moshe weitzman’s picture

Issue tags: +Fields in Core

Status: Reviewed & tested by the community » Needs work

The last submitted patch failed testing.

yched’s picture

Rerolled.

yched’s picture

Status: Needs work » Needs review
FileSize
13.14 KB

- removed unneeded code to figure out the actual formatter used for the build mode in field_default_prepare_view(), now that #535034: Clean-up how fields and instances are prepared for runtime. fills that in cached field_info_instances() for us.
- added field_attach_prepare_view() call to comment_build_multiple() (like was previously done for node_build_multiple).

Which outlines an issue: direct calls to [comment|node]_build($single_object) do not trigger field_attach_prepare_view().
[comment|node]_build_multiple() should probably set an $object->_prepared = TRUE flag after calling field_attach_prepare_view($objects). Then, [comment|node]_build() calls field_attach_prepare_view($single_object) if empty($object->_prepared).

No time for this tonight, but feedback welcome.

Leaving to CNR for the bot.

Status: Needs review » Needs work

The last submitted patch failed testing.

yched’s picture

Status: Needs work » Reviewed & tested by the community

Rerolled. Back to RTBC where it was in #31.

yched’s picture

Patch :-p

Status: Reviewed & tested by the community » Needs work

The last submitted patch failed testing.

yched’s picture

Status: Needs work » Needs review
FileSize
14.39 KB

Rerolled, + should fix tests.

Status: Needs review » Needs work

The last submitted patch failed testing.

webchick’s picture

Status: Needs work » Needs review

Oh, testing bot...

catch’s picture

Priority: Normal » Critical
Status: Needs review » Reviewed & tested by the community

Back to RTBC. I'm also marking this critical, since it's the only way we can have different formatters for reference fields without either caching inconsistencies or performance issues, and it needs to go in before freeze.

Status: Reviewed & tested by the community » Needs work

The last submitted patch failed testing.

catch’s picture

Status: Needs work » Needs review
FileSize
13.67 KB

Straight re-roll for the test bot.

This is needed in core to replace taxonomy_field_load(). It will also be necessary for node reference, user reference (and any other reference field) in contrib - to avoid either infinite recursion in hook_field_load() or lots of little direct queries in formatters themselves.

mattyoung’s picture

.

Status: Needs review » Needs work

The last submitted patch failed testing.

catch’s picture

Status: Needs work » Needs review
FileSize
13.73 KB

Should fail less spectacularly this time.

catch’s picture

FileSize
17.92 KB

Now with taxonomy_field_load() converted to the new hook, added field_attach_prepare_view() to comment.module since that's fieldable now, used taxonomy_field_formatter_prepare_view() in field.api.php for the example, where there was previously none.

This is ready for review now. taxonomy_field_formatter_prepare_view() can be optimized to use taxonomy_term_load_multiple() directly now (to take advantage of things like entity caching) - but that's hopefully a post-freeze task, whereas this API change isn't.

catch’s picture

For issue summary, my original post, and yched and frando's followups in #1 and #2 still describe why this is needed.

This was RTBC for a couple of weeks back in August, all I've done is updated it for translatable fields, and apply it to the main intended use-case in core which is the taxonomy field.

moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community

This all looks proper to me, and yched/frando have already positively reviewed.

Status: Reviewed & tested by the community » Needs work

The last submitted patch failed testing.

catch’s picture

Status: Needs work » Needs review
FileSize
17.92 KB

Hm.

bjaspan’s picture

Status: Needs review » Needs work

I have a concern about field_default_prepare_view():

* It builds $modules into an array of formatter modules, one module per $instance, and then groups objects, instances, and items by module. Then, it calls $module_field_formatter_prepare_view() for every module in $module.

* If $instances contain multiple instances using formatters from the same module, won't $module_field_formatter_prepare_view() be called multiple times instead of a single time?

* I think this can be fixed with "$modules[$module] = $module".

field_attach_prepare_view() is documented to add additional data items to field data items. In the test case, nothing *explicitly* tests that 'additional_formatter_value' gets added to the field's data; it only implicitly tests it by checking the themed output. I'd prefer an explicit test, though this isn't a biggie.

If I'm wrong about the $modules bug, or once it is fixed, I'll RTBC this.

catch’s picture

Status: Needs work » Needs review

I went for $modules[$module] since I'd also had a nagging feeling similar to bjaspan's when looking at earlier revisions of this patch, but I've not verified it either way, however it's a tiny change which does no harm.

The explicit test I'm hoping we can do after code freeze panic.

catch’s picture

FileSize
17.93 KB
bjaspan’s picture

Status: Needs review » Reviewed & tested by the community
catch’s picture

This is now required for RDF.

Pasting from an multi-recipient e-mail from Scor (on the assumption he doesn't mind):

- we are struggling with taxonomy RDF integration because hook_taxonomy_term_load() not invoked during a node view (it seems the terms are cached somewhere).

That's due to the direct query hack here to avoid infinite recursion - which can be quickly fixed as a bug fix once this is in.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Hm. I can see the utility of this hook. I'm a little worried about the name which associates it with a 'view' operation, rather than a 'load' operation, and load comes up at times when view is not involved, which will probably be a DX WTF. However, since this has the thumbs-up from all of the Field API maintainers, and since I can't think of anything better, I can't really justify holding up commit over this.

Catch gave me a documentation snippet on IRC to add to hook_field_load() to help other modules not get into this infinite recursion situation. Added that and committed to HEAD!

yched’s picture

Status: Fixed » Active

Sorry for not chiming back earlier.
In #33 I outlined :
"direct calls to [comment|node]_build($single_object) do not trigger field_attach_prepare_view().
[comment|node]_build_multiple() should probably set an $object->_prepared = TRUE flag after calling field_attach_prepare_view($objects). Then, [comment|node]_build() calls field_attach_prepare_view($single_object) if empty($object->_prepared)."

(which also means I should probably not have marked RTBC 10 days later - er... :-/)

Not easy for me to check wheter the committed patch adressed that concern, but turning back to active

yched’s picture

Marked #632188: Notices on comment preview when node has terms as duplicate of "direct calls to [comment|node]_build($single_object) do not trigger field_attach_prepare_view()"

moshe weitzman’s picture

this issue is deprecated by recent formatter as array patch?

catch’s picture

I don't think so - we need the field_attach_prepare_view() calls to be put into the singular node_build_content() as well as node_build_multiple(), but also to do a similar thing to the entity recursion patch with $object->field_prepared = TRUE so they only get called once.

yched’s picture

Status: Active » Fixed

No, not deprecated. the formatters as array patch builds on this hook. This issue was kept open to address "direct calls to [comment|node]_build($single_object) do not trigger field_attach_prepare_view()." See #58.

However, that's best as a separate followup thread. I opened #661494: direct calls to node_view() do not trigger f_a_prepare_view().
Setting this one to 'fixed'

Status: Fixed » Closed (fixed)
Issue tags: -Performance, -Fields in Core

Automatically closed -- issue fixed for 2 weeks with no activity.