Also see #1346204: [meta] Drupal 8 Entity API improvements and #1540656: [META] Entity Serialization API for web services (e.g. content staging).

Overview

This is the meta-issue for adding an entity property API/entity field API as discussed during the WSCCI Web Services Format Sprint (report) and first proposed at the core conversations in London. In huge parts its about decoupling Field API systems and taking them to entity properties - so one could call it also “re-factor Field API”.

The goal is to provide a unified and simple API for working with entity properties - any non-property data on an entity is not supported in any way (except for IDs). For improved DX, properties should become classed objects and provide a sane default handling for language.

Planned features:

  • Properties become classed objects and include what is currently fields plus what is currently raw entity data (e.g., {node}.uid).
  • Properties can define custom classes for improving DX by adding further suiting methods.
  • Introspection capabilities are built in the Entity and Property objects, i.e. you call a method to get back the definitions of the contained properties. Make introspection available without any concrete data as well (i.e. look up the properties of a certain entity type, bundle, field, ..)
  • Handle entity references in a meaningful way.
  • Support computed properties, which store no data but provide a mechanism to lookup the property value. I.e. add support for entity back-references that can be looked up that way.
  • Entity / Property based validation of values and access checks.

Note: The initial name "Entity Property API" got changed to "Entity Field API".

This can serve as foundation for

  • making every entity property/field translatable (d8mi)
  • the WSCCI Web Services format implementation for entities
  • content staging
  • adding semantic mappings
  • further entity-api improvements, e.g. sub-systems for rendering or editing entities.
  • contrib developments like Rules or Search API which can work solely based upon property definitions

Roadmap

Step1: Implement the base: Property objects and interfaces, introspection

Step2: Convert all entities to make use of the new API
#1818580: [Meta] Convert all entity types to the new Entity Field API:

Also, for the conversion to be complete we need to cover:

Step 3: Implement entity-level APIs based upon field-level APIs

Step 4: Refactor field API

Once step 4 is completed, field and properties should be the same API-wise. Morever, it would be nice to be able to use field API components with entity fields also:

Bonus step: Port field API components to entity fields

Misc related issues (without any particular order):

Let’s use this issue for high-level architecture discussion and update the roadmap accordingly.

Issues

Related issues are tagged with Entity Field API.

Comments

Issue summary:View changes

Updated issue summary.

Certainly a point to discuss is the difference between properties and fields. Here are my thoughts:

  • Properties are purely code-driven, thus provided by modules. They don’t necessarily appear in the UI (e.g. manage display) at all and are just defined in the info hook.
  • Fields are user-configurable, driven by configuration. But fields are properties too, e.g. field module should probably expose property information for all fields, register display/form components...
  • Storage. Is there a difference in storage? In the beginning, entity property storage has to be managed by modules “manually”. Later though, we might support automatic handling of property storage just as for fields (limited to certain entity-storage-controllers).

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Issue tags:+D8MI

Widgets would be vital for property translation. Looking forward to participate in the property translatability aspects and in ways to get there.

Subscribed, obviously - although, as I announced in a couple places, I won't be able to be deeply involved in this task in the D8 cycle.

Some roadblocks that come to my mind about having widgets and formatters on non-fields :
(for the sake of the discussion, I'll use the term 'renderers' to avoid repeating 'widgets and formatters') :

- Which applies where : in the current model, a renderer declares on which field types it applies.
This basically means "which kind of value structures (and set of 'columns') I know I can handle".
Obviously, this also lets us populate options in the "select widget" and "select formatter" dropdowns in the UI.
If we move to "renderers operate on property", we need a notion of "data type" for the properties. I'm not fully sure how this relates to / is different from our current 'field types'

- Consequently, the "values" that the renderer must render needs to be structured the same whether they are a field value or a property value. The answer to this possibly lies in "Field and property values as classed objects with a common interface".
Although : note that renderers can currently chose to operate on "the set of values of a multiple field as a whole" instead of on "each single value independently".

At this point, I'm not sure whether this really differs from "we basically made all properties as fields, with the incurring overhead" :-). OK, properties are out of field storage.

- In their current APIs, renderers receive $field and $instance definitions, so that they can adjust their behavior depending on (for example) a field type, some specific field or or instance settings.
It might just be that the corresponding hooks (or hopefully methods at some point in D8) must be prepared to receive NULL for those.

Also, FYI : I plan to open a meta issue on "Object oriented Field API" when I get a moment. "Widgets as classes" would be a good place to start, and it seems that WSCCI plugins are coming into shape.

Thanks for the feedback. I'm looking forward to "Object oriented Field API" meta issue. :)

If we move to "renderers operate on property", we need a notion of "data type" for the properties. I'm not fully sure how this relates to / is different from our current 'field types'

I agree that we need a notion of 'data type'. In the beginning I thought about having a list of pre-defined data types like the entity api module already does + all field-types as valid data types. Details of that should probably discussed over at #1346220: Add entity property metadata. In the long-term a general data-type hook might make much sense though, so that data-type <=> field-type.

Consequently, the "values" that the renderer must render needs to be structured the same whether they are a field value or a property value. The answer to this possibly lies in "Field and property values as classed objects with a common interface".

I'm not sure whether we need classed objects here - I'd prefer to avoid unnecessary overhead. I strongly agree about the defined structure of data values, i.e. values of a certain type have to be structured in the defined way. However, I think documenting this defined structure should suffice. E.g. "dates have to represented as timestamp." - so every module can rely on that. That's how entity API module in d7 works. Of course that would not mean we cannot use classes for certain data types - we might want to go for "classed representations" of certain data values. But having even simple property values like strings or numbers represented with classed objects does not look good to me.

In their current APIs, renderers receive $field and $instance definitions, so that they can adjust their behavior depending on (for example) a field type, some specific field or or instance settings.
It might just be that the corresponding hooks (or hopefully methods at some point in D8) must be prepared to receive NULL for those.

Good question. Maybe the property info might contain equivalent amount of information at some point. Though I think those renderers should be able usable without entities at all, thus work just with "data property information" and render that data.
So maybe, it would make sense to have a separate entity-aware render-layer for that. E.g. having different renderers like different form element types + different formatters like different widgets.

I have to throw a small wrench in here. Sorry. :-)

Are we intending properties to be used for metadata, or just for data? Because right now we conflate the two, which is becoming an issue in some cases, especially moderation.

The hand-wavy distinction I'd make is that fields are intrinsic to the entity object. Metadata is extrinsic. That is, it is not part of the essence of "being" that object. That means that, for example, it should not be syndicated along with the node if it is transferred to another site (at least by default), and changing it should not (generally) trigger a new revision to be made (and thus invoke moderation issues, such as with Workbench).

Examples:

- Is changing a node's "promote to front page" or "sticky" status really a new revision? No, those are how the node is organized within the site, not the definition of the node itself.
- Is a node's presence as part of an OG really intrinsic to the object? Should changing its OG membership trigger a new revision? I'd argue not.
- Should flagging or unflagging a node create a new revision? Currently it does not, because flag implements its own non-field storage.
- Everything I just said about flag applies to nodequeue, too. :-)
- Is changing a node's position within a menu structure a new revision? Right now it sometimes is and sometimes is not, depending on whether you change it from the node edit form or the menu admin screen. Yeesh.

I'd argue that a differentiation between data and metadata is more important than a distinction between defining-module-originated fields and user-originated fields.

@Crell: that does sound like a wonderul model for translations too :) That would mean though that author, published, etc become fields, which makes access control field based in core, etc. (i was advocating making those properties fields before with no success so far).

Well, arguably author could be per-revision, and published is just weird. :-) Some fields might be a gray area as to whether they are data or metadata, like taxonomy terms. Access control... yeah, talk to agentrickard. He's got all sorts of unpleasant things to say about revisions and access control. :-)

> In short, having those APIs splitted out and separated allows re-using them without having to use fields. So for example a module could provide a custom stored entity property, but make use of translation capabilities and existing form widgets and display components

That is sort of possible at the moment isn't it?
I toyed with the idea of writing a field storage provider module that would store field data as columns on the entity's table -- provided they are single-valued fields only, of course.

On another note, I think one thing we need as part of this is a sort of hook_schema() that describes the properties found on objects.

Thanks for the feedback Larry!

Are we intending properties to be used for metadata, or just for data? Because right now we conflate the two, which is becoming an issue in some cases, especially moderation.

The proposal does not change how we would make use of entity property's data, thus I'd not say it's per se metadata or data. The usage of an entity property totally depends on the implementing module.

The hand-wavy distinction I'd make is that fields are intrinsic to the entity object. Metadata is extrinsic.

I don't think your hand-wavy distinction of fields <-> data, properties <-> metadata gets it. In the end being a field or not is an implementation detail, which should not affect the behavior of the data. I'd argue that module created fields should be able to act as both.

Still, your comment raises valid concerns in particular to revision-handling. Imagine a workflow-state field (published, needs-review, draft,...) - should it be revisionable? I'd say there are use-cases for both ways, what would lead us to the feature-request of "Make properties/fields optionally revisionable.".

Metadata is extrinsic. That is, it is not part of the essence of "being" that object.

So do you say "metadata" like node publishing options should not be part of the object at all? I'm not so sure about that, e.g. I'd not argue that metadata like "is this node published to the frontpage" is per-se extrinsic to the node.

@Examples:
Yep, I'd agree that bookmark-like flags or nodequeues are and should be separate to the entity.
Then, I'd argue that a node being sticky or not, is a property of a node as the language suggests - thus it should be part of the object. Consequently, one could argue that a global flag should be a property too as being flagged or not is a property of the node. Indeed, this would make sense to me.

In the end this boils down to regular data modeling questions - should something be part of an object or not? Anyway, I'd appreciate a better separation of metadata and content of nodes too, i.e. in particular UI wise. That doesn't mean the data has to be extrinsic though, it could be separated in-side the object too.

Spinning the wheel further, I think DX-wise it makes much sense to expose extrinsic data similar to regular entity properties, i.e. make the data accessible in a simple way.

I'd argue that a differentiation between data and metadata is more important than a distinction between defining-module-originated fields and user-originated fields.

I do not think we have to choose between either of those.
Anyway, this issue is not about creating a distinction between module-originated fields and user-originated fields, it's about unification. It's about making fields and properties more equal so developers using the data don't have to care whether it's a field or not - thus the same APIs can be applied to both. Just what remains is the distinction of module-originated fields and user-originated fields.

I toyed with the idea of writing a field storage provider module that would store field data as columns on the entity's table -- provided they are single-valued fields only, of course

Yep, but that shouldn't be necessary. You should be able to make use of a display component or a widget just with $data that matches display component's or widget's requirements.

On another note, I think one thing we need as part of this is a sort of hook_schema() that describes the properties found on objects.

Yep, see #1346220: Add entity property metadata.

@fago : no issue yet, but I have some working code for "Widgets as (WSCCI) Plugins" in the wscci-plugins-fieldapi branch in the WSCCI sandbox. Direct port of the current shape of the widget API for now (tightly relying on $fields and $instances).

See also http://groups.drupal.org/node/195618.

Issue tags:+Entity system

If this unification is accomplished, are fields still needed? or can the more complex fields be refactored as entities?

Issue tags:-D8MI+D8MI-meta

Move to new D8MI-meta tag.

Might be interesting to investigate using annotations for metadata, like Doctrine 2 does.
We could even borrow their parser class.
http://www.doctrine-project.org/docs/common/2.2/en/reference/annotations...

Example from their docs:

<?php
class Product
{
   
/** @Id @Column(type="integer") @GeneratedValue */
   
protected $id;
   
/** @Column(type="string") */
   
protected $name;
}
?>

EDIT: Updated docs link (from doctrine 2.0 to doctrine 2.2)

@bojanz

Other Frameworks are starting to implement this idea as well. My previous exposure to this idea has been with a method of developing ASP.NET MVC called Code First:

http://blogs.msdn.com/b/efdesign/archive/2010/03/30/data-annotations-in-...

It allows for creating the model of your entity in code then allow internal systems to generate the proper tables. Most notably is the use data annotations to help the model generators create the proper structure.

Some of the kind of data annotations they use are
data type annotations : declare the data type of the column
data relationship annotations : declare parent -> child relationship between objects
UI Hint : declare a field level template that should be used when rendering
data validation annotations : declare validation rules (yeeaaaa, in the model. Not always a good thing)

Full list:
http://msdn.microsoft.com/en-us/library/system.componentmodel.dataannota...

To ease with module created fields, I created a ModuleField module for D7 and opened #1426804: Improve API for module-created fields (with and without instances) for D8.

While I agree that anything user-created should be a field and not a property, I think we should support both module-created fields and module-created properties. I think of properties as needing to be very light-weight, and therein lies their challenge. Almost every column in the node table ideally should be translatable, and if a property API is the way to do that, then we need to be ruthless in optimizing this (e.g., maybe they need to stay as columns in the base table). If that creates limitations on what can be done with properties, then so be it: for anything that wants something more, use fields, which is why I think module-created fields will remain needed.

Re #15:

If this unification is accomplished, are fields still needed? or can the more complex fields be refactored as entities?

IMO, while having a special entity type for a collection of fields, as in the Field Collection module, is very cool, I think a single field remains a different concept than a property or entity. A single field can be (and often is) multi-valued and/or multi-columned, whereas I think we'll want to restrict properties to single-valued and single-columned.

@effulgentsia I agree that Field API provides things that are unique beyond what the Entity API will provide, but currently it also provides a lot of things that are overlap. Maybe I can make my point with an example. Let's say that we have two entity types in our system: Person and Address. Now we would like to enter addresses to a person's information. What I envision doing at this moment would be to let the Field API know (or it might be more appropriately named Relation Management API) that Addresses are related to a Person, and that I would like that capability of entering addresses in a person's create/edit form. So what I am proposing is refactoring the Field API in a way that anything that Entity API can do, we let Entity API do it, and everything else can be part of the Field API. The benefit of this would be that any further improvements to the Entity API will be automatically available to the Field API, any entity would be able to be treated as a field, and it would unify how developers deal with entities and fields (well in the proposed case they would all be entities). I created an issue to get some conversation going about this idea: #1412620: Make Fields Entities

So what I am proposing is refactoring the Field API in a way that anything that Entity API can do, we let Entity API do it, and everything else can be part of the Field API.

That's what the issue is about. It's about implementing stuff for all properties that previously was available to fields only.

I think of properties as needing to be very light-weight, and therein lies their challenge. Almost every column in the node table ideally should be translatable, and if a property API is the way to do that, then we need to be ruthless in optimizing this (e.g., maybe they need to stay as columns in the base table).

Agreed. The idea is that the API just defines an "interface" (yes, an API :D). Thus, it's the job of the storage controller to efficiently store and load translatable properties, what we define is just how those can be accessed/updated. That way we do not put any bloat on properties in general. If you don't want translatable properties, don't implement and go with your light-weight controller.

IMO, while having a special entity type for a collection of fields, as in the Field Collection module, is very cool, I think a single field remains a different concept than a property or entity. A single field can be (and often is) multi-valued and/or multi-columned, whereas I think we'll want to restrict properties to single-valued and single-columned.

I agree with the former, but disagree with the latter. I don't think we need to restrict properties in any way to keep them efficient, we can just leave it to the implementation. Forcing developers to provide a field type if they just want to provide some additional data sounds awkward to me, as there would be UI implications. Instead, we should move the storage-related API out of the field module and let the field module care about the differentiation only: the UI.

Issue summary:View changes

Updated issue summary.

Ok, so http://groups.drupal.org/node/237443 outlines the plan to get started with this.

the entity-property-fago branch in fago's sandbox contains some starting code (the interfaces), the plan is to get them in together with a first implementation for the test entity. Will open an issue for this.

Implementation will happen in the sandbox over at http://drupal.org/sandbox/fago/1497344, so details should be figured out in issues there. Partly, some issues are already there which need to be updated, others need to be created. Also, we'll need to update this issue's summary.

Then, once we've a first implementation we'll post it to the core queue. That's the process which has been already successfully used by wscci and others already, so let's follow it.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

updated the roadmap based upon latest wscci plans

Issue summary:View changes

Updated issue summary.

I've updated and revised the summary to reflect our recent plans + link to the sandbox.

Issue summary:View changes

improved structure and added sandbox link

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

created and added some more issues

Title:[META] Entity Property API[meta] Entity Property API
Category:task» feature

Category:feature» task

Over at #1696648-1: [META] Untie content entity validation from form validation, I asked if it would become possible to store updated values for fields and properties without using a form.
Here, I'd like to ask if it'll become possible to not only render a single field as we can today (using drupal_render(field_view_field($entity_type, $entity, $field_name))), but also to render a single property; that is how I interpret this line:

further entity-api improvements, e.g. sub-systems for rendering or editing entities.

Right now (in D7), the rendering of e.g. the author and publishing date (the created property) are entangled in multiple layers: most of it in the theme system, at the theme-specific layer even ($variables['name'], $variables['date'] and $variables['submitted'], with the date being formatted in a different way depending on the theme!), instead of having the ability to override/alter/annotate them in a generic manner like we can for fields. This also makes it effectively impossible to render these properties independently, without having to simulate parts of the theme system.

This is something that would allow us to make the Edit module much cleaner.

Category:task» feature

Unintentional category change — sorry about that.

ad #29:
Yep, I think we need a separate system for rendering entities, which then is implemented on top of properties/fields and other display components. That should be probably developed inline with the blocks & layout initiate and is out of scope for this issue. Still having a uniform property API should help with that a lot as it allows us to generate those display components based upon property definitions, such that those display components are renderable on their own + clearly relate to their source property (if any).

It's not entirely clear to me then if this issue is going to help me or not :) Are you saying that the rendering will not be addressed, but the foundations for making that possible are going to be addressed in this issue? If so, could you please clarify that?

Also: is "display component" a new Drupal term I'm knot familiar with (from the blocks & layouts initiative)? And what are "property definitions"?

I think this is also the appropriate issue to ask the following question: will Entity Property API make it possible to render the form/widget for a given property, much like it's currently possible in Field API?

With Field API, one can use _field_invoke('form', …) to generate the form to edit just that field. E.g. the FAPE module leverages this.

will Entity Property API make it possible to render the form/widget for a given property, much like it's currently possible in Field API?

There is work happening in the plugins-field-api branch of http://drupal.org/sandbox/eclipsegc/1441840 to make widgets and formatters (and the rest of Field API) into plugins. That work has gotten a little stale, but there will likely be a sprint to get it up to date and move it along further before DrupalCon Munich. Partly, the goal there is to make widgets and formatters a little less tightly coupled to the internals of $field and $instance structures.

Per #31, this issue is about making both properties and fields follow a common API for accessing their data and their "definitions" (definitions being similar to what $field and $instance currently are in Field API).

The combination of these two efforts should naturally lead to formatters and widgets being used for all properties (both non-fields and fields), but actually doing that will likely require a separate issue once these pieces are complete.

To add to what @eff said, the work in the plugins-field-api branch was made with the idea that "widgets & formatters on non-fields would be cool" in a corner of my head, but didn't really engage into actual steps in that direction.
Specifically, the code within the actual class of a widget or formatter implementation expects to be able to refer to arbitrary keys within the $field and $instance definitions, and right now I have little clue about how to abstract this out on non-fields.

$field and $instance definition structures will hopefully turn to classed object at some point (they currently are in the branch, and CMI will probably need this, but that change is highly invasive, and an initial 'widgets as plugins' patch will most likely need to come without). This might help abstracting out an interface, but right now it's not sure properties will have a similar $property / $property_instance dichotomy.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

remove issue marked as duplicate of sibling

Title:[meta] Entity Property API[meta] Unified Entity Field API
Issue tags:+Entity Field API

#1696640: Implement API to unify entity properties and fields got committed with new terminology: We refer to entity properties as entity fields now. Also see #1798140: Agree upon terminology for configurable fields.

Issue summary:View changes

Updated issue summary.

Created conversion issues for all entity types and linked them from the summary. Also opened a single major task for it: #1818580: [Meta] Convert all entity types to the new Entity Field API

Issue summary:View changes

added conversion issues

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

updated summary

Issue summary:View changes

added more related issues

updated the summary to add more issues / details

Issue summary:View changes

more issues

Issue summary:View changes

better formatting

Issue summary:View changes

added stuff that has no issue yet

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

added a list

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

I think #1939994: Complete conversion of nodes to the new Entity Field API should be another meta for set of issues to replace direct usage of entity properties to methods

Issue summary:View changes

...minor typo

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

In response to #17 and #18, it may be sounds new in the PHP world, but it is a very old concept and you guys might be interested in looking at the JSR-317 [ref] (equivalent of PHP's PSR) which defines the JPA (Java Persistence API) in its version 2.0. The final draft was accepted in 2009. The JSR-220 defines the JPA in its version 1.0 and was submitted in 2003. So this is a very mature specification.

Seeing how PHP is getting more and more like Java, and doctrine replicates Hibernate with POJO's and annotations on entities, this can be a good example.

In short: JPA is the specification which allows an implementation translate a Conceptual Data Model (CDM) into a Physical Data Model (PDM). Write your POJO (Plain Old Java Object), and an hibernate implementation will create the table for MySQL if you use a SQL dialect with a mysql-connector.

Having the model in code, completely abstract the PDM. In Drupal we have taken the direction of abstracting the PDM so this is good to have. Moreover, it is generally a good practice as it respect the Don't Repeat Yourself (DRY) principle. The API should be then capable of:

  • Generating a PDM from a CDM with a given connector (mysql, postgre, mongodb, etc.),
  • Updates the PDM when the CDM changes,
  • Manage @OneToMany, @ManyToOne (uni-directional and bi-directional), and @ManyToMany relationships.

When speaking about lazy-loading references, I have some experience in this field and this mainly depends on the PDM: for instance imagine a node entity references a user entity because it is its author.

In a SQL driven environment, we can proxy the user with its PK in the node entity on loading. This has the following advantage, if you now want to load the top-K nodes of the same author, you can ask your model to make the relation on the PK of the user without actually load this user because SQL DB systems just need this key to create a WHERE clause. This kind of optimization would not have been possible on another environment such as Object-oriented databases or Document-oriented databases for instance.

Think also as client (JS) serialization of these references: it do-able but not simplistic, you need to have a dataService server-side which will load reference when the client asks for, and a client capable of handling synchronous field access or a framework which handle asynchronous field population.

After having submitted #1817778: Consider using Doctrine ORM for Entities for proposal, I realized that Doctrine ORM comported some flaws and might not cover all our uses cases such as re-visioning and translatability of our entities. We can go in the direction of implementing our own ORM, from an entity manager, with proper annotations (at field-level like in #17).

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Added issue link

Issue summary:View changes

Updated issue summary.

Updated issue summary adding link to #1950632: Create a FieldDefinitionInterface and use it for formatters and widgets to the "Bonus step" section.
Didn't figure out if and where #2004626: Make non-configurable field translation settings available in the content language settings would fit in. Maybe someone else is bold enough to add this one, too.

Status:Active» Fixed

We achieved almost everything of this, yay! Now we need to complete it, there's a new meta for this, see you over in #2095603: [meta] Complete Entity Field API.

Automatically closed -- issue fixed for 2 weeks with no activity.

Issue summary:View changes

Updated issue summary.