[META] Entity Serialization API for web services (e.g. content staging)

I think there will be a lot of devils in the details of Step 3. And I think they will be an issue of increasing complexity for both the Staging support and Preview support.

These are some of the areas that I think need to be considered:

• Dependencies can be multi-directional and ad-hoc.
• The order of entity deployment matters.
• Deployments can be bi-directional, i.e., they may need to be "unstaged" at some point.
• When deploying, where the entity might appear in a site matters because it might need to appear exclusively.

Dependencies can be multi-directional and ad-hoc
So, for example, lets say that I was publishing a new "Case Study" on my website and when I publish a new Case Study I need the Case Study (primarily a PDF file) published along with content (text and images) that will appear on my home page, which will feature the Case Study. And assume this Case Study is about a new "topic" for my website and all my topics are represented through taxonomies, so I have a new taxonomy term that needs to published with it. Assume that for reasons that make sense for this site that these are all separately defined entities or nodes.

I might have a node that, when published, must be published along with three or four other separate, but dependent, entities, perhaps two image entities and a "resource" entity - the Case Study PDF file. It may be that once published, the Case Study entity (pdf file) will live permanently in the Resources section of my website, but the "Featured Article" content could all be removed.

So in the deployment of this content, I would want it all to be packaged together and dependent. If one of the entities could not be deployed, then none of them could be. This requires not only identifying dependencies, but also some kind of transactional deployment handling. And the transactions would need to be at both the database and file system level, so both data and file deployments would be rolled back on a failed attempt.

The deployment order matters
One example: If I am deploying that Case Study that needs to be categorized by a new taxonomy term, then I would need to ensure that the taxonomy term was deployed first or that both the term and the entity were deployed completely before the record tying the entity and term together was added to the table, otherwise the insert would (should) fail.

Deployments can be bi-directional
I assume that deployments can cause stuff to be removed or unpublished from a server, as well as added to the server. So that has to be done in such a way that an entities dependencies, as both a parent and child in dependency relationships, are taken into account.

To explore the Case Study example above further, let's say that at a later date I might decide that the feature content doesn't need to be published any longer. So I might want to deploy changes that would unpublish all those entities, but the Case Study that was part of the original package should be left on the site. So the dependencies for this deployment would need to change.

Consider also that maybe I want to unpublish the feature content that was published along with the Case Study but that one of the images from the original feature content was reused later for some other feature content and that featured content is still published on the site. In this case, I should not be able to deploy changes that would unpublish that image since it is part of a dependency with another entity.

When deploying, where the entity might appear in a site matters because it might need to appear exclusively
Example: If I have blocks on the front page that are used for presenting one item and one item only at any given point time (like my featured Case Study), then the Staging process needs a way to determine if deploying an entity would violate that exclusivity. Publishing rules need to be definable for blocks or regions or pages etc that would check the availability of a block or region for a given entity depending on a variety of conditions, not the least of which would be publication and/or expiration date.

Anyway, some food for thought.

Thanks for getting this organized @_dixon

From the perspective of "Content moderation and content preview on one and the same environment", there is already work being done in the following issues.

• #218755: Support revisions in different states
• #218755: Support revisions in different states

Steps 1 - 4 in this plan also are very valuable to both methods.

(Settings status to "Active" since there is no actual patch to work on)

@Indytechcook, did you intend to link to two separate issues?

Adding link to 218755

Added links to issues and the sandbox currently being worked on.

Linclark started a discussion about workflows on gdo here:

http://groups.drupal.org/node/246748

I wrote a long response about how Deploy works, and the pros and cons of its' approach. This is touching on a lot of the issues raise in #1.

FWIW, I've tagged a whole bunch of issues with UUID:
http://drupal.org/project/issues/search/drupal?issue_tags=UUID

And I also just created:
#1726734: Replace serial entity IDs with UUIDs in URLs, or even globally?

In light of that, it appears to me that we need to solve some much more low-level things first, in order to allow the higher-level content staging efforts to truly take off. That isn't really reflected in the outlined staging roadmap in this issue summary here.

(That said, I'm also not 100% sure whether this issue is actually the roadmap for content staging, or actually something else.)

Also, (no pun intended) is @dixon_ still leading this effort? I wasn't really involved in staging discussions, but I've the impression there wasn't much activity lately, so we overall didn't make much progress in the low-level UUID/entity system field yet :-/

Given the remaining timeline until feature freeze, we should really make sure that someone is actively taking on the lead for the low-level UUID system changes that are required for the staging efforts. I'm trying to push issues where I can (because we're going to use UUIDs for configuration, too), but I can't really take on that lead myself.

Updated issue summary.

Added links to UUID followup issues

Updated the issue summary to point to the new UUID followup issues. Although I'm not sure we need the docs followup here - feel free to remove it.

I've created #1757452: Support config entities in entity reference fields (but won't have time to work on it)

Step 3, Entity property API has just been committed! Would be awesome if someone could tackle Step 4, Entity Reference Field in core. Note that Paris WSCCI writeup has some notes.

Here's an issue for Entityreference in core, from one of the co-maintainers: #1801304: Add Entity reference field.

Note that there's already some sort of an entity reference field that went in with the properties patch: http://drupalcode.org/project/drupal.git/blob/HEAD:/core/lib/Drupal/Core...

It's just an item type (for the metadata, or whatever it's called now). There's no reference field in there. Not sure why this is confusing people across the queue.

Clarified the intention.

Updated issue summary.

We are making great strides here. Step 4 and Step 5 are in progress. I've updated the Issue Summary. Please help Amitaibu and Lin and perhaps architecture/planning should start for Step 6.

Updated issue summary.

Step 5 is done and Step 4 is nearly RTBC. Anyone up for focusing on Step 6? We are so close!

Created #2112799: [meta] Port deploy to Drupal 8 for step 6.

I don't think this is a configuration system issue

Updated issue summary.

Anything left to here for core? :)

I think we can close this issue.

As a follow-up, we are not using entity reference fields in core at all AFAIK. Thats blocked by #2116551: Fix the UX of Entity Reference fields. So help there is welcome.

Another good task would be to start a fresh deploy module for D8, since so much has changed. See https://drupal.org/project/deploy

@Moshe: Yes, taxonomy/file/image fields do not directly extend from the configurable ER field. That doesn't matter much for this, they all extend from the core entity reference field type and can be introspected about what they reference and access those references using a single API. Including base fields like the node author, node type and so on.

@Berdir - good point!. That means that the best place for interested folks to move to is #2112799: [meta] Port deploy to Drupal 8.

FYI, @dixon_ made a terrific presentation at Drupalcon on content replication for D8. See https://austin2014.drupal.org/session/content-staging-drupal-8. Help him!