Problem/Motivation

You may know modules like Services or RESTWS, we want something similar for Drupal 8 core. It should be possible to simply enable a web API that automatically exposes any entity type via a RESTful interface. That means that we can easily apply CRUD operations (Create, Read, Update and Delete) to entities remotely. See the initial g.d.o post for more background.

Proposed resolution

Add rest.module to Drupal 8 core. I think we should break this up into smaller chunks, so this issue will only deal with DELETE operations. Some definitions and terminology:

Resources: Those are the things that are exposed to the outside as web API. Examples: a tweet on twitter, a job ad on a recruiting site etc. CRUD operations can be used to interact with resources. Not all resources might support writing, so it is perfectly fine to have read-only resources. Example: a watchdog resource that exposes log entries for reading, but does not allow any manipulation. This module should allow the easy creation of resources: just implement a class with CRUD methods and rest.module will handle routing and serialization for you.

Drupal resources: the next logical step is to connect Drupal entities (nodes, users, taxonomy terms ...) with resources. A node resource for example allows third party consumers to read nodes in a machine readable serialization format (e.g. JSON-LD), but also to create new nodes (e.g. blog posts) over the remote API. There should be an admin interface to enable which resources/entities are exposed, per default all resources are disabled (admin path: admin/config/services/rest).

Plugins: The new plugin system allows us to register resources in the system by simply using annotations on the resource class. Plugin derivatives are used to automatically register any entity type as resource out of the box, all mapping to just one generic entity resource class.

Content entities: when I say entity I mean content entity here. While there might be a lot of use cases to expose configuration entities as resources this is not a priority now. The plugin system allows us to simply add that later, or we could even unify it with the current entity plugin.

Routing: hook_route_info() and Route instances are used to specify the requirements of a web API request. Operations are tied to HTTP request methods (e.g. GET for reading). It is also planned to move access control (see #1793520: Add access control mechanism for new router system) into the routing system.

DELETE: work is started on the delete operation, because this does not require any serialization, which is also work in progress right now. See #1811510: Enable JSON-LD entity serialization for example.

I will work on this in my REST for Drupal 8 sandbox, you can easily review with the web repository viewer (work in progress).

Remaining tasks

  • Develop a prototype in the sandbox
  • Discussion + reviews

User interface changes

An administration page will be added at admin/config/services/rest to enable entities/resources that should be exposed.

API changes

None.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

klausi’s picture

The code in the sandbox is currently quite ugly, but delete web requests work at least now+ there is a test case.

The interface for a resource controller is work in progress. For a resource there should be exactly one active resource controller, e.g. for the "node" resource there should be a generic entity resource controller be responsible per default. I'm still thinking about the power and scope of resource controllers.

  • god-like resource controllers: they get and return the raw serialized resource representation (the JSON-LD string) and invoke serialization stuff themselves. That is helpful if we want to swap out resource formats for one particular resource type, e.g. if I want application/ld+json to look completely different on the taxonomy term resource.
  • narrow resource controllers: they get and return full entity/resource objects, serialization is done somewhere else. Simplifies the controllers and keeps them more light weight.

Access control is also a problem right now, we don't have that in the routing system yet (see #1793520: Add access control mechanism for new router system for work on that). And we don't have something like entity_access() or access controllers in core (see #1696660: Add an entity access API for single entity access for that). I think I will just expose my own permissions right now ("read resourceXY over REST API", "create resourceXY over REST API" etc.) and check for that.

Resource registration and discovery: I'm still thinking in info hooks. There is something similar in #1763974: Convert entity type info into plugins, so that approach might work here as well. rest.module should expose all entity types as resources while other modules could add more.

URIs: I'm currently using the pattern "entity/$entity_type/$id" as path (see #1803586: Give all entities their own URI for that suggestion).

Anonymous’s picture

That is helpful if we want to swap out resource formats for one particular resource type, e.g. if I want application/ld+json to look completely different on the taxonomy term resource.

As far as I can tell, we don't need resource controllers to be god-like, because that functionality is handled by Serializer.

You would just have two Normalizers, e.g. JsonldNormalizer and TaxonomyTermJsonldNormalizer. Symfony's Serializer component would ask each Normalizer whether it is capable of normalizing the object which is being passed to it. JsonldNormalizer would say yes for any subclass of Entity, while TaxonomyTermJsonldNormalizer would say yes only to TaxonomyTerm objects. So long as TaxonomyTermJsonldNormalizer is first in the list of Normalizers, it would be used.

klausi’s picture

Had a very productive code sprint yesterday with fubhy, we have now a working resource plugin system with derivatives (stolen and modified from layout.module). A resource is represented as annotated plugin class that will implement a CRUD interface. For entities we have one single resource plugin that masks itself with derivatives for every entity type. That translates into:

Resource name     Plugin name           Plugin class
foobar            foobar                FooBarResource
node              entity:node           EntityResource
user              entity:user           EntityResource
taxonomy_term     entity:taxonomy_term  EntityResource
...
Crell’s picture

Some quick comments as I go through:

Don't bother with the rest_resource_manager() function. Just make code access the container directly.

Routes should technically begin with a / now. The path matcher for various reasons is currently accounting for both approaches, but Symfony assumes a leading /. We should do that.

As discussed on G+, the access() method on Resources should probably get removed and handled at the route level instead.

"Router" as a class name is going to confuse people. :-) Also, that class should be made ContainerAware, and then you can call $this->container->get('resource_manager')->getInstance() instead.

For EntityResource, do we want methods of CRUD, or do we want to map those to HTTP methods (GET, PUT, DELETE, etc.)? That could actually simplify the "Router" class quite a bit.

klausi’s picture

Thanks Crell, I have fixed most of your comments. I also updated the issue summary for further clarification.

Not sure what you mean in your last sentence? I would prefer to have a hard-coded mapping of CRUD to HTTP methods:

array(
  'create' => 'POST',
  'read' => 'GET',
  'update' => 'PUT',
  'delete' => 'DELETE',
);

But that might not be desirable for all resources, e.g. one might want to use 'update' => 'PATCH'. So we are experimenting with a routes() method on the ResourceBase class. It defines the above mapping as default, any resource plugin that wants to do it differently can overwrite routes() and do black magic HTTP requirements in there.

Crell’s picture

The list of HTTP methods is fixed. It doesn't change per site. Rather than have that mapping array hard coded somewhere, just call the PHP methods strtolower($http_method). So HTTP POST => public function post(), etc.

I don't see what the extra abstraction layer here buys us. PUT and PATCH are not the same thing and not interchangeable, so if you swap "update" from one to the other you're changing semantics and changing the API, which is bad.

klausi’s picture

Status: Active » Needs review
FileSize
30.5 KB

Time to get some more reviews, attached is the current state of work.

I introduced a watchdog resource plugin to demonstrate alternative resource implementations.

There are still quite some @todos in it and I have completely ignored formats and serialization, which will not happen in this issue.

I want to at least wait for #1793520: Add access control mechanism for new router system to improve the current access handling before this gets near to being committed.

@Crell: currently I try to establish a well defined CRUD ResourceInterface with advanced semantics that is a bit more independent of requests, responses and HTTP methods (well, currently not completely, but that's the plan). What does ->post() mean and what is its method signature? And what is the method signature of put()? The RequestHandler has to know how to invoke the resource methods, it will probably do serialization and pass on the resource ID from the URL path.

Or do you suggest that we just define all 4 methods equally:

public function get($id = NULL, $data = NULL)
public function post($id = NULL, $data = NULL)
public function put($id = NULL, $data = NULL)
public function delete($id = NULL, $data = NULL)

$id is the resource ID, $data is the unserialized resource (e.g, a Node object).
Also ugly, $data does not make sense for get() and delete(). And $id might not make sense for post().

I have right now:

public function create($format, $data);
public function read($id, $format);
public function update($format, $id, $data);
public function delete($id);

Note: I have $format on the ResourceInterface methods now, which will probably go away as I want to do unserialization beforehand.

Status: Needs review » Needs work

The last submitted patch, 1816354-7-rest.patch, failed testing.

Crell’s picture

If those are controllers, then they should not be defined explicitly by an interface.

/node/{node}

That would map to:

public function get(Node $node) {}
public function put(Node $node) {}
public function delete(Node $node) {}

And you just don't implement the rest. Whereas this:

/foo/{node}/{user}/{view}

Maps to:

public function get(Node $node, User $user, View $view) {}
public function post(User $user, View $view,Node $node) {} // Order is irrelevant, remember

And that's all you implement.

Make more sense? (I suppose this means you maybe need a method_exists() check, either at runtime or when generating routes based on the information that call directly to the method.

Crell’s picture

Component: wscci » base system

Meant to refile...

klausi’s picture

Currently resource plugins are not controllers in the Symfony page callback meaning. We currently have the RequestHandler class for that, which is responsible for serialization, access and mapping the request to the correct resource plugin (I know, the WatchdogResource contradicts that right now and does serialization on its own - just for demonstration purposes).

public function get(Node $node) {}

That does not make sense to me. You already pass the loaded node into the resource plugin, so there is nothing left to do? Or do you think the resource plugins should handle serialization themselves and return the finished JSON-LD string? My plan was to keep serialization separate from resource plugins.

public function delete(Node $node) {}

Loading the node just to delete it feels a bit silly. I know, I'm doing that exactly in the code as well, but it would be cool to just call entity_delete($entity_type, $id);

Maybe it is a good idea to not have ResourceInterface at all and just map all methods that can be found to HTTP request methods in the routes. Then it is more clear when you integrate new HTTP methods such as PATCH by simply adding a patch() method. On the other hand: resource plugins can overwrite routes() anyway if they don't like the default from ResourceBase.

Oh, another clarification: HTTP methods should not change per site, but per resource. We might define our fine mapping for entity resources, but any other artificial resource that is added by a contributed module might want to use PATCH instead of PUT or whatever.

The simpletest failures from above occur when the module is enabled. There must be a circular dependency somewhere that tries to access RestBundle while it has not been picked up yet. Will investigate that.

effulgentsia’s picture

This might be moot given #11.1, but just in case it's not:

Loading the node just to delete it feels a bit silly

You can have a controller signature of delete($node) without type hinting it with Node, and then just the raw id will be passed. Or you can make the signature delete($nid), but then the route needs to be /node/{nid} instead of /node/{node}.

klausi’s picture

Status: Needs work » Needs review
FileSize
30.69 KB
1.48 KB

New patch that does not throw errors when enabling the module. hook_route_info() relied on the loaded plugin manager which was not available immediately. Nothing else changed.

effulgentsia’s picture

hook_route_info() relied on the loaded plugin manager which was not available immediately.

That should actually be fixed as of #1708692-51: Bundle services aren't available in the request that enables the module. Might therefore be possible to revert #13.

Status: Needs review » Needs work

The last submitted patch, 1816354-13-rest.patch, failed testing.

klausi’s picture

Status: Needs work » Needs review
FileSize
18.94 KB
26.22 KB

No, #1708692: Bundle services aren't available in the request that enables the module did not fix the problem. The tests failed for #13 because the same problem exists in hook_permission() if the module is not installed yet, so I also added an existence check there.

This patch now implements Crell's idea to map HTTP requests methods directly to plugin methods with the same name, e.g get(), post(), delete(), head() ... The plugin base class provides a set of all currently known HTTP request methods, allowing plugins to simply override it to add other custom non-standard HTTP methods.

Routes are only created if the lower cased HTTP method exists on the resource plugin.

The ResourceInterface has been removed. The RequestHandler has shrunk to a single handle() method that forwards requests to the resource plugin.

Question:
Should the custom ResourceException class be replaced with the existing FlattenException from Symfony?

Status: Needs review » Needs work

The last submitted patch, 1816354-16-rest.patch, failed testing.

klausi’s picture

Status: Needs work » Needs review
FileSize
940 bytes
25.96 KB

Aha, there seems to be a fail related to the entity_test dependency for manual testing purposes. We have simpletests, so I don't need that anymore.

klausi’s picture

To answer my question: we can simply use HttpException from Symfony, does exactly what I wanted to represent with ResourceException. Removed that custom exception.

mitchell’s picture

+   * Plugins can override this method to provide additional custom request
+   * methods.

That's so cool. :-)

+  protected function requestMethods() {
...
  'DELETE'

If request methods can be defined within plugin derivatives, why is it necessary to hard-code any request methods? Could you move these method declarations directly into the plugin derivatives where they are used?

Perhaps this would help to dynamically build the list of available methods when responding to an OPTIONS request of a particular route.

+   *   The list of allowed HTTP request method strings.

If you keep some methods hard-coded, then I think "common HTTP request methods", as expressed in
HTTP/1.1: Method Definitions is a more maleable description.

+          'format' => 'application/ld+json',

Is configurging accepted mime types going to be added in this issue? There's also: #1505080: [META] Content Negotiation for Accept Headers, and a list of mime types is available in ./core/includes/file.mimetypes.inc, so perhaps you could leverage that.

I wonder if there might be another approach that would make it possible to select request methods and mime types in a route configuration UI.

moshe weitzman’s picture

Status: Needs review » Needs work

Looking great to me. Some nits ...

The admin page at admin/config/services/rest is a bit too plain for my taste. I personally would like some grouping of all entity types and descriptions for the enity group. I think some Help for this admin page would be good. You could reuse text from the Issue Summary here. I might also like to see which http methods are supported for each resource. And example CURL commands for invoking them :). And pictures of muscular German developers making DELETE requests :)

I think the watchdog service should be renamed to dblog service and should ship inside of dblog folder. I don't know if there is a more appropriate place for the EntityResource plugin to live.

I'd love for the dblog to be able to return a collection like the most recent n entries of a given type. See `drush --filter=watchdog` for all the operations we could support. We might as well start figuring out the API for this. I agree with doing this in a followup as you suggest.

Typo: defintion in ResourceManager class

Typo: tst in DeleteTest doxygen

Typo: enttiy in entityCreate() in ResourceBase.php

// Clear the static cache, otherwise we won't see the update. in testDelete(). This comment seems not to match the code and can be removed.

I think httpRequest method could go right into WebTestBase class

Can an $entity->delete() fail? We don't handle that in EntityResource::delete

Grayside’s picture

@moshe

I'd love for the dblog to be able to return a collection like the most recent n entries of a given type.

Is there an already resolved decision about things like paging, queries, and collection handling?

Crell’s picture

At this point, my view is that collections and paging are a problem for Views plugins, because Views is how we build collections.

The details of what Views should output in that case in terms of the exact output, I don't know.

mitchell’s picture

#23: Maybe as a follow-up to this issue, #1819760: Add a REST export display plugin and serializer integration., and #23298: Entity bundles should declare a plural form of their label; in #1792828: [META] Provide entity integration with Views, support for entity type machine name as a contextual filter could be added, and then a default route for '/{entity-type-name-plural}' could supply the collection.

Grayside’s picture

Pluralizing things sometimes is not good. Your top-level resource list gets cluttered, and you can't easily extend "node/" to "node/32" to go from collection/factory to entity.

Crell’s picture

Let me reiterate: Collection GET routes are off topic for this thread. :-) We're just dealing with a framework for single-item CRUD. Views is how we create collection listings, and therefore a given site builder could put whatever collection listing he wants wherever he wants.

If we want core to ship with some default views or template views for listing services, that's a February/March question, not a November question.

klausi’s picture

Status: Needs work » Needs review
FileSize
18.07 KB
29.11 KB

Thanks for the feedback Moshe!

Added hook_help() with some rudimentary information. Good idea with cURL command, I generated a commend that is displayed with the cookie settings to test the DELETE functionality. I have split up the checkboxes between entity resources and other resources on the admin page. It would be cool to extend the admin interface further to display path information, HTTP method, all operations, permissions etc., but that could require a table and some usability engineering to get that right. So I would like to go with this simple interface for now and enhance it in a follow up.

Renamed the Watchdog resource to DB Log resource. I would like to keep it in the rest module folder for now, because that makes it easier for me to provide core patches.

Fixed the typos, improved the comment about the static cache flag for entity_load().

RESTTestBase::httpRequest(): not sure this should go into DrupalWebTestBase, because cURL is ugly. I have to do "unset($this->curlHandle);" in the test case if I want to do multiple cURL requests in a row, because there is no way to know the current cURL options and how to set them back to default values (mostly default values simply don't exist). Very poor cURL API.

Added an EntityStorageException try/catch block around entity->delete().

@mitchell: the HTTP request methods are defined explicitly because we cannot know which public PHP methods on the plugin class are supposed to be exposed as controller callback. For example the public getPluginDefinition() method should not be exposed in a route with _method = GETPLUGINDEFINITION. Plugins can easily override the list of HTTP standard methods already defined in ResourceBase::requestMethods().

ygerasimov’s picture

Status: Needs review » Needs work

I think plugins are used very nicely here. Really like this approach.

+++ b/core/modules/rest/lib/Drupal/rest/Plugin/ResourceBase.phpundefined
@@ -0,0 +1,90 @@
+        $route = new Route("/$prefix/{id}", array(
+          '_controller' => 'Drupal\rest\RequestHandler::handle',
+          'plugin' => $this->plugin_id,
+        ), array(
+          '_method' => $method,
+          'format' => 'application/ld+json',
+        ));

For the POST request (to create an entity) we should post to "/$prefix" as we don't have the id.

Maybe we should remove 'format' property for now. As now we aim for delete call to be working it should be ok to do it with any other format.

We can also improve testing (to provide verbose message about headers and body of request plus headers and body of response). This can be done in follow-up issue.

moshe weitzman’s picture

Status: Needs work » Reviewed & tested by the community

format is needed even for delete, in case there are errors. in any case, i think consistency is valuable here. I don't think the tests need to be more verbose, as an error message with a line number will identify exactly what the problem is. we have been removing assert messages in tests recently for this reason.

i looked at the code and I think we can commit it just to keep things moving. we have to work on CREATE/READ/UPDATE soon. We can move the dblog service in a follow-up.

Crell’s picture

Status: Reviewed & tested by the community » Needs work

Note: On routes, non-prefixed defaults and requirements have special meaning. Defaults get used for placeholders, and requirements are treated as a regex for a matching placeholder. That's why there's _method and _controller. plugin and format need to be prefixed as well, I believe.

yched’s picture

+++ b/core/modules/rest/lib/Drupal/rest/Plugin/Derivative/EntityDerivative.phpundefined
@@ -0,0 +1,50 @@
+    return $this->derivatives[$derivative_id];

I might be wrong, but shouldn't there be an isset() check ?

+++ b/core/modules/rest/lib/Drupal/rest/Plugin/ResourceBase.phpundefined
@@ -0,0 +1,90 @@
+          'title' => t('Access @method on resource %label', array('@method' => $method, '%label' => $definition['label'])),

Nitpick, but grammatically shouldn't this be "Access @method on %label resource" ? ("Access POST on node resource")

+++ b/core/modules/rest/lib/Drupal/rest/Plugin/Type/ResourceManager.phpundefined
@@ -0,0 +1,48 @@
+class ResourceManager extends PluginManagerBase {

I think the current usage in core is to name the classes FooBarPluginManager.
(same way we do FooBarCacheBackend - the name of the generic 'thing' that is implemented is PluginManager, CacheBackend, not Manager, Backend)

+++ b/core/modules/rest/rest.installundefined
@@ -0,0 +1,15 @@
+/**
+ * Implements hook_uninstall().
+ *
+ * @todo is this needed when using the config system?
+ */
+function rest_uninstall() {
+  config('rest')->delete();

I don't think this is needed, or at least there's an open issue for it, and other modules in core currently don't bother with this, so I guess that could be left out of this patch.

+++ b/core/modules/rest/rest.moduleundefined
@@ -0,0 +1,101 @@
+        $plugin = $manager->createInstance($key);

The usual Plugin API flow is to call getInstance(), while createInstance() is internal.

At least, ResourceManager ::getInstance() currently contains some logic wrapping createInstance(), but is never actually called in the patch.

+++ b/core/modules/rest/rest.moduleundefined
@@ -0,0 +1,101 @@
+        $plugin = $manager->createInstance($key);

Ditto

22 days to next Drupal core point release.

klausi’s picture

Status: Needs work » Needs review
FileSize
7.63 KB
43.48 KB

Removed the format setting on the routes for now, it is indeed not needed for this patch. I'm also not special-casing any POST request here without {id}, => follow-up stuff.

@Crell: but I need the plugin ID as parameter on RequestHandler::handle(), so I provide it as default value in the route. Does not seem wrong to me? How would I retrieve the plugin ID if it is stored in _plugin?

Thanks yched, cleaned up the plugin code. There was some old unused stuff in getInstance().

klausi’s picture

FileSize
28.35 KB

Oh, forgot to rebase my development branch, ignore the patch from #32. Updated patch attached.

Crell’s picture

klausi: Request attributes that have magic special meaning to the system (e.g., for routing purposes) should have a _ prefix. Request attributes that are just values intended for the controller to take as a parameter should not have a prefix.

I'm not 100% sure which category plugin would fall into here, but _format definitely needs a prefix since that will be used for routing.

ygerasimov’s picture

Here is updated patch adding comments about how we can use _format after #1793520: Add access control mechanism for new router system committed.

klausi’s picture

Thanks Yuriy. Added some more documentation improvements. Interdiff also contains Yuriy's changes.

Status: Needs review » Needs work
Issue tags: -WSCCI

The last submitted patch, 1816354-36-rest.patch, failed testing.

klausi’s picture

Status: Needs work » Needs review
Issue tags: +WSCCI

#36: 1816354-36-rest.patch queued for re-testing.

moshe weitzman’s picture

Status: Needs review » Reviewed & tested by the community

All recent feedback is in. RTBC.

Dries’s picture

Status: Reviewed & tested by the community » Fixed

This looks great, and has buy-in from all the right people. Committed to 8.x. Yay! :-)

klausi’s picture

Work on reading resources with GET requests continues in #1834288: RESTfully request an entity with JSON-LD serialized response.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

effulgentsia’s picture

Title: Add a REST module, starting with DELETE » Change notice: Add a REST module, starting with DELETE
Priority: Normal » Critical
Status: Closed (fixed) » Active
Issue tags: +Needs change record

Moving the change notice requirement from #1935538-22: Switch REST to default to HAL to here, since I don't think we have any change notice at all yet for rest.module. The bulk of the info should be added as pages within http://drupal.org/developing/api, with a change notice just announcing the new module and pointing there.

effulgentsia’s picture

Category: feature » task
Crell’s picture

Assigned: Unassigned » linclark
klausi’s picture

Assigned: linclark » Unassigned
Status: Active » Needs review

Change notice created: http://drupal.org/node/1975444

What else should we add there?

moshe weitzman’s picture

Priority: Critical » Normal
Status: Needs review » Fixed

Thanks klausi.

effulgentsia’s picture

Title: Change notice: Add a REST module, starting with DELETE » Add a REST module, starting with DELETE
Issue tags: -Needs change record

:)

Automatically closed -- issue fixed for 2 weeks with no activity.

Anonymous’s picture

Issue summary: View changes

expanded proposed solution

Wim Leers’s picture

Component: base system » rest.module