Currently RestWS only allows to access entity via it's id, but in many cases you don't know the id, so it would be cool to have features like to retrieve all entities of a certain type or search for entities with certain properties using EntityFieldQuery.
This functions need to have some sort of paging system, in case there are to many Enities and the question is how should this system work?
One possibility is, that that every result has a link to the next page. Another would be, to specify the pages in the query...
What do you think, RestWS Community?

Comments

klausi’s picture

Issue tags: -gsoc2012 +GSoC 2012

Tagging

sepgil’s picture

Klausi and I discussed how I should implement this task and what we came up with, is that to make a basic query which simply returns all nodes, the client has to use HTTP GET method on the base uri of the resource.
So if you want to get all nodes, you'll have to retrieve them with the GET method on the uri node, node.js or node.xml.
If want to do a query you'll have to specify the query parameter via a question mark.
For example, if you want to get all nodes of the type article from the author foo the uri has to look like this:
node?type=article&author=foo
If the there is an entity property colliding with a name used by restws, you'll have to use the prefix "property_".

To prevent DoS attacks and we'll also have to use some sort of paging(like described in the post above). I did some research and found out that it would be best to implement the system described here:
http://tools.ietf.org/html/rfc5005
We will have four key elements called "first", "last", "prev", "next" which will help you navigate through several pages, with each page having an uri like "[entity]?page=[n]".
This means, we will have to change the current output format a little bit to distinguish actual values from restws metadata.
My suggestion would be to have an element/object which contains all values, and have all metadata about paging on the first level of the hierarchy.
Json example for nodes:
node?type=article&page=2

{
  "first":"node?type=article&page=1",
  "last":"node?type=article&page=3",
  "prev":"node?type=article&page=1",
  "next":"node?type=article&page=3",
  "resources": [
    {
      "title" : "foo",
      .
      .
      .
    }
    .
    .
    .
  ]
}

XML example for users:
user?page=4

<node>
  <first>user?page=1</first>
  <last>user?page=15</last>
  <prev>user?page=3</prev>
  <next>user?page=5</next>
  <resources>
    <item>
      <uid>5</uid>
      .
      .
      .
    </item>
    .
    .
    .
  </resources>
</node>
sepgil’s picture

Another thing I noticed, is that restws should by default only output a reference to the entities like:

{
  "first":"node?type=article&page=1",
  "last":"node?type=article&page=3",
  "prev":"node?type=article&page=1",
  "next":"node?type=article&page=3",
  "resources": [
    {
      "uri: "node/4",
      "id" : "4",
      "resource" : "node"
    }
  ]
}

and only if the client uses the parameter "full=TRUE" all properties of the entities should be outputted.
That would reduce the load on the server.

sepgil’s picture

Status: Active » Needs work
StatusFileSize
new6.64 KB

I've created a basic patch to test out the functionality.
Currently it simply lists all resources if the either /resource.json/xml is called or if the right http header are set.

klausi’s picture

+++ b/restws.entity.inc
@@ -70,6 +70,19 @@ interface RestWSResourceControllerInterface {
+   * If a format doesn't wants to implement queering, then it should throw an
+   * exception.

queering should be querying. Exception type RestWSException.

+++ b/restws.entity.inc
@@ -70,6 +70,19 @@ interface RestWSResourceControllerInterface {
+   * @param array $parameters

better name: filters?

+++ b/restws.entity.inc
@@ -70,6 +70,19 @@ interface RestWSResourceControllerInterface {
+   * @return

@return array

+++ b/restws.entity.inc
@@ -70,6 +70,19 @@ interface RestWSResourceControllerInterface {
+  public function query($parameters);

additional parameter: "meta_controls" for sorting, paging etc.

+++ b/restws.entity.inc
@@ -156,6 +168,21 @@ class RestWSEntityResourceController implements RestWSResourceControllerInterfac
+    $query_result = $query->execute();

query limit is missing: variable restws_query_max_limit 100

+++ b/restws.entity.inc
@@ -156,6 +168,21 @@ class RestWSEntityResourceController implements RestWSResourceControllerInterfac
+    $query_result = $query_result[$this->resource()];

this should be checked for emptyness

+++ b/restws.entity.inc
@@ -156,6 +168,21 @@ class RestWSEntityResourceController implements RestWSResourceControllerInterfac
+    $result = array();
+    foreach ($query_result as $entity_reference) {
+      $result[] = $entity_reference->$id_key;
+    }

couldn't you use array_keys($query_result)?

+++ b/restws.module
@@ -122,7 +122,7 @@ function restws_handle_request($op, $format, $resource_name, $id = NULL, $payloa
+        if ($op == 'create' || $op == 'query') {

not related to create, use your own if branch.

+++ b/restws.module
@@ -216,6 +216,41 @@ function restws_menu_alter(&$items) {
+        $items[$resource . '.json']['page arguments'] = array();

you need to do that for all formats available

+++ b/restws.module
@@ -224,12 +259,17 @@ function restws_menu_alter(&$items) {
+  else if (($pos = strpos($base_uri, '.')) && $format_name = substr($base_uri, $pos + 1)) {

should be "elseif"

+++ b/restws.module
@@ -276,7 +316,13 @@ function restws_page_callback($resource, $page_callback = NULL) {
+        if (!empty($id))
+        {

"{" should be on the same line as if

Otherwise looks pretty good so far.

klausi’s picture

Oh, and simpletests please!

sepgil’s picture

StatusFileSize
new8.6 KB

New version of the patch is out, now with simpletests ;-)

sepgil’s picture

StatusFileSize
new8.57 KB

Forgot to delete a debug statement...

klausi’s picture

+++ b/restws.entity.inc
@@ -71,16 +71,16 @@ interface RestWSResourceControllerInterface {
+  public function query($filters, $meta_controls);

@param docs for meta_controls is missing. both parameters should have array() as default.

+++ b/restws.entity.inc
@@ -168,18 +168,19 @@ class RestWSEntityResourceController implements RestWSResourceControllerInterfac
+  public function query($filters, $meta_controls) {

same array() defaults here

+++ b/restws.formats.inc
@@ -60,6 +60,8 @@ interface RestWSFormatInterface {
+   * If a format doesn't wants to implement querying, then it should throw an

should be "want"

+++ b/restws.formats.inc
@@ -129,7 +131,7 @@ abstract class RestWSBaseFormat implements RestWSFormatInterface {
+    $values = $resourceController->query($parameters, NULL);

use array() instead of NULL

+++ b/restws.module
@@ -122,12 +122,16 @@ function restws_handle_request($op, $format, $resource_name, $id = NULL, $payloa
+          drupal_add_http_header('Status', '201 Created');

WTF?

+++ b/restws.module
@@ -216,40 +220,27 @@ function restws_menu_alter(&$items) {
+      $format = drupal_strtolower($format['label']);

do not use the label, use the array keys as identifier for formats

+++ b/restws.module
@@ -216,40 +220,27 @@ function restws_menu_alter(&$items) {
+          'access callback' => TRUE,

That is fine, but we need to check for the query operation in restws_handle_request() and use view instead.

+++ b/restws.test
@@ -166,6 +166,30 @@ class RestWSTestCase extends DrupalWebTestCase {
+    for($i=0; $i<5; $i++) {

100000 white space issues.

+++ b/restws.test
@@ -166,6 +166,30 @@ class RestWSTestCase extends DrupalWebTestCase {
+    $result = $this->httpRequest('node.json', 'GET', $account);

same test for XML please

sepgil’s picture

Issue tags: -New feature, -GSoC 2012

Beside fixing the issues mentioned above, this patch also changes the output format so it fits the description in the posts above.

sepgil’s picture

Issue tags: +New feature, +GSoC 2012
StatusFileSize
new9.61 KB

wtf? I don't know what happened with my post above, but here is the patch.

klausi’s picture

+++ b/README.txt
@@ -60,7 +60,8 @@ Design goals and concept
+ * The module supports full CRUD (Create, Read, Update, Delete) and quering for

should be querying

+++ b/restws.entity.inc
@@ -70,6 +70,21 @@ interface RestWSResourceControllerInterface {
+   * If a resource doesn't wants to implement querying, then it should throw an
+   * RestWSException with the 501 HTTP status code.

should be "throw a RestWSException"

+++ b/restws.entity.inc
@@ -70,6 +70,21 @@ interface RestWSResourceControllerInterface {
+   * @param array $parameters

should be "filters"

+++ b/restws.entity.inc
@@ -70,6 +70,21 @@ interface RestWSResourceControllerInterface {
+   *   An array containing the ids of the resources;

should end with a simple dot.

+++ b/restws.entity.inc
@@ -86,7 +101,6 @@ interface RestWSResourceControllerInterface {
 }
-
 /**

unrelated change

+++ b/restws.entity.inc
@@ -156,6 +170,22 @@ class RestWSEntityResourceController implements RestWSResourceControllerInterfac
+    $id_key = $entity_info['entity keys']['id'];

unused?

+++ b/restws.formats.inc
@@ -58,6 +58,19 @@ interface RestWSFormatInterface {
+   * @param RestWSResourceControllerInterface $resourceController
+   *   The controller used to update the resource.

description is wrong

+++ b/restws.formats.inc
@@ -58,6 +58,19 @@ interface RestWSFormatInterface {
+   * @param array $parameters
+   *   The parameters for the query or NULL all existing resources should be
+   *   returned.

I think the default should be an array() not NULL.

+++ b/restws.module
@@ -119,13 +119,20 @@ function restws_handle_request($op, $format, $resource_name, $id = NULL, $payloa
+    // Since there is now access callback for query we need to use view.

should eb "no" instead of "now"

+++ b/restws.module
@@ -119,13 +119,20 @@ function restws_handle_request($op, $format, $resource_name, $id = NULL, $payloa
+        else if ($op == 'query') {

should be "elseif"

+++ b/restws.module
@@ -224,12 +252,17 @@ function restws_menu_alter(&$items) {
+  $base_uri = arg(0);

better name: resource_arg to match id_arg

+++ b/restws.test
@@ -166,6 +166,37 @@ class RestWSTestCase extends DrupalWebTestCase {
+    for($i = 0; $i < 5; $i++) {
+      $this->assertRaw("<title>node$i</title>", 'XML has been generated.');
+    }

you could also assert if the outer list element is present.

sepgil’s picture

Status: Needs work » Needs review
StatusFileSize
new9.18 KB

Fixed all the problems, I think it should be ready to commit now. I'll open follow up issues for the additions features like sorting, paging and query filters.

fago’s picture

+1 for going with next/previous links.

In particular for XML, pls try to avoid coming up with something new, but better follow the atompub specs: http://tools.ietf.org/html/rfc5023#section-10.1

I don't think there is something as widespread for json, but at least google has a direct json port from atompub, so just going with a similar structure in json is probably a good idea as well.

klausi’s picture

Status: Needs review » Fixed

Improved some comments and details and committed it, thanks!

@fago: paging will be a separate issue, this one is just about very basic querying without any filters and pagers.

@sepgil: please open follow-up issues for filtering, paging and sorting. And link them here.

sepgil’s picture

The first follow can be found here: #1697514: Querying - Filters

sepgil’s picture

I made a separate issue for Meta Controls: #1698644: Querying - Filters & Sort Meta Controls

sepgil’s picture

Paging is beeing implemented in this issue: #1700098: Querying - Paging

Automatically closed -- issue fixed for 2 weeks with no activity.