i'd like to import feed items to local nodes, from a service that outputs feeds in JSON- can this be done easily?

Comments

alex_b’s picture

Title: JSON support » JSON parser

You'd need to build a JSON parser (see plugins/FeedsCVSParser.inc for an example parser). Should be quick and easy to do. The parser will be a custom parser as the format of the JSON feed is proprietary - correct?

mchelen’s picture

subscribing

gravelpot’s picture

I may end up working on this as I need to import popular queries from a Google custom search engine as nodes, and that is output as JSON.

BenK’s picture

Subscribing...

planet4sale’s picture

Hi gravelpot,

Did you get anywhere with this?

Code I have borrowed from a couple of sources which I am hoping to use for a JSON parser module:


/**
 * Parses a given file as an JSON file.
 */
class FeedsJSONParser extends FeedsParser {

  /**
   * Implementation of FeedsParser::parse().
   */
  public function parse(FeedsImportBatch $batch, FeedsSource $source) {
    // Use JSON Parser which doesn't exist yet to parse the file and return a formatted PHP array.
    $rows = json_parser_json_to_array(realpath($batch->getFilePath()));
    
    // Populate batch.
    $batch->setItems($rows);
  }
  
}

function  json_parser_json_to_array($url){

// The following lines will access the API, and then parse the JSON into an associative array
$json_output = file_get_contents($url);
$json_object = json_decode($json_output);
$myinfo = objectToArray($json_object);
// The function 'objectToArray' can be found below

return $myinfo;
	
}

// Function to convert the JSON into a PHP Array.
function objectToArray($object) {
if(!is_object($object) && !is_array($object)) {
return $object;
}
if (is_object($object)) {
$object = get_object_vars($object);
}
return array_map('objectToArray', $object);
}

alex_b’s picture

Version: 6.x-1.0-alpha12 » 6.x-1.x-dev
Status: Active » Needs work

Good start.

- Don't use $batch->getFilePath(), but $batch->getRaw(), this will give you the actual content of the document.
- I think its not useful to convert the result of json_decode() into an array. I think we should expect an array of items as input in the first place. If this is not the case, converting doesn't do much good. We should rather bail (throw an exception).

planet4sale’s picture

Thanks,

That looks like it might be easier to work with (looping through arrays was getting a bit mad here).

twistor’s picture

I do not think it's safe to assume that an array will be the input type. In fact, it's a security flaw in javascript. Not a problem here but, I think that most JSON services should be returning objects; usually with a results attribute. Seeing as this is a custom parser, it depends on the data that is returned.

alex_b’s picture

#6/#8: Feeds is architected to import series of items - which need to be arrays or array-like structures.

If we want to support importing single, unwrapped items, the way to go would be not to convert them (as in #5), but to wrap them into an array to make a single item look like a series of items with one entry.

twistor’s picture

I don't think I explained myself correctly. I was trying to say that the result from a typical JSON service is indeed an array, but it is usually attached to a base object as an attribute named something like 'results'.

gravelpot’s picture

This was my implementation for parsing the JSON feed that a Google Custom Search Engine will give you for your popular search terms. I've rolled it into a very small module that just implements the parser plugin. Hope it's helpful!

/**
  * Class definition for Google CSE popular queries parser
  *
  * Parses a JSON string returned by Google.
  */
class GoogleCseQueriesFeedsParser extends FeedsParser {

  /**
   * Implementation of FeedsParser::parse().
   */
  public function parse(FeedsImportBatch $batch, FeedsSource $source) {
    $search = array('\x2F', '\x26');
    $replace = array('/', '&');
    $raw = $batch->getRaw();
    $result = str_replace($search, $replace, $raw);
    $decoded_result = json_decode($result, true);
    $batch->setTitle($decoded_result['title']);
    $batch->setItems($decoded_result['popularQueries']);
  }

  /**
   * Return mapping sources.
   *
   */
  public function getMappingSources() {
    return array(
      'query' => array(
        'name' => t('Query'),
        'description' => t('The search query term.'),
      ),
      'num' => array(
        'name' => t('Number'),
        'description' => t('Number of queries.'),
      ),
      'href' => array(
        'name' => t('Search URL'),
        'description' => t('URL of the search query.'),
      ),
     );
  }
}
xjm’s picture

Tracking.

discursives’s picture

subscribing

DarrellHQ’s picture

Following

fereira’s picture

I've been thinking about writing a json parser for awhile but never got around to it. Although this version is for Google CSE the basic approach should work pretty well. I have a collection of very simple modules which consume json, use json_decode to transform the string to a php array, then some theming functions to theme the content. One of the issues I've run into is that, for the 5.2.* version of PHP the json_decode function will silently return a null if it couldn't decode the json. The 5.3.* version will return an error code. I ran into some problems with json which contained UTF characters that would blow up json_decode(). It would be a good idea to check the results of the json_decode function before attempting to set times from the array.

criz’s picture

Yes, json_decode() is tricky. I had success with the following:

<?php
  $raw = $batch->getRaw();
  //some cleaning
  $json = preg_replace('/,\s*([\]}])/m', '$1', $raw);
  //check utf8 encoding and decode
  $json_array = json_decode(utf8_encode($json), TRUE);
?>
twistor’s picture

You might want to check out my new module Feeds JSONPath Parser. The syntax is pretty easy to pick up.

alex_b’s picture

Just found this JSON document spec: http://tools.ietf.org/html/draft-zyp-json-schema-02 . This seems to be the standard to implement for a useful JSON document parser.

maruan’s picture

Edited:
I am new to feeds, and after reading some more, I see that my earlier question did not belong here, but rather to services.
cheers !

Hanno’s picture

Status: Needs work » Closed (fixed)

As the JSONPath parser is launched, I set this issue on fixed. Please feel free to reopen if there is a need.