I am following the recipe at http://www.drupaltherapy.com/feedapi to import YouTube videos. However, the YouTube feed only returns the first 25 results, and I would like to pull the rest of the results during cron runs. Is there a way to perform pagination on a feed? If not, where can I start looking to add that feature?

CommentFileSizeAuthor
#4 feedapi-pagination.patch6.76 KBinfojunkie
#3 feedapi-pagination.patch6.9 KBinfojunkie

Comments

infojunkie’s picture

infojunkie
Location Vancouver
commented

Some more information:

The Atom format allows for paged feeds using the syntax

<link rel="next" href="..." />

YouTube feeds (e.g. http://gdata.youtube.com/feeds/base/users/GibsonGuitarCorp/uploads) support this feature. In the previous feed, you can find

<link rel='next' type='application/atom+xml' href='http://gdata.youtube.com/feeds/base/users/GibsonGuitarCorp/uploads?start-index=26&amp;max-results=25'/>
aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

To consume such paginated feeds, you definitely need to write your own parser.
The recipe is easy: handle those 'next' links and download them as well, and process them one-by-one.

infojunkie’s picture

infojunkie
Location Vancouver
commented
Status: Active » Needs review
StatusFileSize
new feedapi-pagination.patch6.9 KB

I went ahead and created a patch for FeedAPI core to handle paginated feeds. The main idea is to refresh one 'next' page whenever the first page remains unchanged. The implementation is as follows:

* Parsers simplepie and common_syndication now return an additional array $feed->options->pager consisting of 'first', 'next', 'previous' and 'last' links.
* The 'next' page is stored in field feedapi.url_next and is loaded into $feed->url_next.
* When function _feedapi_invoke_refresh finds that no new items are present, it checks for $feed->url_next. If it's present, it invokes the parser again with this URL. The 'next' page is then updated with the result of this parsing.

Thanks for your consideration.

infojunkie’s picture

infojunkie
Location Vancouver
commented
StatusFileSize
new feedapi-pagination.patch6.76 KB

Sorry, reversed source and destination in the patch. Fixed now.

infojunkie’s picture

infojunkie
Location Vancouver
commented
Status: Needs review » Closed (won't fix)

Cleaning up my issue queue.