Problem with aggregating news from Google

Feed as an example: http://news.google.com/news?um=1&cf=all&ned=us&hl=en&q=drupal&cf=all&out...

Feed items are added to MN but when one clicks on a title of an aggregated article one invariably gets:

Redirect Notice
The previous page is sending you to an invalid url.

If you do not want to visit that page, you can return to the previous page.

This only happens for Google RSS

Comments

haisam’s picture

haisam commented
Priority: Normal » Critical
lennart’s picture

lennart commented

This is because the equal signs in the URL are transformed into something else. It is *not* only with Google RSS. It is with all redirecting URLs with equal signs in them.

haisam’s picture

haisam commented

It looks like this was already noted as an issue here http://drupal.org/node/611972

haisam’s picture

haisam commented
Status: Active » Closed (fixed)

I'm closing my ticket so http://drupal.org/node/611972 can be the issue to be resolved.

haisam’s picture

haisam commented
Status: Closed (fixed) » Closed (duplicate)
haisam’s picture

haisam commented
Status: Closed (duplicate) » Active

I'm re-opening this because the new feeds (6.x-1.0-alpha5) module doesn't solve this problem so I don't think it's a duplicate of http://drupal.org/node/611972

whurleyvision’s picture

whurleyvision commented

Hey everyone, forgive me if I seem out of place here but I think this is a google issue. I am pulling feeds from several other searches and they work correctly. As fas as I can tell, Google seems to be taking the referring website, and if it's not *.google.com then they redirect you to that error page. i.e. Google is free to take your feed, but if you try to take theirs, no dice.

I did a little research and found that Google does block several RSS aggregation tools such as Yahoo! Pipes. So, I'm curious as to how this is an issue with the feeds module. Please correct me if I'm wrong, I am new to both Drupal and RSS aggregation.

haisam’s picture

haisam commented

You bring up a good point. So I did the following test.

I went here:

http://localhost/managingnews-1.0-beta5/admin/content/aggregator/add/feed

and I added:

http://news.google.com/news?pz=1&cf=all&ned=us&hl=en&q=drupal&cf=all&out...

then I did 'update items' via:

http://localhost/managingnews-1.0-beta5/admin/content/aggregator/update/2 (the number 2 may be a different number for you)

then I went here

http://localhost/managingnews-1.0-beta5/aggregator

Now select any link and you will see that one is correctly redirected to the source.

Now if you perform similar steps via:

http://localhost/managingnews-1.0-beta5/node/add/feed

you will note that one gets the redirect error that I initially talked about.

whurleyvision’s picture

whurleyvision commented

I've skipped over this issue by using Yahoo! instead of Google and running everything through Yahoo! Pipes to create a highly customized pre-processed feed BEFORE bringing it into Managed news.

So far this has worked with the notable exception of one particular piece of content that I have to get from Google. In that case, the RSS feed works and when I import it I get "30 new items added", but nothing shows up on the site (i.e. in that new node).

haisam’s picture

haisam commented

Instead of skipping can you please confirm that my conclusion is correct, that /node/add/feed has a problem with google but that /admin/content/aggregator/add/feed does not?

whurleyvision’s picture

whurleyvision commented

Sorry, but I can't. I'm using a hosting provider and the path "/admin/content/aggregator/add/feed" does not exist. Since I have no access to root, I'm limited as to what digging around I can do. Sorry :(

haisam’s picture

haisam commented

First make sure you are logged in as admin (UID=1) and you will need to enable the Core Aggregator module.

So go here /admin/build/modules and click on Aggregator and save now your will be able to find

/admin/content/aggregator/add/feed

whurleyvision’s picture

whurleyvision commented

Ok, followed those directions and yes, i appears that conclusion is correct (as near as a total n00b can tell). When I went to /admin/content/aggregator/add/feed/ and put in my google based feed the items did show up (and correctly) at aggregator/sources/2 :)

alex_b’s picture

alex_b commented
Title: Google News Aggregation: "sending you to an invalid url" » Google News Feeds: "sending you to an invalid url"
Project: Managing News » Feeds
Version: 6.x-1.0-beta5 » 6.x-1.x-dev
Priority: Critical » Normal

I did some digging and unfortunately this issue seems to run deep:
http://github.com/skyzyx/simplepie/issues/#issue/1

As a stop gap you could use the patch on the thread and apply it to simplepie.inc [edit: additionally you will have to remove the html_entity_decode() from $item['url'] = html_entity_decode($simplepie_item->get_link()); in FeedsSimplePieParser.inc]

Removing the release critical flag as we are using this flag only for issues that need to be addressed for the immediate release. Moving to Feeds as this is rather a Feeds issue than a Managing News issue.

alex_b’s picture

alex_b commented
Status: Active » Closed (duplicate)
Issue tags: -RSS, -google, -news

#898428: SimplePie does not decode queries in RSS item links

mikelaw1’s picture

mikelaw1 commented

This is a potential solution: Tools > Options > UNCHECK "Open article link automatically"

Has helped.