I'm not sure where this breaks down, but an rss feed from twitter doesn't seem to parse completely.
I have tested using another RSS feed such as http://drupal.org/node/feed, and that gets converted to nodes.
But when I use a twitter feed, such as http://twitter.com/statuses/user_timeline/17000175.rss , it does not parse into nodes. (However, the "Link to site" link does link properly.)
Finally, the core aggregator module does parse the twitter user timeline feed properly.
I don't know if it's a bug in the format that twitter is using, or if it's a problem with the parser_common_syndication module.
| Comment | File | Size | Author |
|---|---|---|---|
| #9 | parser_cs_intra_page_cache.patch | 1.45 KB | aron novak |
Comments
Comment #1
aron novakWell, I just debugged to hunt this bug.
The twitter uses really strange method: if you download the same feed twice, for the second download, the feed will be empty.
It's hard to answer what to blame. Twitter does not seem to handle If-Modified-Since, parser_common_syndication tries to use it to avoid double downloading. SimplePie works fine. I also used twitter feeds with FeedAPI, as I remember the authenticated feeds worked fine. So two choices:
- switch to authenticated feeds
- switch to SimplePie
Comment #2
nadavoid commentedAron, thanks for the suggestions. I was able to get it to work, but I had to switch to SimplePie. Using authenticated feeds with parser_common_syndication still didn't work for me.
Glad I've got that working on a test site now. thanks again!
(authenticated feeds = http://myusername:mypass@twitter.com/etc/etc.)
Comment #3
niklp commentedI've been thrashing with *exactly* the same problem all afternoon.
I can't even figure out what Twitter is up to - I can see my favorites on a NON-authenticated feed just fine, but cannot get them into feedapi for love or money.
I can wget them from any server, see them in any browser (except when I'm logged in, which is REALLY weird) but nothing in feedapi. Very annoying!!
Comment #4
gsnedders commentedTwitter responds to a request with If-None-Match/If-Modified-Since with a feed with no items in it if nothing has changed, and doesn't return status 304.
Comment #5
niklp commentedThat's all well and good, but how does it know whether or not something has changed if I've just added the feed??
Surely there must be a way to force it to return some values? Otherwise this is just odd...
Comment #6
thsutton commentedContrary to the suggestions above I disabled SimplePie parser and switched to Common syndication parser and unauthenticated Twitter feeds started working again.
Comment #7
webchickSubscribing.
Comment #8
Inte commentedsubscribing
Comment #9
aron novakHere is a patch to prevent parser common syndication to download the same stuff twice if the feed does not support etag/not modified since correctly.
I tried it out and for me it helped in the case of twitter timelines at parser common syndication.
Please give it a try and do feedback here :)
Comment #10
aron novakI committed the patch because it did not break simpletest tests :)
Please reopen the ticket if you have still problems with twitter + parser_common_syndication
Comment #12
swentel commentedTwitter feeds are imported when you click on refresh items on the node, however, when running cron nothing is imported at all, even after a few hours and really do now there should be new content .. applied the patch on latest official release, should I try the dev release or not ?
Comment #13
niklp commentedI've been using dev with no problems. Not completely sure about the twitter issue, but it *seems* to be fixed...
Comment #14
Shai commentedI was having the same problem. No items being returned for a Twitter feed. I found the following solution which is working flawlessly for me. I have no idea why it is working and I totally stumbled upon it.
What you need to do is remove all items from the feed. Then refresh after that.
Why do you need to remove all items when there weren't any there in the first place? I don't have a clue, but it works.
p.s. This is working for "refresh" but cron isn't pulling in new. I'll report back.
Shai
Comment #15
swentel commentedYeah, the 'remove items and refresh strategy' works perfectly, but refreshing them manually like say once per day is not an option. I'm also looking into it more, will report back too if I find some solution.
Comment #16
Shai commentedHere's a workaround that works with cron. It's a workaround... that works.
Don't get the feed url from twitter.com but rather from search.twitter.com. Go to the advanced search option and fill out the form putting the user name whose feed you desire in the "from" field. The results from this search are exactly the seem as what you would get from the regular feed for a user at twitter.com.
The only difference I can tell is that the search feed doesn't output the user name in the body of the tweet like the twitter.com feed does. In a way, it is better this way if you are pulling in tweets from just one user. The user name thus doesn't repeat for each item.
But if you do want the username, it is mappable via feedapi_mapper.
Shai
Content2zero
Comment #17
Shai commentedI don't believe that my workaround listed in #16 works, or works consistently. This is a serious problem.
Twitter feeds work absolutely fine in Aggregator module... not that I'd consider going back to that :)
Shai
Comment #18
peter panes commentedI'm having the problem where everything works fine except the feed not getting updated by cron.
Can anyone confirm if the dev version of feedapi fixes the cron problem?
Comment #19
niklp commentedIMO there's not much point *not* using the dev version - I'd give it a whirl; of late, I don't seem to have had any problems with it.