Pulling activity from exteral sources [#896850]

This is a feature when talking to Hernani at DrupalCon Copenhagen in the hotel :)
He had created a stream building that would pull data from a twitter alike system. So, it would be very nice when having various external sources like facebook, twitter, status.net, etc ... being pulled into the heartbeat activity streams.

This could be a submodule with UI configuration or by hooks. Here i have not fully made up my mind but it would be nice if people had their opinion shouted here.
So in short, pulling and analysing the xml or json strings from external sources for an authentication, provided by user profile setting or something, the fields should be mapped into the heartbeat activity table fields. So this is what i ment with hook or UI with a mapper. The latter is far more work of course.

Comments

Comment #1

yeeloon CreditAttribution: yeeloon commented 1 September 2010 at 17:25

+1 subscribing

Comment #2

hernani CreditAttribution: hernani commented 2 September 2010 at 22:18

Hi Stalski, sorry for my lateness, i arrived back two days ago and I was still a bit offline.

As twitter and status.net now both support oAuth i started to rewrite my module in order to fully use it, and i have still some advances with status.

It would be really interesting if we could use heartbeat to show the info retrieved by this module and if somehow we would be able to push updates to these related platforms.

What would be needed to assure that heartbeat is able of doing it?

Hernani

Comment #3

Stalski CreditAttribution: Stalski commented 3 September 2010 at 07:25

I now it can work . So big assurance ;)

However, the pushing is another matter. I would not include this in this issue (feature request) as it has more reprecussions.

Comment #4

Andreas Radloff CreditAttribution: Andreas Radloff commented 5 November 2010 at 09:25

+1! I would love the feature of being able to pull twitter streams into the heartbeat feed.

Comment #5

Stalski CreditAttribution: Stalski commented 9 November 2010 at 18:48

Status:

Active

» Closed (won't fix)

Just wanted to let you know that it already can be done very easy with custom code. We did that in some other projects.
For instance a module that pulls the activity from somewhere else, and you just log the activity in the heartbeat_activity table. That does it.
Or some code that invokes a rule, so you can add the activity by rules ...
I think this one - if it is a module worth - is maybe better to have as separate contribute. The problem is especially that their would be just too lot of code in heartbeat to do this here.
So i think this is a very good suggestion, but not in core heartbeat for drupal6.

Comment #6

ifuyivara CreditAttribution: ifuyivara commented 3 March 2011 at 16:34

I am about to create a sub-module that does this. It will depend on feeds module to fetch the data from an Atom RSS and save it into heartbeat structure. I'm also planning on implement PubSubHubbub compatibility to improve performance and scalability. Any recommendations you may have based in your experience that will make my life easier during the development process?

Thanks!

Comment #7

Stalski CreditAttribution: Stalski commented 3 March 2011 at 16:48

Not much actually. I did it a couple of time custom and it did not make much time. It seems you are far enough to know heartbeat's structure and its objects. You could do it at runtime without saving a local copy or by triggering a cron pulling a feed to save it local.

I don't know PubSubHubbub, but it does not seem to hard. I found that any caching layer adding to heartbeat , or activity what so ever, if is very hard. It seems like the time you get to set a cache it needs to be purged already (well in bigger sites with lots of users ofcourse).

I tried one time with jabber / xmpp but did not get the time to fully see where i could get. I am now checking out node.js , maybe there is a future there so caching would not be an issue as the real activity could live elsewhere then.

Comment #8

ifuyivara CreditAttribution: ifuyivara commented 3 March 2011 at 20:44

I do need to save the data locally (and in heartbeat, to be able to comment, flag, share, etc) triggering cron to save data locally its a good option for small - medium sites. But I'm talking about many users pulling information from feeds (think about thousands). I don't want to be slammed in the face with the "poll problem" in the future when cron is trying to fetch thousands of feeds (been there). This seems to be a problem even if you use the batch implementation.

As you mentioned cache is not a good option but PubSubHubbub seemed like a good approach. So yeah this is definitely not going to be in the heartbeat core but rather in a contributed module. I'll create a thread for that latter after experimenting with this.