Currently, the Twitter Module only imports those twitter posts that the user posts himself/herself. However, if the user has retweeted someone else, those retweets are not imported even though they show up on the user's timeline on twitter.com.

Can we have an option to import such Twitter retweets as well? I see that the Twitter API supports this.

I have already made some changes to the code on my site to import these. But if a more formal feature is included in the Twitter module, it would be useful.

Details of the changes I have made to get this working:

  /**
   * Fetch a user's timeline
   *
   * @see http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-statuses-user_timeline
   */
  public function user_timeline($id, $params = array(), $use_auth = FALSE) {
	if (is_numeric($id)) {
	  $params['user_id'] = $id;
	}
	else {
	  $params['screen_name'] = $id;
	}
        //Added so that native retweets are included in the import
	$params['include_rts'] = "true";
	$params['count'] = "3200";

	return $this->get_statuses('statuses/user_timeline', $params, $use_auth);
  }

In addition, I have made the following change in the _twitter_convert_status function which I re-introduced to twitter 3.0 from twitter 2.6 (read more about this on the 5th comment on the thread #985544: {twitter}.twitter_id incompletely stored (final digits are zeroes) due to json_decode limitation in PHP<5.3):


  //*** Introduced from version 2.6 - Start ***//

  protected function _twitter_convert_status($status) {
	$result = (array)$status;
	$result['twitter_id'] = $result['id'];

	if (!empty($result['user']) && is_object($result['user'])) {
	  $result['account'] = $this->_twitter_convert_user($result['user']);
	  $result['user'] = $result['account'];
	  $result['screen_name'] = $result['account']['screen_name'];
	}
	else {
	  $result['screen_name'] = NULL;
	}

        //Added so that native retweets are included in the import - Start
	if (!empty($result['retweeted_status']) && is_object($result['retweeted_status'])) {
	  $result['orig_status'] = $this->_twitter_convert_status($result['retweeted_status']);
	  $result['text'] = 'RT @' . $result['orig_status']['screen_name'] . ': ' . $result['orig_status']['text']; //This is necessary as the twitter API truncates the retweeted text in $result['text'] if it goes beyond 140 characters!
	}
        //Added so that native retweets are included in the import - End

	$result['created_time'] = strtotime($result['created_at']);

	// These come in as objects rather than strings IF they are empty, curiously
	// enough. We want nulls, so we'll special case them.
	foreach (array('in_reply_to_status_id', 'in_reply_to_user_id', 'in_reply_to_screen_name') as $key) {
	  if (is_object($result[$key])) {
		$result[$key] = NULL;
	  }
	}
	return $result;
  }

  //*** Introduced from version 2.6 - End ***//
CommentFileSizeAuthor
#2 import-retweets-1182170-2.patch3.29 KBmichaek

Comments

michaek’s picture

michaek commented

Using include_rts as a parameter is indeed what is _meant_ to work, according to the API documentation. http://dev.twitter.com/doc/get/statuses/user_timeline It works great for manual retweets (those starting with RT) but it has no effect on native retweets (from Twitter.com, forexample).

It may be best to also hit retweeted_by_me to include retweets. http://dev.twitter.com/doc/get/statuses/retweeted_by_me However, that method requires authentication, so it can't be used unless you've enabled OAuth.

michaek’s picture

michaek commented
Version: 6.x-3.0-beta3 » 6.x-3.x-dev
Status: Active » Needs review
StatusFileSize
new import-retweets-1182170-2.patch3.29 KB

I'm providing a patch to provide an option to import retweets for each Twitter account. This patch does not resolve the problem with native retweets from my earlier comment, but that can be raised as a separate issue.

michaek’s picture

michaek commented
Status: Needs review » Closed (fixed)

This is committed to dev, working toward a stable release, and I'm going to consider it closed until issues are raised.

kjholla’s picture

kjholla commented

@michaek: I don't think there is a problem importing native RTs using the include_rts parameter. The native retweets also get imported once you include include_rts = true. However, in the resulting data, the native RTs also show up as though they were manual RTs, i.e. they show up with an "RT @username:" prefixed to the retweeted tweet.

Hence, there is no need to hit retweeted_by_me just for these.

Regards,
KH

michaek’s picture

michaek commented

When I tested it, I definitely wasn't getting the native retweets, though I would get RT rewteets. It could be a difference between using the module with OAuth and without - though I was pretty sure I tested it both ways. I really want to get some test coverage for this, to reduce the informality of determining whether something's supported or not!

kjholla’s picture

kjholla commented

OK. That might be the case. I was testing it with OAuth enabled and hitting user_timeline with the parameter include_rts = true did the trick for me.

It brought it the native retweets with the timeline, albeit with an "RT @username:" prefixed.

The other problem I noticed was that the status text would get truncated at 140 characters, hence in my code you will notice that I am explicitly getting the text from the retweeted status:

        //Added so that native retweets are included in the import - Start
    if (!empty($result['retweeted_status']) && is_object($result['retweeted_status'])) {
      $result['orig_status'] = $this->_twitter_convert_status($result['retweeted_status']);
      $result['text'] = 'RT @' . $result['orig_status']['screen_name'] . ': ' . $result['orig_status']['text']; //This is necessary as the twitter API truncates the retweeted text in $result['text'] if it goes beyond 140 characters!
    }
        //Added so that native retweets are included in the import - End

Regards,
KH

kjholla’s picture

kjholla commented

I think I found another bit which may contribute to the native retweets not coming up on user_timeline.

It is to do with the URL that is used to call the twitter API.

The URL needs to be http://api.twitter.com/1/statuses/user_timeline.xml?screen_name=username...

The Twitter Module currently uses http://twitter.com/statuses/user_timeline.xml?screen_name=username&inclu...

I checked this on a browser and using http://api.twitter.com/1/statuses/user_timeline does seem to work without need for authentication.

Regards,
KH

michaek’s picture

michaek commented
Assigned: Unassigned » michaek
Status: Closed (fixed) » Active

Goodness! Nice catch - the API host was definitely something I thought I'd taken a look at. I'm reopening this until I update the module to reflect this.

kjholla’s picture

kjholla commented
Version: 6.x-3.x-dev » 6.x-3.0-beta4

@michaek

I recently downloaded the beta4 and installed it on my site.

I noticed that though the API host URL has been changed to api.twitter.com from twitter.com.

However, this still doesn't work. I discovered that we will need to include the version number of the API as well.

Hence, the API host URL needs to be api.twitter.com/1

Regards,
KH

Anonymous’s picture

Anonymous (not verified) commented

I popped into twitter.lib.php and changed

'api' => 'api.twitter.com',

to

'api' => 'api.twitter.com/1',

Now it works perfectly. :)

steinmb’s picture

steinmb commented
Version: 6.x-3.0-beta4 » 6.x-3.x-dev
Category: feature » bug
Priority: Normal » Major

#867906: Ability to show full public feed with retweets claim this got fixed but it looks like there still things that are not working properly. Let us keep the discussion in here and make sure it get fixed, once and for all. Anyone tested D7, does it also have this issue?

And yes, from the look of https://dev.twitter.com/docs/api/1/get/statuses/user_timeline does the API version need to be included to get the retweets included in the XML. I also verified this manually doing the tests in #7. Was unable to figure out if this always have been like this or this is do to changes in the API.

rene_w’s picture

rene_w commented

I was also missing retweets running twitter-6.x-3.0-beta9 on Drupal 6.22; with the api change to /1 it started working for me as well. Remaining problem is that some tweets are cut off due to the added RT @... as described above.

Snoopy Pfeffer’s picture

Snoopy Pfeffer commented

I found out that it is no good idea to add '/1' at the end of the 'api' parameter. This only works for the non OAuth requests of the Twitter module. It is better to add that missing '1' in the call function, to not break OAuth:

  /**
   * Method for calling any twitter api resource
   */
  public function call($path, $params = array(), $method = 'GET', $use_auth = FALSE) {
    $url = $this->create_url('1/' . $path);

After that change retweets are shown properly.

Beside that it is possible to use Twitter 6.x-3.0-beta9 together with OAuth 6.x-2.02. For that you need to do the following changes of the Twitter module:

diff ./twitter.inc ../original/twitter.inc
11c11
<   module_load_include('php', 'oauth','oauth.lib');
---
>   module_load_include('php', 'oauth_common','lib/OAuth');
diff ./twitter.lib.php ../original/twitter.lib.php
218c218
<     $url = $this->create_url('1/' . $path);
---
>     $url = $this->create_url($path);
diff ./twitter.module ../original/twitter.module
260c260
<   if (!module_exists('oauth')) {
---
>   if (!module_exists('oauth_common')) {
diff ./twitter.pages.inc ../original/twitter.pages.inc
13c13
<     '#access' => module_exists('oauth'),
---
>     '#access' => module_exists('oauth_common'),
314c314
<   module_load_include('php', 'oauth', 'oauth.lib');
---
>   module_load_include('php', 'oauth_common', 'lib/OAuth');
355c355
<   module_load_include('php', 'oauth','oauth.lib');
---
>   module_load_include('php', 'oauth_common','lib/OAuth');

It would be nice to see these fixes in the new version of the Twitter module.

juampynr’s picture

juampynr commented
Status: Active » Closed (won't fix)

The new Twitter REST API 1.1 returns retweets automatically when requesting the user timeline.

I am working in a new branch (6.x-4.x) and will release soon a new version of the Twitter module for Drupal 6 that uses the new API. The old one will be turned off on March 2013, so there is no point in working on a solution.