tweets are inconsistently imported into tapatio nodes

I want to give a more detailed answer, but the short of it is that you don't need to send @ messages. The assumption was that most people using the system will have a hard time figuring out how to follow a group let alone send @ messages. Also there are two different ways that taptio goes and pulls messages. This is configureable on a site wide basis admin/settings/comms/twitterAPI.

I do think however that you may have found a bug and that this issue should be looked into more and documented, its been so long since I looked at that piece of code.

Yeah please do, also the function you are looking for is comms_cron(). That is where all of the twitter scrapping starts from.

marlon0,

Any chance you will be able to have this documented by Tuesday night?

-evoltech

Right. I understand the issue.

The logic with keeping track of the most recent twitter message id in comms_twitterfrommsgmaxid is to help decrease the amount of load we put on the twitter.com and the tapatio site. For example if we request all messages for a user since the begining of time, this could be a very long list that tapatio would have to parse. If instead we update the max message id after a message is added we are able to say in following requests something like, "give me all messages for user x greater then
".

But there is two race condition problems here. The first is the one you point out. That when a new group is added its followers messages will not be added. I argue that this is by design because we supposedly do not want any messages from users following this group until after we add it.

The second race condition is where user x an y both tweet since the last time we polled twitter. User x tweets after user y, but tapatio searches for tweets for user x. User x's tweet is pulled into the system, and comms_twitterfrommsgmaxid is updated which is greater than the message id for user y. When we poll for user y we will never pull its tweet.

A propose a solution where we track comms_twitterfrommsgmaxid on a per user basis. I will look into making this change unless you are feeling gung-ho marlon.

Feedback?

This is a performance hit for the drupal installation and increased network traffic. This is however the solution I am considering for #570170. There is a general problem with collecting tweets in that it is really resource intensive, and if it is done via cron through drupal I agree that we should be as conscious about this as possible, especially when a system is configured to be highly responsive (cron running every minute).

This said, we need some solution. Maybe we could benchmark and see if this is a significant concern?

Thanks! My drupal brainwashing tells me that each users maxmsgid should be kept in the user profile, but I guess it doesn't make a difference that it is stored in the variable table. I will test this patch out in more detail later.

FYI, please make future patches against the development branch.

OK. I am applying this patch right now, and will be attempting to benchmark it to see if I can make some conclusions about the capacity of this solution.

In the benchmarking I specifically tried to answer the following questions about the cron function, where the majority of the twitter interaction happens (pulling in new tweets and checking for new followers for registered groups).

1) What functions are taking the most time? Hands down this is $twitter->getFollowers and $twitter->getTweetsForUsers, as expected because they are making requests to twitter.com through the API. On my box and network $twitter->getTweetsForUsers was taking .1205 to .3604 seconds per call (depending on if tweets are pulled in or not) and $twitter->getFollowers was taking .1611 seconds per call. It should be noted that both of these times are most likely when a request via the API is refused. The actual addition of a comms node takes .0178 seconds.

2) Is there a significant performance hit by implementing a new tweet check for each follower vs performing a tweet check for 6 followers at a a time? Not really. There definitly is a performance increase for the first run though. The old $twitter->getTweetsForUsers, which tried to get tweets for 6 users at a time could take up to .1218 to 2.4575 (the larger number being an initial startup number).

3) Assuming that cron will be called once a minute what is the most number of followers that can be queried in a minute? Also assuming that times above are consistent, the same hardware, the same network latency, and the same twitter latency (I know, a lot of "ifs" right?) then we have the following capacity per minute:

old method (6 users at a time):
%74 percent time spent trying to get followers: 329 groups.
%20 percent of the time spent getting tweets: 99 followers.

new method (1 user at a time);
%21 percent time spent trying to get followers: 83 groups.
%77 percent of the time spent getting tweets: 136 followers.

We should as a result be able to service more followers, but obviously this is not enough capacity for any kind of real use, hence #570170: Add a standalone daemon for scrapping twitters search api..

This patch, along with a few changes I made while doing the benchmarking are now applied to both the stable branch and dev branches. Thanks again marlon0!

It turns out there is a Rate limit on the search API (scroll down to the section titled "Search API Rate Limiting"). So this patch increases the load that comms_cron hits the searc API 6 fold. I ran into this on a system with 146 followers running cron once a minute (146*60=8760 requests per hour, where each request is assigned to an ip addr).

Acording to the doc above, there are work arounds, such as including a unique User-Agent header, and respecting the HTT 503 message Retry-After header's value.

Then there is also white listing for a deployment per ip address. I am going to be doing some more looking into this over the next couple of days and will post back here.

Another problem that is raised with storing the max messid only with in the user profile is that when the system finds a new follower of a group it will then try and pull in all the tweets for that user. I can't think of a use case where this would be desireable. Maybe when we pull in tweets for a new user (one without a maxmsgid) then we only take the most recent tweet? Or maybe we save a system wide maxmsgid and if a user does not have one then we use the system one?

Here is a patch for TwitterSearchAPI.php that will allow using proxies for requests since twitter rate limits by ip. This is incomplete because comms_cron.php needs to get patched to make use of this by providing an array of proxies to the API constructor. If we are to do this then we should probably implement some admin interface to add proxies.

Subscribing, greetings, Martijn

I am closing this issue, since it's for a Drupal version no longer supported.