On myfeed.gr we are using the module activitystream and we have cron problems cause on every cron job module works for 100 + user’s feed.

My question is on every cron job called the function _activitystream_save and if there are new feeds the function save them as nodes but if feeds exist the fuction update the node,right?

	if ($new) {
		node_object_prepare($node);
		node_save($node);
	  $actions = db_query('INSERT into {activitystream} (nid,module, guid, link, data) VALUES (%d,"%s", "%s","%s","%s")',$node->nid,$name,$activity['guid'],$activity['link'],$activity['data']);		
	} else {
		node_save($node);
	}

So on every cron job the module update 3000 nodes and we get time out error for alloweded memory size.

If we remove the last "else { node_save($node); }" ,we will save some bytes from memory,right?

In that case, module will running without any problem?

Comments

HorsePunchKid’s picture

HorsePunchKid commented

I'm coming up against this issue, too, I think. I pretty much want Activity Stream to pull the nodes in and leave them alone once they're there. For example, I've hooked up a Google Reader feed, which has put a couple of nodes up on my site. I'm not quite ready for them to be showing up yet, so I demote them or unpublish them. But the next time cron runs, they revert and are up on the front page again.

akalsey’s picture

akalsey commented

The module does this to pick up updates in the feed. Consider this use case...

1. You add an item to Flickr.

2. Activity Stream updates and pulls the item into your stream.

3. You notice a typo in the Flickr title, so you update it at Flickr.com

If Activity Stream didn't save every item on update, it wouldn't ever pick up the typo fix. node_save() doesn't take any additional memory. The node object is already populated by that point, and the node object is passed by reference to node_save. If you're running out of memory, there's perhaps a different issue. Do you have the devel module installed? That's a common cause of memory issues during bulk operations. Devel keeps track of all the db queries and slowly sucks up memory with that log.

Ad for HorsePunchKid, if you want to set the items to unpublished, go into your content types and change the setting for the activity stream content type to make all nodes unpublished.

akalsey’s picture

akalsey commented
Status: Active » Closed (works as designed)
skullJ’s picture

skullJ commented

News aggregator module does not seem to update items with node_save() and its really good module and work perfectly!

Temporary i fix the memory limits errors but its not good idea to give it in public...

www.myfeed.gr updates feed items of 60 users on every 2 minutes. Original Activitystream module have problem to update up to 25 activitystream_modules' accounts and this is the weak point of it!

akalsey’s picture

akalsey commented

Sucking down the feeds from third party sites every two minutes is a good way to get blocked by the remote servers. Generally, anything more than every 15 minutes is bad etiquette.

You're not going to have that problem with Activity Stream, though. For stream sources that come from feeds (in the included modules, that's everything but Flickr), Activity Stream's feed parser caches the feeds for 15 minutes. No matter how often you run cron, it gets the source feed no more than four times an hour.

If you truly want to pull feeds in near real time, you're going to run into all sorts of issues. A single feed parser has a limit to the number of feeds it can download and parse in a minute. At each cron run, Activity Stream goes through each feed, downloads the source feed (if it's not cached), parses the feed and inserts it into the database. The downloading part is what takes longest and is unpredictable. For large commercial sites, you can assume that the whole connect and download process takes around 1 second. The slower the server, the longer this will take, just like browsing anything on a web site.

So if you were to fetch every feed fresh from the server, you'll only be able to get 50-60 feeds per minute. With 60 users, each using 10 different sources, you can expect the cron run will take about 10-15 minutes to complete. Running every two minutes means that one cron run hasn't completed before the other one has a chance to start.

Solving this problem would require distributing feed fetching across multiple parallel processes. This is WAY outside the scope of Activity Stream. A massively parallel feed parsers similar to those I've designed at Pheedo and Feed Crier would require Activity Stream to become a module that couldn't be installed by mere mortals. You'd have to set up forking processes, job queues that run across many servers, and generally have a bit more access to your server than the average Drupal user has.

As for memory usage, it's unlikely that Activity Stream is directly causing the problem. I've run it with several hundred feeds and encountered no issues. Profiling memory usage of the module by itself reveals no memory issues, other than a few hundred bytes more in usage during each pass.

skullJ’s picture

skullJ commented

I changed all cron function in Activitystream module and a huge part from system to cron runs every low-limit that php.ini can handle!
Im using Poormanscron to delay if a cron run not complete and starts after delay from the spot that stops. Somehow my cron function checking for updates and if there is a big one, delay gives as much time as module needs.

2 minutes is the lowest limit that i can update my site, its complicate to tell exactly the time to updating my site and depends of new feeds for downloading. The max limit is 15 minutes.

Of course this is a test and its working fine for 15 days now...i didnt met any server problem yet!

akalsey’s picture

akalsey commented
Status: Closed (works as designed) » Closed (fixed)

You changed the module, and then filed a bug report against the module?

There's no way at all that I can offer support for code that I didn't write and have never seen. If you've altered the module, you're on your own. I'd look to the code you altered for memory issues since on a stock drupal install with 500 feeds in Activity Stream I'm unable to produce any memory leaks.

You should also realize that the 15 minute cache on the feeds doesn't have anything to do with php.ini or cron. You can ask Drupal to get them every 5 seconds if you want, but you're wasting your time. You're only fetching them off the local disk. The feeds are stored on disk for a minimum of 15 minutes before the module will fetch from the live site again.