Is anyone running a site with hundreds of feeds, or even thousands? I have a site that will start with around 100+ feeds, but within a year the hope is to be processing 1,000+ feeds every 15 to 30 minutes.. Is FeedAPI capable of handling this type of action yet? Will the cron execution time be the death of it?

Thanks for your time

Comments

aron novak’s picture

aron novak
Primary language Hungarian
Location Hungary, Budapest
commented

Well, i can suggest you to benchmark it. Create a test site with 1000+ feeds, provide enough resources and see what happens.
And it really depends on the server environment.

alex_b’s picture

alex_b commented
Status: Active » Closed (fixed)

Mike: it all depends on your hardware and the types of feeds your processing. the more feeds, the higher frequency of new stuff on the feeds, the better hardware and DB optimization you will need.. at some point you will hit a limit, 1000 feeds should be fine. 2000 too.

You will need to benchmark yourself.

virtualdrupal’s picture

virtualdrupal commented

Thank you that's a perfect answer, it's on a dedicated quad core with 4gb of ram, and the server power can be increased. My main concern was whether I'd start experiencing failed cron runs that couldn't be overcome with more power, I saw the other issues regarding cron time outs and they were using far less feeds.. probably a shared server environment though.

Thanks again

iandit’s picture

iandit commented

I'm running on a dual quad core with 4 gigs of ram and I'm still experiencing alot of failed cron. I've built few "mappers" for our site so I'm trying to optimize them but anything I do so far hasn't helped me in the dreaded cron timeouts.

So will keep you posted on how I'll be doing. Who can tell me how they've done to make the cron timeout work? or how they've optimized their system to make large quantity of feeds work properly. so far I think I have around 1000 feeds and it does fail once or twice a day. Not good as I always have to check since i sucks if the feeds aren't updated.

Thanks for any help.

iandit’s picture

iandit commented

alex_b: what would you suggest as DB optimization? Are you using MyISAM or innoDB? Please share us some thoughts on how you would optimize the DB, that would be greatly appreciated.

Thanks.

jday’s picture

jday commented

I am trying to pull in posts from two blogs, each with about 100 posts. Only the last 25 posts from each blog have been imported as nodes, I've unchecked the 'update existing nodes' option and run cron but no new imports...am I missing something?

virtualdrupal’s picture

virtualdrupal commented

That's my exact setup, scary.. Please keep us posted, hopefully this thread can revolve around DB optimization to circumvent feed count restrictions.. Having 1,000+ feeds is probably going to become more and more common...

cedarm’s picture

cedarm commented

As for mysql database engines, we've found InnoDB to be a better choice for our needs. Again, benchmark it yourself, on your hardware. This is how we determined what was best for us.

Also, keep in mind concurrency issues. Requesting thousands of pages sequentially is quite different from ten jobs requesting hundreds, or a hundred jobs requesting tens of pages. You'll likely see the biggest difference between MyISAM and InnoDB here.

For DB tuning, try out the script from http://www.day32.com/MySQL/. If using InnoDB, make sure your innodb_log_file_size is large enough. If you need to change this on a running mysql server, look at http://dev.mysql.com/doc/refman/5.0/en/adding-and-removing.html. We have innodb_buffer_pool_size=1G and innodb_log_file_size=250M.

izmeez’s picture

izmeez commented

subscribing because this discussion on db optimization is interesting, Izzy.