When refreshing a feed Aggregator should check whether hash of feed data is changed or not. This will save Aggregator from invoking the parser and querying the database for duplicates for an unchanged feed. A patch is attached.

CommentFileSizeAuthor
#4 aggregator-feed-hash-291064-4.patch4.56 KBmustafau
aggregator-feed-hash.patch4.04 KBmustafau

Comments

dries’s picture

dries commented

Did this solve a performance issue on your website? The patch looks good but I'd be interested to learn how much queries this saves and whether this could add up to a significant performance improvement.

alex_b’s picture

alex_b commented

We did the same hash method in FeedAPI and saw huge performance improvements.

If a feed hasn't changed, using a hash essentially saves one or more expensive checks on uniqueness per feed item of a feed. With news feeds and 2 queries for uniqueness checking that can be up to 10 X 2 or 20 X 2 queries against a large set of data per feed.

The same algorithm has been implemented by SimpleFeed and by the patch for aggregator rework over here #236237.

dries’s picture

dries commented

Sounds good. I'd be happy to commit this patch -- it's a lot easier to review and commit than #236237: Aggregator rework: extensible API, SimpleXML parser, use taxonomy for categorization is.

mustafau’s picture

mustafau commented
StatusFileSize
new aggregator-feed-hash-291064-4.patch4.56 KB

Small update: Clean hash column when removing feed items.

mustafau’s picture

mustafau commented

@alex_b: Can you RTBC this?

dries’s picture

dries commented
Status: Needs review » Fixed

I gave it another review and committed this to CVS HEAD. Thanks.

alex_b’s picture

alex_b commented

Thank you. Great to see this in.

Anonymous’s picture

Anonymous (not verified) commented
Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.