It seems that the crawler keeps resetting and crawling way more links than necessary. I am trying to setup Boost so that everything stays cached indefinitely unless a new node has been created or an existing node has been updated. After that, only the pages that have been flushed should be recrawled. I have cron set to run every 15 minutes. I have the following option checked, thinking it might prevent the crawler from hitting stuff that doesn't need to be re-cached:

Check database timestamps for any site changes. Only if there has been a change will boost flush the expired content on cron.

It doesn't seem that the crawler checks the database. It just crawls every time cron runs.

Also, I keep getting this message in the logs:

Crawler - Restarting with 1 thread, to try & get the stubborn urls cached.

As an example, I just updated a node that has the following taxonomy terms: Red, Apple. After saving, the cached node page and the term pages for Red and Apple should be flushed. This works as expected. Nothing else has changed on the site. On next cron run, the crawler should only hit those pages and not the entire site. Otherwise, it is extremely resource intensive. I received an email from my hosting provider about using too many resources and this is on a VPS server.

Is there any way to configure the crawler so that it only hits pages that have been flushed after insert/update/delete?

Comments

szy’s picture

szy commented

'Indefinite cache' - you mean interminable? How have you defined it?

Have you unchecked 'Crawl All URL's in the url_alias table'?

Szy.

Coupon Code Swap’s picture

Coupon Code Swap commented

Crawl All URL's in the url_alias table is unchecked. I have tried setting expiration to one hour and also one day. the crawler continually queues all the URLs everytime cron runs.