Hello all, i'm using boost with the crawler enabled, and i was thinking, can the crawler increase the bandwith usage? This because i'm having way much usage then normal! The normal is about 2GB/day, now it's about 6GB/day! This without an increase in page views and visitors.

Comments

demon326’s picture

demon326 commented
Category: support » bug

I did some testing, and it seems that the boost crawler is ignoring the not index setting for some pages, in detail, the calender module pages. I told boost not to index /event/*, but it still tried to crawl those url's, because the drupal log said: "crawler already running", and this only on the event pages. and because of this loop, my site was using to much bandwidth.

Please test it out with the event module....

it's the Event module thats doing strange with boost....