Hi Mike
Multi-site -- been using Boost more or less successfully for the last few months, still have that odd non-matching gzip mod_rewrite annoyance but I remain a fan!

Boost works well for the most part, it handles multiple sites and seems to play well in normal activity ... except for boost_crawler which I have no end of problems with. Now before I talk about my issues I want to first make sure I do understand how this is INTENDED to work -- with multiple sites sharing a database (including {node} and other content), do you expect the boost_crawler to be called by a single site or by individual sites?

I have two problems right now,
1) some sites just wont function with the boost crawler at all, I just get "page not found" immediately after the crawler starts. I've seen this on and off for various sites, usually refreshing the boost admin page fixes things but today I'm having no luck at all; I've not yet cleared cache but I am trying to avoid that (there are some 300k urls in the cache that I'd rather not lose)

2) I am getting "duplicate entry" records in boost_crawler where one site is trying to put into boost _crawler URLs from another site (that are already there)

So, the second error makes me wonder whether I am just doing things wrong (hence my first question), however I only noticed this with 1.16 ... the first problem is definitely a bug only a problem for me if I am supposed to have multiple crawlers.

In the end, I can work around both issues with an external spider without any real problem, and perhaps that is the Right Answer in any case, but it would be preferable to do this within Drupal (it is then self-contained).

Comments

mikeytown2’s picture

mikeytown2 commented

Crawler is designed to be run once per database. So if your using i18n or domain access then only one of those sites should be calling the crawler and because of the shared database, the crawler will only allow for one instance of it (you don't have to worry about it, if you have multiple cron calls only one crawler will be activated). If your using multisite with different databases then you can have mutiple crawlers running at the same time; this isn't necessarily a bad thing, just be aware of the number of CPU cores on your box; it would be a good idea to set cron to run at different times for each of those sites, due to the increased load a crawler brings.

One thing I've been thinking about is the url_alias setting for the crawler. For your situation it might be best to not use that setting. Right now it will only load url's from the base url of the calling cron. So example.com/en/node/1 will be added to the crawler index, but example.com/fr/node/1 will not be for a cron that was called via example.com/en/cron.php; if using the url_alias setting. Doing this correctly will require tighter integration with the i18n module and the Domain Access module, so each entry in the url alias table will get the multiple bases added to it; 5 sub sites means each url alias gets 5 entires in the crawler table.

Entries in the boost_cache table should be added to the crawler, avoiding the above situation I just described. This means that if that page was hit in the past, it will be crawled in the future. New pages or pages that have yet to be hit will not be crawler until they have been accessed. So running without crawling the url_alias table is not ideal, but it should allow for a much smoother operation.

I just get "page not found" immediately after the crawler starts

Can I get more details on this error?

"duplicate entry" records in boost_crawler

Turn off url_alias setting; if this doesn't fix the issue please let me know. Once I know this is causing the problem I can start to look into a smarter solution.

Dave Kinchlea’s picture

Dave Kinchlea commented
Status: Active » Postponed

Thanks for the reply! Just for the record, I'm not using i18n or Domain Access modules but my setup is similar in concept to Domain Access. There is shared content but distinct system, variables, session and cache tables (and a few unrelated module tables) for each site. I am going to go with the one run approach and see how that goes.

Given your last comment I'm thinking that I need to clear the crawler table and restart. While I had turned off the url_alias settings awhile ago, I didn't clear the table so there are a lot of aliases in there.

Regarding the "page not found", I'm not sure I can give you more information but a lot more testing confirms it is just one of my sites and it is the most minimalistic in it's module choice ... I'm going to assume that problem is entirely my configuration and given that I don't need to nor should I be running boost_crawler from that site, I think we should both ignore the problem. I'm actually really pleased to hear that it need only be run once, to be honest I was going to have to abandon the crawler soon because of the scaling issues, my whole approach to hosting here is based on a sort of master hosting community that smaller, lesser-functional communities are run from. Hearing that boost wants to support that more or less by design just reinforces my belief in your code!

I'm going to clear the boost_crawler table and start from scratch, this time I will not include the url_alias option ... I'm going to set the status to postponed but I will update when I have something definitive to say.

ianchan’s picture

ianchan commented

subscribe

Dave Kinchlea’s picture

Dave Kinchlea commented

Clearing the cache did not solve the problem, I have disabled the crawler and instead crawl my sites externally. I may revisit this at some later date but for now, at least, I'm not putting any more effort into this.

bgm’s picture

bgm commented
Issue summary: View changes
Status: Postponed » Closed (cannot reproduce)

Closing old support request with no activity in a long time.