My current test site is a Drupal 6.13 multi-site setup with only a few pages. I am running the Aug. 6 dev version of Boost.
I enabled Preemptive Cache HTML on the Performance admin page. I then enabled Preemptive Cache: Crawl URL's with the boost block setting to yes on the Boost Settings admin page.
After running cron a few times, the Stop Crawling button on the Boost Settings admin page shows 75 out of 11 URL's done. 11 pages were in the cache, but the 75 out of 11 does not make sense. I don't have 75 pages. I expected the first number to be less than the total URLs to be crawled.
I ran cron again, and the Stop Crawling button shows 75 out of 1 URL's done. The boost_crawler table has only one row.
I ran cron again, and the Stop Crawling button shows 0 out of 0 URL's done. which makes sense because all my pages were cached at this point.
Cron ran automatically and now I show 50 out of 0 URL's done.
The crawling/caching part seems to be working OK, but the button display is off. I am assuming the second number in the button is the total number of URLs queued up to be crawled since the last cron and the first number is the number of URLs that have been actually crawled out of the second number.
| Comment | File | Size | Author |
|---|---|---|---|
| #5 | boost-544476.patch | 4.64 KB | mikeytown2 |
Comments
Comment #1
mikeytown2 commentedyeah, it's way off, and I'm not sure of a good way to make it accurate without hurting performance. Each crawler thread grabs 25 urls at a time and then updates that value so the next thread will grab the next chunk. Also I should report the crawler status, if its running or if it stopped, and change the buttons text accordingly. Setting the button so it caps at the total # of urls would be the way to fix the ludicrous number report. Making a note that it is inaccurate will be helpful as well.
Comment #2
mikeytown2 commentedI should also make it only spin up 1 thread if you have less then 50 urls to crawl, 2 if less then 75, ect...
Comment #3
Tally commentedI personally do not need a button on the admin page to show the current detailed status of crawling. I vote for getting rid of the boost_crawler_position number on the button.
I think the button display would be fine if it indicated that crawling is active and maybe the total number of URLs to be crawled. That way, you will know that clicking the button will do something. After clicking, the 0 URLs listed will inform you that crawling has stopped.
Maybe have the button display "Stop Crawling. x URL's to crawl" when crawling is active and "Crawling not active. 0 URL's to crawl" otherwise.
If someone wants an up-to-the-minute status of crawling, then that would be something to put in a block.
Comment #4
mikeytown2 commentedIt's useful when doing 10k + pages. Lets you see the progress; allows you to stop it if needed since it could run forever; or if you see that it is taking too long. Making it do a countdown is an idea I like. Up to the minute stats would be a waste of processing power.
Have a variable that holds the number of running threads. Once crawler is done decrease that by 1.
Make
variable_set('boost_crawler_position', ($from + $step));atomic. This is not in its current form.Comment #5
mikeytown2 commentedhere's your request as well as all of my proposed changes.
Comment #6
Tally commentedYou are awesome! I tried doing a patch yesterday. I started along the same path as your patch, but I was in way over my head. :) This is a great learning experience for me.
I am reviewing your patch. I found some minor spelling errors. I will continue reviewing and testing it over the next few days.
Comment #7
mikeytown2 commentedcommitted this since it has some other important changes that I need to be in. Please open a new issue for spelling errors and other things.