all normal, crawler starts, then access denied [#563380]

Comment	File	Size	Author
#35	boost-563380.patch	4.4 KB	mikeytown2
#34	boost-563380.patch	920 bytes	mikeytown2
#25	boost-563380.patch	965 bytes	mikeytown2
#2	robots.txt	1.66 KB	dbeall
	htaccess.txt	7.12 KB	dbeall

Comment #1

mikeytown2 commented 29 August 2009 at 23:15

Here's my guess...
Crawler grabs all url's from your url_alias table, inside that are some non-published url's that also have an alias. Crawler hit's it and gets a 403 as expected. Can you check to see if this is the case; is the access denied message legitimate? Looks like I need to make the url_alias table import only get published url's.

Log in or register to post comments

Comment #2

dbeall commented 29 August 2009 at 23:18

Status	File	Size
new	robots.txt	1.66 KB

could this be from the robots file?

addition info: normal install, single domain, shared host, installed in root.
running poormanscron.dev

edit: I will have a look, but I am almost positive all aliases are published and available to all users

Log in or register to post comments

Comment #3

mikeytown2 commented 29 August 2009 at 23:23

eh scratch that... for some reason the menu hook is not working for anonymous users.

access callback inside boost_menu() isn't happy...

Try doing a flush_all_caches on the performance page at the bottom.

Log in or register to post comments

Comment #4

dbeall commented 29 August 2009 at 23:23

There was one page not published as it was a base path for views tabs page, changed it to published. Have cron set to every hour for now, will see.

Log in or register to post comments

Comment #5

dbeall commented 29 August 2009 at 23:25

flushed caches on performance page. you want in there...

Log in or register to post comments

Comment #6

dbeall commented 30 August 2009 at 00:47

still same result:
Db log:
access denied 08/29/2009 - 7:42pm boost-crawler Anonymous
boost 08/29/2009 - 7:42pm Crawler Start Anonymous
boost 08/29/2009 - 7:42pm Expired stale files from static page cache. Anonymous

user 08/29/2009 - 6:24pm Session closed for batman. batman (this is last log out)
boost 08/29/2009 - 6:24pm Flushed ALL files from static page cache. batman (cleared cache performance page)
content 08/29/2009 - 6:22pm page: updated Service BASE PAGE. batman view (set page to published)

Log in or register to post comments

Comment #7

mikeytown2 commented 30 August 2009 at 00:54

would you mind running this in a php block?

echo db_result(db_query("SELECT access_callback FROM {menu_router} WHERE path LIKE 'boost-crawler'"));

It should return 1

Log in or register to post comments

Comment #8

dbeall commented 30 August 2009 at 01:04

I put it in a block, on front page rt, bottom.. It just shows the php code.. howto?

Log in or register to post comments

Comment #9

mikeytown2 commented 30 August 2009 at 01:05

Input format should be PHP code. Select below the text area

Log in or register to post comments

Comment #10

mikeytown2 commented 30 August 2009 at 01:06

you might have to enable the php filter, on the module page

Log in or register to post comments

Comment #11

dbeall commented 30 August 2009 at 01:13

I added an input format as php with no filters, block shows, but no code and no characters.
..never have used php format,.. am i doing something wrong... can you say noob...

edit... ok module page...

Log in or register to post comments

Comment #12

mikeytown2 commented 30 August 2009 at 01:14

ok that means for some reason that didn't get set right... Can your try pressing the 'Clear cached data' button at the bottom of the performance page again? Then check the output from the block again.

Log in or register to post comments

Comment #13

dbeall commented 30 August 2009 at 01:16

damit. added module, shows php code selection, selected. now block shows nothing or no block

Log in or register to post comments

Comment #14

mikeytown2 commented 30 August 2009 at 01:19

Do this to keep your self sane. It should print output before the result from the database.

echo 'output: ' . db_result(db_query("SELECT access_callback FROM {menu_router} WHERE path LIKE 'boost-crawler'"));

Log in or register to post comments

Comment #15

dbeall commented 30 August 2009 at 01:26

cleared cache, php input format is set to default, all users, no block shows.

Log in or register to post comments

Comment #16

mikeytown2 commented 30 August 2009 at 01:28

what region is the block located in? footer is a good place.

Log in or register to post comments

Comment #17

dbeall commented 30 August 2009 at 01:30

out put==1
.. I hate being stupid...

Log in or register to post comments

Comment #18

mikeytown2 commented 30 August 2009 at 01:33

go ahead and disable the block, the database has the correct setting. Run cron and let me know; after this I'll have to think of another reason why it wouldn't be working. I hoping the 'Clear cached data' this time did the trick. It's now in my list of things to do #563408: flush all caches before releasing RC3

Log in or register to post comments

Comment #19

dbeall commented 30 August 2009 at 01:34

block is right, bottom, front page.. not an issue, my client(friend) is just tickled...

Log in or register to post comments

Comment #20

dbeall commented 30 August 2009 at 01:37

ok, back to normal... This is not a priority, as in this second... a work in progress.
Did I say thank you...

Log in or register to post comments

Comment #21

dbeall commented 30 August 2009 at 01:43

cleared all caches, including the performance page and ran manual cron before logout.

Log in or register to post comments

Comment #22

dbeall commented 30 August 2009 at 01:57

don't know if this would help, I do have access to set php.ini
I assume clearing Boost database(clean boost database, use caution) is left as is or(not cleared)

Log in or register to post comments

Comment #23

mikeytown2 commented 30 August 2009 at 02:12

yeah you don't need to clear the boost database

Log in or register to post comments

Comment #24

dbeall commented 30 August 2009 at 03:17

update, no change.
cron 08/29/2009 - 10:13pm Cron run completed (via poormanscron). Anonymous
cron 08/29/2009 - 10:13pm Cron run completed. Anonymous
warning access denied 08/29/2009 - 10:13pm boost-crawler Anonymous
boost 08/29/2009 - 10:13pm Crawler Start Anonymous
boost 08/29/2009 - 10:13pm Expired stale files from static page cache.

Thank You.
Will be checking out your new stuff tomorrow..

Log in or register to post comments

Comment #25

mikeytown2 commented 30 August 2009 at 07:32

Status:

Active

» Needs review

Status	File	Size
new	boost-563380.patch	965 bytes

here is the patch. going to commit it, so it should be in the latest dev

Log in or register to post comments

Comment #26

mikeytown2 commented 30 August 2009 at 08:40

once this issue if fixed, expect RC3 :)

Log in or register to post comments

Comment #27

dbeall commented 30 August 2009 at 12:54

been up sudo-ing all night(lol), finally got my first networked linux(lamp) server running(rough, but running)named it DRU.
by the time i get sleep, your commits will be cycled or i will apply patches where needed.
-The php(block) server test you did, checks to see that mysql is responding properly to boost? yes.
I noticed this was the URL listed in details at access denied time, http://bexleymarine.com/boost-crawler?nocache=1

Log in or register to post comments

Comment #28

mikeytown2 commented 31 August 2009 at 03:33

@dbeall, have you tried the latest dev yet?
ps this is how I do it with XAMPP
http://drupal.org/node/505974

Log in or register to post comments

Comment #29

dbeall commented 31 August 2009 at 09:08

well, still getting same result, access denied. It just has me thinking that I am doing something wrong. ?
You know what they say, It's not the computer, It's the idiot using it(that would be me).

Log in or register to post comments

Comment #30

mikeytown2 commented 31 August 2009 at 09:23

@dbeall
what modules are installed? Out of those what has to do with access control?

$result = db_query('SELECT name FROM {system} where status = 1');
while ($output = db_result($result)) {
  echo $output . "<br>\n";
}

Log in or register to post comments

Comment #31

dbeall commented 31 August 2009 at 10:20

have block visible www.bexleymarine.com front page, rt side, bottom. No access controls that I am aware of other than core permissions(which are mostly open to all). There are only 2 auth users, admin(me) and the business owner.

also running test on my site db.com

Log in or register to post comments

Comment #32

dbeall commented 31 August 2009 at 14:21

This may or may not be an issue:
noticed that the access denied has my host IP address listed.. Is there a way to assign an IP other than the host for crawler?
Trying this:
I have set an 'allow host rule' in the Drupal access settings and added the host IP address into htaccess for 'allow'.
It has to be something, because it faithfully tries at each cron run.

Log in or register to post comments

Comment #33

dbeall commented 31 August 2009 at 17:05

nope, adding rules to allow host IP address didn't do anything.

Log in or register to post comments

Comment #34

mikeytown2 commented 31 August 2009 at 22:27

Status	File	Size
new	boost-563380.patch	920 bytes

K figured it out... now I need a solution. Your guesses at the IP address where correct, but for another reason. I do a test to make sure the crawler can only be called by its self, which does a simple IP address comparison. Your server doesn't set the $_SERVER['SERVER_ADDR'] variable. I might end up doing something similar to a cron key, thats now in drupal 7 core, in order to protect access from outsiders http://api.drupal.org/api/function/system_update_7001/7. I should also put in a test on the requirements page to make sure drupal allows access to this url.

Attached is a patch that removes this logic, but it's not the correct solution. It's a way to confirm that this is the problem. Apply the patch or remove-manually/comment-out the 4 lines and let me know if this fixed it.

Log in or register to post comments

Comment #35

mikeytown2 commented 31 August 2009 at 23:36

Title:

all normal, crawler starts, then access denyed

» all normal, crawler starts, then access denied

Status	File	Size
new	boost-563380.patch	4.4 KB

this should do it

Log in or register to post comments

Comment #36

dbeall commented 31 August 2009 at 23:55

Status:

Reviewed & tested by the community

» Needs review

!!! You nailed it !!!
mikeytown2 == Boost Drupal PHP server wizard

Log in or register to post comments

Comment #37

mikeytown2 commented 31 August 2009 at 23:47

Status:

Needs review

» Reviewed & tested by the community

Log in or register to post comments

Comment #38

mikeytown2 commented 1 September 2009 at 00:13

Status:

Needs review

» Fixed

committed

Log in or register to post comments

Comment #39

dbeall commented 1 September 2009 at 00:23

Possibly the final post here, and it should read something like this;
Thank You mikeytown2 for your patients, persistence, goals, knowledge and know-how.
The developers do play a huge role in making Drupal the best it can be, what a crowd.
Boost is awesome, get yours today!

Log in or register to post comments

Comment #40

15 September 2009 at 00:30

Status:

Fixed

» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Log in or register to post comments

all normal, crawler starts, then access denied

Comments