By tarbidian on

When my website is viewed using the Lynx browser, I get a 404 error. The links in the menus show up, but the page does not. I am using the Search 404 module, so the Search page shows up.

However when the page is viewed with Firefox the pages seem to show up with no problem. The website is www.howgreenisyourvote.ca.

Does anyone know what the problem might be?

I am concerned about this, because I suspect this problem is preventing google search bots from crawling the site. I have had a few visits from googlebot, but my site is not being indexed.

thanks!

⋅ Categories: Drupal 6.x

Comments

themphill’s picture

themphill commented

Using lynx, if I tell it to send a user agent string of "none" $ lynx -useragent=None http://www.howgreenisyourvote.ca/ I get the proper page. Sending an explicitly set null value results in the 404 page. It seems like there may be some browser sniffing going on. You are wise to have checked your site with lynx, and it may well be that you are turning away the search engine crawlers.

tarbidian’s picture

tarbidian commented

Thanks for getting back to me. It seems that Live and Yahoo have indexed the page, but google still has not. I'm starting to think that maybe my site was sandboxed by google for some other reason (or hopefully, their crawlers just haven't reached me yet).

I am using a pretty basic installation of Drupal and haven't purposely enabled anything to block Lynx. Does Drupal by default do something to block certain browsers (I would be extremely surprised to find out it does, it seems very anti-Drupal to do so)?

gforce301’s picture

gforce301 commented

Drupal blocks no user agents that I am aware of. I do not think core even has the capability built in.

themphill’s picture

themphill commented

On the original poster's site I was able to reproduce the lynx browsing problem. Not so with any other Drupal sites I've checked. Use of the "Search 404 module" could explain the difference.

Google Webmaster Central may shed more light on the original poster's concern over that search engine.

jazzitup’s picture

jazzitup commented

I have the exact same problem (D5 with or without Search 404 module) and even if I've indexed my site with sitemap.xml (by being aware of Google Webmaster tools), I'm being penalized with a low pagerank on Google with no apparent reason, so I assume this could cause that problem as well. Any ideas?

--
www.jazzit.hr

shelleyp’s picture

shelleyp commented

Lynx isn't able to access my content either. I noticed that this wasn't resolved. Does anyone know what could be causing this? I'm not blocking based on user agent.

shelleyp’s picture

shelleyp commented

My ISP has mod_security enabled, and it, by default, filters lynx and curl access.

Either you'll need to negotiate with your ISP, or tell Lynx and curl users to spoof a different user agent.

benjarlett’s picture

benjarlett commented

Getting my ISP to switch off the apache Mod Security module fixed it for me too. Thanks to this thread.