Crawlers hitting /%20 can replace the home page with a 404

I have the same problem on one of my Drupal 6.24 sites. But it doesn't help when I clear cache. Simply disabling caching is the only thing that makes the front page work (i.e. not throw 404).
I think this started happening fairly recently but cannot pinpoint exactly when. It would be "nice" to be able to use caching again..

I am also encountering the same error on a Drupal site. This is critical as the home page of the site shows the error, but none of the other pages do. I also noticed that accessing my site through the http://wwww.mysite.com shows the error, while accessing it through http://mysite.com (no www) does not. You can reach me through my profile page for any clarification needed.

Any idea if the issue could simply be a series of conflicting URL Aliases? Just throwing that out there.

I see from http://drupal.org/node/1097528 that there's a lot of talk about Drupal's optimized CSS files go missing (ex: sites/default/files/css/css_1f1a2a7caec3083a8b9c49b26f39e9b4.css). Are these files actually missing from the file server when the home page 404's? Even when these CSS files go AWOL your pages should still load as opposed to 404.

Any idea if this could be a web host configuration issue related to it's own caching layer (varnish, memcached, cache router, APC, etc.)? Though still not sure why the home page is the only site page being affected.

_rs
aka: @webuxr

There is just too little information. A stack trace by XDebug would be a useful point to start from and see where the problem can lie: a buggy 3rd party module, a cache corruption, an encoding / compression issue, a db misconfiguration...

I just upgraded from an old version of D6 and the home page is indeed showing a 404 while all other pages are working. I turned off caching but that hasn't fixed it. :/

[edit] My temp hack is to make my 404 page the same as my home page and enable the blocks404 module. Bleh!

i have the Cache Exclude module installed set to exclude caching for the front page of our site. I'm not sure if this is the cause of the issue, but i'm sure it's not a solution in my case, as the page not found cached homepage still happens intermittently... and there doesn't seem to be any pattern. I've now completely disable Drupal's built-in page caching on my website to see if it doesn't happen again.

I'm both encouraged and discouraged to see so many threads of people having this same issue on their sites. What's most discouraging is that there seems to be no good way to track/debug this. Any help on what steps i can take to get some sort of error or trace or debug information if/when this might possibly happen so we could start debugging would be awesome!.

Also, i'm pretty sure this is not a server issue, as on our server we have many many drupal 6 sites, and this is the only one having this issue. The only difference between this and the others, mainly, is that this one has LOTS of modules, LOTS of views, and utilizes memcache.

We are also having this problem--randomly, but 2-3 times per week, home page requests return the custom 404 page. We aren't running Boost or Varnish or any other server caching solution. . . . I'm reluctant to turn off Drupal's caching, but I may have to. SchwebDesign, did that work for you?

PS. We're running Core 7.22 on IIS.

I am also having this same problem - randomly the home page is showing a page not found error and the menu is missing. I have my home page set to a specific node number. The home page has a bunch of different views blocks assigned to it. I am not running boost. I have the cache set to normal. Only the home page shows the error. Two of the blocks on the home page use views slideshow. Drupal core and all modules are updated to their latest versions.

We are seeing this happen on a Drupal 7.23 site. It happens infrequently, and we can't figure out why it is happening. Some of the major modules and their versions that may/may not be related:

cacheexclude 7.x-2.3
civicrm 4.3.5
ctools 7.x-1.3
entity_token 7.x-1.2
i18n 7.x-1.9
views 7.x-3.7

Cacheexclude is used to exclude the home page from the cache for an unrelated reason, which I would think should avoid this issue altogether being that it seems everyone else's issue is that the 404 page response gets cached somehow for the homepage.

Can someone confirm whether the 404 page response is getting cached for the homepage? Is that the issue here?

I have a similar situation :

In one of our Drupal 6 multilangual Website, sometime, in some languages (not all) the homepage is replaced by the 404 custom page. Empty the caches resolve the situation for some times.

What i do know :
- When on the 404 pages, the website links are set on default language and not the current language
- I use cache from the performance module, i've APC on the server and that's all (no varnish and no boost module)
- I use views and a lot of views block are cached
- We do not have that problem on the preview server (obviously cache is not activated)
- We have two front servers (8001 & 8002), when on the 404 page rise for example on spanish version -> The 404 rise on the homepage www.mysite.com/es but not on www.mysite.com:8001/es and www.mysite.com:8002/es
- Sometime (but strangely not always) if i add a / after my url, the homepage is good, so with a www.mysite.com/es redirect to 404 page, the www.mysite.com/es/ is good.

I started a new issue under 7.x (https://drupal.org/node/2099807), since this one is marked as 6.x. Hoping that will get someone's attention.

Still an issue for 7.x closing other issue as duplicate.

Wanted to update the thread with our latest findings on this issue…

The problem seemed to occur when Drupal received a request for www.domainname.com/%20 (i.e., a space character after the home page URL). For some reason, Drupal could consider this a 404 page not found, but cache it in the cache-page table under the regular home page URL (i.e., without the space character). With the core page caching turned on, any future requests for the home page would use the cached custom 404 content.

The next cron run would empty the cache and "fix" the problem.

We next learned that the culprit repeatedly asking for www.domainname.com/%20 was the Google Search Appliance bot indexing our site (not a generic Google bot, but the paid-for instance used for internal site searching). Somehow www.domainname.com/%20 had gotten stuck in the index, so every time the bot crawled the site, it was requesting that URL.

We had the GSA reindex the site…. and the www.domainname.com/%20 URL was gone. We haven't gotten the home page 404 issue since.

Obviously, this isn't a definitive proof of solving the problem. We still don't understand exactly how Drupal's core page caching is handling URLs with spaces or why the custom 404 content would get cached under the home page URL. But maybe this will help someone else make further progress.

This was happening under Drupal 7.23 running on IIS.

Please do not change versions

I am experiencing an issue with Drupal 6 on a site that I have taken over, and I think it could be related to this. It is extremely hard to track down why it's happening because the site has a large number of modules and a lot of code and thousands of pages.

The setup:

• Caching for anonymous users enabled
• Aggregated CSS enabled
• Aggregated JS enabled
• The homepage is a page created with panels specified with a path, containing various nodes, views and blocks

The issue:
For anonymous users (i.e. those who get cached pages), they sometimes get no CSS or JS. It's fine for logged in users. It only seems to be a problem for the homepage. If it does happen on other pages, it goes away after refreshing a few times. On the homepage the CSS and JS is getting a 404 error.

Steps to reproduce:
I can reliably reproduce the problem simply by enabling or disabling any module. It doesn't matter what the module is. Flushing the cache manually fixes it and causes the pages to be rebuilt with references to newly built CSS and JS files that exist.

I have found that the reason why enabling or disabling a module causes the problem is because drupal_clear_css_cache() and drupal_clear_js_cache() are called by system_modules_submit(). So submitting the modules form page causes those functions to be called, which deletes the cached aggregated CSS and JS. When an anonymous user then visits the homepage, they get served with the cached version of the page which references CSS and JS files that no longer exist.

I've been investigating how the cache system is supposed to work, and as I understand it the cache files that Drupal creates have a filename that is an MD5 of the list of aggregated files and the first letter of the css_js_query_string variable. This should in theory be the same, so a file with the same name should be regenerated. This happens in the drupal_build_css_cache() function, which is called by drupal_get_css(), which is called by template_preprocess_page() in theme.inc. The problem is that template_preprocess_page() doesn't get called if Drupal has a cached version of the page to serve.

Flushing the cache manually fixes it because it will then create the page again, calling the preprocess functions, and creating the CSS and JS files. The page will then look fine until those files next get deleted by drupal_clear_css_cache(). I'm guessing this must happen when cron runs or some other module is causing it to occur, which is why the homepage breaks about once a day.

Can anyone who knows more about Drupal 6's caching please advise me on what is supposed to happen here so that I can further debug and track down what is going wrong with my site. Shouldn't Drupal check whether the CSS and JS referenced by a cached page exist, and generate them if they don't?

Has anyone found a cause for this problem? I've just experienced it for the first time on my Drupal 6 website and it was a bit of a worry because all the advertising that supports the cost of the site is on the home page!

Anyway, it resolved this time by clearing the cache and rebuilding the permissions, but I'd like to get to the root cause and fix it.

Agreed, however people can still post from their experience and solutions/fixes they tried/found.

For example... one site I had this issue on within the past year the solution was:

uninstall Search 404 module
and instead install Fast404.

I don't completely understand why, but it likely had something to do with memory issues + caching as a side-effect of the Search 404 module taking longer/higher resources to complete and /or how it works as compared to the fast404 module,

Hope this helps someone!

Follow-up: We switched away from the GSA that had been indexing our site, but a few months later, the problem resurfaced. Someone somewhere coded a link to our site that ended in a blank space, and if the home page cache had just been cleared, Drupal would return a 404 page and cache that as the home page.

We added a rewrite rule in web.config to strip out trailing spaces (and trailing %20), and we haven't been hit since.

Can someone post the steps to reproduce this, if there are any? #25 seems reproducible.

I tried to reproduce in 7.x-dev what is described in #17, but although the page ending in example.com/%20 was cached, it did not replace the home page for anonymous browsers.

This seems like a huge problem, and if we can catch it in the act consistently, that would be great. In addition to getting the steps to reproduce sorted out, this issue needs a title and summary update, because it seems to be something like "Crawlers hitting /%20 can replace the home page with a 404" or similar.

I presume this happen when bot users is trying to access the page. I'm using the pingdom to alert me when the website is down. The website wasn't down, but the 404 appeared was on the home page.

I think i'm right. Today after 404 error i checked the report log and i have found theat the ccs file was not found. Optimise css is enabled. I am assuming people (most likely robots) are hitting the site using a cached page (such as google cache). If only the page is cached, the users browser will try to download the "old" (pre-cache cleared) JS and CSS files.

Affected people: have you got $base_url set?

Yes, $base_url is set still getting the same issue.

Steps to Reproduce:
1) Clear all your cache.
2) Then visit your homepage_url/%20 (www.website.com/%20) or any URL/%20 before visiting any other page.
you will get 404 page(This depends on your server mostly IIS gets 404)
If you get 404 then only you can reproduce this issue.
3) Then visit your homepage or that URL(entered in 2) You will get 404.

@nileshlohar Steps to reproduce! Can you please put those in the issue summary and remove the tag? Thank you!

After clearing the cache initially cache tables becomes empty.

immediately after that when we first visit http://www.homepage.com/%20 and get page not found error.
so, that 404 page gets cached in 'cache_page' table with cid as "http://www.homepage.com/ " (Note the trailing space)
by this homepage is not yet cached.

and then when we try to visit homepage it fetches the earlier page from cache as it ignores that space.
As the MySQL ignores trailing spaces with = or IN operator (https://bugs.mysql.com/bug.php?id=64772)

I am attaching the patch for it, not the final one but it solves the issue.
I guess will need more work on it

There is a patch, so changing to "Needs review".

I followed the steps in the issue summary and I cannot reproduce the issue as written.

@cliefen as i said it depends on your server and server configurations.
I am getting it on iis
On which server you tried?

Are you getting 404 after hitting /%20 after any url?

You should get 404 with /%20 to reproduce this issue.

It is Apache 2.4.

Are you getting 404 after hitting /%20 after any url?

Yes

Can you please try with any other inner page or on iis server.

Important thing is after clearing all cache you should first visit url/%20 and then that url.
And url/%20 should give 404 page.

Is anyone getting the same issue?

I am getting it more frequently coz we are not caching the front page

@nileshlohar Switch an issue to "Active" status only if we are starting over completely.

@nileshlohar: On Nginx I am unable to reproduce the issue following your steps. On IIS it is giving me homepage after adding /%20 with url

@uzma: Then this issue must be server config specific....
/%20 was giving 404 on my setup.

Actually, there is not a working patch yet.

Using rules module can you write rule to redirect user on front page on action user logged out.

Updated the title as requested in #28.

Updated the issue summary.

I don't think this should be flagged as a critical, given it only seems to impact IIS servers, but I'll leave that to the D7 maintainers to decide.

(55)

Pretty sure this isn't critical, as it's IIS only, and potentially only older IIS versions.

This wasn't IIS-specific although one of the steps to reproduce may have been, was fixed in Drupal 10+ in #3368537: Ensure trailing whitespace at the end of a cache ID results in a unique cache item.