I was testing out a 6.2 upgrade for the last few days.

Everything seemed to be working fine, and I was nearly close to moving my production site to 6.2 today, when my test site suddenly went into serious convulsions.

Hard to describe what happened, near as I can tell, I clicked on the Modules link on the admin page to go to admin/build/modules, and I got an "Internal Server Error". Errors log at my site merely state "premature end of headers". From that point on, all admin pages show only the statement "You do not have any administrative items."
Going to visit a node, such as node/222, I end up viewing the full front page of my site, the default view showing nodes recently added.

I can log out and log back in, but other than that, nothing else is working as it should, and debug information is pretty non-existent.

I had a backup from yesterday, so I restored both the database, and the 6.2 kit files. This seemed to restore the site, and I could change the theme, see admin pages. I changed theme back to bluemarine, and then clicked on admin/build/modules - bam! site was busted again - got the Internal Server Error message, and "premature end of script.." in the error logs. and site was messed up from that point on.

This seemed strange, so I restored the db again, and this time, I have not been able to reproduce the problem. So seems to me that the database is fine now? But some PHP timeout or error in admin/build/modules causes the DB to be corrupted? All pure guesses at this time. Too scary to upgrade at this time, so will stay put until find out more about this.

Comments

bwooster47’s picture

Component: base system » menu system

Possibly related information: now that I can go to admin/build/modules (though with trepidation!), with the help of the devel module, I noticed that each time I go to the modules admin page, I see a huge number of database updates:
Executed 1676 queries in 4289.78 milliseconds.
See many repeated commands from the _menu_router_build, and menu_link_save functions:
INSERT INTO menu_router
and
UPDATE menu_links
and finally
INSERT INTO menu_links (taking nearly 2 seconds).

Could a failure to complete those nearly 5 seconds of db updates cause the site to be corrupted as described in the issue?

bwooster47’s picture

Given that the above problem may not be easy to comment on - can someone in the know help with the following questions:

1) If my site gets corrupted like that again - is there a workaround?
Can I blow away the menu_router and menu_links tables and have some way to recreate them under Drupal 6?

2) Would another workaround be to go to admin/build/modules again? Not sure why it is rebuilding all the menus, but it does that all the time, and on my shared hosting site I've seen it take from 2 to 10 secs to display that page (when it works, the original issue arose in the three instances when it failed to complete all the work to display the page).

3) In case anyone is interested, I can attach the HTML section of the huge database activity as reported by the devel module, in this instance, going to admin/build/modules did not fail so captured this data, but that html file is over 2MBytes, so did not want post it unless someone thinks it could be useful - let me know, and I can attach it.

bwooster47’s picture

Fourth case of site-corruption now, everyone upgrading to 6.2 should beware the Modules admin page! Going there can destroy your site, only fix I know of is to restore database. This time, can't view any page at all without the "Internal Server Error" message.

I'll update this comment with the corruptions I see, to keep track of ways things go bad - maybe there will be pattern and easier workaround.

May 11 - one case was somewhat easy to restore.
In this case the corruption occured when the Theme Developer module was enabled at the admin/build/modules page.
After the corruption occurred, I used phpMyAdmin to hack the system table - reset the 1's to 0's in the devel_themer module row.
Then - I could go back to the site, most pages seem to be displayed correctly.
Note that this case does not always happen - there have been instances when the Theme Developer module enabling works fine, and I can browse the site, and the key corruption always has been admin/build/modules page display - going there is risky and prone to causing the database to be corrupted.

bwooster47’s picture

Status: Active » Closed (fixed)

Closing issue for now - may be related to devel module, http://drupal.org/node/257233 and other similar issues.
Possible workarounds - try disabling the devel module (remove the modules folder?), if that is not possible, then go into system table in the database and disable devel and devel_themer rows, and then revisit admin/build/modules, this may fix the problem.

bwooster47’s picture

Component: base system » menu system
Status: Active » Closed (fixed)

I closed this, thinking that all problems were due to enabling devel module - but no.
After a day of using the site, this time changed the theme to bluemarine, and all pages ended up with blank body - "The requested page could not be found."
Got the blocks on the left side on some pages, and the text "You do not have any administrative items.".

Again, restoring from database only way to get back online.

I now find it hard to believe that people are using Drupal 6.2 on a regular basis! Keep backups, is all I can say...

The pattern of problem and characteristics:
1) site is shared hosting, which places some sort of limits on PHP execution, as well as long db accesses. There is no memory_limit problem - PHP can use lots of memory, more than 32M.
So the http://drupal.org/node/31819 (modules page is blank) issue is not relevant. Also, in this case, the modules pages has a White Screen of Death - but it is not empty, it says "Internal Server Error", and it also corrupts the database. For me, the WSOD is the "Internal Sever Error" default Apache error page, only seen on Drupal 6.2 and not on 4.7.
2) Drupal 4.7 works just fine with exactly the same content.
3) Same content on Drupal 6.2 shows pages load more slowly, and get frequent "Internal Server Error" messages - indicating host killed PHP process, most likely. Never see this on the same content for Drupal 4.7
4) Have same set of modules in 6.2 - image, token, faq. Views is new in 6.2, and it is being used for displaying taxonomy in a block.
5) Visiting the admin/build/modules page is a reasonable sure way to corrupt the database. But have seen db corruption once when visiting just a regular taxonomy term list pages, as well as visiting the main admin page at admin/ So it is not just the admin/build/modules page that is the key problem, other pages also have the same DB corruption effect.

All this probably works fine in non-shared hosting environments, and maybe that is what most people have?

sharifudinrizal’s picture

I'm in the same boat, and it doesn't have to do with the menu system specifically. 6.2 seems so janky that using it for production is suicide. This version should have never left the RC stage..

bwooster47’s picture

Component: menu system » base system
Status: Closed (fixed) » Active

See comment http://drupal.org/node/254616#comment-841538 above that shows the cases when the DB is corrupted. Since this continues to happen, and memory is not an issue, and unlike the other reports on blank pages this time the database is corrupt, this is a debilitating issue, so keeping it back open.

bwooster47’s picture

Component: menu system » base system
Status: Closed (fixed) » Active

I believe I have confirmed the source of problems - it is high CPU usage.

So, anyone on a shared host, that is limiting CPU - i.e., killing php if the CPU usage goes over a limit, then it is highly likely that the DB will be corrupted.

The CPU usage I've seen for going to pages like admin, or admin/build/modules, according to the resource monitor at my site, has been around 11% (enough for them to terminate php). In two instances, saw numbers of 110% which is probably an calculation error, or multiple-cpu issue). Seeing this error on admin/ page is not fatal - you can just retry later, and the DB will be ok. But if you see this error on going to admin/build/modules, then it is fatal - the database will be corrupted.

This CPU issues is not seen at all with Drupal 4.7

So, the CPU usage in Drupal 6.2 is far more than in Drupal 4.7 - the performance has dropped in this newer release.

bwooster47’s picture

Final comment!

Conclusions:
1) Drupal 6.2 uses more CPU and memory resources than Drupal 4.7. Performance has gone down in the newer releases.
2) Drupal 6.2 is not suited for shared hosting sites that use resource limits. Typical limits may be 10% CPU maximum, and 16M limits for processes. The memory limit is the bigger problem - it is exceeded easily, with just a small set of modules (image, pathauto, views), it can go upto 21M. By the way, changing php memory_limit may not solve this - many shared hosting sites use their own resource watching scripts that kill php if it goes above 16M - which will result in HTTP Error Code 500 - Internal Server Error.
The biggest culprit are the admin/ pages. Going to admin/ will result in 500 code, but a re-load will show the page, usually. Going to admin/build/modules is a killer - if that ends up taking more memory (it does not always use more than 16M, many times it is below that limit), and you get a HTTP Code 500, then your Drupal database is corrupted. No re-loads will help, have to restore the db at this point.
Shared hosts with higher limits - like 24M for process memory, and 20% CPU allowance (peak), will probably work fine.
Or stick with Drupal 4.7 - it works fine under the 16M/10%CPU limits.

nukeemusn’s picture

Ok, I'm having this issue too, and dont' have a pre-corruption backup. Is there anyone who knows what they're doing better than I that knows if (and which) database tables I could export/import into a fresh install of Drupal 4 or 5? I'm specifically thinking of content (forums and story) and my user list.

rblindore’s picture

Dear Sir,
I am facing the same problem on local host also. What is the reason?
In addition what should I do to fix the problem, if I don't have db backup?
Rgds
Mahesh

kylamees’s picture

Have the same problem. "You do not have any administrative items" at admin and some node pages don't show (redirecting to front page). And "Create Content" page is empty.

Is it possible to restore some defaults in database tables and save node contents? And how to avoid this database corruption next time with v. 6.2?

danielb’s picture

I have this same problem. Nothing on the admin page. What.

harishu’s picture

Version: 6.2 » 6.3

same problem here. without a db backup what can i do?

cdale’s picture

Also had this problem, not sure what caused it, but managed to fix it by going to admin/build/menu, and then going through each menu and hitting 'reset' for all menu items I could find that actually had this option.

I lost any customizations I had for those items, but after getting the menus back the way I wanted them, I had no more issues. Weird.

alan d.’s picture

+1 for this issue in hosted environments.

Site5 hosting has a fixed 30 sec limit, a fairly common limit in these setups. This can not be changed.

One of the issues is that the rebuild menu function is called by multiple modules, including devel, rather than being included as part of the core process. As a result, my menu is rebuilt 6 times during a clear cache call.

arhak’s picture

If you can't find a workaround to solve this kind of problems (neither devel's rebuild menu, patches, etc) you will always be able to get the site working back if you enable the "admin/settings/performance" entry in the menu_router table access that page and clear the cache (if it results possible without timeout), this is a brute solution and certainly not the best one neither the fastest one.
Follow the issues below and find out how to create a php page to rebuild the menu, how to patch D6 for ALTER instead of rebuild the menu_router table, and many other alternatives.

- here I started a kind of index, it links to the real discussions: #289618: menu_router table truncated and site goes down: Transaction or Lock required for D6
- the development discussion, some patches to improve the chances but not real fix, developers talking about to solve the issue in D7 first and maybe backport it later to D6 (IMO it won't be fix unless a LOT of user complains about it, and the better the hardware the less chances to hit this issue): #238760: menu_router table truncated and site goes down
- a different symptom to the same issue: #246653: Duplicate entry in menu_router: Different symptoms, same issue
- the hart of the problem: #249185: Concurrency problem with variable caching leading to cache inconsistency
- more discussion marked as duplicated: #248739: Duplicate entry errors - lots of them
- being afraid to use D6 in production: #254616: SIte corrupted with no administrative blocks any more after going to admin/build/modules

arhak’s picture

Version: 6.3 » 6.x-dev
Status: Active » Closed (duplicate)

sorry, I forgot to change the status: read above

alan d.’s picture

A quick fix was to place a simple check on the menu rebuild function itself.

<?php
function menu_rebuild() {
  static $menu_built = FALSE;
  if ($menu_built) return;
  $menu_built = TRUE;
?>

This prevented 4 of the 5 calls, doubling the speed. This was being called by content_type_update($info) for each content type when invoked using module_invoke_all('node_type').

Not sure on side effects but sure beats the white screen of death. :)

harishu’s picture

if i go and do an update again (www.site.org/update.php) everything goes back to normal. i didnt have to restore the database. can s'one pls expalin this.

vicentefoxxx’s picture

I installed Drupal 6.3 in a test server and i got the same problems. All the time was when I was installing modules. Right now I am getting "You do not have any administrative items" i can not instert any new content or administrate anything.

harishu’s picture

did u try an update to drupal again ?

kenorb’s picture

kasha_x’s picture

we were having the same problems and were slack on our backups. i thought all was lost but did the update and regained my sanity.

do the update people, it just might help you. props to harishu :-)

torcuato’s picture

Hi everybody, I have the same problem on local, no memory problem, just before I changed all memory limits to 64M, it happened when I was trying panels on my drupal 6.12 and I tried update.php but it doesn't work ,and also no backups, please help!!

alan d.’s picture

1) Try increasing your memory even more, and run phpinfo() to make sure that the system is actually allowing you to increase your memory limit. Some have restrictions on the max or simply ignore requests to increase it.

eg: just after the ini_set() type in the following:

phpinfo();
exit();

Look for something like max memory limit.

2) Are you sure that it is not some error in panels version? If it was a dev version, this would be the first thing i would disable and see what happens. (see 3)

3) If you have lightbox2 + cck installed, and high # imagecache / content types, turn lightbox2 off in the systems table using phpmyadmin or something similar (change status from 1 to 0) and flush the cache tables manually. These all start with cache_* or just cache.

This combo is about the most resource hungry of any modules that I have seen. #409354: Performance issues with large number imagecache presets and content types

4) Otherwise, disable some of the modules in the system table (take care with dependencies) and clean the cache and run updates. Then try running update.php again.

Good luck

arhak’s picture

apologizes to Alan D. @#26, but
what are you talking about?

it is not a memory problem, it is a Drupal problem #251792: Implement a locking framework for long operations

@torcuato: read above #17
it is a detailed explanation and links to alternatives to solve your "right now problem"
many of those solutions are equivalent and will bring your site back to live (then do a backup immediately)
the problem might come back and hit you again when using several tabs at the same time and/or views/panels or any intensive module

this should be gone when Drupal 6.16 comes out (according to the looking framework)

alan d.’s picture

@arhak I guess that this is really a "no transaction rollback to handle fatal errors when updating core tables". You eleminate the error cause (memory), you do not have the error. Sorry, I was treating this as a request for help rather than an issue update.

I was assuming worst case, which has happened a lot to people. e.g. The entire site has been nuked. No admin functions, et al. Just a white screen or massively reduced functionality, which steps 3 - 4 come in.

Step 3 is an unusual case caused by this module combo. These settings are in the variables table and effect the performance no matter where or what you are doing. (Eg: these 1MB + variable cache entries were always loaded)

arhak’s picture

@#28 no, I wasn't criticizing your help support, rather questioning the proposed solution

you can increase memory to avoid it to happen, but IMO it is not a memory limit problem, rather a concurrency one
If you start opening several tabs in a browser and several of them attempt to rebuild the menu, you get stuck with a broken menu nevermind how much memory you had

moreover, if the site mentioned @#25 is down, what good might bring increasing memory?, that won't get it back to work

you suggested a "preventive" measure, but he asked for a way to bring the site back to live, right?
and most of the subscribers of this thread land here trying to figure out how to get it back
what causes it might not even matter, how to avoid it will matter, but after having the site back
that's why I provided referenced info @#17 (the best I could)

PS: I apologize again, but started #27 with an apology to state that I wouldn't agree with you, nothing else, the users having the problem will be thankful of receiving as many support and alternatives as possible, please don't get me the wrong way

alan d.’s picture

Hate forums, there is never a way to express you tone - no offense taken / given :)

The increase of memory does help when running "update.php" way to help reset things. I've "reset" at least 6 sites using the magic that running update.php does. I think that this usually triggers a cache flush and rebuild of the menu but I've never looked into it.

Yet this does fail on many hosted environments due to server based restrictions on the max php memory.

arhak’s picture

see, now I got your point, and it actually address recovering the broken site
I wasn't aware of that, whenever I ran into it (luckily I haven't in a long while) I solved it procuring a way to manually invoke the menu rebuild, mostly manually inserting the menu router row in the table via phpMyAdmin (it isn't a big deal at all) and then got admin/settings/performance page back to rebuild everything

you approach would be more friendly though

evansdave’s picture

Version: 6.x-dev » 6.16
Component: base system » database system

Re #27 above and "should be solved w 6.16..."

I am running 6.16, and installed/enabled CCK and Views on an otherwise working site. AFter about a minute, the browser went blank (as in "plain white screen in browser (Chrome) window"). When I went back to the starting page...the site was dead. I had the teaser portion of my most recent content, and clicking "Administrative" in navigation produced badly mangled screen with "You do not have any administrative items."

Tried removing modules, removing entries from db (e.g., views tables and entries in SYSTEM table)...after an hour simply dropped the db, reinstalled 6.16...copied back the modified theme templates and was running again.

It would be awesome if a rollback/cat recovery, etc. was available....but as others have noted the key to an evolving platform like Drupal is "back up early, back up often." (Or, as we say in rock climbing "Don't climb any farther past your last set anchor than you are willing to fall.")