Send email notification to administrator upon database failur (eg corrupt .MYI files)
Christoph C. Cemper - June 19, 2006 - 10:38
| Project: | Database Administration |
| Version: | 5.x-1.x-dev |
| Component: | Code |
| Category: | support request |
| Priority: | critical |
| Assigned: | Unassigned |
| Status: | active |
Jump to:
Description
Just today I noticed that 2 days ago on the weekend my blog at http://www.marketingfan.com/
Fatal error: Can't open file: 'sessions.MYI' (errno: 145) query: SELECT u.*, s.* FROM users u INNER JOIN sessions s ON u.uid = s.uid WHERE s.sid = 'dfad75d9fbb0d4bf3692e7251578ef95' AND u.status < 3 LIMIT 0, 1 in /home/mf/public_html/includes/database.mysql.inc on line 66
Is there a repair quick option somewhere in Drupal?
Is there a way to email such fatal errors to my email or SMS pager?
thanks c.cemper

#1
not a Drupal problem.
#2
Try running a repair on the sessions table in your database via phpMyAdmin.
#3
In cases like this, google is your friend.
http://www.zen-cart.com/forum/archive/index.php/t-15226.html
#4
Well, thanks for the MYSQL mini-tutorial...
my only problem is that a connect error is only displayed to the user, who is *at best* informed and goes away...
every usual user would be turned off big times from such an error handling!
what I need - in addition to the exisitng great watchdog/logging framework
is a way to email such fatals to an email/sms router to that I'll get informed when such outages occur!
see
http://weblog.cemper.com/a/200605/29-great-reasons-for-using-drupal-part...
so instead of a dumb
$connection = mysql_connect($url['host'], $url['user'], $url['pass'], TRUE) or die(mysql_error());
mysql_select_db(substr($url['path'], 1), $connection) or die('unable to select database');
the "or die(mysql_error());"
should be replaced by something nicer with
A) a qualified error message to the user
B) an email to the admin
C) a throttle to avoid 1000 mails per minute when the MYSQL goes down
clear?
so IMHO this IS a drupal problem - affecting it's operability by bad error/alert handling
regards,christoph
#5
a patch against cvs is welcome, then :)
#6
I just had this problem again... and my hack already informed me...
anytime the server is rebooted under overload, this happens
(overload due to cache issue in my other request)
A patch would be just an additional file to create with that "patch"-utility, right?
I'm thinking that the script could to an MYSQL repair itself, but not sure how that would work for other DBs...
also as the sessions table is HEAVY updated, a lock congestions could be reduced by switching this table to InnoDb... which other tables are updated with sessions? do you know?
best,christoph
#7
about patches, see the documentation here:
http://drupal.org/node/22568
About your other questions, I am afraid I don't know. I am only a beginner, after all ;)
Good luck.
#8
issue http://drupal.org/node/70252 is a duplicate of this one.
see also: http://drupal.org/node/69049 .
Those who are having this problem: try to discuss what you have in common, so that you have this problem but not others...
#9
Good to see that others are having the same problem.
I am currently running a hack that sends me a warning email about such problems,
so that I can do it by hand...
while the quick fix is to to a "repair table" on the mysql files,
definately a great system like Drupal should do that by itself...
What are your thoughts... is implemenation of a "repair table" statement too dangerous for load, DDOS, etc.etc.
thanks christoph
#10
Still not a Drupal prob.
#11
Has anyone been able to confirm that using innoDB on sessions (and cache? watchdog?) is an effective way to avoid this corruption problem?
#12
@chk: agree - it's a MYSQL problem.... the drupal problem is the bad /lack of/ error alerting of this critical situation
@robert: InnoDB has translaction logs that do a recover after a database instance crash (just like oracle) also it's row level locking vs. full table locking should even speed up access to Drupal overall - in terms of response time... the incurrect CPU overhead is there, but since we would only have it on the session table this could be a great thing... I might try this out later today...
christoph
#13
FYI - not only the session table is problematic... look at these corruptions
| mf_drupal.cache | repair | info | Key 1 - Found wrong stored record at 2962696 |
| mf_drupal.cache | repair | info | Key 1 - Found wrong stored record at 2962696 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2963152 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2963180 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2963620 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2963820 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2963892 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2964012 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2964088 |
| mf_drupal.cache | repair | info | Found block with too small length at 2964140; Skipped |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2964152 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2964912 |
| mf_drupal.cache | repair | info | Found block with too small length at 2964940; Skipped |
| mf_drupal.cache | repair | info | Found link that points at -6055548285182294948 (outside data file) at 2965196 |
| mf_drupal.cache | repair | info | Found block that points outside data file at 2965464 |
| mf_drupal.cache | repair | warning | Number of rows changed from 951 to 967 |
It might be a good thing to issue constant "repair tables" in the drupal code after encountering an access error
#14
#15
I'm preparing a patch to contribute that handles it the way I currently do (sending warning emails)
#16
cemper, I think that sending such a warning email should be integrated into the database administration module. it already offers options to repair tables, IIRC.
In general, I think that this is not Drupal's task, but the task of some database monitoring application.
#17
Changing project to Database Administration to keep people from closing the issue prematurely. Changing title to reflect intention of work to be done.
#18
@kills: the db admin module might be a good place logically,
BUT if the initial mysql connect in database.mysql.inc fails,
then we won't get any further... and the visitor would be turned away by an ugly bug...
Actually THAT server already runs a regular "repair table" daily, but if you crash the machine during day, the whole site stays down until the next cron job runs...
and while running an optimize/repair tabel pro-actively daily, it's certainly not a good idea to this every hour imho - performancewise...
@robert: thanks for the good title