I ask because typical MySQL commands don't seem to work and because I keep killing the MariaDB server when trying to add a node to nodequeue when running the latest version of BOA HEAD as of 4/27/2012. (See http://drupal.org/node/1553158 for the issue about the nodequeue.)

Thanks for any and all help in advance. I truly appreciate it!
Steve

Comments

omega8cc’s picture

omega8cc commented

There is nothing BOA specific, so standard service mysql restart will work, but if it crashed, it may take a while until it will recover from crash, so please be patient and monitor syslog in another terminal window with tail -f /var/log/syslog | grep mysql until you will see the lines saying something like:

/usr/sbin/mysqld: ready for connections.
socket: '/var/run/mysqld/mysqld.sock'  port: 3306
Checking for insecure root accounts.

Now db server should be available.

wickwood’s picture

wickwood commented

After the db server crashed while adding a node to nodequeue (see http://drupal.org/node/1553158 ), I did what you said and the db server never restarted.

Here's the output I got from service mysql restart

 * Stopping MariaDB database server mysqld                                                                                                          [ OK ] 
 * Starting MariaDB database server mysqld                                                                                                          [fail]

And in the other terminal where I ran tail -f /var/log/syslog | grep mysql I just got output like this over and over and over again at about 3 minute intervals. 

Apr 30 20:10:27 wwm001 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 [Note] PrimeBase XT (PBXT) Engine 1.0.11-7 Pre-GA loaded...
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 [Note] Paul McCullagh, PrimeBase Technologies GmbH, http://www.primebase.org
Apr 30 20:10:27 wwm001 mysqld: InnoDB: The InnoDB memory heap is disabled
Apr 30 20:10:27 wwm001 mysqld: InnoDB: Mutexes and rw_locks use GCC atomic builtins
Apr 30 20:10:27 wwm001 mysqld: InnoDB: Compressed tables use zlib 1.2.3.3
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27  InnoDB: Initializing buffer pool, size = 483.0M
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27  InnoDB: Completed initialization of buffer pool
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27  InnoDB: highest supported file format is Barracuda.
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 Percona XtraDB (http://www.percona.com) 1.0.17-13.0 started; log sequence number 74930223888
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 [ERROR] /usr/sbin/mysqld: unknown variable 'lc_messages_dir=/usr/share/mysql'
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 [ERROR] Aborting
Apr 30 20:10:27 wwm001 mysqld: 
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27  InnoDB: Starting shutdown...
Apr 30 20:10:28 wwm001 mysqld: 120430 20:10:28  InnoDB: Shutdown completed; log sequence number 74930223888
Apr 30 20:10:28 wwm001 mysqld: 120430 20:10:28 [Note] PrimeBase XT Engine shutdown...
Apr 30 20:10:28 wwm001 mysqld: 120430 20:10:28 [Note] /usr/sbin/mysqld: Shutdown complete
Apr 30 20:10:28 wwm001 mysqld: 
Apr 30 20:10:28 wwm001 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Apr 30 20:10:41 wwm001 /etc/init.d/mysql[19986]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Apr 30 20:10:41 wwm001 /etc/init.d/mysql[19986]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Apr 30 20:10:41 wwm001 /etc/init.d/mysql[19986]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Apr 30 20:10:41 wwm001 /etc/init.d/mysql[19986]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Apr 30 20:10:41 wwm001 /etc/init.d/mysql[19986]:

I let this go for about 20 minutes and then I tried running service mysql stop waited a few minutes and then ran service mysql start.
The out put I got from this was basically the same:

wwm001:/home/wickwood/boa# service mysql stop
 * Stopping MariaDB database server mysqld                                                                                                          [ OK ] 
wwm001:/home/wickwood/boa# service mysql start
 * Starting MariaDB database server mysqld                                                                                                          [fail]

I also got the same output as above over and over again at about 3 minute intervals.

I let this go for another 30 minutes and the db server never recovered. I don't know what else to do at this point but recover my system from backup as I have done previously, so that's what did to get everything back online.

Thanks again for your help on this! I really appreciate everything you're doing for me and everyone else!
Steve

wickwood’s picture

wickwood commented

BTW, I also tried rebooting the server before I restored from backup, and the db server did not restart this way either.

I then tried running the service mysql restart, and got the same results as above.

So now I'm going to restore the system from backup to the when it was running BOA HEAD from 3/31/2012.

omega8cc’s picture

omega8cc commented

The problem is that you have still old MariaDB version for some reason and new config.

Please remove lines from your /etc/my.cnf:

lc_messages_dir         = /usr/share/mysql
lc_messages             = en_US

This will allow you to start db server

Also, what is the result of mysql -V?

omega8cc’s picture

omega8cc commented
Component: Miscellaneous » Code
Category: support » bug
Priority: Normal » Critical
Status: Active » Fixed

Fixed in commit: http://drupalcode.org/project/barracuda.git/commit/c800978

Thanks!

wickwood’s picture

wickwood commented

Output from mysql -V was:
mysql Ver 15.1 Distrib 5.3.6-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

I had posted that in comment #2 of the issue Adding a node to nodequeue causes error with older version of BOA and shuts down databases for all sites with new version of BOA, but neglected to post it here, too. Sorry about that.

As soon as my system is restored, I will try this. I was planning to go to bed, but I won't be able to sleep if I don't try this first.

Thanks for your help, I'll let you know what happens.

Steve

wickwood’s picture

wickwood commented

So my system just finished restoring, and I opened /etc/mysql/my.cnf to make the edits you suggested in #4, but I don't have those lines in this file. (BTW, the comment said the file was /etc/my.cnf and that does not exist. I figured this was just a typo.)

Would these lines be there with my older BOA HEAD system (3/31/12) or only after I upgrade to the last HEAD? I haven't run the upgrade yet.

omega8cc’s picture

omega8cc commented

You no longer see those lines in the /etc/mysql/my.cnf file after another upgrade to head, because they are now automatically removed, as per patch from #5 above. They were there before, hence the error causing MariaDB to never start: 

Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 [ERROR] /usr/sbin/mysqld: unknown variable 'lc_messages_dir=/usr/share/mysql'
Apr 30 20:10:27 wwm001 mysqld: 120430 20:10:27 [ERROR] Aborting

Another problem is, why it didn't upgrade your db server to the latest MariaDB 5.5.23 so it is still 5.3.6, and we will try to reproduce this.

wickwood’s picture

wickwood commented

OK, I think I understand you now. I thought I was suppose to remove those variables before I upgraded. I will run the upgrade this morning and report back.

wickwood’s picture

wickwood commented

1. I upgraded to BOA HEAD with commit in comment #5. Everything went perfectly with the upgrade.

2. After all new platforms added by Octopus were verified, I then tested whether or not I could reboot by Linode VPS, and that worked fine as well. So I took a manual backup of the VPS at this point.

3. I ran grep mysql /var/log/syslog
This was the output:

May  1 09:57:21 wwm001 mysqld: 120501  9:57:20 [Warning] Access denied for user 'intntnllyInvalid'@'wwm001.wickwoodmarketing.com' (using password: NO)
May  1 09:57:21 wwm001 mysqld: 120501  9:57:21 [Warning] Access denied for user 'intntnllyInvalid'@'wwm001.wickwoodmarketing.com' (using password: NO)
May  1 09:57:33 wwm001 mysqld: 120501  9:57:33 [Warning] Access denied for user 'intntnllyInvalid'@'wwm001.wickwoodmarketing.com' (using password: NO)
May  1 09:57:33 wwm001 mysqld: 120501  9:57:33 [Warning] Access denied for user 'intntnllyInvalid'@'wwm001.wickwoodmarketing.com' (using password: NO)
May  1 10:56:33 wwm001 named[2279]: built with '--prefix=/usr' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc/bind' '--localstatedir=/var' '--enable-threads' '--enable-largefile' '--with-libtool' '--enable-shared' '--enable-static' '--with-openssl=/usr' '--with-gssapi=/usr' '--with-gnu-ld' '--with-dlz-postgres=no' '--with-dlz-mysql=no' '--with-dlz-bdb=yes' '--with-dlz-filesystem=yes' '--with-dlz-ldap=yes' '--with-dlz-stub=yes' '--with-geoip=/usr' '--enable-ipv6' 'CFLAGS=-fno-strict-aliasing -DDIG_SIGCHASE -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS='
May  1 10:56:35 wwm001 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
May  1 10:56:36 wwm001 mysqld: 120501 10:56:36 [Note] PrimeBase XT (PBXT) Engine 1.0.11-7 Pre-GA loaded...
May  1 10:56:36 wwm001 mysqld: 120501 10:56:36 [Note] Paul McCullagh, PrimeBase Technologies GmbH, http://www.primebase.org
May  1 10:56:36 wwm001 mysqld: InnoDB: The InnoDB memory heap is disabled
May  1 10:56:36 wwm001 mysqld: InnoDB: Mutexes and rw_locks use GCC atomic builtins
May  1 10:56:36 wwm001 mysqld: InnoDB: Compressed tables use zlib 1.2.3.3
May  1 10:56:36 wwm001 mysqld: 120501 10:56:36  InnoDB: Initializing buffer pool, size = 483.0M
May  1 10:56:36 wwm001 mysqld: 120501 10:56:36  InnoDB: Completed initialization of buffer pool
May  1 10:56:36 wwm001 mysqld: 120501 10:56:36  InnoDB: highest supported file format is Barracuda.
May  1 10:56:37 wwm001 mysqld: 120501 10:56:37 Percona XtraDB (http://www.percona.com) 1.0.17-13.0 started; log sequence number 72720344026
May  1 10:56:38 wwm001 mysqld: 120501 10:56:38 [Note] Event Scheduler: Loaded 0 events
May  1 10:56:38 wwm001 mysqld: 120501 10:56:38 [Note] /usr/sbin/mysqld: ready for connections.
May  1 10:56:38 wwm001 mysqld: Version: '5.3.6-MariaDB-mariadb115~lucid-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (MariaDB - http://mariadb.com/)
May  1 10:56:39 wwm001 /etc/mysql/debian-start[2785]: Upgrading MySQL tables if necessary.
May  1 10:56:40 wwm001 /etc/mysql/debian-start[2788]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May  1 10:56:40 wwm001 /etc/mysql/debian-start[2788]: Looking for 'mysql' as: /usr/bin/mysql
May  1 10:56:40 wwm001 /etc/mysql/debian-start[2788]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May  1 10:56:40 wwm001 /etc/mysql/debian-start[2788]: This installation of MySQL is already upgraded to 5.3.6-MariaDB, use --force if you still need to run mysql_upgrade
May  1 10:56:40 wwm001 /etc/mysql/debian-start[2799]: Checking for insecure root accounts.

4. I then tried to add a node to a nodequeue which was killing my db server before. The node was not added to the nodequeue, and it looks like the db server is crashing but it is also recovering. See comment #4 Adding a node to nodequeue causes error with older version of BOA and shuts down databases for all sites with new version of BOA for the syslog reports after crash and further discussion of this other issue.

5. I ran service mysql restart and got this output:

 * Stopping MariaDB database server mysqld                                                                                                                                  [ OK ] 
 * Starting MariaDB database server mysqld                                                                                                                                  [ OK ] 
 * Checking for corrupt, not cleanly closed and upgrade needing tables.

6. While doing step 5. I also ran tail -f /var/log/syslog | grep mysql in another terminal window and this was the output for that:

May  1 13:06:46 wwm001 mysqld: 120501 13:06:46 [Note] /usr/sbin/mysqld: Normal shutdown
May  1 13:06:46 wwm001 mysqld: 
May  1 13:06:46 wwm001 mysqld: 120501 13:06:46 [Note] Event Scheduler: Purging the queue. 0 events
May  1 13:06:46 wwm001 mysqld: 120501 13:06:46  InnoDB: Starting shutdown...
May  1 13:06:46 wwm001 mysqld: 120501 13:06:46  InnoDB: Shutdown completed; log sequence number 72773045408
May  1 13:06:46 wwm001 mysqld: 120501 13:06:46 [Note] PrimeBase XT Engine shutdown...
May  1 13:06:46 wwm001 mysqld: 120501 13:06:46 [Note] /usr/sbin/mysqld: Shutdown complete
May  1 13:06:46 wwm001 mysqld: 
May  1 13:06:46 wwm001 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
May  1 13:06:47 wwm001 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
May  1 13:06:47 wwm001 mysqld: 120501 13:06:47 [Note] PrimeBase XT (PBXT) Engine 1.0.11-7 Pre-GA loaded...
May  1 13:06:47 wwm001 mysqld: 120501 13:06:47 [Note] Paul McCullagh, PrimeBase Technologies GmbH, http://www.primebase.org
May  1 13:06:47 wwm001 mysqld: InnoDB: The InnoDB memory heap is disabled
May  1 13:06:47 wwm001 mysqld: InnoDB: Mutexes and rw_locks use GCC atomic builtins
May  1 13:06:47 wwm001 mysqld: InnoDB: Compressed tables use zlib 1.2.3.3
May  1 13:06:47 wwm001 mysqld: 120501 13:06:47  InnoDB: Initializing buffer pool, size = 483.0M
May  1 13:06:48 wwm001 mysqld: 120501 13:06:48  InnoDB: Completed initialization of buffer pool
May  1 13:06:48 wwm001 mysqld: 120501 13:06:48  InnoDB: highest supported file format is Barracuda.
May  1 13:06:48 wwm001 mysqld: 120501 13:06:48 Percona XtraDB (http://www.percona.com) 1.0.17-13.0 started; log sequence number 72773045408
May  1 13:06:48 wwm001 mysqld: 120501 13:06:48 [Note] Event Scheduler: Loaded 0 events
May  1 13:06:48 wwm001 mysqld: 120501 13:06:48 [Note] /usr/sbin/mysqld: ready for connections.
May  1 13:06:48 wwm001 mysqld: Version: '5.3.6-MariaDB-mariadb115~lucid-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (MariaDB - http://mariadb.com/)
May  1 13:06:49 wwm001 /etc/mysql/debian-start[7342]: Upgrading MySQL tables if necessary.
May  1 13:06:49 wwm001 /etc/mysql/debian-start[7345]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May  1 13:06:49 wwm001 /etc/mysql/debian-start[7345]: Looking for 'mysql' as: /usr/bin/mysql
May  1 13:06:49 wwm001 /etc/mysql/debian-start[7345]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May  1 13:06:49 wwm001 /etc/mysql/debian-start[7345]: This installation of MySQL is already upgraded to 5.3.6-MariaDB, use --force if you still need to run mysql_upgrade
May  1 13:06:49 wwm001 /etc/mysql/debian-start[7352]: Checking for insecure root accounts.
May  1 13:07:07 wwm001 mysqld: 120501 13:07:07 [ERROR] mysqld: Table './wickwoodonlineco/cache' is marked as crashed and should be repaired
May  1 13:07:07 wwm001 mysqld: 120501 13:07:07 [ERROR] mysqld: Table 'cache' is marked as crashed and should be repaired
May  1 13:07:07 wwm001 mysqld: 120501 13:07:07 [Warning] Checking table:   './wickwoodonlineco/cache'
May  1 13:07:08 wwm001 mysqld: 120501 13:07:08 [ERROR] mysqld: Table './wickwoodonlineco/variable' is marked as crashed and should be repaired
May  1 13:07:08 wwm001 mysqld: 120501 13:07:08 [ERROR] mysqld: Table 'variable' is marked as crashed and should be repaired
May  1 13:07:08 wwm001 mysqld: 120501 13:07:08 [Warning] Checking table:   './wickwoodonlineco/variable'
May  1 13:07:09 wwm001 mysqld: 120501 13:07:09 [ERROR] mysqld: Table './wickwoodonlineco/semaphore' is marked as crashed and should be repaired
May  1 13:07:09 wwm001 mysqld: 120501 13:07:09 [ERROR] mysqld: Table 'semaphore' is marked as crashed and should be repaired
May  1 13:07:09 wwm001 mysqld: 120501 13:07:09 [Warning] Checking table:   './wickwoodonlineco/semaphore'
May  1 13:10:12 wwm001 mysqld: 120501 13:10:12 [ERROR] mysqld: Table './guilderlanddemoc/semaphore' is marked as crashed and should be repaired
May  1 13:10:12 wwm001 mysqld: 120501 13:10:12 [ERROR] mysqld: Table 'semaphore' is marked as crashed and should be repaired
May  1 13:10:12 wwm001 mysqld: 120501 13:10:12 [Warning] Checking table:   './guilderlanddemoc/semaphore'
May  1 13:10:13 wwm001 mysqld: 120501 13:10:13 [ERROR] mysqld: Table './guilderlanddemoc/variable' is marked as crashed and should be repaired
May  1 13:10:13 wwm001 mysqld: 120501 13:10:13 [ERROR] mysqld: Table 'variable' is marked as crashed and should be repaired
May  1 13:10:13 wwm001 mysqld: 120501 13:10:13 [Warning] Checking table:   './guilderlanddemoc/variable'

Conclusion
I have some problems with my databases that need to be fixed and there may be some issue with the Nodequeue module too, but this issue of restarting and recovering the MariaDB server is fixed. (Interestingly, none of the tables marked with errors above have anything to do with sites are having problems with the Nodequeue module, but they are both Drupal 6 sites and 2 of the first sites that I moved to my BOA system about a year ago.)

I'll continue the discussion with the potential issue with the Nodequeue module in the issue Adding a node to nodequeue causes error with older version of BOA and shuts down databases for all sites with new version of BOA

Thanks again Omega8cc for all your help with this and all your hard work on the BOA system in general. Your generosity and patience is beyond compare!

Steve

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.