Closed (fixed)
Project:
Barracuda
Component:
Miscellaneous
Priority:
Normal
Category:
Support request
Assigned:
Unassigned
Reporter:
Created:
17 Sep 2012 at 12:00 UTC
Updated:
2 Nov 2012 at 18:21 UTC
Jump to comment: Most recent file
Started to experience this issue today -- similar to #1779626: MariaDB 5.5.27 breaks install and upgrade because of its mytop related bug
We did not do anything prior -- server was running just fine -- 24 hours ago the machine that hosts our site was rebooted by linode as a result of software issues they reported.
Steps we have taken with no luck yet.
- Killed the hanging
boa_wait.pid - Removed mytop with
apt-get remove mytop,apt-get purge mytopandrm -f /usr/bin/mytop - Ran
dpkg --configure -a - Ran
run barracuda up again - Scripts run but stops with
MariaDB server not running! - Checking of
/var/backups/barracuda-upgrade-120917-1714.logshows no errors -- posted file here http://pastebin.com/K4uE5CnX - Trying to start mysql manually by
service mysql force-reloadgives the following result:
Reloading MariaDB database server: mysqld/usr/bin/mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)' Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Looked at #1553166: How does one restart services like MariaDB on the BOA system? and see that there /etc/mysql/my.cnf contains :
lc_messages_dir = /usr/share/mysql
lc_messages = en_USTried to remove both lines and restart with same error message. as #8.
Running tail -f /var/log/syslog | grep mysql results in:
Sep 17 18:15:55 server mysqld: ??:0(??)[0xb6ce8955]
Sep 17 18:15:55 server mysqld: ??:0(??)[0xb69565ee]
Sep 17 18:15:55 server mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Sep 17 18:15:55 server mysqld: information that should help you find out what is causing the crash.
Sep 17 18:15:55 server mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Sep 17 18:16:06 server /etc/init.d/mysql[22745]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Sep 17 18:16:06 server /etc/init.d/mysql[22745]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Sep 17 18:16:06 server /etc/init.d/mysql[22745]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Sep 17 18:16:06 server /etc/init.d/mysql[22745]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Sep 17 18:16:06 server /etc/init.d/mysql[22745]:
Running tail -f /var/log/syslog | grep mysql at the same time as service mysql start results in http://pastebin.com/wLarEnZc
mysql -V == mysql Ver 15.1 Distrib 5.5.27-MariaDB, for debian-linux-gnu (i686) using readline 5.1
Help please.
Thank you
Hg
Log files:
/var/aegir/config/includes/barracuda_log.txt -- http://pastebin.com/ZJ6NzQuy
/var/aegir/install.log -- http://pastebin.com/UFKbFUw9
/root/.barracuda.cnf -- http://pastebin.com/uPhKVjND
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | update-head.txt | 17.46 KB | hyperglide |
| #5 | ps axf.txt | 4.99 KB | hyperglide |
| #5 | syslog.txt | 2.86 MB | hyperglide |
Comments
Comment #1
hyperglide commentedComment #2
hyperglide commentedChecking
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!Can see that the file does not exist. How to recreate please?
Comment #3
AquaticDisorder commentedI came across the mytop issue during upgrade process.
I carried out the steps above but still had issues with database connection.
I managed to get my boa back up by configuring BARRACUDA.sh.txt to:
_SQL_FORCE_REINSTALL=YES
Ran the BARRACUDA.sh.txt script and had to restart mysql.
This worked for me all be it guess work :)
Comment #4
omega8cc commentedPlease post the result of commands:
$ grep mysql /var/log/syslog$ ps axfYou probably have to kill this stuck mysql daemon, so it will start again and recover from this crash. However, if this was the reboot what caused the problem, I'm not sure what else can be broken there.
Comment #5
hyperglide commentedHere are the 2 requested outputs.
Warning the syslog is long and over 3 MB so cannot post to d.o. in full
chopped allot of reteat lines out.
The errors started about 13 hours ago it would appear. so not likely related to the server restart unless a damaged table was being written to
Comment #6
omega8cc commentedOK, so it is a known mariadb-5.5.25, percona-5.5.25 and percona-5.5.27 problem - see our Twitter timeline: http://twitter.com/omega8cc
Now, there is no mysql running and also cron is not running - probably because as a result of not completed upgrade.
Could you try to *start* mysql with
service mysql start? There is nothing to reload/restart since it doesn't run at all.The next steps: add the
_SQL_FORCE_REINSTALL=YESline in your /root/.barracuda.cnf file and then runbarracuda up-head. Yes, please upgrade to head, not stable, just to avoid any further issues. It is pretty stable and we are using the head in production.Comment #7
hyperglide commentedThanks for the replies.
I did try
service mysql startResponse was
Starting MariaDB database server: mysqld . . . . . . . . . . . . . . failed!Trying upgrade to head now.
ty
Comment #8
hyperglide commentedAdded
_SQL_FORCE_REINSTALL=YESthen ranbarracuda up-headStill encountered an error as noted below in the second try.
Full screen output is in the attached file
Comment #9
omega8cc commentedSomething is broken there. Could you try to run manually what it says to run now? In this order:
$ dpkg --configure -a$ apt-get update$ apt-get upgrade$ aptitude full-upgradePlease don't update any config files manually (like for mysql).
Comment #10
hyperglide commenteddpkg --configure -ayields:apt-get updateyields:apt-get upgradeyields:aptitude full-upgradeyields:Nothing updated manually.
Comment #11
omega8cc commentedIt looks like it may have issues with broken config file, not sure.
If this will not help, uninstall mariadb first:
$ apt-get remove mariadb-server-5.5Then run
barracuda up-headagain.Comment #12
hyperglide commentedshould I also run
apt-get purge mariadb-server-5.5?Comment #13
omega8cc commentedNo,
removeshould be enough. We just need to get rid of broken binaries etc.Comment #14
hyperglide commentedStill not able to get MariaDB running:
Comment #15
omega8cc commentedCould you add our ssh key and send us your IP via contact for so we could take a look? This guessing is not going to help.
Comment #16
hyperglide commentedSent - Thank you for the support!
Comment #17
hyperglide commentedWanted to say thank you to the Drupal community and Omega8cc for the outstanding support and help.
Always a big fan of AEGir and BOA!
For those asking -- the issue was related to a failed recovery from a very hard server crash.
Always pay it forward -- http://trotc.org/
Comment #18
omega8cc commentedThanks for the update!
Comment #20
naurisr commentedCan you tell me how you fixed this? It seems I have the same problem
Comment #21
hyperglide commented@Naurisr -- Our issues was determined to be caused by a server crash and data corruption.
We resolved it by restoring to an earlier database backup. had some slight data loss.
Comment #22
omega8cc commentedComment #23.0
(not verified) commentedadded log files.