site offline (mysql stopped??) [#2081911]

Hi all,

I am using BOA-2.0.9 / KVM and this problem has occurred several times. I made a few small edits to one of my BOA sites (Drupal 7) and then suddenly I get:

The site is currently not available due to technical problems. Please try again later. Thank you for your understanding.

All sites including the hostmaster are down. When I look at $ htop I can see that there is no user called mysql so I wonder if this means that the mysql service has stopped. Also I get

$ mysql -v
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
$

I can fix the problem by rebooting but any ideas please what is causing this problem?

Thanks...

$ cat /var/log/barracuda_log.txt
Sun Jun  2 17:35:15 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.0 / PHP 5.2.17 and 5.3.25 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES
Tue Jun  4 18:55:28 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.0 / PHP 5.2.17 and 5.3.25 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES
Tue Jun 11 18:21:13 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.1 / PHP 5.2.17 and 5.3.26 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES
Fri Jun 28 23:52:07 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.1 / PHP 5.2.17 and 5.3.26 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES
Mon Jul  8 12:27:57 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.26 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.31 localhost / Wildcard YES
Fri Jul 19 19:05:25 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.32 localhost / Wildcard YES
Wed Jul 24 18:59:48 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.32 localhost / Wildcard YES
Thu Aug  1 10:29:02 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.32 localhost / Wildcard YES
Fri Aug  2 19:42:21 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.32 localhost / Wildcard YES
Wed Sep  4 16:45:00 UTC 2013 / Ubuntu.precise i686 XEN / Aegir BOA-2.0.8 / Barracuda BOA-2.0.9 / Nginx 1.5.2 / PHP 5.2.17 and 5.3.27 / MODERN-YES / FPM 5.3 / CLI 5.3 / MariaDB-5.5.32 localhost / Wildcard YES
$

$ cat /root/.barracuda.cnf
###
### Configuration created on 130602-1720
### with Barracuda version BOA-2.0.9
###
### NOTE: the group of settings displayed bellow will *not* be overriden
### on upgrade by the Barracuda script nor by this configuration file.
### They can be defined only on initial Barracuda install.
###
_HTTP_WILDCARD=YES
_MY_OWNIP="12.345.6.78"
_MY_HOSTN="***.co.uk"
_MY_FRONT="***.co.uk"
_THIS_DB_HOST=localhost
_SMTP_RELAY_TEST=YES
_SMTP_RELAY_HOST=""
_LOCAL_NETWORK_IP=""
_LOCAL_NETWORK_HN=""
###
### NOTE: the group of settings displayed bellow
### will *override* all listed settings in the Barracuda script,
### both on initial install and upgrade.
###
_MY_EMAIL="***@mail.com"
_XTRAS_LIST="PDS CSF CHV FTP WMN"
_AUTOPILOT=YES
_SYSTEM_UPGRADE_ONLY=YES
_AEGIR_UPGRADE_ONLY=NO
_DEBUG_MODE=NO
_DB_SERVER=MariaDB
_DB_BINARY_LOG=NO
_DB_ENGINE=InnoDB
_SSH_PORT=22
_LOCAL_DEBIAN_MIRROR="ftp.debian.org"
_LOCAL_UBUNTU_MIRROR="archive.ubuntu.com"
_FORCE_GIT_MIRROR=""
_DNS_SETUP_TEST=YES
_NGINX_WORKERS=AUTO
_NGINX_DOS_LIMIT=300
_BUILD_FROM_SRC=NO
_PHP_MODERN_ONLY=YES
_PHP_FPM_VERSION=5.3
_PHP_CLI_VERSION=5.3
_PHP_FPM_WORKERS=AUTO
_NGINX_EXTRA_CONF=""
_NGINX_LDAP=NO
_PHP_GEOS=NO
_PHP_MONGODB=NO
_PHP_EXTRA_CONF=""
_LOAD_LIMIT_ONE=1444
_LOAD_LIMIT_TWO=888
_CUSTOM_CONFIG_CSF=YES
_CUSTOM_CONFIG_SQL=NO
_CUSTOM_CONFIG_REDIS=NO
_CUSTOM_CONFIG_PHP_5_2=NO
_CUSTOM_CONFIG_PHP_5_3=NO
_SPEED_VALID_MAX=3600
_USE_MEMCACHED=NO
_NEWRELIC_KEY=
_USE_STOCK=NO
_EXTRA_PACKAGES=
_STRONG_PASSWORDS=NO
###
### Configuration created on 130602-1720
### with Barracuda version BOA-2.0.9
###
$

Comments

Comment #0.0

juc1 commented 4 September 2013 at 19:19

Issue summary:

View changes

Comment #0.1

juc1 commented 4 September 2013 at 19:22

Issue summary:

View changes

Comment #0.2

juc1 commented 4 September 2013 at 19:40

Issue summary:

View changes

more detail

Comment #0.3

couturier commented 4 September 2013 at 20:09

Issue summary:

View changes

Comment #1

couturier commented 5 September 2013 at 23:03

When you have problems this major, it is really time to start upgrading to Drupal 7. You also should take a close look at your server capacities. Is it adequate for Drupal? Sometimes increasing the php memory helps a lot. I am not sure what your specific problem is based on your error message, but my previous two suggestions could go a long way to resolving your issue.

You said this error is in Drupal 7, but you've filed the issue against the Drupal 6 module?

Comment #2

omega8cc commented 8 September 2013 at 05:49

Category:

bug

» support

You probably want to look for hints in your system logs, typically grep mysql /var/log/syslog, plus, you don't need to reboot, just check what happens when you run service mysql restart while monitoring syslog with tail -f /var/log/syslog | grep mysql

Comment #3

juc1 commented 8 September 2013 at 16:23

@ omega8cc yes the problem is fixed by the restart mysql command. I have pasted some extracts from /var/log/syslog from the relevant day below (full file is here) - can you please tell me whether this looks like a server issue or is it mysql settings such as my.cnf (which I think are set by BOA). I am pretty sure that the server did not run out of memory because I can see about 100MB of free memory in the resource log and also because this is a test server with almost zero traffic / mysql requests.

Thanks...

/var/log/syslog =

Sep 4 15:25:17 badger kernel: [2767350.508257] [ 3984] 0 3984 1050 13 0 0 0 sleep
Sep 4 15:25:17 badger kernel: [2767350.508260] Out of memory: Kill process 28118 (mysqld) score 339 or sacrifice child
Sep 4 15:25:17 badger kernel: [2767350.509131] Killed process 28118 (mysqld) total-vm:548252kB, anon-rss:172016kB, file-rss:0kB
Sep 4 15:25:17 badger mysqld_safe: Number of processes running now: 0
Sep 4 15:25:17 badger mysqld_safe: mysqld restarted
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: The InnoDB memory heap is disabled
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: Compressed tables use zlib 1.2.3.4
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: Using Linux native AIO
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: Initializing buffer pool, size = 61.0M
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: Completed initialization of buffer pool
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: highest supported file format is Barracuda.
Sep 4 15:25:18 badger mysqld: InnoDB: The log sequence number in ibdata files does not match
Sep 4 15:25:18 badger mysqld: InnoDB: the log sequence number in the ib_logfiles!
Sep 4 15:25:18 badger mysqld: 130904 15:25:18 InnoDB: Database was not shut down normally!
Sep 4 15:25:18 badger mysqld: InnoDB: Starting crash recovery.
Sep 4 15:25:18 badger mysqld: InnoDB: Reading tablespace information from the .ibd files...
Sep 4 15:25:20 badger mysqld: InnoDB: Restoring possible half-written data pages from the doublewrite
Sep 4 15:25:20 badger mysqld: InnoDB: buffer...
Sep 4 15:25:20 badger mysqld: 130904 15:25:20 InnoDB: Waiting for the background threads to start
Sep 4 15:25:21 badger mysqld: 130904 15:25:21 Percona XtraDB (http://www.percona.com) 5.5.32-MariaDB-30.2 started; log sequence number 6123085518
Sep 4 15:25:21 badger mysqld: 130904 15:25:21 [Note] Plugin 'FEEDBACK' is disabled.

Sep 4 18:39:19 badger kernel: [ 5842.219014] Out of memory: Kill process 12940 (mysqld) score 241 or sacrifice child
Sep 4 18:39:19 badger kernel: [ 5842.219331] Killed process 12940 (mysqld) total-vm:498368kB, anon-rss:122252kB, file-rss:0kB
Sep 4 18:39:19 badger mysqld_safe: Number of processes running now: 0
Sep 4 18:39:19 badger mysqld_safe: mysqld restarted
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: The InnoDB memory heap is disabled
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: Compressed tables use zlib 1.2.3.4
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: Using Linux native AIO
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: Initializing buffer pool, size = 61.0M
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: Completed initialization of buffer pool
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: highest supported file format is Barracuda.
Sep 4 18:39:19 badger mysqld: InnoDB: The log sequence number in ibdata files does not match
Sep 4 18:39:19 badger mysqld: InnoDB: the log sequence number in the ib_logfiles!
Sep 4 18:39:19 badger mysqld: 130904 18:39:19 InnoDB: Database was not shut down normally!
Sep 4 18:39:19 badger mysqld: InnoDB: Starting crash recovery.
Sep 4 18:39:19 badger mysqld: InnoDB: Reading tablespace information from the .ibd files...
Sep 4 18:39:21 badger mysqld: InnoDB: Restoring possible half-written data pages from the doublewrite
Sep 4 18:39:21 badger mysqld: InnoDB: buffer...
Sep 4 18:39:21 badger mysqld: 130904 18:39:21 InnoDB: Waiting for the background threads to start
Sep 4 18:39:22 badger mysqld: 130904 18:39:22 Percona XtraDB (http://www.percona.com) 5.5.32-MariaDB-30.2 started; log sequence number 6143828009
Sep 4 18:39:22 badger mysqld: 130904 18:39:22 [ERROR] mysqld: Out of memory (Needed 131014656 bytes)
Sep 4 18:39:22 badger mysqld: 130904 18:39:22 [Note] Plugin 'FEEDBACK' is disabled.
Sep 4 18:39:22 badger mysqld: 130904 18:39:22 [Note] Server socket created on IP: '0.0.0.0'.

Sep 4 18:40:15 badger mysqld: 130904 18:40:15 InnoDB: Initializing buffer pool, size = 61.0M
Sep 4 18:40:15 badger mysqld: InnoDB: mmap(64913408 bytes) failed; errno 12
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 InnoDB: Completed initialization of buffer pool
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 InnoDB: Fatal error: cannot allocate memory for the buffer pool
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 [ERROR] Plugin 'InnoDB' init function returned error.
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 [ERROR] mysqld: Out of memory (Needed 131014656 bytes)
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 [ERROR] mysqld: Out of memory (Needed 98254848 bytes)
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 [ERROR] mysqld: Out of memory (Needed 73678848 bytes)
Sep 4 18:40:15 badger mysqld: 130904 18:40:15 [Note] Plugin 'FEEDBACK' is disabled.
Sep 4 18:40:16 badger mysqld: 130904 18:40:16 [ERROR] Unknown/unsupported storage engine: InnoDB
Sep 4 18:40:16 badger mysqld: 130904 18:40:16 [ERROR] Aborting
Sep 4 18:40:16 badger mysqld:
Sep 4 18:40:16 badger mysqld: 130904 18:40:16 [Note] /usr/sbin/mysqld: Shutdown complete
Sep 4 18:40:16 badger mysqld:
Sep 4 18:40:16 badger mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

Comment #4

juc1 commented 8 September 2013 at 16:28

You said this error is in Drupal 7, but you've filed the issue against the Drupal 6 module?

@ couturier - yes it is an Aegir thing - there is no D7 version of Aegir / Barracuda yet but they manage D7 sites...

Comment #5

omega8cc commented 8 September 2013 at 17:17

@Juc1 The syslog clearly says that the parent system kills mysql. This is rather weird, because BOA is trying to use some very modest defaults, depending on the amount of RAM available. You would need to adjust limits in /etc/mysql/my.cnf until it stops killing it, and then set _CUSTOM_CONFIG_SQL=YES in the /root/.barracuda.cnf to avoid overwriting custom config on upgrade.

What is the name of your provider and the amount of RAM you have there?

This looks very typical for 512 MB droplet on digitalocean, by the way.

Comment #6

omega8cc commented 8 September 2013 at 17:18

Also, please paste the /etc/mysql/my.cnf you have there now.

Comment #7

omega8cc commented 8 September 2013 at 18:46

Status:

Active

» Fixed

We have improved mysql limits tuning on low-memory systems (like 512 MB -- tested on digitalocean) so I would recommend to upgrade to head and see how it works: barracuda up-head system

Comment #8

juc1 commented 8 September 2013 at 18:52

Yes you are correct it is 512 MB Digital Ocean

Here is my.cnf below.

Thanks...

$ cat /etc/mysql/my.cnf
[client]
port                    = 3306
socket                  = /var/run/mysqld/mysqld.sock
default-character-set   = utf8

[mysqld]
user                    = mysql
pid-file                = /var/run/mysqld/mysqld.pid
socket                  = /var/run/mysqld/mysqld.sock
port                    = 3306
basedir                 = /usr
datadir                 = /var/lib/mysql
tmpdir                  = /tmp
default_storage_engine  = InnoDB
lc_messages_dir         = /usr/share/mysql
lc_messages             = en_US
character_set_server    = utf8
collation_server        = utf8_general_ci
skip-external-locking
skip-name-resolve
connect_timeout         = 60
join_buffer_size        = 1M
key_buffer              = 2M
key_buffer_size         = 61M
log_warnings            = 2
max_allowed_packet      = 32M
max_connect_errors      = 30
max_connections         = 30
max_user_connections    = 30
myisam_sort_buffer_size = 1024K
query_cache_limit       = 128K
query_cache_size        = 64M
query_cache_type        = 1
read_buffer_size        = 8M
read_rnd_buffer_size    = 4M
sort_buffer_size        = 128K
bulk_insert_buffer_size = 128K
table_open_cache        = 64
table_definition_cache  = 512
table_cache             = 128
thread_stack            = 256K
thread_cache_size       = 128
thread_concurrency      = 8
wait_timeout            = 3600
tmp_table_size          = 64M
max_heap_table_size     = 128M
low_priority_updates    = 1
concurrent_insert       = 2
max_tmp_tables          = 16384
server-id               = 8
myisam-recover          = BACKUP
#log_bin                 = /var/log/mysql/mariadb-bin
#log_bin_index           = /var/log/mysql/mariadb-bin.index
#expire_logs_days        = 1
#max_binlog_size         = 100M
open_files_limit        = 196608
#slow_query_log          = 1
#long_query_time         = 10
#slow_query_log_file     = /var/log/mysql/sql-slow-query.log
#log_queries_not_using_indexes


# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
sql_mode                = NO_ENGINE_SUBSTITUTION
# you can't just change log file size, requires special procedure
#innodb_log_file_size   = 50M
innodb_buffer_pool_size = 61M
innodb_log_buffer_size  = 4M
innodb_file_per_table   = 1
innodb_open_files       = 256
innodb_io_capacity      = 512
innodb_flush_method     = O_DIRECT
innodb_flush_log_at_trx_commit = 2
innodb_thread_concurrency = 8
#innodb_lazy_drop_table  = 1
innodb_lock_wait_timeout = 120


[mysqld_safe]
socket                  = /var/run/mysqld/mysqld.sock
nice                    = 0
open_files_limit        = 196608
syslog

[mysqldump]
quick
max_allowed_packet      = 32M
quote-names

[mysql]
no-auto-rehash

[myisamchk]
key_buffer              = 1M
sort_buffer_size        = 64K
read_buffer             = 4M
write_buffer            = 4M

[isamchk]
key_buffer              = 1M
sort_buffer_size        = 64K
read_buffer             = 4M
write_buffer            = 4M

[mysqlhotcopy]
interactive-timeout

!includedir /etc/mysql/conf.d/
$

Comment #9

omega8cc commented 8 September 2013 at 19:06

OK, then please make sure you have _CUSTOM_CONFIG_SQL=NO and then run barracuda up-head system

Comment #10

juc1 commented 9 September 2013 at 19:04

@ omega8cc ok I have done that. I will report back if it is not fixed.

Thanks...

Comment #11

23 September 2013 at 19:11

Status:

Fixed

» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Comment #11.0

(not verified) commented 23 September 2013 at 19:11

Issue summary:

View changes

site offline (mysql stopped??)

Comments

Comment #0.0

Comment #0.1

Comment #0.2

Comment #0.3

Comment #1

Comment #2

Comment #3

Comment #4

Comment #5

Comment #6

Comment #7

Comment #8

Comment #9

Comment #10

Comment #11

Comment #11.0

News items

Our community

Documentation

Drupal code base

Governance of community