Daily we receive notice from linode of both High I/O and CPU usage.

Looking at PS during peak load shows:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
 root      3549  0.0  0.1  17652  1268 ?        Ss   Nov06   4:08 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
 root     13824  0.9  1.4  19656 10820 ?        D    09:15   0:00  \_ /usr/share/webmin/webmincron/webmincron.pl
 mysql     2761  1.8 38.7 765764 297604 ?       Sl   Dec08 542:07  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --open-files-limit=196608 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
 root      2762  0.0  0.0   3220   268 ?        D    Dec08   0:05  \_ logger -t mysqld -p daemon.error
 root     11886  0.0  0.0   3868   452 ?        Ss   Dec25   0:09 /usr/sbin/cron
 root     12942  0.0  0.1   4020   808 ?        S    09:14   0:00  \_ /USR/SBIN/CRON
 root     12963  0.0  0.0   1804   352 ?        Ss   09:14   0:00  |   \_ /bin/sh -c bash /var/xdrago/second.sh >/dev/null 2>&1
 root     12965  0.0  0.1   4240  1112 ?        S    09:14   0:00  |       \_ bash /var/xdrago/second.sh
 root     14445  0.0  0.0   3232   244 ?        S    09:15   0:00  |           \_ sleep 10
 root     12943  0.0  0.1   4020   808 ?        S    09:14   0:00  \_ /USR/SBIN/CRON
 aegir    12949  0.5  2.7  56264 21248 ?        Ds   09:14   0:00  |   \_ /usr/local/bin/php /var/aegir/drush/drush.php @hostmaster hosting-dispatch
 root     13446  0.0  0.1   4020   892 ?        S    09:15   0:00  \_ /USR/SBIN/CRON
 root     13781  0.0  0.0   1804   500 ?        Ss   09:15   0:00  |   \_ /bin/sh -c bash /var/xdrago/manage_ltd_users.sh >/dev/null 2>&1
 root     13786  0.1  0.1   4308  1460 ?        S    09:15   0:00  |       \_ bash /var/xdrago/manage_ltd_users.sh
 root     14447  0.0  0.0   1748   248 ?        D    09:15   0:00  |           \_ symlinks -dr /data/disk/dev/clients
 root     13448  0.0  0.1   4020   892 ?        S    09:15   0:00  \_ /USR/SBIN/CRON
 root     13780  0.0  0.0   1804   504 ?        Ss   09:15   0:00  |   \_ /bin/sh -c bash /var/xdrago/minute.sh >/dev/null 2>&1
 root     13785  0.0  0.1   4200  1184 ?        S    09:15   0:00  |       \_ bash /var/xdrago/minute.sh
 root     14448  0.0  0.2   5024  1576 ?        S    09:15   0:00  |           \_ perl /var/xdrago/monitor/check/hackmail
 root     14450  0.0  0.0   3260   508 ?        D    09:15   0:00  |               \_ tail --lines=999 /var/log/mail.log
 root     13449  0.0  0.1   4020   892 ?        S    09:15   0:00  \_ /USR/SBIN/CRON
 root     13777  0.0  0.0   1804   504 ?        Ss   09:15   0:00  |   \_ /bin/sh -c bash /var/xdrago/second.sh >/dev/null 2>&1
 root     13784  0.0  0.1   4244  1272 ?        S    09:15   0:00  |       \_ bash /var/xdrago/second.sh
 root     14442  0.0  0.0   3232   248 ?        S    09:15   0:00  |           \_ sleep 1
 root     13450  0.0  0.1   4020   892 ?        S    09:15   0:00  \_ /USR/SBIN/CRON
 aegir    13776  1.4  3.1  56264 24496 ?        Ss   09:15   0:00      \_ /usr/local/bin/php /var/aegir/drush/drush.php @hostmaster hosting-dispatch
 redis    29312  0.5  0.7  72420  6012 ?        Ssl  04:56   1:22 /usr/bin/redis-server /etc/redis/redis.conf
 dev      11657  0.2  0.3  60740  2996 ?        S    09:12   0:00 /usr/local/bin/php /data/disk/dev/tools/drush/drush.php --items=1 --quiet --root=/data/disk/dev/aegir/distro/017 --uri=dev.server.domain.com hosting-advanced_cron --backend
 dev      11867  0.0  0.0   1804   360 ?        S    09:12   0:00  \_ sh -c /data/disk/dev/tools/drush/drush.php  @dev.w13-mm.domaintour.com cron --backend  2>&1
 dev      11868  4.3 10.8 159852 83444 ?        D    09:12   0:08      \_ /usr/local/bin/php /data/disk/dev/tools/drush/drush.php @dev.w13-mm.domaintour.com cron --backend
 dev      12414  0.3  3.0  60740 23120 ?        S    09:13   0:00 /usr/local/bin/php /data/disk/dev/tools/drush/drush.php --items=1 --quiet --root=/data/disk/dev/aegir/distro/017 --uri=dev.server.domain.com hosting-advanced_cron --backend
 dev      13030  0.0  0.0   1804   420 ?        S    09:14   0:00  \_ sh -c /data/disk/dev/tools/drush/drush.php  @dev.w13-mm.domaintour.com cron --backend  2>&1
 dev      13032  1.9  7.9  96032 61416 ?        D    09:14   0:01      \_ /usr/local/bin/php /data/disk/dev/tools/drush/drush.php @dev.w13-mm.domaintour.com cron --backend
 aegir    13426  1.0  2.8  55680 22156 ?        S    09:14   0:00 /usr/local/bin/php /var/aegir/drush/drush.php --items=5 --quiet --root=/var/aegir/host_master/018 --uri=admin.domain.com hosting-tasks --backend
 root     14409  0.3  0.5  40568  4388 ?        Ds   09:15   0:00 nginx: master process /usr/sbin/nginx
 aegir    14454  0.0  0.0   8112   168 ?        D    09:15   0:00 /usr/local/bin/php /var/aegir/drush/drush.php --items=5 --quiet --root=/var/aegir/host_master/018 --uri=admin.domain.com hosting-tasks --backend

It appears that multiple tasks are running simultaneously. Related to DB Backups and CRON.

Logs are available for view at:

Is there a way to sequence these tasks to run over a long period of time, disable the DB backups on selected domains or is this a bug?

Thank you,
Hg

Comments

omega8cc’s picture

omega8cc commented
Status: Active » Closed (works as designed)

This is by design. We have previously paused all drush tasks during mysql backup, but since it may take some time, we no longer do that to avoid confusion for user waiting for tasks to run. Default alerting settings/limits at Linode are simply way too low. Consider disabling them to avoid this false alarm.

You can also pause all drush tasks while mysql backups run, by adding touch /var/run/boa_wait.pid line at the top and rm -f /var/run/boa_wait.pid line at the bottom of /var/xdrago/mysql_backup.sh script. It will be overwritten on every upgrade, though.