Sometime last month MariaDB was corrupted by the Barracuda upgrade script only upgrading it partly. Not knowing that using apt-get to uninstall MariaDB would be a bad idea, we went ahead and uninstalled MariaDB. We then ran the upgrade script so that it would reinstall MariaDB. I did not have any issues until we ran the upgrade script on 11-23-12.

After 11-23-12, the following issues appeared:

  • php-fpm shutting down randomly and causing the sites to return 502 bad gateway errors. If you restart php-fpm, the websites come back online, and this seems to work for about 24-48 hours.
  • After about a week, the main aegir hostmaster returns a 502 bad gateway error and will only work for a few minutes after an upgrade.

I found this in the 11-23-12 upgrade log(I have attached the full log, if that is helpful):

Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
The following NEW packages will be installed:
libaio1{a} mariadb-client-5.5 mariadb-client-core-5.5{ab}
mariadb mariadb-server-core-5.5{ab}
0 packages upgraded, 5 newly installed, 0 to remove and 26 not upgraded.
Need to get 29.8 MB of archives. After unpacking 104 MB will be used.
The following packages have unmet dependencies:
mariadb : Conflicts: mysql-server-core-5.5 but 5.5.24-0ubuntu0.12.04.1 is installed and it is kept back.
mysql : Breaks: mysql-server-core-5.1 which is a virtual package.
mariadb : Conflicts: mysql-client-core-5.5 but 5.5.24-0ubuntu0.12.04.1 is installed and it is kept back.
mysql : Breaks: mysql-client-core-5.1 which is a virtual package.

The following actions will resolve these dependencies:

Keep the following packages at their current version:

1)     mariadb-client-5.5 [Not Installed]
2)     mariadb-client-core-5.5 [Not Installed]
3)     mariadb-server-5.5 [Not Installed]
4)     mariadb-server-core-5.5 [Not Installed]

No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 26 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Writing extended state information...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...

If you could point me in the right direction, that would be most helpful. :)

CommentFileSizeAuthor
barracuda-upgrade-121123-2022.txt53.36 KBg33kg1rl

Comments

omega8cc’s picture

omega8cc commented

You have a broken system because you have played with some extra, not supported packages probably - example:

Processing triggers for libapache2-mod-php5 ...
 * Reloading web server config apache2       #[80G Processing triggers for libc-bin ...

What Apache is doing here?

Also, you have installed standard MySQL, so it must cause issues when BOA expects MariaDB:

The following packages have unmet dependencies:
 mariadb-server-core-5.5 : Conflicts: mysql-server-core-5.5 but 5.5.24-0ubuntu0.12.04.1 is installed and it is kept back.
 mysql-server-core-5.5 : Breaks: mysql-server-core-5.1 which is a virtual package.
 mariadb-client-core-5.5 : Conflicts: mysql-client-core-5.5 but 5.5.24-0ubuntu0.12.04.1 is installed and it is kept back.
 mysql-client-core-5.5 : Breaks: mysql-client-core-5.1 which is a virtual package.

Please uninstall the extra stuff you have added with commands like:

$ apt-get remove mysql-server-core-5.5 mysql-server-core-5.1 mysql-client-core-5.5 mysql-client-core-5.1 libapache2-mod-php5
$ apt-get autoremove

You may want to run above two lines twice.

Then add forced SQL mode:

$ echo "_SQL_FORCE_REINSTALL=YES" >> /root/.barracuda.cnf

Now run upgrade again:

$ barracuda up-stable

Then restart mysql to force it to apply tables repair/upgrades:

$ service mysql restart

Finally apply hot-fixes:

wget -U iCab http://files.aegir.cc/update/global-fix.sh.txt
bash global-fix.sh.txt
rm global-fix.sh.txt
g33kg1rl’s picture

g33kg1rl commented

I am doing the upgrade now. Thank you so much for looking into this for me.

The upgrade is hanging at restarting MariaDB. If I look in the log file it says:

DROP all opt -- in * out * 80.80.173.170 --> 0.0.0.0.0/0

It has up to 2000 lines of different ip addresses and the list is growing. I don't know if this is normal or odd behavior.

At this point, I am just going to let it continue working and hope that it finally finishes the upgrade.

g33kg1rl’s picture

g33kg1rl commented

I finished the upgrade. The erroneous packages have been uninstalled and the hot fix applied.

php-fpm is still crashing every morning. The hostmaster website stayed up for a couple hours instead of a couple of minutes. As of this morning, it is returning 502 bad gateway errors.

What is the next step to fix this?

Thank you again for helping me out with this. I swear I'll never use apt-get on the server again!

omega8cc’s picture

omega8cc commented
Status: Active » Postponed (maintainer needs more info)

No idea. Please attach (as a file) the full log of the last upgrade - I mean the last barracuda-upgrade-DATE-TIME.txt file.

g33kg1rl’s picture

g33kg1rl commented
Status: Postponed (maintainer needs more info) » Closed (fixed)

We were able to troubleshoot this issue. The trouble was php-mysql, which is used by both MariaDB and the mysql packages. We had to purge MariaDB, reinstall mysql, purge mysql and then run barracuda up-stable with the force sql option.

So the only change to your steps would be to add apt-get purge instead of apt-get remove.

We would have never figured this out without your help. Thank you so much!

omega8cc’s picture

omega8cc commented

Just for the record, I didn't include the apt-get purge for mysql packages because it could also delete all your databases if you didn't realize what happens when you say Y on its all prompts, but if that helped and you didn't delete databases, that is OK.