After updating Barracuda successfully, get the following error when upgrading from BOA-2.0.5 using:

octopus up-stable all

ctopus [Tue Apr  2 13:19:30 UTC 2013] ==> UPGRADE B: Running hostmaster-migrate, please wait...
Octopus [Tue Apr  2 13:20:59 UTC 2013] ==> UPGRADE B: Hostmaster STATUS: upgrade completed
Octopus [Tue Apr  2 13:20:59 UTC 2013] ==> UPGRADE B: Simple check if Aegir upgrade is successful                                       Octopus [Tue Apr  2 13:21:01 UTC 2013] ==> UPGRADE B: FATAL ERROR: Required file /data/disk/o1/aegir/distro/003/sites/o1.server.sizeshape.com/settings.php does not exist
Octopus [Tue Apr  2 13:21:01 UTC 2013] ==> UPGRADE B: FATAL ERROR: Aborting AegirSetupB installer NOW!
Octopus [Tue Apr  2 13:21:01 UTC 2013] ==> UPGRADE A: FATAL ERROR: AegirSetupB installer failed
Octopus [Tue Apr  2 13:21:01 UTC 2013] ==> UPGRADE A: FATAL ERROR: Aborting AegirSetupA installer NOW!
Octopus [Tue Apr  2 13:21:01 UTC 2013] ==> FATAL ERROR: AegirSetupA installer failed
Octopus [Tue Apr  2 13:21:01 UTC 2013] ==> FATAL ERROR: Aborting Octopus installer NOW!
CommentFileSizeAuthor
#9 patch_commit_4e87f87ea3ff.patch785 bytesomega8cc
#2 Terminal log140.04 KBD.Holmlund
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

omega8cc’s picture

Status: Active » Postponed (maintainer needs more info)

Please follow guidelines.

D.Holmlund’s picture

Status: Postponed (maintainer needs more info) » Active
FileSize
140.04 KB

I am having the exact same problem with UPGRADE B: FATAL ERROR: Required file /data/disk/fub11/aegir/distro/004/sites/o1.XXXXXXX/settings.php does not exist when running octopus up-stable after upgrading to BOA-2.0.6 stable. I have tried removing incomplete 00x-directories and clearing /opt/tmp/ without any luck.

The logs are also giving me the following errors (see the attached terminal log for more info):

UPDATE {hosting_ip_addresses} SET id=19 WHERE nid=4 AND ip_address='127.0.0.1' [40.11 sec, 18.01 MB]
UPDATE {hosting_ip_addresses} SET id=20 WHERE nid=4 AND ip_address='127.0.0.1' [40.11 sec, 18.01 MB]
Drush was not able to start (bootstrap) the Drupal database.

It has affected two separate servers after upgrading from 2.0.5 to 2.0.6 so it does not seem to be just a local problem.

omega8cc’s picture

Category: support » bug

@D.Holmlund - Thanks. The problem is shown here:

Executing hosting_server_update_6200 [34.09 sec, 17.97 MB]                                                                                                  [success]
Duplicate entry '19' for key 'PRIMARY'                                                                                                  [warning]
query: UPDATE hosting_ip_addresses SET id=19 WHERE nid=4 AND ip_address='127.0.0.1' database.mysqli.inc:169 [34.09 sec, 17.97 MB]
Duplicate entry '20' for key 'PRIMARY'                                                                                                  [warning]
query: UPDATE hosting_ip_addresses SET id=20 WHERE nid=4 AND ip_address='127.0.0.1' database.mysqli.inc:169 [34.09 sec, 17.97 MB]
UPDATE {hosting_ip_addresses} SET id=19 WHERE nid=4 AND ip_address='127.0.0.1' [34.1 sec, 18.01 MB]                                               [error]
UPDATE {hosting_ip_addresses} SET id=20 WHERE nid=4 AND ip_address='127.0.0.1' [34.1 sec, 18.01 MB]                                               [error]

Basically, hosting_server_update_6200 is trying to upgrade the hosting_ip_addresses table, but it looks like for some reason you have had there duplicate entries for 127.0.0.1 already, so it fails because there are new keys added, but not sure.

Note that it looks like a duplicate of:

#1930670: Duplicate entry 0 for key PRIMARY in hosting_ip_addresses when installing / upgrading
#1932616: IPs deleted from hosting_ip_addresses table on server verify

At least the errors are the same, but with our version of hosting_server_update_6200, well tested on many servers already, it shouldn't happen.

Could you post a result of this command?

su -s /bin/bash - fub1 -c "drush @hostmaster sqlq \"SELECT * FROM hosting_ip_addresses\""
omega8cc’s picture

Also:

su -s /bin/bash - aegir -c "drush @hostmaster sqlq \"SELECT * FROM hosting_ip_addresses\""

omega8cc’s picture

Priority: Normal » Critical

At any rate, it is critical.

D.Holmlund’s picture

Priority: Critical » Normal

Here goes (1.1.1.1 is anonymized):

su -s /bin/bash - fub1 -c "drush @hostmaster sqlq \"SELECT * FROM hosting_ip_addresses\""

vid	nid	ip_address
586	551	1.1.1.1
521	509	1.1.1.1
643	577	1.1.1.1
550	534	1.1.1.1
563	540	1.1.1.1
577	546	1.1.1.1
2	2	1.1.1.1
769	643	1.1.1.1
847	678	1.1.1.1
659	584	1.1.1.1
10	10	1.1.1.1
607	562	1.1.1.1
622	566	1.1.1.1
722	618	1.1.1.1
753	638	1.1.1.1
821	666	1.1.1.1
856	681	1.1.1.1
863	685	1.1.1.1
888	700	1.1.1.1
4	4	127.0.0.1
4	4	127.0.0.1

su -s /bin/bash - aegir -c "drush @hostmaster sqlq \"SELECT * FROM hosting_ip_addresses\""

vid	nid	ip_address
2	2	1.1.1.1
10	10	1.1.1.1
4	4	127.0.0.1
4	4	127.0.0.1
omega8cc’s picture

Priority: Normal » Critical
Status: Active » Needs work

Right, so you have duplicate entries in both Master and Satellite instances already and this is what breaks the upgrade. I have never seen anything like that on any server we manage.

However, it should also break the Master instance (barracuda) upgrade, but it worked (?)

I guess we need to run some de-duplicate on this table as a part of the hosting_server_update_6200 upgrade to avoid this edge case.

omega8cc’s picture

Project: Octopus » Hostmaster (Aegir)
Version: » 6.x-2.x-dev
Component: Aegir Hostmaster » Code
Status: Needs work » Needs review

This simple patch should do the trick, I guess.

I'm moving this to the correct upstream queue, btw.

omega8cc’s picture

Uploading the patch for review again.

D.Holmlund’s picture

Patch works like a charm, thank you!

omega8cc’s picture

Status: Needs review » Needs work

That patch alone is not enough. It "works" in BOA now only because we have added some hot-fix to avoid this issue, but the correct and complete patch still needs to be submitted for review.

omega8cc’s picture

linksync’s picture

Status: Closed (duplicate) » Needs work

Worked for me also.

Sever has default setup: Debian squeeze and BOA-2.0.4, which was upgraded to BOA-2.0.5. No additional packages or customisation on the server.

omega8cc’s picture

Status: Needs work » Closed (duplicate)