We want Aegir to be able to backup and migrate itself. Right now, we assume that it is installed in sites/default which makes it impervious to tasks and things like that (see #453540: consider the 'default' site like a regular site, with exceptions for that particular issue). Aegir should be installed in a real sites/fqdn directory and processed like the others, with some exceptions. In particular, "restore" actions should be very carefully performed, with lots of warning, and maybe followed by a system-wide platform verification. "Disable" should be forbidden altogether.

The biggest hurdle in getting hosting to upgrade itself is how we manipulate the task log. When the log is written to the database, the DB may have changed names or the table may have changed structure. The first part has an easy fix: reload the DB credentials before writing the log. The second part is more tricky: I don't think there's an easy solution and **some** upgrades will have to be performed manually.

But it would be important to be able to upgrade from (say) Drupal 5.14 to 5.16 (without changes to the hosting DB structure) automatically.

Comments

anarcat’s picture

Hosting setup needs to be rerun (and rerunnable) when migration of aegir is be finished so that the cronjob gets fixed.

adrian’s picture

Status: Active » Fixed

http://drupal.org/node/319277#comment-1634440

SO it's kind of working.

We still need to change setup to replace the arguments to hosting dispatch.

s+(.*)'(.*)/drush.php'(.*)--root='(.*)'.*+$1 '/path/to/drush.php' --root='$4' $3 > /dev/null)+

That regex will handle it for us.

I just committed the re-runnable setup command, so the master site is now fully manageable, for everything but installs (which we won't be able to do until d6).

Closing this.

anarcat’s picture

Version: 5.x-0.2-alpha1 » 5.x-0.2-rc1
Status: Fixed » Needs work

So that's nice: we can deploy a new site using the backend, but what about the frontend? We have yet to be able to migrate Aegir itself, from the frontend. I think it's a reasonable goal.

anarcat’s picture

So I'll be posting the migration instructions in the UPGRADE.txt file. I found two bugs during testing, in the backup system, which i fixed in those commits:

http://drupal.org/cvs?commit=217648
http://drupal.org/cvs?commit=217654

I also had to verify the d6 platform through the web interface first, because otherwise it would fail like this:

sh-3.1$ /var/hostmaster/drush/drush.php --debug --verbose=2 provision verify
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.051 sec]   [bootstrap]
Initialized Drupal 6.12 root directory at /var/hostmaster/drupal-6.12   [notice]
[0.094 sec]
Found command: provision verify [0.097 sec]                          [bootstrap]
Initializing drush commandfile: provision_apache [0.097 sec]         [bootstrap]
Initializing drush commandfile: provision_drupal [0.103 sec]         [bootstrap]
Initializing drush commandfile: provision_mysql [0.103 sec]          [bootstrap]
Including                                                            [bootstrap]
/var/hostmaster/.drush/provision/web_server/verify.provision.inc
[0.104 sec]
Including                                                            [bootstrap]
/var/hostmaster/.drush/provision/platform/verify.provision.inc [0.112
sec]
Including                                                            [bootstrap]
/var/hostmaster/.drush/provision/db_server/verify.provision.inc
[0.118 sec]
Could not connect to the master database. [0.155 sec]                [error]
An error occurred at function :                                      [error]
drush_provision_mysql_provision_verify_validate [0.156 sec]
Command dispatch complete [0.156 sec]                                   [notice]
anarcat’s picture

Issue tags: +sentient, +self-hosting, +skynet

We are likely to need the #500362: dispatcher locking if we want to run concurrent frontend installations (which is probably required to have the frontend upgrade itself.

anarcat’s picture

adrian’s picture

Here are the patches for the new hostmaster migrate command.

These patches add a new command to provision which can be used as :
drush --root=/path/to/old/platform hostmaster migrate example.com /path/to/new/platform --debug

I'd suggest having exact copies of the hostmaster platform on two different versions of drupal core, for testing right now.

ie: migrate hostmaster HEAD on drupal 6.10 to hostmaster HEAD on drupal 6.14

to handle migrations from previous versions of hostmaster to ones after this patch lands, we will need to (for now) be tolerant of the 'hostmaster park' command not being available, and possibly fall back on just disabling the queue by hand (it's not perfect but it should work in almost all cases).

i'm also pondering the idea of shipping a drush_make file alongside provision, so it can roll out the latest version of the interface.

anarcat’s picture

So the current state of affairs is this:

1. we can migrate a hostmaster site using the hostmaster-migrate command, in the backend
2. the UPGRADE.txt recommends the hostmaster-migrate process
3. we cannot yet provision-install hostmaster sites (see #711740: allow install of hostmaster sites in provision-install)
4. the install process can setup only provision or the whole thing, but lacks some glue to run hosting-setup and other things (#711760: separate backend and frontend install process)
5. we cannot yet migrate or clone hostmaster sites from the frontend (see #711746: allow upgrade of hostmaster sites from the frontend)

Once all this is done, we can change the project name to Skynet, a few expert systems, neural networks and new AI thingies, hook it up to some nukes, and watch the fireworks. ;)

Anonymous’s picture

Only 5.) is left now :)

anarcat’s picture

Just a note here to mention that #500362: dispatcher locking was implemented, which theorically makes it possible to run the frontend dispatcher on multiple servers at once.

steven jones’s picture

Status: Needs work » Postponed

  • Commit fe0539e on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by adrian:
    #454312 - hosting setup is now re-runnable. for post migration of...
  • Commit a5cf660 on 5.x, 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by anarcat:
    adjust upgrade instructions: don't migrate, but stage using 'deploy'...
  • Commit e81bd23 on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by anarcat:
    now that #454312 is fixed, we don't need an extra verify (which wasn't...

  • Commit fe0539e on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by adrian:
    #454312 - hosting setup is now re-runnable. for post migration of...
  • Commit a5cf660 on 5.x, 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by anarcat:
    adjust upgrade instructions: don't migrate, but stage using 'deploy'...
  • Commit e81bd23 on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by anarcat:
    now that #454312 is fixed, we don't need an extra verify (which wasn't...

  • adrian authored fe0539e on 7.x-3.x-2345987
    #454312 - hosting setup is now re-runnable. for post migration of...
  • anarcat authored a5cf660 on 7.x-3.x-2345987
    adjust upgrade instructions: don't migrate, but stage using 'deploy'...
  • anarcat authored e81bd23 on 7.x-3.x-2345987
    now that #454312 is fixed, we don't need an extra verify (which wasn't...
ergonlogic’s picture

Issue summary: View changes
Status: Postponed » Closed (duplicate)