Upgrade from Civi 3x to 4x fails

Based on your patch, I am currently working on a version that separate the verify and the deploy task.

It's working but it need more work to find the good version of Drupal to put in the civicrm.settings.php (ie. Drupal6 only if Drupal6 and Civi >= 4.1)

I should have beta code to propose soon.

Sorry it took me so long. I'm new to provision.
I have tested the patch on some empty civicrm site and it's working well. I will now test it on production web site.
The separation of deploy and verify is far from perfect and the code is duplicated and things like civicrm file settings creation is done several times as we can see in this log :

Backed up site up to /var/aegir/backups/civi-d6-4-20120421.173236.tar.gz.                                                          [success]
Created civid64_2 database                                                                                                         [success]
Settings file generated: /var/aegir/platforms/civi-4.1.1-d6/sites/civi-d6-4/civicrm.settings.php                                   [ok]
CiviCRM: Generated config civicrm.settings.php file                                                                                [ok]
CiviCRM: Fixing permissions                                                                                                        [ok]
CiviCRM: CiviCRM upgrade was successful.<br />WARNING! CiviCRM 4.1 introduces an improved way of handling cron jobs. However [ok]
the new method is NOT backwards compatible. <strong>Please notify your system administrator that all CiviCRM related cron
jobs will cease to work, and will need to be re-configured (this includes sending CiviMail mailings, updating membership statuses,
etc.).</strong> Refer to the <a
href="http://wiki.civicrm.org/confluence/display/CRMDOC41/Managing+Scheduled+Jobs">online documentation</a> for
detailed instructions.<br />The CiviCRM Administration menu structure has been re-organized during this upgrade to make it
easier to find things and reduce the number of keystrokes. If you have customized this portion of the navigation menu - you should
take a few minutes to review the changes. You may need to reimplement or move your customizations.<br />Yahoo recently
discontinued their geocoding and mapping API service. If you previously used Yahoo, you will need to select and configure an
alternate service in order to continue using geocoding/mapping tools.<br />We have integrated KCFinder with CKEditor and
TinyMCE, which enables user to upload images. Note that all the images uploaded using KCFinder will be public.
No database updates required                                                                                                       [success]
'all' cache was cleared                                                                                                            [success]
Finished performing updates.                                                                                                       [ok]
Generating civicrm settings file ..
Settings file generated: /var/aegir/platforms/civi-4.1.1-d6/sites/civi-d6-4/civicrm.settings.php                                   [ok]
CiviCRM: Generated config civicrm.settings.php file                                                                                [ok]
Config successfully updated.                                                                                                       [completed]
CiviCRM: Fixing permissions                                                                                                        [ok]
Cache cleared.                                                                                                                     [ok]
Generating civicrm settings file ..
Settings file generated: /var/aegir/platforms/civi-4.1.1-d6/sites/civi-d6-4/civicrm.settings.php                                   [ok]
CiviCRM: Generated config civicrm.settings.php file                                                                                [ok]
Config successfully updated.                                                                                                       [completed]
CiviCRM: Fixing permissions                                                                                                        [ok]
Cache cleared.                                                                                                                     [ok]
Generating civicrm settings file ..

Thanks for the patch!

Questions:

* in _provision_civicrm_get_cms(), why not use drush_drupal_major_version() ?
* as far as I understand, you are proposing to remove the upgrade code from 'verify' in order to have it in 'deploy', which is run on import/migrate? Just to be sure, is this also how drupal does?
* did you test in a non-CiviCRM platform?

I started integrating some patches, so the upgrade path is working, but will release a beta first, and then apply the patch, so that we can get a stable release out before pushing a bit change.

For the $cms, I applied the following patch upstream.

In fact, for the Drupal6 'cms' issue, see: #1549850: civicrm.settings.tpl should generate "Drupal6" not "Drupal" for CIVICRM_UF in D6 environment.

* in _provision_civicrm_get_cms(), why not use drush_drupal_major_version() ?

Just inexperience :) the solution you propose is clearly better.

* as far as I understand, you are proposing to remove the upgrade code from 'verify' in order to have it in 'deploy', which is run on import/migrate? Just to be sure, is this also how drupal does?

Yes i think so, but as you can see, i'm not really an expert on drush / provision

* did you test in a non-CiviCRM platform?

Not yet but i will do some more test today... on real site and as you propose on some non-CiviCRM sites.

New patch that includes the changes of bgm.
There is still one big bug that i need to resolve - we should remove drush_bootstrap in deploy and find another way to find out if civicrm module is enabled.
Otherwise, it's working fine and i'm going to do some more test (non civicrm site for example).

To summarise offline discussions with samuelsov and anarcat:

* the civicrm DB upgrade should be done in deploy only
* when doing a Migrate task: aegir runs deploy, then verify, so careful to redundant operations, but some tasks such as regenerating civicrm.settings.php or checking permissions need to be done in verify (and some of those must be done before calling the DB upgrade)
* is OK to bootstrap Drupal, not many alternatives.

Patch committed (with a bit of refactoring). Please do more tests!