i ran into trouble today trying to install Aegir using hosting and provision HEAD revisions, which (with anarcat's help) was tracked down to residual hostmaster.profile and hosting module dependencies. i had attempted to install following these instructions, provided by anarcat in irc:

cd /var
sudo cvs -z6 -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal co -d aegir/drupal-5 -r DRUPAL-5-12 drupal # whatever, maybe 5-13 now
cd aegir/drupal-5/
sudo cvs co -d profiles/hostmaster contributions/profiles/hostmaster # no -r
sudo cvs co -d sites/default/modules/hosting contributions/modules/hosting # same
# now we change the process
cd ~
sudo cvs co -d drush  contributions/modules/drush
sudo mkdir .drush
sudo cvs co -d .drush/provision  contributions/modules/provision

anarcat's grep matches on what needs to go:
http://anarcat.ath.cx/~anarcat/pastes/2009-02-16T180721

Comments

spiderman’s picture

spiderman
he/they
Primary language English
Location Halifax, NS
commented 
18:03 <@anarcat> so i guess a workaround is to install 0.1 provision into sites/default/modules, just to get those 
                 functions
18:04 <@anarcat> but setup the platform using the provision in .drush
18:04 <@anarcat> not sure it's going to work, but it's worth the try

so that seemed to work, but then i got to the stage of running 'drush hosting setup' from within the /var/aegir/drupal-5 dir (as directed by the hosting wizard at hosting/wizard/hosting/init), and got:

Drush was not able to start (bootstrap) Drupal.                          [error]
Hint: This error can only occur once the database connection has already
been succesfully initiated, therefor this error generally points to a
site configuration issue, and not a problem connecting to the database.

Drush was attempting to connect to : 
  Drupal Root       : /var/aegir/drupal-5
  Site Path         : sites/default
  Site URI          : 
  Database Driver   : mysql
  Database Hostname : localhost
  Database Username : aegir
  Database Name     : aegir_02
anarcat’s picture

anarcat commented
Priority: Normal » Critical

So after discussion with Adrian, I realized this was a pretty serious issue.. Hosting *does* need some functions from provision so those need to either be duplicated into provision (meh) or hosting must be able to load provision.

Now there are two ways hosting is called: from the web interface, through Apache/PHP and from the commandline through Drush. The former will not have access to the provision module so all the functions that call provision in that code path need to be fixed. The latter *will* have access to drush so it will be able to call parse_output() and things like that.

Basically, the IPC layer can be written in provision.module and hosting can use it since it's also called from drush.

Regarding my earlier paste of "what needs to be fixed", here's a filtered output based on the above criterias:

./db_server/hosting_db_server.module:265:  $path = _provision_config_path();
./web_server/hosting_web_server.module:29:  $mkdir_cmd['@provision_link'] = url('admin/settings/provision');
./web_server/hosting_web_server.module:44:    but you can change it change them in the <a href="@provision_link">provisioning section</a>.', $mkdir_cmd);
./web_server/hosting_web_server.module:266:    if (!provision_user_in_group($user, $group)) {
./platform/hosting_platform.module:279:  $path = _provision_config_path();
./platform/hosting_platform.module:282:  if (!provision_check_path($path . '/drush', "is_dir")) {
./platform/hosting_platform.module:283:    provision_check_path($path, "chmod", "770");
./platform/hosting_platform.module:284:    if (!provision_check_path($path . '/drush', "mkdir")) {
./platform/hosting_platform.module:289:  if (provision_check_path($path, 'writeable')) {
./hosting.templates.inc:15:      'template' => variable_get('provision_drupal_settings_template', _provision_drupal_default_template()),
./hosting.templates.inc:48:      'template' => variable_get('provision_apache_vhost_template', _provision_apache_default_template()),

Those lines are OKAY as they are in the drush code path (i think):

./task/hosting_task.module:75:    $values = array_merge($values, module_invoke_all('provision_args', node_load($node->rid), $task));
./task/hosting_task.module:457:  global $provision_errors;
./task/hosting_task.module:460:      return array(PROVISION_QUEUED => $provision_errors[PROVISION_QUEUED]);
./task/hosting_task.module:463:      return array(PROVISION_SUCCESS => $provision_errors[PROVISION_SUCCESS]);
./task/hosting_task.module:466:      foreach ($provision_errors as $error => $message) {
./itk/hosting_itk.module:16: * Implementation of hook_provision_args()
./hosting.queues.inc:152:  $data = module_invoke_all('provision_args', $ref, $command); 
./hosting.queues.inc:235:  global $provision_errors;
./hosting.queues.inc:257:    $proc = provision_proc_open($cmd);
./hosting.queues.inc:261:    if (!$data = provision_parse_output($return)) {
./hosting.module:350:  if (_provision_setup()) {
./hosting.module:360:      provision_log("message", t("Dispatch command was run successfully"));
./hosting.module:364:      provision_log("error", t("Dispatch command could not be run. Returned: \n@return", array('@return' => $return))); 
./hosting.module:365:      provision_set_error(PROVISION_FRAMEWORK_ERROR);
./hosting.module:369:  // @TODO use provision_output for this, but we need pretty print first.
./hosting.module:370:  $logs = provision_get_log();
./hosting.module:375:  if (provision_get_error()) {
./hosting.module:387:        provision_log("Notice", t("Existing hosting dispatch cron entry was found. Not replacing"));
./hosting.module:394:    provision_log("message", t("No existing crontab was found"));
./hosting.module:411:    provision_log("Notice", t("Installed hosting dispatch cron entry to run every minute"));

... although some will probably have to be ported to use drush_ instead of provision_.

Those I'm not sure about at all:

./hosting_help.inc:136:function provision_help($section) {
./hosting_help.inc:365:    $mkdir_cmd['@provision_link'] = url('admin/settings/provision');
./hosting_help.inc:456:function provision_elements() {

And the DNS module probably needs a bit of a cleanup:

./dns_server/README.txt:6:   * this currently implies manually activating the provision_dns modules (see
./dns_server/hosting_dns.api.inc:92:/* replace provision_dns_zone() multi-op API call to manage soa table */
./dns_server/hosting_dns.api.inc:97:/* replace provision_dns_rr() multi-op API call to manage rr table */
./dns_server/hosting_dns.install:54:      // The provision_dns_soa table defines zones
./dns_server/hosting_dns.install:74:      // The provision_dns_rr table defines Resource Records
./dns_server/hosting_dns.module:8: * corresponding provision_dns.module in the provision project complements this
./dns_server/hosting_dns.module:103: * Implementation of hook_provision_args
./dns_server/hosting_dns.module:613:    $tlds = split("\n", variable_get('provision_dns_tlds', ''));
./dns_server/hosting_dns.admin.inc:228:    #provision_dns_zone('update', $form_values);   
./dns_server/hosting_dns.admin.inc:231:    #provision_dns_zone('add', $form_values);
./dns_server/hosting_dns.admin.inc:233:  return 'admin/build/provision_dns';
./dns_server/hosting_dns.admin.inc:247:  #provision_dns_zone('delete', array('zid' => $form_values['zid']));
anarcat’s picture

anarcat commented
Status: Active » Needs work

So this commit fixed some issues, but we still have some remaining:

./web_server/hosting_web_server.module:29:  $mkdir_cmd['@provision_link'] = url('admin/settings/provision');
./web_server/hosting_web_server.module:44:    but you can change it change them in the <a href="@provision_link">provisioning section</a>.', $mkdir_cmd);
./platform/hosting_platform.module:279:  $path = _provision_config_path();
./platform/hosting_platform.module:282:  if (!provision_check_path($path . '/drush', "is_dir")) {
./platform/hosting_platform.module:283:    provision_check_path($path, "chmod", "770");
./platform/hosting_platform.module:284:    if (!provision_check_path($path . '/drush', "mkdir")) {
./platform/hosting_platform.module:289:  if (provision_check_path($path, 'writeable')) {
./hosting.templates.inc:15:      'template' => variable_get('provision_drupal_settings_template', _provision_drupal_default_template()),
./hosting.templates.inc:48:      'template' => variable_get('provision_apache_vhost_template', _provision_apache_default_template()),

Those probably still need to go:

./hosting_help.inc:136:function provision_help($section) {
./hosting_help.inc:365:    $mkdir_cmd['@provision_link'] = url('admin/settings/provision');
./hosting_help.inc:456:function provision_elements() {

And the DNS module still needs cleanup.

adrian’s picture

adrian commented

I removed all those instances. Help now also works again.

Now the tricky part is going to be the provision constants.

I think hostmaster needs to change to only create the initial user / client.

The first page of installation will then be to add the server node, which will instruct the user to use a command that will set up the hosting
platform (ie: hosting setup).

This will populate the server specific set of provision functions. These will then be used as defaults to manually create the platform node.

All instances of the error handling constants will need to be migrated from.

hosting.module:    if ($code == PROVISION_SUCCESS) {
hosting.module:      provision_set_error(PROVISION_FRAMEWORK_ERROR);
hosting.queues.inc:  exit(PROVISION_SUCCESS);
hosting.queues.inc:  exit(PROVISION_SUCCESS);
hosting.queues.inc:  $result = db_query("SELECT nid FROM {hosting_task_queue} WHERE status=%d ORDER BY timestamp, nid ASC LIMIT %d", PROVISION_QUEUED, $limit);
hosting.queues.inc:  return (db_result(db_query("SELECT nid FROM {hosting_task_queue} WHERE status=%d AND nid=%d", PROVISION_QUEUED, $nid))) ? TRUE : FALSE;
hosting.queues.inc:      $code = PROVISION_FRAMEWORK_ERROR;
hosting.queues.inc:    if ($code == PROVISION_SUCCESS) {
hosting.queues.inc:  $drush_path = sprintf("%s/drush.php", PROVISION_DOCROOT_PATH);
hosting.queues.inc:      escapeshellarg(PROVISION_DOCROOT_PATH));
hosting.wizard.inc:  if ($node->task_status & PROVISION_SUCCESS) {
hosting.wizard.inc:  if (!($node->task_status & PROVISION_SUCCESS)) {
hosting_help.inc:  $docroot = PROVISION_DOCROOT_PATH;
hosting_help.inc:  $uri = PROVISION_BASE_URL;
hosting_help.inc:  $username = PROVISION_SCRIPT_USER;
hosting_help.inc:  $username = PROVISION_SCRIPT_USER;
hosting_help.inc:    $username = PROVISION_SCRIPT_USER;
hosting_help.inc:    $group = PROVISION_WEB_GROUP;
hosting_help.inc:    $backup_path = PROVISION_BACKUP_PATH;
hosting_help.inc:  $username = PROVISION_SCRIPT_USER;
hosting_help.inc:  $path = PROVISION_DOCROOT_PATH;
hosting_help.inc:  $help['suggestion'] = t('Based on your server configuration, we have determined that your user account should be @script_user', array('@script_user' => PROVISION_SCRIPT_USER));
hosting_help.inc:    $username = PROVISION_SCRIPT_USER;
hosting_help.inc:    $group = PROVISION_WEB_GROUP;
site/hosting_site.module:  if ($code & PROVISION_SITE_INSTALLED) {
site/hosting_site.module:  if ($node->status & HOSTING_SITE_DELETED || ($task_status != PROVISION_QUEUED && $task_status != PROVISION_SUCCESS)) {
task/hosting_task.module:  if (!($status & PROVISION_SUCCESS)) {
task/hosting_task.module:    db_query("UPDATE {hosting_task_queue} SET STATUS = %d WHERE nid =%d", PROVISION_QUEUED, $task_id);
task/hosting_task.module:    $node->task_status = PROVISION_QUEUED;
task/hosting_task.module: $task->task_status = PROVISION_QUEUED;
task/hosting_task.module:    db_query("INSERT INTO {hosting_task_queue} (nid, timestamp, status) VALUES (%d, %d, %d)", $node->nid, mktime(), PROVISION_QUEUED);
task/hosting_task.module:    && !in_array($node->task_status, array(PROVISION_QUEUED, PROVISION_SUCCESS))
task/hosting_task.module:define('PROVISION_QUEUED', 0);
task/hosting_task.module:define('PROVISION_SUCCESS', 1);
task/hosting_task.module:define('PROVISION_INSTALL_ERROR', 2);
task/hosting_task.module:define('PROVISION_PERM_ERROR', 4); 
task/hosting_task.module:define('PROVISION_WEB_ERROR', 8);
task/hosting_task.module:define('PROVISION_FRAMEWORK_ERROR', 16);
task/hosting_task.module:define('PROVISION_SITE_NOT_FOUND', 32);
task/hosting_task.module:define('PROVISION_SITE_INSTALLED', 64);
task/hosting_task.module:define('PROVISION_DB_ERROR', 128);
task/hosting_task.module:  PROVISION_SUCCESS => t("Successful"),
task/hosting_task.module:  PROVISION_QUEUED => t("Queued"),
task/hosting_task.module:  PROVISION_DB_ERROR => t("Database error"),
task/hosting_task.module:  PROVISION_INSTALL_ERROR => t("Drupal installation error"),
task/hosting_task.module:  PROVISION_PERM_ERROR => t("File permission error"),
task/hosting_task.module:  PROVISION_WEB_ERROR => t("Web server error"),
task/hosting_task.module:  PROVISION_FRAMEWORK_ERROR => t("Provision framework error"),
task/hosting_task.module:  PROVISION_SITE_NOT_FOUND => t("Site not found"),
task/hosting_task.module:  PROVISION_SITE_INSTALLED => t("Site has already been installed"),  
task/hosting_task.module:    case PROVISION_QUEUED : 
task/hosting_task.module:      return array(PROVISION_QUEUED => $provision_errors[PROVISION_QUEUED]);
task/hosting_task.module:    case PROVISION_SUCCESS : 
task/hosting_task.module:      return array(PROVISION_SUCCESS => $provision_errors[PROVISION_SUCCESS]);
task/hosting_task.module:        if (!in_array($error, array(PROVISION_QUEUED, PROVISION_SUCCESS))) {
task/hosting_task.module:  if ($status == PROVISION_QUEUED) {
task/hosting_task.module:  if ($status & PROVISION_SUCCESS) {
task/hosting_task.module:       if (!in_array($node->task_status, array(PROVISION_QUEUED, PROVISION_SUCCESS))) {
web_server/hosting_web_server.module:  $username = PROVISION_SCRIPT_USER;
web_server/hosting_web_server.module:  $group = PROVISION_WEB_GROUP;
web_server/hosting_web_server.module:  $vhost_path = PROVISION_CONFIG_PATH;
web_server/hosting_web_server.module:  $vhost_path = PROVISION_VHOST_PATH;
web_server/hosting_web_server.module:  $username = PROVISION_SCRIPT_USER;
web_server/hosting_web_server.module:  $cmd = trim(str_replace("sudo", '', PROVISION_RESTART_CMD));
web_server/hosting_web_server.module:      '#default_value' => ($node->restart_cmd) ? $node->restart_cmd : PROVISION_RESTART_CMD,
web_server/hosting_web_server.module:    '#default_value' => ($node->script_user) ? $node->script_user : PROVISION_SCRIPT_USER,
web_server/hosting_web_server.module:    '#default_value' => ($node->web_group) ? $node->web_group : PROVISION_WEB_GROUP,
web_server/hosting_web_server.module:    '#default_value' => ($node->config_path) ? $node->config_path : PROVISION_CONFIG_PATH,
web_server/hosting_web_server.module:    '#default_value' => ($node->backup_path) ? $node->backup_path : PROVISION_BACKUP_PATH,
adrian’s picture

adrian commented

I have some code in my new checkout to use the default drush return codes.

Here is a list of the stuff i couldn't change :

db_server/hosting_db_server.module:21:   $username = PROVISION_SCRIPT_USER;
dns_server/hosting_dns.module:36:  $username = PROVISION_SCRIPT_USER;
dns_server/hosting_dns.module:37:  $group = PROVISION_WEB_GROUP;
dns_server/hosting_dns.module:38:  $path = PROVISION_CONFIG_PATH;
dns_server/hosting_dns.module:62:#  $username = PROVISION_SCRIPT_USER;
dns_server/hosting_dns.module:63:#  $cmd = trim(str_replace("sudo", '', PROVISION_RESTART_CMD));
dns_server/hosting_dns.module:218:      '#default_value' => $node->zonedata ? $node->zonedata : PROVISION_CONFIG_PATH .'/named',
hosting.queues.inc:307:  $drush_path = sprintf("%s/drush.php", PROVISION_DOCROOT_PATH);
hosting.queues.inc:309:      escapeshellarg(PROVISION_DOCROOT_PATH));
hosting_help.inc:177:  $docroot = PROVISION_DOCROOT_PATH;
hosting_help.inc:178:  $uri = PROVISION_BASE_URL;
hosting_help.inc:180:  $username = PROVISION_SCRIPT_USER;
hosting_help.inc:200:  $username = PROVISION_SCRIPT_USER;
hosting_help.inc:356:    $username = PROVISION_SCRIPT_USER;
hosting_help.inc:357:    $group = PROVISION_WEB_GROUP;
hosting_help.inc:358:    $backup_path = PROVISION_BACKUP_PATH;
hosting_help.inc:377:  $username = PROVISION_SCRIPT_USER;
hosting_help.inc:378:  $path = PROVISION_DOCROOT_PATH;
hosting_help.inc:389:  $help['suggestion'] = t('Based on your server configuration, we have determined that your user account should be @script_user', array('@script_user' => PROVISION_SCRIPT_USER));
hosting_help.inc:397:    $username = PROVISION_SCRIPT_USER;
hosting_help.inc:398:    $group = PROVISION_WEB_GROUP;
itk/hosting_itk.module:37:    $user = posix_getpwnam(PROVISION_SCRIPT_USER);
itk/hosting_itk.module:55:    $user = posix_getpwnam(PROVISION_SCRIPT_USER);
web_server/hosting_web_server.module:25:  $username = PROVISION_SCRIPT_USER;
web_server/hosting_web_server.module:26:  $group = PROVISION_WEB_GROUP;
web_server/hosting_web_server.module:27:  $vhost_path = PROVISION_CONFIG_PATH;
web_server/hosting_web_server.module:50:  $vhost_path = PROVISION_VHOST_PATH;
web_server/hosting_web_server.module:67:  $username = PROVISION_SCRIPT_USER;
web_server/hosting_web_server.module:68:  $cmd = trim(str_replace("sudo", '', PROVISION_RESTART_CMD));
web_server/hosting_web_server.module:216:      '#default_value' => ($node->restart_cmd) ? $node->restart_cmd : PROVISION_RESTART_CMD,
web_server/hosting_web_server.module:227:    '#default_value' => ($node->script_user) ? $node->script_user : PROVISION_SCRIPT_USER,
web_server/hosting_web_server.module:238:    '#default_value' => ($node->web_group) ? $node->web_group : PROVISION_WEB_GROUP,
web_server/hosting_web_server.module:249:    '#default_value' => ($node->config_path) ? $node->config_path : PROVISION_CONFIG_PATH,
web_server/hosting_web_server.module:261:    '#default_value' => ($node->backup_path) ? $node->backup_path : PROVISION_BACKUP_PATH,

The only one that needs special attention is :

site/hosting_site.module:205:  if ($code & PROVISION_SITE_INSTALLED) {

The code does not represent if the site has been installed, we need to check the returned array of errors, for which provision needs to be modified to use drush_set_error and not provision_set_error.

adrian’s picture

adrian commented
Status: Needs work » Fixed

Most of those could only be defaulted from within the site itself (ie: the web group etc), so i have copied the little stuff that was needed into hosting, and changed all the defaults.

Hostmaster now installs clean 0.2

anarcat’s picture

anarcat commented
Title: hosting should not depend on provision » hostmaster should not depend on provision
Project: Hosting » Hostmaster (Aegir)
Version: 5.x-0.2.x-dev »
Status: Fixed » Needs work

Well, we still have work to do here. Either hostmaster should not depend on provision/drush, or we should change the way we setup platforms right now, because the install profile explicitely depends on those modules. Even if i remove the dependencies from the profile, I still get this:

Fatal error: Call to undefined function _provision_mysql_can_create_database() in /var/hostmaster/drupal-5.x/profiles/hostmaster/hostmaster.profile on line 80
anarcat’s picture

anarcat commented

I fixed brutally a few issues in the hostmaster profile (not committed yet), but I still stumble upon errors in hostnig.module:

Fatal error: Call to undefined function provision_log() in /var/hostmaster/drupal-5.x/profiles/hostmaster/modules/hosting/hosting.module on line 380
anarcat’s picture

anarcat commented

I committed two fixes to both hosting and hostmaster:

http://drupal.org/cvs?commit=176206
http://drupal.org/cvs?commit=176204

anarcat’s picture

anarcat commented
Status: Needs work » Fixed

I feel like i fixed all the issues with the latest commits above.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 221fa87 on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by anarcat: 
    remove some provision_log still remaining

See: #375360

  • Commit 221fa87 on 599758_edit_main_site, 640952_client_preview, dev-dns, dev-features, dev-headless_install, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newsiteform, dev-nginx, dev-platform_management, dev-ports, dev-purgebackup, dev-relationships, dev-restore, dev-server_nodetype, dev-services, dev-site_rename, dev-ssl, dev_dns, prod-koumbit, ssl, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-588728-views-integration, dev-1403208-new_roles, dev-helmo-3.x authored by anarcat: 
    remove some provision_log still remaining

See: #375360