nameserver provisioning

It looks like we're aiming for the first mechanism (rsync) at the moment, in conjunction with an abstracted double API to accomplish this elegantly.

First, Aegir will have a generic "DNS Server" node type, which will be associated with a site and handle management of the zonefile/RRs. Underneath that, there will be a "DNS API" which implements the actual distribution/maintenance of the zonefiles on behalf of Drupal/Aegir. Different "engines" will implement the integration with different types of DNS software (BIND, djbdns, MyDNS, etc.)

Initially, we will only support BIND, but the idea is to define an abstracted DNS management tool that Aegir can integrate with, independent of the underlying DNS software.

More to come on this shortly..

subscribing

After much ado, I've got a very rough outline of the code required to achieve this feature ready for review. Please note that
this code is currently not working properly for me, and I'm only posting it for development review and debugging!

The attached patches have been rolled against the latest (D5) revisions of both the hosting and provision modules.

A quick outline:

* Minor patches to both provison and hosting incorporate the concept of a DNS server that is associated with each platform.
* The new hosting/dns_server/hosting_dns.module is the frontend node module that allows the admin to setup all the defaults for newly-created zones and records
* The new provision/dns_server/provision_dns.module provides an abstract implementation of DNS management facility for Aegir, and uses an "engine" (currently only provision/dns_server/engines/provision_bind.module exists) to do the heavy lifting
* Many of the provision hooks are not implemented, but I've made an attempt at the all-important "verify" hook, which is what is currently broken- it doesn't appear to run at all when I install a new site
* Once installed, you will need to:
* create a DNS server node, and setup appropriate defaults for it
* edit your platform node, to set the DNS server as your default for the platform
* add a line to your sudoers file similar to the apache2ctl one, except for the rndc utility which the aegir user will use to restart BIND

Also note that I've implemented a simple admin interface at admin/build/provision_dns, but it will only work if you break the secure permissions on the /var/aegir/config/named directory and files so that the web server can write to these directly.

Obviously this code will require some heavy revisions, but I'm hopeful that this is a good start to adding this feature to an increasingly powerful project :)

with help from adrian and anarcat, i've managed to fiddle some of the core provision stuff and re-rolled this patch so it is now working (relatively speaking) on my testbed. attached are the newpatch files. same caveats apply as above, except you should no longer see segfaults ;)

testing and feedback hugely appreciated :)

Here's a fresh patch with some bugfixes and a more complete implementation of the provision hooks. Important notes for making this work. Eventually, I'll figure out how to roll these things into the install/configuration process,but for now, you need to:

1. Hit update.php to get the DNS-specific updates to provision tables
2. Manually enable the provision_dns and provision_bind modules from the admin/build/modules page, and configure/save the defaults on the admin/settings/provision_bind page
3. Create a DNS server node (node/add/dns_server) after the configuration wizard has completed. Make sure you set a default IP, email address and primary/secondary NS records, or your zonefiles will be malformed.
4. Edit and re-save the default platform node (node/6/edit) so that it will store a default DNS server for the platform ( the DNS server doesn't show up in the form if there is only one DNS server node in the system)
5. Re-verify the platform in order to setup the new perms on $config and $config/named dirs. Note: The verify task is currently failing as per #343428, but it should still do everything required here

I think that's all, but please let me know if you have any trouble getting the patch to work- I'm very eager to get feedback on this! :)

Thanks to ac for finding some logic errors in the FQDN parsing routine (_provision_dns_split_url) which determines what zone to provision the site within. I've added a new settings page (currently at admin/settings/provision_dns) to allow the admin to define valid TLDs for Aegir to accept. These are used to validate site nodes on creation, and subsequently used to determine the zone at provision time.

One open question for me is: can we assume that the 'zone' for a given URL is simply the TLD + the last component preceding it? ie. what is the right 'zone' for a FQDN like www.foo.bar.com?

Anyway, fresh patch attached, with replacement _provision_dns_split_url function, error-checking in the _provision_pre_install and _provision_verify hooks, along with a hook_nodeapi implementation in the hosting_dns_server.module to validate sites upon creation. Also moved the BIND provision settings to a local task under the provision_dns ones, so they're now found at admin/settings/provision_dns/bind.

@anarcat: thanks for these comments. i've begun incorporating changes to my patch, but will leave this issue open until a decision is made on the server-type abstraction direction.

1. I've moved the define()s out of hosting.module and provision.module and put them in hook_init()s instead.

2. leaving this one until a hook exists ;)

3. I've moved the hook_help() into the hosting_dns_server.module and out of hosting_help.inc

4. To avoid patching hosting_platform, I've removed the dns_server field from hosting_platform.install for the moment, and used your suggestion of incorporating the ALTER TABLEs into hosting_dns_server.install instead.

However, this still leaves a 5 chunks in the patch of hosting_platform.module, which all relate to the new dns_server field, so I'm not sure this move makes much sense, until we abstract the 'servers' and how they relate to the platform better.

new patch forthcoming, after I go through your other set of comments, attempt to debug the error you got, and get my network connection stabilized again ;)

As i've just mentioned on irc.

Do not use alter to modify other modules' tables. There's no way to keep track of that stuff, and will cause problems later on.

Instead use a join table to keep the records, and write your own nodeapi implementation.

http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/hosting/ali...

That is an example to do it.

I created a hook_hosting_summary, and updated the existing modules to use that.

Here is an example :

function hosting_web_server_hosting_summary() {
  $summary = array();
  $web_servers = _hosting_get_web_servers();
  $summary['web_servers'] = theme('item_list', array_map('_hosting_node_link', array_keys($web_servers)), t('Web servers'));
  return $summary;
}

@Adrian: thanks for the feedback and new hook. I've rejigged hosting_dns_server.module to use a join table and nodeapi implementation for keeping track of each platform node's DNS server. I've also implemented the new hosting_summary hook.

I believe this addresses all of the comments in anarcat's first set of feedback (#8 above), and leaves my patch with only one small change to provision.module:

-    _provision_create_dir(PROVISION_CONFIG_PATH, t('Provision configuration'), 0700);
+    _provision_create_dir(PROVISION_CONFIG_PATH, t('Provision configuration'), 0755);

Which was something Adrian and I discussed on IRC as a way to allow BIND to see the $config/named contents generated by the provision_dns.module. Everything else is now self-contained in my new modules. Yay!

I'm still working on re-rolling the patch with all the other comments and feedback incorporated, as well as trying to reproduce or track down the error anarcat saw during his testing. More to come soon..

i've done my best to incorporate most of the changes in #9 above, so here's a breakdown of what i've done:

1. replaced ns1/ns2 with a 'ns' field in hosting_dns_server, a text field in the table corresponding to a textarea in the form, which takes one DNS server per line.

2. matched the maxlength for the form fields with the corresponding fields in the hosting_dns_server table.

3. renamed hosting_site_zone to hosting_dns_server_zone, although I realized I'm not actually using this table as yet, and I've kinda lost track of how/where to do so :P

4. replaced the 'type' enum with a regular string, which defaults to 'A'

5. altered ttl field in the RR table to default to NULL, as suggested.

6. added error-checking to all the routines that write/manipulate the zone/named files.

7. renamed provision_bind_create_zone to provision_bind_create_zone_config as suggested.

8. I've refined the provision_bind_update_zone routine with respect to the unlinking comments above. The reason for the unlinking was in case the origin had changed on a new revision of the zone, so I've made this logic explicit: if the zone exists, and there is an 'old_origin' field (set by the caller of this func), then unlink the old zonefile and remove the zone from named.conf before re-creating it.

9. Added useful phpdoc strings to all the functions (at least, I hope I got 'em all!)

10. Finally, and perhaps most importantly, I've tweaked the "zone" parsing logic in _provision_dns_split_url to match that discussed in the IRC snippet anarcat pasted above. The 'host' is now determined to be the first component of the FQDN provided, unless removing that component leaves us with a TLD. I haven't looked at setting up delegation/ownership for the cases where the "zone" is actually a subdomain of a larger zone (but not a TLD), so this will naively produce a 'foo.bar.com' zonefile for the url 'www.foo.bar.com' as it stands.

I have not dealt with the following:

1. default_ip: will need some help and further guidance to relate this to the web_server nodes, but hopefully this will do as-is for now.

2. the extra "default" fields in hosting_dns_server table are left alone for now

3. for the time being, i'm leaving the ns1/ns2 in the provision modules, just for simplicity. clearly this needs to be refined further, but i'm not sure exactly how that should work as yet. in any case, we can handle arbitrary numbers of NS records via the admin UI which lets you create records of any type within a given zone.

That's all for now- some quick tests seem to indicate all of this is working reasonably well, but I'll do a more thorough review in the morning, and post a revised patch for review.

attached is a more recent version of the patch, with most of the feedback above applied (as described in my previous two comments). unfortunately, this is not quite working for me yet, but i'm posting it in hopes that somebody can help me work out the kinks ;)

known issues: the weighting of the two new modules in the system table (provision_bind and provision_dns) need to be manually altered to '4' and '5' respectively, otherwise you end up with dir a dir called PROVISION_CONFIG_PATH/named in your drupal root, instead of $config/named (i'm not totally sure how/where to make this change automatic, but i'm guessing in my install hooks?)

beyond that, this code seems to significantly slow (and sometimes stall) the verify and install tasks when run through drush. not sure why, but it seems to take an inordinately long time for the drush process to restart apache.

finally, the node/add/site form (after the patch is applied) seems to lose any installed 'profile' packages in the system, making it impossible to validate said form and create a new site.

all of this started happening after Vertice's most recent commits (http://drupal.org/project/cvs/196005 see Dec 11) on provision.module related to proc_open. before that, things seemed to mostly be working, except that the zonefile's NS records were slightly malformed.

Has this been removed from cvs for RC1?

Attached is a fresh version of this patch, which breaks the hosting_platform/package functionality that detects modules/themes/profiles for new site creation.

When applied after checking out the hosting and provision modules, but before installing the system with the hostmaster install profile, the subsequently installed system fails to detect the "packages" properly, resulting in no profile/module/theme nodes appearing in the admin/content/node page. For me, this seems to happen even if I don't enable the DNS-related modules at install time.

I've no idea why or how this is happening, so I'm hoping someone can provide some insight into what my code is doing to cause such havoc!

I've committed this latest patch, having tested an install on a pristine server and confirmed that nothing breaks provided the modules aren't turned on. The code still needs a lot of work, but at least we can do that in cvs and collaborate more easily now.

The commits are here:
http://drupal.org/cvs?commit=164899
http://drupal.org/cvs?commit=164901

Ok. I fixed the issue you were having with package breaks.

You were initializing the $data array in the verify task.

As it stands atm, this does nothing because the tables need to be migrated up.

This code looks sound, apart from the issues related to the migration of things to hosting.

The one problem i noticed is lack of documentation. It doesn't instruct you how to add the necessary lines to your named.conf etc.