0.4: multiple web server support

I'm having to figure out how to do this at the moment, and am using a web services approach as I don't like the intergration between the application and the provisioning, it's very tightly coupled & interwoven from what I can see.

Splitting it all out to a web service removes the dependency on application and location, it also infers that the local tying together of Hostmaster and Drupal can be separated; as well as the actual need for Hostmaster for some commands at all.

Having a local agent with a WS and a call to consume it, as long as the call is valid and authenticated it dosen't matter what application (and it shouldn't in an architecture) called it as long as it has permissions too.

Also am moving the reporting, status updates, remote admin commands and some content generation to WS so they can be managed in a much larger scale system.

Is there any plan/thinking about how to go about this ?

Maybe I can do some prelim work; as multiple server support is something we have to figure out here in the next few months, as well as the system wide (mass site) reporting tools. Which I've mocked up with JpGraph as I couldn't find a graphing/reporting part of Drupal.

I see that a lot of what Hostmaster does is drush, so are you thinking some kind of ssh connection between the sites (as per http://groups.drupal.org/aegir/roadmap), or a remote agent that handles the incoming requests and drush's locally to that site ? (then you could even use it on the local site).

I've already integrated LDAP for the cross site membership (though would love to help integrate LDAP for the server/site management in Aegir), and have done a lot of work with Astereisk in the past and see them on your todo/extension modle list.

Things I'll have to do anyway (so preferably can help todo for Aegir ):

• Remote provisioning & management for upgrades
• System (all site) reporting

Done so far :

• LDAP - Central resignation of users & organic groups
• SSO between all sites - Using LDAP shared secret
• CMS feeds using secure site from source and FeedAPI for readers

Just need to check out profiles now to make sure I can wrap it all up into one install ..........

@anarcat

Backup between sites

See as the vast majority of sites have remote mysql access disabled and rightly so, that wouldn't be an option though it might be worth checking if the remote user has GRANT permissions. As then the target server could GRANT for the host/IP of the source and MySQL it straight over.

Though if you want to keep everything over SSH then there will be the 2.4 G file size limit, unless ssh/scp is compiled with 64 bit file support.

I like the idea of MySQL -> MySQL, though GRANT is a rare one for most hosts to give out due to security. Though as I understand it Aegir is more for people who have that kind of access to their servers anyway, could make having GRANT a requirement for this kind of action.

I can foresee the situation with myself and the system we're building, that we'll start sites on a shared server and the ones that make it will be migrated off to their own server(s).

With your suggestion of hub-spoke (move via the master) or mesh (slave-slave), I agree on the security vs overhead. With our set-up I think the ones that get moved are likely to be quite large so the overhead argument will win. Even if it's the master ssh'ing to the reviving server, then pulling the backup and restoring it. So the local server doesn't have permissions or access, it's a session from the master tunneling though to the source server via the target.

Seems a bit convoluted, though tunneling and onion security ideas are fairly well laid out and understood.

#338621: Create a hosting daemon for quicker response times

This I like as I already have a LDAP set up that all the server will have secure access to, so they can all use a shared secret approach. Even without LDAP you can have a cron job cycling shared keys between accepted and set up servers that will allow the security to be in place. I've set it up so far to use a system akin to : http://en.wikipedia.org/wiki/Secure_remote_password_protocol to allow a SSO approach at the moment (mainly so users can browser between organic groups they are members of across multiple servers, taking they auth token with them etc).

Statistics tool

I think I'll have to write this as a module for a local site and leave some kind of API open, so something like the hosting daemon (or agent) could query it, get the XML/stats/data, then have the master collate and render the graphs.

Not sure about licensing, though I'm guessing JpGraph can't be used because of licensing : http://www.aditus.nu/jpgraph/jpdownload.php

Though I've seen this : http://drupal.org/project/swfcharts and having something like this :

http://www.maani.us/xml_charts/index.php?menu=Gallery&submenu=Parallel_3...

For all servers would nice I think, graphical & browsable etc.

Though this is down the list on priorities for us at the moment.

LDAP

Personally I'm no fan of OpenID at all after watching a recent presentation on it. It's also not suitable for the project we're building as the membership is closed and SSO is needed just within the system domain. I'm using ApacheDS at the moment, and have use the usual Drupal LDAP modules to integrate the sign-on, and am putting in the og membership listings so they can be pulled from any where, then using a shared secret and token to move between sites.

Outcome

Will look at the ssh work done so far and moving sites about, as that's what we need :)

(Once I get this sorted : http://drupal.org/node/442238#comment-1841666)

Well I have the og & views thing sorted now I think, so I'll be looking at this next week I hope.

Subscribe

75% of the way though the site build now so this is getting rather close [ i.e. I've not forgotten ;) ] !!

Hello all.

I was trying to set up the multiple server thing following http://groups.drupal.org/node/21966 that seems a bit old.

I found out it was a good base but Lacked all the database part that is needed to set up the other server but since I am not sure about the architecture that is aimed at.

For me the goal is to set up a staging environment (nearly there and I'll report the result as promised as soon as I am a bit satisfied with it) where I can move sites from dev, then test, on the same server for me now, and prod that is on a completely separated (jailed in freebsd term) environment with the same platforms (drupal, pressflow, aquia, OpenAtrium ...) as the one used in dev and test but dedicated to public exposure.

I had to create the SAME master database with the same credentials I used on the first server to get the platform to validate on the "production/public exposed" other server.

It validated fine but now I can't migrate the sites between the local and "remote" platform, but maybe it is not quite there already ?

I want to say that I am "strongly" (as in as strong as a loosy contributor I am ) opposed to opening the mysql database to remote access.
I really don't trust my databases to be remote secure, and they are too big targets to ever feel safe with them, and too crucial to risk seeing them tampered with. I strongly think it should be accessed through local sockeds only and secured being a good firewall (pf in my case).
I understand that remote access if useful and feasible in a controlled environment, cluster configuration and the like but I think it should not be a REQUIREMENT for migration.
Besides for me to migrate a server is really an potentially heavy load operation that either happens when you publish a site or when you really want to offload a site to another server, in that case you really want to move the database along.
And like you I like the "Push" architecture, where one server can upload a backup and send command to another one without giving any rights to the target on the "master" one.

As for http://groups.drupal.org/node/21966, I would really love to edit it and at least expands it on the ssh setup since the link it refers too is a complicated mess and borderline "wrong" but I did not want to "steal" Briwood work with maybe inferior work as long as I am sure about the best way to do it.

Maybe I should do another page with my edits and wait for a review ?

I will also do a report on the freeBSD fresh install I did recently with the new procedure to help fellow freebsd guys and help further evolve the install process.

Thanks a lot for your time all.

Here is a patch for this.

Confirmed that we did end up with two databases. Otherwise, the sites look okay. Vertice said he is concentrating on drush this week, so maybe something to handle that will happen.

In general, commands need to know if they should run on one or all servers. For install, you do want all the web stuff to run in parallel. I ran reset password, the UI caught the first link, but the second link was what I wanted with how DNS was working (using hosts to target a site at a time for debugging).

In the future, some admin screen summarizing what servers are running what may be useful for humans seeing platforms have the right servers and how much each server is doing. And/or organize your hostnames.

We should be able to break out the Provision end of this into a separate issue. anarcat or vertice- can you write up a plan for that so we can help out in that issue?

Grouping things around the platform makes a lot of sense to me as Aegir currently has no way (or even a formally roadmapped way) to handle the deployment of code to a server (or servers) for the purpose of constituting a platform. Indeed, the notion of what goes into a platform is pretty out of scope of the current iteration (or two).

However, the simple "multi-server" platform needs to be a bit smarter, in that it should at least not call all the provision tasks every time. Notably, right now two databases are created.

Added a feature request for Provision: #730892: Adding a --no-database or --web-only option

subscribe

This has been completed in HEAD thanks to drumm and myself.

you can even migrate sites between web servers and manage clusters of web servers.