Leverage a container-based infrastructure

I am currently in the unique position of modernizing a ten-year-old Drupal provisioning tool some of you may have heard of called Aegir. Aegir started 10 years ago powering a Drupal-as-a-Service business, and is still in use today.

It's primary use case is exactly the same as SimplyTest.me: Build a codebase, launch a Drupal site.

Like I said in some tweets, most criticism of Aegir is valid. DId I say "modernizing"? I meant rewriting from scratch.

I'm focusing only on the CLI at this point, creating a new standalone tool called Provision 4.x. Symfony Console based, composer powered, using Robo as a framework... it's been amazing to work with. See /src folder at https://github.com/aegir-project/provision

Provision has always supported Apache & NGINX, but now it works with Docker-compose out of the box. However, being OOP, you can really wire up any service (like kubernetes) using simple classes.

In a few short months I've got it functional. It will launch a working docker stack when you run provision verify SERVER. It will build a Drupal codebase from a makefile or a git repository when you run provision verify PLATFORM. And finally, it will create a database, user with permissions, and a virtualhost configuration when you run provision verify SITE.

Technical merits aside, this project has community and history behind it. I'd estimate at least 10,000 sites already run on Aegir, including more than 500 sites being hosted by NASA.

My goal with Provision 4.x is to make it a Universal DevOps CLI: It should make it easy to run Drupal on any computer, local, testing, production, or otherwise. (It already does localhost well. I was even able to use MacOS' built-in Apache.)

Any community investment in this tool will have huge payoffs across the pipeline.

Provision 4.x is a blank slate. Ready to support whatever the community deems important. I'd love to figure out if there's potential for collaboration here.

Here's some GIFs showing how we're trying to make it as easy to digest as possible: https://twitter.com/jonpugh/status/955472442033569792

I'm really encouraged by @JonPugh's work with Provision. It's looking very promising.

Regarding the accessibility of Kubernetes, local development can be done using minikube (https://github.com/kubernetes/minikube), and there's also a free tier for Kubernetes on GCE, which would be suitable for tinkering during dev work. See https://cloud.google.com/free/

Once you have a tool that is running using Docker, it isn't hard to deploy it on Kube. See the following hack I did a short while back as a POC that runs as a Docker container on Kubernetes:

https://github.com/pantheon-systems/composer-lock-docker

Generally speaking, I think that moving to a Docker / Kube infrastructure is the right way to go for most use-cases where you need to scale out. (Did you notice that the load balancer in the POC comes for free, and the scale-out is a single setting in a .yml file?)

That said, there are still a couple of barriers:

- Familiarity: Lots of people are using Docker, but Kube is still mysterious. While Kube makes it easy to do a lot of things, there's a lot to learn and understand. I do encourage you to study simple examples, like the above, and not get too bogged down in all of the kube documentation.
- Payment methodology: I'm not really sure how simplytestme is being funded right now. If someone is donating a bunch of more traditional servers for free, maybe it's most expedient to keep using that. If there's a source of funding that's currently going towards the existing servers, then maybe Kube could be a drop-in replacement at some point. If this is all coming from Bank of Nerdstein, then you need to be comfortable with how the costs are incurred and scaled.

Collaboration among different tools e.g. provision sounds like a winning proposition to me.

@JonPugh Provision sounds like a really useful project, will keep an eye on progress there!

To provide some background to the project referenced in the IS. It was made as a PoC at the time when the projects future was uncertain to basically see how hard it would be to setup a "quick and dirty" replacement to simplytest.me, so big thanks to @nerdstein for taking over the reins!

Also, a massive thanks to @patrick_d for all his work building and maintaining the service all these years!

It is important to note that these container-based infrastructure solutions were not really an option when the project started and TBH it's only recently that a somewhat "stable" container-based solution has been viable.

With this in mind the challenge was how to build a similar solution leveraging as many upstream sources as possible.
There's lots of potential to optimise it a lot more but decisions were made to reduce any ongoing maintenance.
Unfortunately, there's cost implications with this approach as it assumes the use of a KaaS providers such as GKE (Google), AKS (Microsoft) or EKS (Amazon).

Having discussed with @nerdstein on slack today, it's clearly preferable to continue to leverage the existing infrastructure rather than using a cloud provider with additional costs. It would be great if we could get sponsorship from a cloud provider as it would eliminate the need for any infra maintenance. @nerdstein wonder if it's worth asking, they're all currently pushing this tech and a service like simplytest.me is an excellent example of how it works, especially since the project is open source with an appropriate audience.

The alternative is to build a self managed cluster, my concern with this is the maintenance/support burden it would add to the project.

@greg.1.anderson - working on some docs for helm but trying to make the setup as easy as possible and discovering some edge cases with the new kube/docker integration. Looks like I'll probably stick to minikube which seems to be more stable at this stage.

Still blows my mind what can be achieved these days with 193 lines of code! -- https://github.com/thom8/simplytest

It is important to note that these container-based infrastructure solutions were not really an option when the project started and TBH it's only recently that a somewhat "stable" container-based solution has been viable.

The alternative is to build a self managed cluster, my concern with this is the maintenance/support burden it would add to the project.

IMO it's really all about how the resources are managed/paid for. If we could get cheap/free/inexpensive hosting from a platform that manage k8s for us, then that's one thing—the community operating a k8s cluster on bare metal hardware or cheap dedicated servers is something else entirely.

It's not that k8s an unstable platform at this point—it's just we're in the same sort of situation with k8s (and OpenShift, etc.) right now that I think Docker was in ~4-5 years ago, where (nearly) everyone sees how it's the best technical solution and will be awesome, but not many people want to risk putting their legacy apps (e.g. Drupal) into production with it yet (unless using a managed platform, e.g. GKE, EKS, etc.).

Anyways, since I know I can't devote much to this project, I would encourage @nerdstein to move towards whatever tech stack / setup he's most comfortable with (regardless of popularity or buzzworthiness), since he's the one who has his name in the maintainer's list :)

The nicest thing about simplytest.me is its ephemeral nature—it makes it a lot easier to deal with databases and such when you don't have to deal with containers and persistence!

I think it would be worth considering building/maintaining a single Docker image (or tiered set of images) that can start up a disposable Drupal site given a project name or other argument. This would provide a very simple and clean API that requires nothing except Docker running on the host and could work on both the current bare metal servers as well as container-as-a-service providers (e.g. ACI and hyper.sh, which I think would likely be most cost effective, hardware donation aside) and/or a Kubernetes cluster in the future.

Running multiple services per-container is counter to Docker normal practice, however as far as I can see all of the primary reasons for doing this don't actually apply in our case. The benefits of splitting containers by service are things like (a) process/resource management, (b) scaling, (c) security isolation and (d) state management. (a) is fixed at 1 DB and 1 web server per-site with almost identical management needs and resource usage between sites, (b) is not an issue (no need to scale web servers separately etc), (c) is also not an issue, since it's disposable and (hopefully) sufficiently locked down at the configuration level and (d) is also not an issue (disposable).

In other words I am thinking the an API like `docker run simplytest/drupal8 panels` that would contain all resources needed running under a lightweight process supervisor (e.g. multirun). Having everything in a single image would make the image build slightly more complex, but once built the software/config wouldn't change that fast.

The benefits of a single image are that it becomes more accessible/useful for people to test locally (no need to download a docker-compose file or anything at all) and the simplytest.me scheduling aspect becomes much simpler and more agile - easier to integrate into the current framework as well as refactor into more modern architectures.. Specifically one site = one container means no networking, service discovery or dynamic configuration to deal with.

Running multiple services per-container is counter to Docker normal practice

And yet 99.9% of real world use I've observed do just that :)

It's a tired trope, and unless you're building greenfield systems or have the time/ability/staff to support the logging, monitoring, debugging, etc. of container-per-service, you're going to use something like supervisord or even systemd to manage multiple services in a given container.

In the case of something like simplytest.me, I agree 100% that I'd rather have a monocontainer (like how Drupal VM's official docker image is configured) to manage 1,000 instances of, than running 3x or 4x that number of instances to split webserver + php-fpm + database + drush + whatever other tooling is needed per container.

Tools like k8s make managing the more complex scenario easier, but IMHO the complexity (of the service-per-container approach) wouldn't be worth it for a community/freely-supported project like this. You want devs who want to fix things to be able to jump in and fix a problem quickly and without having to be deeply invested in the Docker ecosystem.

If this were an SaaS product being managed by some company in the community with at least a part time dedicated engineer, my thoughts would be much different, but it's a community project that needs to be as simple, maintainable, and grokable as possible, so I agree with @Owen above.

Another thing we can do (significantly more easily) with a single container is actually install Drupal core during the container image build (heck, we could even build a separate image with every different project pre-installed!). This is one less thing that has to happen on start-up which should reduce both the start-up time and complexity. I have done similar things in the past and am working on a proof-of-concept we could play with.

I have put up a proof of concept at https://github.com/grugnog/simplytest - this image comes with Drupal pre-installed (including the database) so the entrypoint is very simple/fast. You can install additional projects with composer by passing the project name as an argument.
You can also easily pre-build an image for a project, which can be used as a "lazy-loading" system, so project sites can be spun up quickly after first run. Popular projects/versions could also be pre-built in advance (of course theoretically every project/version could be built in advance given sufficient time and disk space).
See the README for examples of all these and to try it out.

The image is reasonably small (650MB, including base image) and I made a start on tweaking the config to lock-down PHP (using the config from the project) and limit memory (uses about 200M RAM for MySQL and 50MB for Apache).

Still would be quite a bit to do to get this production ready:

• Needs a frontend that can take a project name and redirect to a known URL
• Needs some proxies to make the containable accessible on a namespaced public URL. A bunch of ways of doing this: Træfik could do this trivially on a single Docker server or local Kubernetes cluster - a distributed setup would be more work though.
• Needs something to spin-up containers and hold the browser in a wait-state until it is ready. There are bunch of ways of doing this too: I would suggest a simple web application that sits on the default Træfik backend and when a HTTP request is received uses the Docker or Kubernetes APIs to lazy build (if needed) and start a container, serving a static "please wait" page that just refreshes (the user will automatically get the project site once it comes up and registers itself with Træfik).
• There is a bunch of logic in SimplyTest.me to handle different dev versions, patches, installing distributions and themes etc - any of this that should still be supported would also need to be ported (mainly into /entrypoint.sh). Right now it only enables a module/theme if the name is the same as the project
• Would need a Drupal 7 version (if desired). These versions can probably inherit from a shared base image.

Sorry for coming late to the party, and I may be too late to say this, but I thought I'd mention it because it looks like things may have stalled. (I could be wrong. If so, please enlighten me.)

I think we may be going overboard with containers. Would they really add additional value over Aegir 3, the currently stable version? On Simply's front-end dashboard site, we could make use of Aegir Site Subscriptions. This has the added benefit of communicating with Aegir over its Web-services API, which means we can swap out the back-end (Aegir 3) and replace it with a new one (Aegir 5) when it's ready. By the time it's stable, we'll have a clean migration/import path.

Aegir 3 already supports Composer in a nice clean way via Aegir Deploy.

There's a bit more work to do on Aegir 5, but we're close to an alpha. See the architecture for more information. And yes, it'll be able to manage containers too (because Ansible can do anything).

Thoughts?

Great discussion everyone!!
We have switched frameworks, so I am closing this issue as fixed. Thanks!

Woo hoo!