Automatic Updates meeting on 5 September 2019

This meeting:
➤ Is for core developers, initiative contributors, the Drupal Association and anyone interested in the initiative.
➤ Usually happens on the first Thursday of the month at 20:00 UTC / 16:00 EST.
➤ Is done over chat.
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to: https://www.drupal.org/project/automatic_updates/issues/3072104
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us. (edited)

7 replies
Last reply 11 hours agoView thread
dts 10:02 PM
waves.
I haven't had a chance to follow up on any issues since my trip that ended earlier this week, but I'm happy to chat now about anything. (edited)
dstol 10:03 PM
:one: @dts gives a status update
dts 11 hours ago
I'm not sure I have much to share unless folks want an overview of the MWDS work.

dstol 11 hours ago
That'd be helpful, I wasn't there so I feel a bit out of the loop

dstol 11 hours ago
I'd also be curious how your signing discovery is going too

dts 11 hours ago
Sure. At MWDS, we confronted the problem of how we can validate data Drupal receives over the network without relying on resources that will be inconsistent on many hosts.

dts 11 hours ago
For example, we don't want to rely on a recent OpenSSL version, an up-to-date CA certificate list, a GPG CLI tool in the path, or deprecated tools like mcrypt.

dts 11 hours ago
So, I fielded the question to various folks who work on similar problems. The author of the Signed Exchanges IETF proposal (which I was considering) suggested instead that we look at OpenBSD's Signify work.

dts 11 hours ago
Signify relies only on Ed25519, which is built into modern PHP versions via Sodium and is available on every PHP version supported by Drupal 7 or 8 via a polyfill.

dts 11 hours ago
Leaning on Signify's design had other benefits. For example, we get interoperable CLI tools with moderately broad availability. This unblocked work to create test fixtures. More importantly, it helped us validate that our crypto is probably properly implemented (given that it works bi-directionally with OpenBSD's tools). (edited)

dts 11 hours ago
There were two gaps for us to plug in Signify's capabilities, though.

dts 11 hours ago
First, we wanted to be able to validate the data in pure PHP (no CLI/exec calls). We accomplished this.

dts 11 hours ago
Second, we wanted to support a signing hierarchy so that we could protect the root secret by keeping it offline in an HSM. We extended the Signify format to support chained signing, including implementations in PHP and shell script (as well as format documentation). We also found documentation online of someone using a YubiHSM (which I have acquired for dev purposes) to work with Signify. (edited)

dts 11 hours ago
We also talked quite a bit with Narayan and Neil to ensure the design is viable for Drupal.org infrastructure.

dstol 11 hours ago
Chained signing is work in progress still?

dts 11 hours ago
It's fully implemented in php-signify.
:heavy_check_mark:
1

dts 11 hours ago
There's work to be done for testing with YubiHSM devices, but that doesn't affect the implementation of anything for Drupal.org infrastructure or Drupal itself.

dstol 11 hours ago
That's https://github.com/drupal/php-signify for the record (edited)

dts 11 hours ago
Yes. :slightly_smiling_face:

dts 11 hours ago
So, what this gets us is confidence in the integrity of data we receive over the network. This becomes far more important as we start obtaining code over the network to update the current, running code.

dts 11 hours ago
Because of the design, the approach to that can be mostly consistent all the way from Drupal 7 on PHP 5.6 all the way to the latest Drupal 8 on PHP 7.3.

dts 11 hours ago
And the crypto design is super modern, so almost no concessions are made to get that broad compatibility. It's a crypto design that should be good for 10-20 years at least.

dts 11 hours ago
(The Signify design also accounts for updating the crypto in the future, as well.)

dstol 11 hours ago
What's next here? Infra team work?

dts 11 hours ago
The two biggest pieces for the infra side are the offline root (which we can postpone doing properly for now) and the tools for signing builds.

dts 11 hours ago
One of the signing challenges is that we want intermediate keys to have an expiration, so that means we need to be able to re-sign builds periodically, even if nothing has changed.

dts 11 hours ago
If we, say, rotate intermediate keys quarterly, then we might want to re-sign builds older than 60 days.

dstol 11 hours ago
I assume the infra team is on top of that, but do you have any links to infra issues related to that work?

hestenet (he/him) 11 hours ago
I can grab a few links..

hestenet (he/him) 11 hours ago
https://www.drupal.org/project/infrastructure/issues/3076191
Drupal.orgDrupal.org
Build a signing oracle server & HSM infrastructure
With upcoming automatic updates, there will be higher stakes for packages and other artifacts hosted on Drupal.org. Just using HTTPS everywhere doesn’t protect against all attack scenarios. Any manual verification that site builders might have been doing is skipped with automatic updates, so more assurance that code is relatively trustworthy is needed. Signed artifacts will be delivered as “chained signatures,” described at https://github.com/drupalassociation/php-signify#chaining-with-csig, along with the BSD Signify-based PHP implementation for verification. In practice:
Aug 20th

hestenet (he/him) 11 hours ago
https://www.drupal.org/project/drupalorg/issues/3053199
Drupal.orgDrupal.org
Add SHA512SUMS file of all files
Problem/Motivation For the readiness checks and even during applying of updates, we want to make sure we aren't modifying a file that is already changed from its default. Call it patched, hacked, whatever. To do that, we should run a hash check against the files and compare it against... something. It should look like: http://mirrors.us.kernel.org/ubuntu-releases/19.04/SHA256SUMS Proposed resolution
May 7th

hestenet (he/him) 11 hours ago
https://www.drupal.org/project/drupalorg/issues/3061250
Drupal.orgDrupal.org
Sign packages and hash files for automatic updates
Problem/Motivation For #3053199: Add SHA256SUMS file of all files and #3061248: Package quasi-patch for automatic updates we need to sign the artifacts. Proposed resolution Use chained signatures as described at https://github.com/drupalassociation/php-signify#chaining-with-csig, which can be verified with that library. An example signed hash list is at https://github.com/drupalassociation/php-signify/blob/master/tests/fixtu... Remaining tasks
Jun 12th

hestenet (he/him) 11 hours ago
https://www.drupal.org/project/drupal/issues/2367319
Drupal.orgDrupal.org
Implement automatic background updates for highly critical security issues
Problem/Motivation Many software programs (including CMS software such as WordPress) support automatic updates, in which the site applies an update on its own with no intervention from the site administrator. We need to continue exploring this for Drupal. Here are some pros and cons identified in this and related feature requests: Pros
Oct 31st, 2014

31 replies
Last reply 11 hours agoView thread
dstol 10:11 PM
:two: Discuss how Automated Updates could/should/will work with Composer
https://www.drupal.org/project/ideas/issues/2940731#comment-13245802

grasmash 11 hours ago
So, first question. Will automated updates, as it is planned today, work on a site that uses Composer to manage dependencies?

greg.1.anderson 11 hours ago
If I may skip over the first question, an important point for the Composer in Core initiative is, if a site is NOT YET using Composer, and applies automated updates, then we want to make sure that the result after the update remains "composer ready".

greg.1.anderson 11 hours ago
As long as the update "quasi-patch" files omit no changed files, then the result of the update should be equivalent to the new Drupal version as encapsulated in the download tarball.

greg.1.anderson 11 hours ago
(We are presuming that the quasi-patches are going to be generated from the tarballs)

greg.1.anderson 11 hours ago
Once auto updates is using Composer, the only real option is to run composer update (or some variant thereof), and that should always be Composer compatible

grasmash 11 hours ago
I do understand the value of keeping things Composer ready.

grasmash 11 hours ago
but it only underscores the point that we value Composer

grasmash 11 hours ago
and yet we will stop supporting you via automatic updates the second you start using it (edited)

grasmash 11 hours ago
that feels very strange.

greg.1.anderson 11 hours ago
Well, I think what we have here is a staged implementation plan. Ideally Composer is supported from the first release, but if it is not, well, I don't think it would be right to stop the autoupdates release to wait for Composer.
:heavy_plus_sign:
1

hestenet (he/him) 11 hours ago
Not sure if @heddn is available to chime in today - but I’d like to get his two cents on these questions specifically.

greg.1.anderson 11 hours ago
Given infinite time and funding, sure, let's add more resources to Composer + autoupdates.

dts 11 hours ago
We're pretty obligated to ship basic auto-updates for something this year. It will likely not support Composer for those.

dts 11 hours ago
However, it also will not make the situation worse. My understanding is that we plan to ship changes in a way that you end up converging on what you'd get by extracting a fresh tarball to update.
:+1:
1

drumm 11 hours ago
Since the Composer initiative wasn’t complete when this round of work started, the first focus has been tarball-like installs. With Composer use getting standardized now, attention can go into automating composer-installed updates.

dts 11 hours ago
Just as importantly, it will help us exercise mechanisms for pre-flight update checks and signing infrastructure, both of which will be useful for any Composer auto-update support. (edited)

greg.1.anderson 11 hours ago
Yeah, the signing infrastructure will be very useful

drumm 11 hours ago
This first round has also been targeting the masses, people who don’t necessarily have much of a deployment pipeline, and there’s a good chance they’ve avoided Composer.

heddn:mtech: 11 hours ago
hey, weighing in here

heddn:mtech: 11 hours ago
much of the work to date is not contrary to use w/ composer

heddn:mtech: 11 hours ago
meaning, in a lot of cases it will work

heddn:mtech: 11 hours ago
there’s some caveats. big ones.

heddn:mtech: 11 hours ago
if after using composer on your site and you run composer require or anything that changes things so they aren’t any longer what d.o tells us they should be for a specific version of core or contrib module, auto update won’t blow those updates away.

heddn:mtech: 11 hours ago
let’s have a scenario or 2 to demonstrate…

heddn:mtech: 11 hours ago
scenario one: tarballs and drush or gzip’ed downloads of modules. 100% fully supported. no issues.

heddn:mtech: 11 hours ago
scenario two: site converted to composer ready project. Yeah!!! no additional modifications to anything besides what comes out of the gate from the conversion to composer. this should work in most cases. except when vendor or other files have changed in a security update (think Phar vulnerability) and those files where changed already. they don’t match the hash from d.o and we’d fail to update you. because we’d be potentially causing more harm than good by overwriting some hacked or patches version of those files.

heddn:mtech: 11 hours ago
scenario three: commerce site with lots of changes and additions via composer. this is basically scenario two but being more concrete. this should also mostly just work, unless you’ve updated to a later version of symfony or twig or something than what d.o. says you should have. if updated something from vendor, then we wouldn’t want to do harm and replace it with something potentially older or contrary to what you already have installed.

greg.1.anderson 11 hours ago
n.b. For scenario 2, drush pm-update to Drupal 8.8.0 should be sufficient to convert a non-Composer site to a Composer-ready site.

dts 11 hours ago
The preflight checks should prevent us doing damage when we shouldn't mess with the installation, too.

heddn:mtech: 11 hours ago
for scenarios 2/3… we can’t read minds. so we assume the worst and fail the update and fail loudly with lots of notifications.

hestenet (he/him) 11 hours ago
Thanks, @heddn those scenarios are helpful illustrations.

greg.1.anderson 11 hours ago
Scenario 2 should not be different than scenario 1 (presuming the site has not yet used composer require or composer update

heddn:mtech: 11 hours ago
and the whole time we’ll be learning. learning right and left what folks are doing and what folks are seeing with updates
:heavy_plus_sign:
1

dts 11 hours ago
A+ to that! We're about to learn a whole lot about the installations that violate our assumptions.
:heavy_plus_sign:
1
:plus_one:
1

dts 11 hours ago
I imagine eventually looking at some way to submit failed preflight reports for us to collect more data on weirdness in the field.

greg.1.anderson 11 hours ago
@heddn Is there something in scenario 2 you were expecting to have happened composer-related? Just want to make sure I'm understanding your adjectives. To me, "composer ready" means site is ready to use with composer require and composer update, but those commands have not been used yet

heddn:mtech: 11 hours ago
@greg.1.anderson is there a composer.lock or not in that scenario?

greg.1.anderson 11 hours ago
Yes, there is a composer.lock, but it contains whatever was bundled from drupal/drupal.

heddn:mtech: 11 hours ago
got it. then it isn’t any different than scenario 1. true enough.

hestenet (he/him) 11 hours ago
I think - if that answers @grasmash’s question about what would be covered in the scope of the current initiative, the next logical question would be:
After this year’s EC-funded work concludes- is there a logical architectural direction to deepen support for situations where there is more heavy composer use/lockfile changes/etc ?
i.e: How much more insight into composer dependencies could be built in, to address more edge cases for sites that have made changes via composer with lockfile/vendor/etc?
And does that logically expand on this work - or is it more of a separate code path?

greg.1.anderson 11 hours ago
Logically, the answer is simple: composer update fixes all. In practice, though, there are complications.

greg.1.anderson 11 hours ago
The main complication is that composer update can take gigabytes of RAM to complete successfully.

dts 11 hours ago
I've certainly proposed the A/B partitioning option so we can gain flexibility to manipulate the code without having to worry about leaving it in an inconsistent state, but that's sort of on-hold pending the 2019 auto-updates work.
:star:
2

greg.1.anderson 11 hours ago
I would guess that the first step would involve building a "you can use it if you can use it" version for Composer, and build from there.

mbaynton 11 hours ago
I was quite pleased with the roadmap we put together back in Seattle, which as I recall incorporates A/B and a gradual introduction of Composer.

dts 11 hours ago
A/B partitioning has value without Composer, as well, but it's really hard to support Composer properly if you have to worry about every file change possibly receiving a web request.

dts 11 hours ago
So, I see it as a nice foundation for Composer work.

greg.1.anderson 11 hours ago
I made a prototype some time back that makes a client script called "composer" that calls through to the real Composer client for most commands, but forwards require and update to a remote server for processing. The remote server generates a composer.lock file and sends it back; the client side runs composer install to apply it. This prototype works fine, but the big issue with a service-based solution is that it's really hard to have a big enough service to keep up with events like Drupal security releases.

greg.1.anderson 11 hours ago
I also have an idea for a prototype that I have not coded on how we might be able to update sites using much less RAM:
https://github.com/g1a/reposer
g1a/reposer
Experimental composer plugin to provide a high-speed / low-memory lock update
Stars
8
Language
PHP
g1a/reposer | May 11th | Added by GitHub

dts 11 hours ago
I have interest in finding a way to have lean Composer builds without remote dependency processing, but that's also a hard problem.

dstol 11 hours ago
and I assume it doesn't support private composer repos?

greg.1.anderson 11 hours ago
What is "it"? The remote service prototype?

greg.1.anderson 11 hours ago
The prototype doesn't.

dstol 11 hours ago
yeah,, sorry

greg.1.anderson 11 hours ago
You'd have to have some way to pass credentials to do that.

greg.1.anderson 11 hours ago
Anyway, the "parent project" idea in reposer is something that would work very well for Drupal sites.

hestenet (he/him) 11 hours ago
On the subject of the A/B concept - I’ve been trying to dig up my notes on that recently - but if anyone has notes or links to share - please drop ’em my way

greg.1.anderson 11 hours ago
We might even be able to implement that (parent sites) with the existing composer update solver.

dts 11 hours ago
For A/B, I tend to point to how CoreOS handle(d) partitions: https://coreos.com/os/docs/latest/sdk-disk-partitions.html
coreos.comcoreos.com
CoreOS
CoreOS provides Container Linux, Tectonic for Kubernetes and the Quay image registry; key components to secure, simplify and automatically update your container infrastructure.(114 kB)
http://coreos.com/assets/images/brand/CoreOS-twitter-embed.jpg
:high5:
1

dts 10 hours ago
Essentially, it's taking the approach they use for /usr and applying it to the Compoer/PHP code.

dstol 10 hours ago
Could we perhaps move a/b discussion to another thread?

hestenet (he/him) 10 hours ago
:thumbsup:

greg.1.anderson 10 hours ago
Anyway, to get back to the question about heavy lockfile updates. If we added the concept of a "parent project" to Composer, then drupal/drupal (or drupal/core) could be the "parent" of the Drupal site. When a new version of Composer comes out, you download the lock file from the parent project. Those dependencies are fixed. You then resolve the dependencies for everything that is not part of the parent.

greg.1.anderson 10 hours ago
That should be able to happen pretty easily using the existing solver algorithm.

greg.1.anderson 10 hours ago
There are some discussions in the Composer 2 issues about doing "partial updates", but it's unclear what the definition of "partial" is here; perhaps it's whatever is mentioned on the commandline.

greg.1.anderson 10 hours ago
In any event, I think we could potentially extend that idea to provide fast and efficient updates for Drupal sites on small hosts.

+3
67 replies
Last reply 10 hours agoView thread
dstol 10:35 PM
:three: other topics we should be discussing?
[Note, no other topics were raised in this thread]
dstol 10:56 PM
:four: A/B discussion
Example of this approach as applied to /usr for a Linux distribution: https://coreos.com/os/docs/latest/sdk-disk-partitions.html

heddn:mtech: 10 hours ago
A/B will require some changes to the front controller and possibly to many other things in core. its also the only way that auto update will make it into drupal core directly. <= based on comments from jess and other core maintainers

mbaynton 10 hours ago
@hestenet (he/him) in terms of links for this, I did some rough mockups before our Seattle presentation, https://github.com/mbaynton/drupal-project-ab. So there's that.
mbaynton/drupal-project-ab
Project template for Drupal 8 projects using an active/inactive layout with composer
Language
PHP
Last updated
5 months ago
mbaynton/drupal-project-ab | Apr 7th | Added by GitHub

hestenet (he/him) 10 hours ago
Thank you!

dts 10 hours ago
I also produced a PoC over a year ago that updates the inactive partition and then switches once it's ready to activate.

hestenet (he/him) 10 hours ago
Which reminds me that this is also a good resource to put in my list of links: https://events.drupal.org/seattle2019/sessions/drupal-core-auto-update-a... (edited)

heddn:mtech: 10 hours ago
symlinks are a *nix thing. won’t run on windows. we’d probably want to make sure we had a dedicated front controller that could make some of these decisions

mbaynton 10 hours ago
yes. The symlinks were a kludge to allow the experimenting and learning to happen. Main problem it pushes under rug is static assets.

greg.1.anderson 10 hours ago
I thought there was some symlink-like (symlink-lite?) thing you could use on Windows. Composer has true, false and default values for "symlink", the last of which uses the Windows native thing on Windows. Or so I recall.

heddn:mtech: 10 hours ago
i could be speaking from old history: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/
Windows Developer BlogWindows Developer Blog
Symlinks in Windows 10! - Windows Developer Blog
Overview Symlinks, or symbolic links, are “virtual” files or folders which reference a physical file or folder located elsewhere, and are an important feature built in to many operating systems, including Linux and Windows. The Windows’ NTFS file system has supported symlinks since Windows Vista. However, it hasn’t been easy for Windows developers to create …
Dec 2nd, 2016
(86 kB)
https://46c4ts1tskv22sdav81j9c69-wpengine.netdna-ssl.com/wp-content/uplo...

mbaynton 10 hours ago
My understanding was it was an elevated permission thing to create them in Windows for some reason. But it's probably not crucial to nail down now, because either they turn out not to be an issue and we're done, or we face the brunt of static asset management through moving files around or serving through php as a next step.

mbaynton 10 hours ago
Having to code for the static assets thing will be a better place to be than where we are now, and one we can be confident of success in if we know it's the only missing piece.

heddn:mtech: 10 hours ago
serving static assets via PHP is an issue. we’re facing that in a hosting provider we are using right now where they manage the php-fpm workers. and there simply aren’t enough workers to handle static assets via php

heddn:mtech: 10 hours ago
we’re to the point where we’re setting up a second “application” that is a clone of the main drupal site to serve up those assets. just to get around running out of workers.

mbaynton 10 hours ago
caching the stuff in front of the backend server an option?

dts 10 hours ago
I don't think we can depend on that.

mbaynton 10 hours ago
Generally, I do respect that for the audience we're targeting, we can't depend on that.

heddn:mtech: 10 hours ago
we digress, but no it isn’t an option because they’d have to give up their dns to cloudflare and for various reasons aren’t able to do that.

mbaynton 10 hours ago
and also that's good feedback to have, I wouldn't have expected that result.

dts 10 hours ago
It may be okay to do the image styles-type approach and lazily copy over assets (and serve them from PHP the first time).

dts 10 hours ago
I have to head to another meeting.

hestenet (he/him) 10 hours ago
I suppose the other implicit assumption is that we’re largely talking about attended automated updates with the A/B model.

mbaynton 10 hours ago
Although the moving files around answer is also an answer, as far as I know.

heddn:mtech: 10 hours ago
i’m not making that assumption

heddn:mtech: 10 hours ago
attending or unattended, there could be a health check in place

dts 10 hours ago
I am also not making that assumption. I think a big benefit to A/B is low-risk unattended updates.

hestenet (he/him) 10 hours ago
Oh I suppose that’s true. So in some scenarios the health check of B could be considered sufficient tos wap over unattended.

heddn:mtech: 10 hours ago
and a settings toggle to yes/no i want attended

hestenet (he/him) 10 hours ago
Gotcha gotcha

29 replies
Last reply 10 hours agoView thread
dstol 11:11 PM
:checkered_flag: Without any other topics, we can call the synchronous portion of the meeting over. If you are still chatting through topics above, no need to stop! If you're new and have ideas on any of the topics we discussed in the threads above please add your thoughts! Thanks everyone for attending!