Automatic updates meeting on 7 November 2019

This meeting:
➤ Is for core developers, initiative contributors, the Drupal Association and anyone interested in the initiative.
➤ Usually happens on the first Thursday of the month at 20:00 UTC / 16:00 EST.
➤ Is done over chat.
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to: https://www.drupal.org/project/automatic_updates/issues/3085755
➤*Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.
:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.
heddn:mtech: 11 hours ago
heddn, Lucas and developer funded by EC to work on automatic upates module in Phase I
:wave:
1

dstol 11 hours ago
Hi, Dave here

dts 11 hours ago
Hi, David here. :slightly_smiling_face:
:wave:
1

greg.1.anderson 11 hours ago
@greg.1.anderson lurking
:wave:
1

mlhess 11 hours ago
Michael Hess. Happy it is Thursday
:wave:
1

dstol Yesterday at 10:01 PM
:one: Call for manual testing. Please
6 replies

dstol 11 hours ago
We need your help with testing autoupdates phase 1.

dstol 11 hours ago
https://www.drupal.org/project/automatic_updates/issues/3085793

Drupal.orgDrupal.org
[META] Find more people / agencies to test out autoupdate
Per a comment in #3079622: Automatic updates meeting on 3 October 2019, it would be helpful to get more people, and possibly agencies, to test out the autoupdate functionality. Goals: Have more usage of the module Have folks review the PSA messaging
Oct 4th

dstol 11 hours ago
If you're able to or interested in installing the autoupdates module on your site today, we'd love your feedback

heddn:mtech: 11 hours ago
I got feedback from one person in the issue queue yesterday trying to install the module on a composer site. So I've updated the docs to make it more clear the module is hobbled to non-composer managed sites

heddn:mtech: 11 hours ago
The other feedback was here in slack from m_herchel around wording/UX improvements.

heddn:mtech: 11 hours ago
https://www.drupal.org/project/automatic_updates/issues/3093072
https://www.drupal.org/project/automatic_updates/issues/3093069
Drupal.orgDrupal.org
Add README.md
Problem/Motivation Add a readme file. Use https://www.drupal.org/docs/8/update/automatic-updates as a starting place for wording. Proposed resolution Remaining tasks User interface changes API changes Data model changes Release notes snippet
Yesterday at 8:46 PM

dstol Yesterday at 10:04 PM
:two: Status update of Phase 1 of automatic updates project
20 replies

heddn:mtech: 11 hours ago
alphas are now released that run in place updates on D7 and D8 for Drupal core

heddn:mtech: 11 hours ago
Some of the automation on the drupal.org side that is supporting all this if a WIP.

heddn:mtech: 11 hours ago
Namely the automation that generates the SHA256s of everything in a release to make sure nothing is hacked.

heddn:mtech: 11 hours ago
Manually generated hashes for every version of Drupal core through 8.6.0-8.7.LATEST and 7.60-7.67 are out there right now.

heddn:mtech: 11 hours ago
For realz public/private keys with yubi keys and HSM is also a WIP on drupal.org side. Until then, we have temporary and very publicly shared key pairs.

greg.1.anderson 11 hours ago
Are the hashes still valid for modules installed via Composer, or do they hash bits that only exist in the download tarballs?

dts 11 hours ago
The hashes only cover core at this point.

heddn:mtech: 11 hours ago
we've intentionally hobbled the module at this point to only work for core. less un-intended consequences :slightly_smiling_face: (edited)

greg.1.anderson 11 hours ago
It's possible to install core via Composer. :slightly_smiling_face:

dts 11 hours ago
Also, it's usually the core updates that people around the globe get frustrated with the timing of.

dts 11 hours ago
Updating Composer-based sites is also explicitly out of scope right now.

dts 11 hours ago
I'm not sure if the hashes would match. I suspect they would.

heddn:mtech: 11 hours ago
correct, composer based workflows are explicitly not supported. its unfortunate (edited)

greg.1.anderson 11 hours ago
They should, unless they include info files modified by d.o. infrastructure, etc.

dts 11 hours ago
I mean, it should include those files.

greg.1.anderson 11 hours ago
I mean match, replying to @dts

heddn:mtech: 11 hours ago
there's 2 versions of the hashes

heddn:mtech: 11 hours ago
one is "packaged"

heddn:mtech: 11 hours ago
one is "non-packaged"

greg.1.anderson 11 hours ago
Ah, cool

dstol Yesterday at 10:11 PM
:three: Next steps for automatic updates after EC funding runs out
3 replies

heddn:mtech: 11 hours ago
I think I heard rumors that some funding was in progress. Has anyone heard anything conclusive?

dstol 11 hours ago
@hestenet (he/him) maybe has some details?

heddn:mtech: 11 hours ago
TIm's offline for the next few days still. He'll have to confirm/deny at a later point.

dts Yesterday at 10:11 PM
:four: Bonus Item: External security reviews.
4 replies

dts 11 hours ago
I've been in touch with Yubico, which had some engineers review the design (and some of the code). The conclusion relayed to me was "no red flags," but I don't have details yet.

dts 11 hours ago
I recall other requests being in progress (but being pushed by others here).

heddn:mtech: 11 hours ago
I've lined up some time from @bdragon this week to scan/review things.

heddn:mtech: 11 hours ago
in the last scrum call, we decided that between the time spent yubico and bdragon would probably be enough.

dstol Yesterday at 10:17 PM
:five: Other topics to discuss (I'll create threads for them)

2 replies

heddn:mtech: 11 hours ago
A/B and composer... I'd like to spend some time noodling on this

heddn:mtech: 11 hours ago
since we seem to have time

dstol Yesterday at 10:21 PM
:six: A/B and composer
79 replies

dts 11 hours ago
So, we could do A/B before Composer, and it's probably the right order.

dstol 11 hours ago
Not disagreeing with you, but could you lay out why?

dts 11 hours ago
Doing A/B would enable lower-risk enablement of non-security updates and, possibly, some module updates (without Composer).

dts 11 hours ago
Manipulating a Composer-based build without A/B seems risky, but A/B seems to still yield value without Composer.

heddn:mtech: 11 hours ago
Would this require a new FE controller? (I'm assuming yes)

dts 11 hours ago
This is not a call to hold off on work on Composer, just a call to get A/B done as a foundation. (edited)

dts 11 hours ago
Yes.

heddn:mtech: 11 hours ago
anyone know if such a core issue exists? We can at least start the conversation now

dts 11 hours ago
I'd love to participate in a remote-centered hack-fest around A/B.

dts 11 hours ago
I already have work that I did on it at last year's MWDS.

dstol 11 hours ago
@heddn I don't think so.

dstol 11 hours ago
@dts repo available some place?

heddn:mtech: 11 hours ago
quick search yields nothing

heddn:mtech: 11 hours ago
https://www.drupal.org/project/drupal/issues/3093093
Drupal.orgDrupal.org
Replace single index.php with a Front End controller that supports A/B updating (automatic updates)
Problem/Motivation To support automatic updates, we need a front end controller that can do what is essentially know as A/B switching. Proposed resolution Remaining tasks User interface changes API changes Data model changes Release notes snippet
Yesterday at 10:28 PM

heddn:mtech: 11 hours ago
I chuckled when selecting 'base' for component. I don't think you can get more base then index.php

dts 11 hours ago
https://github.com/php-rot
Repositories
3
@php-rot | May 5th, 2018 | Added by GitHub

dts 11 hours ago
That's where I put the work. (edited)
:+1:
1

heddn:mtech: 11 hours ago
Can you paste an orientation of the repos into that issue por favor?

dts 11 hours ago
Sure. PHP-Rot is separated into a packager and a runtime.

dts 11 hours ago
The main php-rot repo has both of these main components.

dts 11 hours ago
What the packager does is take a "normal" app that uses index.php as the entry point and wraps it.

dts 11 hours ago
It then uses its own index.php as a transparent request router.

dts 11 hours ago
This is the part that's inspired by GRUB and other bootloaders. Most OSes can boot directly, but tools like GRUB provide the ability to select an OS (or options for the OS) in a way that then hands off booting mostly as if the OS directly booted.

dts 11 hours ago
The PHP-Rot model is based on the PHP application mostly being unaware of the A/B switching, though the idea is to provide an API so that an application that's aware of PHP-Rot can chat with it to do things like get status or trigger rollbacks.

dts 11 hours ago
The PoC I did also pulls a manifest of files with hashes and uses those to update the partition that's inactive.

dts 11 hours ago
It can amortize the update (spread the implementation of the changes over many requests), and it only switches after it's done.

dts 11 hours ago
So, you could use it almost directly with our existing PHP-Signify work.

dts 11 hours ago
The Composer support is less baked, as I ran into barriers trying to get the builds to run efficiently.

dstol 11 hours ago
The memory problem?

dts 11 hours ago
Basically.

heddn:mtech: 11 hours ago
composer support is going to be tricky. itlead me to do some really ugly things for the contrib projects. like taking all the composer requirements, and doing require_once on the resulting autoload.php (edited)

dts 11 hours ago
That's partly why I think A/B first might be the right call.

heddn:mtech: 11 hours ago
without that, composer.json/composer.lock changes by just installing the contrib project

dts 11 hours ago
A/B also gives us a playground to handle Composer stuff, so we don't have to worry about breaking the active site.

heddn:mtech: 11 hours ago
but after 8.8... its going to be so easy to manage the site via composer and the usefulness without composer support starts to dimish

dts 11 hours ago
So, part of how I'd see this playing out is:
(1) A/B implementation
(2) A Composer implementation that isn't capable of low memory
(3) Refinements to the Composer support

heddn:mtech: 11 hours ago
agreed, A/B is important.

dts 11 hours ago
Once A/B is in place, we can even do Composer runs that might fail spectacularly.

heddn:mtech: 11 hours ago
I'm not so sure that (2) won't help a lot of folks

heddn:mtech: 11 hours ago
digital ocean has its smallest droplets at 1GB

dts 11 hours ago
Can you rephrase? I want to be sure I'm parsing the negatives correctly.

dts 11 hours ago
Are you saying #2 is or is not useful?

heddn:mtech: 11 hours ago
I'm saying #2 is very useful. that the number of sites that don't have enough memory are dimishing (edited)

dts 11 hours ago
Ah

dts 11 hours ago
I agree.

dts 11 hours ago
But I also think that #2 without A/B might be quite risky.

heddn:mtech: 11 hours ago
++

dts 11 hours ago
It would suck to have Composer runs fail in the middle of running and leave the site in a funk.

heddn:mtech: 11 hours ago
for sites that have "zaporylie/composer-drupal-optimizations": "^1.0" added to their composer.json, I doubt that memory is as much the issue as failed composer conflicts

dts 11 hours ago
Hmm.

heddn:mtech: 11 hours ago
no evidence. just feelings here

greg.1.anderson 11 hours ago
composer update finishes with a composer install. If something fails, it's more likely to be the update operation. Of course, composer install can fail (network down), so I agree A/B is quite important.

greg.1.anderson 11 hours ago
Hopefully it's rare to need to rely in it though.

heddn:mtech: 11 hours ago
today, d.o's packagist went down and caused a bunch of my tests to start falling on their face. it will happen (edited)

dts 11 hours ago
I think the primary challenges around A/B are for the static and uploaded files.

dts 11 hours ago
I guess static isn't too bad if it's already packaged.

dts 11 hours ago
But, we were looking at some pretty funky directory structures depending on whether we could use symlinks.

greg.1.anderson 11 hours ago
Also, at DrupalCon Amsterdam Eric from Violinist was woring on a hack to make Composer update more efficient. He got time and memory down to about 30% of typical.
:banana-dance:
1
:wave:
1

heddn:mtech: 11 hours ago
yes, even a static assets proxy gets to be an issue. i ran into that recently where there weren't enough apache/nginx threads to handle all the static assets for a site that had a PHP proxy for its assets

heddn:mtech: 11 hours ago
php proxy--

heddn:mtech: 11 hours ago
symlinks do work on windows. at least recent versions of the OS

heddn:mtech: 11 hours ago
and linux/mac is not an issue for symlinks

heddn:mtech: 11 hours ago
uploaded files... we're talking public:// ? That should be fixable via symlinks.

dts 11 hours ago
I thought Windows symlinks required admin rights to create or change (or something equally silly).

dts 11 hours ago
Maybe that's just NTFS hardlinks, though.

heddn:mtech: 11 hours ago
it might. but that doesn't mean it can't be done. it just means windows is complicated.

heddn:mtech: 11 hours ago
complicated != not possible

dts 11 hours ago
Well, PHP and cron runs probably don't (and shouldn't) run with admin rights. (edited)

heddn:mtech: 11 hours ago
windows is already in that camp. I don't worry about making it any worse

heddn:mtech: 11 hours ago
me googles symlinks on windows

heddn:mtech: 11 hours ago
https://www.wintellect.com/non-admin-users-can-now-create-symlinks-windo...
WintellectWintellect
Non-Admin Users Can Now Create Symlinks in Windows 10 - Wintellect
With the Windows 10 Creators Update, Microsoft is making it easier for developers to create symlinks, virtual files that link to actual physical files located somewhere else. The change should speed development in Windows 10, making use of symlinks on the platform as seamless as it already is on Linux and OSX. As Microsoft notes in…
Dec 6th, 2016

heddn:mtech: 11 hours ago
https://support.microsoft.com/en-us/help/13853/windows-lifecycle-fact-sh...
support.microsoft.com
Windows lifecycle fact sheet - Windows Help
Learn about End of Sales, End of Support, Service Packs, Windows Downgrade Rights, and other important Windows lifecycle dates.

heddn:mtech: 11 hours ago
I read this that we don't need to support < windows 10. :shrug: (edited)

dts 11 hours ago
Presumably, the capabilities of the server releases may matter.

dts 11 hours ago
Not sure.

dts 11 hours ago
Some of the active Windows Server releases were pre-Windows 10.

dts 11 hours ago
Anyway, I need to head off.

heddn:mtech: 11 hours ago
seems like there's a GPO permission that can be granted for symlinks on windows server 10 w/o need for full admin access

dts 10 hours ago
Windows 10 doesn't strictly have a server version. There is Windows Server 2019 (and 2012, and more).

dstol 10:44 PM
:checkered_flag: Without any other topics, the synchronous portion of the meeting over. If you are still chatting through topics above, no need to stop! If you're new and have ideas on any of the topics we discussed in the threads above please add your thoughts! Thanks everyone for attending!
dstol 10:54 PM