Allow updating core with the update manager

Subscribing...

Being able to upgrade contrib modules is the biggest pain and we've take a huge leap forward already. Realistically I can live with pushing core upgrades to D8 if we need to since core doesn't have releases all that often, and drush will probably figure out how to do it in the meantime. :)

Sounds pretty critical, that by default plugin manager can't upgrade its biggest use case?

What precisely makes it complicated? I'm scanning the other issues in vain for a clue :(

I disagree about B). I think assuming that the webroot is not the only dir the client has is okay. Perhaps for people with only one directory - has anyone ever seen this? backups would fail.

C). I don't think we need to worry about core hackers, just let them know that they will lose any changes they made

D). As the jack-ass who brought this up and proposed / researched the patch solution, I like Derek's idea more: Don't worry about it. At least for now, it's unlikely to be a problem. If need be, we can update our core updater probably with a minor update if we realize it will cause a problem later. Not ideal, but possible.

E). Is actually a larger problem than specified because we want to run updates in a separate batch process, not as part of the file moving page load (generally). So we would have to bake it in separately into the process.

I actually think the only ways to do E) with absolute certainty is to redirect to update.php immediately after the upgrade, to make sure it is run immediately OR run the updates in the same page request as the core update (which is probably cool, but if an index changes on the node table or something, it might cause an execution timeout or something).

-J

Run for your lives!

Something to add here as a special consideration - preserving "special" files. I suspect many people modify the .htaccess file and some also robots.txt, so when the site is upgraded, such files will need to be somehow protected - one method could be to simply have an upgrade version of drupal that does not include such files...

Okay, getting very close to a prototype here.

One problem we've got to solve though:

In update.manager.php around :425

foreach ($projects as $project => $url) {
      $project_location = $directory . '/' . $project;

This is kinda hacky to begin with, but when you DL drupal core, it unpacks into drupal-7.x-dev or whatever so it doesn't match up, unlike modules and themes. I'm thinking we should probably refactor this so that we are just grabbing the first directory out of our tarball and calling that the project directory. We then have to make sure we can track this throughout the project so we don't have to guess it based on the name of the thing we are updating later.

Hey Derek,

Sorry I'm getting this so late. I've got a million kids running around the house, work is just not going to happen.

Here is a patch I sent to someone who was having issues with copy-contents vs copy directory. The upgrade core patch also addresses this, so I thought it might solve his problem. I haven't messed with this at all in over 2 weeks, so it might not even apply, but much of the useful bits will.

Last time I ran it, it worked, although warning: you may blitz your install.

Best,
Jacob

Sub.

me too

Strange idea that might make "upgrade core" easier...

There is always an upgrade patch that fixes security holes, but does not include "additional bug fixes" -- what if we supported that kind of an upgrade using some sort of code that is capable of applying a patch to core files?

From the drupal.org announcement to upgrade: "To fix the security problem, you can either (1) upgrade Drupal or (2) patch Drupal. We strongly recommend you do a full upgrade (which is also detailed in the security announcement) as the patches do not contain the additional bug fixes that went into the releases. Applying the patches will leave your site in an unversioned state and confuse the update status module, which will keep reminding you to upgrade to 6.16 or 5.22. Please read the announcement for details on the patch."

Just a thought?

Also, #22336: Move all core Drupal files under a /core folder to improve usability and upgrades seems like it will be D8 and would make this much easier (as dww originally pointed out).

Josh

After talking to webchick moving this to Drupal 8, simply to much work left to be done

Downgrading all D8 criticals to major per http://drupal.org/node/45111

1. I don't think this needs to wait until http://drupal.org/node/22336 is fixed. It's just an excuse, and this could be fixed w/out that.
2. Subscribe
3. I think simply extracting Archive.tar.gz/drupal-7.0 -> DRUPAL_ROOT would be perfectly fine.
4. I think we should make sure all files are is_writable before extracting the archive.
5. I'm unassinging this, as I don't think Jacob Singh is going to do any work on this issue.

But that's just my opinion.

Subscribing.

Bump...

Uploading a partial patch for this.

This one is more improved.

Subscribe! Drupal 8 needs this!

+ $archive_errors = array(); //update_manager_archive_verify($project, $local_cache, $extract_directory);

Yes, that was kind of a hack (update_manager_archive_verify() was saying that it didn't have a .info file in it).

I'll try to work on this more today.

Sub

A) authorize.php runs in a reduced bootstrap, so broken contrib can't harm it, but it's still bootstrapping and running parts of core: (Form API, Batch API, system.module, user.module, maintenance theme system, language system, plus the FileTransfer and Updater classes). Updating those things via authorize.php while it's running is slightly scary and hard to get right.

We could do this:

1. Copy all system modules to a tmp backup dir. Verify integrity of ea. copied module immediately after it is copied.
2. Exit and go to the next iteration of the batch.
3. Copy all includes to a tmp backup dir. Verify integrity of ea. copied include immediately after it is copied.
4. Exit and go to the next iteration of the batch.
5. Copy the new authorize.php, then copy modules and includes from archive location into place. Copy the ones that authorize.php uses first. Verify integrity of ea. copied file/module after it is copied. If we run out of time, authorize.php will be able to run on the next iteration of the batch.
6. Copy the rest of the files.

The idea is this: if something fails, authorize.php will still be able to run and revert changes/ finish the job/ etc. Since we're upgrading core, if anything goes wrong, we should abort immediately. We may also calculate beforehand (maybe in step 3) if the server is fast enough to copy the files required by authorize.php before the script times out.

In fact, we could use get_included_files();, then copy those files from the archive first.

Here is a little further.... I still can't seem to get drupal to pick up CoreUpdater.

+++ b/modules/system/system.updater.incundefined
@@ -144,3 +144,55 @@ class ThemeUpdater extends Updater implements DrupalUpdaterInterface {
+    public function getInstallDirectory() {
+
+        return DRUPAL_ROOT;

you can leave out these blank lines, they are not needed.

There are a couple of them in the code

Powered by Dreditor.

Would still love to have this. A huge win for security.

However, this remains a feature request, not a task.

This feature is vital for devs who ever want to be able to hand off a site built with Drupal, train the client to maintain it, and be done with it. I build stuff with Drupal for community organizations and the like who don't have the budget to have a full-time IT person, and you can usually find and train a generally computer-literate person to do module upgrades or other basic in-interface tasks, but that leaves core un-updated and eventually vulnerable. And so I eventually have to go back and help basically everyone I've ever built a site for, which is just not sustainable in the long run. This really, really needs to get fixed in 8.

Agree entirely with jddunn in #38. One of the main reasons to use a CMS is to allow this hand off.

No one disputes its importance, but unfortunately people who need it arent stepping up to work on it. :( if this is an itch you possess, then best thing to do is dig in and see how you can help get the ball rolling with a patch, or some funding for someone else to take it on.

Is anyone working on this feature? - Upgrading core with the update manager automatically without Drush or manual upgrades?
I'm sure there are many people out there (including me) that will be happy to fund this.
How much will it cost? and who wants to take this job?.
I'm sure that all of us together can make it happen.
I guess not in time for Drupal 8 release but it could be added later.

Thanks for your reply.
Maybe others here want to help me fund this feature?

If we can get it into 8.x (even through a contrib module), I'd fund some of this.

Will need to be 8.1 at least I think.

Allow me to revive this by boldly paraphrasing Dries, from drupalcon Barcelona:

To improve the User Experience (UX), Drupal needs to:

• Focus on the non-coders: builders, authors, and editors.
• Improve the UX, making it easier to use and more seamless.
• Focus on the 80% of the world that does not use a content management system, and convince them that Drupal is right for them.

D8 needs this!

Yes, it does and we can probably create a feature branch to start work on this. There are a few core updater modules out there already.

But it certainly won't get done and approved in 7 days, so it will have to wait until something later than 8.0.

There are a few core updater modules out there already.

Aside from Aegir and Drush which are rather like projects, I am not aware of any other modules.

If you know of any others please list them here.

CMS Updater (https://www.drupal.org/project/cms_updater)

And maybe: Drop Guard (https://www.drupal.org/project/dropguard)

Yes, these are all commercial gateways that only update your site if you paid for the license. That being said, I'm pretty sure that the update code should be somewhere in the d.o GPL module.

Hi,

Can someone, please, enlighten me over my doubt: This feature would mean that Drupal would be able to also Upgrade itself via WebUI (e.g.: D8->D9) pretty much like Joomla (something like Notification to Admin About New Version>Press Button "Upgrade">Upgrade in Process>"Upgrade done"), right?

(If so, this would be Security's Paradise for me while also being able to have the latest features - of course this upgrade option would only be available to Admin)

This feature would mean that Drupal would be able to also Upgrade itself via WebUI (e.g.: D8->D9)

This would likely be for minor versions D8.1->D8.2

:( That's sad.

If Drupal is able to upgrade between minor versions why not between major versions?!

And that old story that the site Admin's has to know what he is doing, test the site before upgrade and that, if he's the Admin, then he'll find it easy to manually upgrade it, does not invalidate this. Really, it doesn't stick!!!

There is absolutely no point at having to do this manually except that, if this is done manually, it will scare newbies away (it's more of a tool to separate us from newbies).

I think that Drupal does not need to scare anyone away... But maybe that's just me...

I'm sorry, i do not intent to criticize anyone (specially not you MediaFormat). i'm just tired of this mambo-jambo talk about the "it's better to do in manually" old talk...

@MavMan first, Drupal is not currently able to update between major versions. There's some migration support, but it's clearly something that shouldn't be done without extensive testing. To do it from inside the site would in 99.99999% of all cases result in a broken site. Simply because D9 will break compatibility with D8 (that's why it's called D9 and not simply 8.x+1), and any modules using old APIs will stop working. You'd need a site running with 100% Drupal core and 0% custom or contrib code or themes.

Upgrades between 8.0.x, should be possible (as these will be bug fixes and security releases), and maybe an opt-in to enable updates between 8.x versions.

As to security paradise.. Please realize that to have this possible, your server will need to be able to re-write your PHP files. If you lock down Drupal, that shouldn't be possible, and no attacker should then be able to inject whatever code they please into your site. This is clearly not 'security paradise'. I'd call it 'security limbo'.. You'd be able to upgrade your site by pressing a button (or even automatically), but your site would be always vulnerable to code injection.

Hi jcnventura,

Then how are WP and Joomla guys doing it...?

How can they do this and we can't???

They are embracing "Security limbo" and making people bask in the warm embrace of the god of cluelessness.

But I honestly agree that 8.0.x->8.0.x+1 and 8.x to 8.x+1 should eventually be done. Drupal 8 is not capable of doing this in core. No doubt there will be contrib modules implementing such services, as there are for Drupal 7 already.

It would probably be one of the first settings to be disabled by larger agencies, but for small clients running on hosts that don't allow them to harden their sites, this would be better than nothing.

I started a GoFundMe campaign for this issue.

There's a catch 22 for this issue. The people who are able to code this are technical enough not to need this feature. Those who would benefit greatly from this feature are not technical enough to code it. I'm hoping funding will resolve it. I put $20 in. Anyone else want to contribute?

Feedback from developers is also welcome.

Great initiative, thank you philbar!

@philbar

To clarify: The feature being funded is destined to allow updates (e.g.: 8.1->8.2) or Upgrades (E.g.: 8.x->9) ?

Tks

@MavMan, the funding will be tied directly to this issue. It depends on the core developers.

Keep in mind, Drupal 8 starts a shift in development cycles (#2135189).

@MavMan: please understand that you need an AI-level system to upgrade between D8 to D9. It's not a question whether such an update may or not break your site. It's GUARANTEED to break it. Core update, if ever, will only be between minor versions.

And the $200 funding just out of the blue? This is something on the level of a feature branch.. Just the discussion on this thread has already used more than that..

I've decided to end the funding campaign and refund the money, since there wasn't the support I was hoping regarding the funding campaign.

This was more of an experiment for an idea I thought could take hold in the Drupal community.

I donated $20 and someone else matched it. $40 isn't much incentive and there is no guarantee this issue would even be fixed within the year since this issue is over 5 years old. I don't want to be responsible for managing the funding without more certainty this model would be applied to other Drupal issues. I thought GoFundMe would hold the funds indefinitely, but they have rules were they don't retain funds longer than 30 days (to avoid banking regulations, since they don't want to be a bank). The the funds ended up mixed with my personal funds. I can keep track of it in the short term, but I don't want to manage it long term. Because of this, the funds have been refunded and the campaign ended.

I think composer and the drupal-template in 8.x kind of resolve this issue. (you can update the drupal core with composer). If it's just a matter of upgrading code and database: composer update && drush updb will do the job.

However, IMHO upgrading minor (and major..) version always need to be done first on a testing environment because every drupal site is kind of unique and a bug can throw only on combination of custom code, contrib modules, server configuration, etc..

So..., a "safe" automated update can be done only on an industrialised drupal product (with good tests and their own custom update-testing-deploy script). Even in that case a testing environment would be used to try first the deployment of each version...

Regarding composer update && drush updb, upgrading core look like upgrading a composer library today, and the upgrade manager, that is asked here to be responsible of the core update, is for managing the module of the "drupal" library...

(If you look at Jenkins for example, update of core are shell commands :O, updater/manager of the plugins are inside the UI.)

Maybe this issue is asking the wrong question. Maybe it's more a question of support for a safe way to update drupal core :

• D7: https://www.drupal.org/node/1494290
• D8: not found...

just my 2 cents..

I was wondering what was going on with the auto-update feature in Drupal 8?
I heard that this was going to be a feature when Drupal 8 was released.
I was very disappointed after installing Drupal 8 when it was released to find out that the feature was not included. So I've done nothing more with Drupal 8.
I get so fed up every time I have to upgrade revisions of Drupal 7 every few weeks as the updates come out. Which is why I was looking for the Drupal 8 being released.
I was told that the update feature would come 'later'. Presumably, that meant it would show up in a Drupal 8 update.
Haven't heard a word since then.
An update would be nice.

@fatkinson EXACTLY the same feeling here.

Wasting time updating something that should be done is so ridiculous that I hope to leave Drupal soon, and I guess that we are not alone !

But maybe this is what they want, or they would have done something smart 10 years later.

Drupal was ok 10 years ago but they must be really stupid not to have improved the update process 10 years later...

I know that some are passionate about technical useless things, but some others are just interested in using drupal because it is powerful, not in having to cry every time an update is needed !

@fatkinson @MakeOnlineShop: Where did you hear this was a planned feature of Drupal 8? I have been involved with the project for a few years and I did not ever hear of it.

Drupal 8 can add new features in point releases, so something like this could be added before Drupal 9. For my own self, I do not want the site code to be writable by the web server so I may never use this.

Oh sure we had discussions for this feature stretching back a decade or more. We could extend the classloader to support loading from phpstorage for fun and profit.

What's the state of this feature?

Is it abandoned??? We're kind of "waiting" for this to migrate to Drupal...

The remaining task in this issue is
#509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages

Which is dependent on, (active)
#1538118: Update status does not verify the identity or authenticity of the release history URL

It could take a while...

I don't see why this issue would need to depend on #509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages.

Looks like it was added to the issue summary in https://www.drupal.org/node/606592/revisions/view/9121946/9122132, but with no explanation as to why.

If it doesn't depend on it... then it's inclusion is only delaying the feature. Why not just get on with this?! It's a very important feature (even if it only upgrades between minor versions, it's better than nothing...)

I've been working on it for a month or two now. I'm more interested in providing an avenue for sites to opt in to receive fully automated security updates (#2367319: Implement automatic background updates for highly critical security issues), but being able to update core over http is pretty much a prerequisite for that.

I was told at Drupalcon in Austin that this feature would be in Drupal 8.
I almost jumped for joy when I heard it. When Drupal 8 was released, I immediately installed Drupal 8 on an inactive Web site I owned and logged in to find out that update button wasn't there. When I made a post about it, I was told it would be included with a future release.
I stopped development of Drupal on my sites because I spend more time upgrading firmware to my various sites.
I do not plan to use Drush because of the arrangements I have with the hosting provider I use. So core updates through Drush is not a solution that would appease me.
This was the single most important improvement to Drupal as far as I was concerned. Without it, Drupal 8 fell short.
I suspect others feel the same way. There is just too much time spent updating the core.

@fatkinson - i feel the same way!

I have absolutely no idea why someone would've said this feature would be included in Drupal 8. This feature was never touted on any official communication.

There also seems to be some pretty big misunderstanding in recent comments of the "do-ocracy" nature of open source projects such as Drupal. There is no company/organization/dedicated workforce behind Drupal to complain to when things aren't moving fast enough. The developer community here is a collection of individuals who work on what they are motivated to work on, whether that's because it's a particular problem they have, because they like the fun of the challenge of it, or because they're being paid to work on it, either via their boss or via a customer.

So if you care about any particular feature getting done, and it's not getting done fast enough to your liking through the open source community, then the way forward isn't to make demands of others to work on it (this tends to demotivate people from working on your problem). It's to put some of your own skin in the game, either with code, with $, with mockups, and so on.

Since @mbaynton seems motivated to work on this, a good first step might be finding out what help he might need and see where you can pitch in.

Regardless, it is a feature that is badly needed. I am not going to spend hours updating firmware on all of my Web sites every time there is a security patch or a bug fix. And I suspect others feel the same way!
Are the developers going to continue blindly down the path they are?
Or are they going to recognize the need and act!
For me, Drupal development is on hold!

@fatkinson: this is exactly the kind of reply why you could read #80 again :)

I read it, sir!
And my statement remains unchanged.
If they don't develop this, people like me are going to continue to drop Drupal.
I tried to develop a module for Drupal 7 once.
I spent a great deal of time and money developing a module for Drupal.
I did evertyhing they asked me to do.
Every time I finished their required changes, they saddled me with more required changes.
That module was never completed and is still sitting on the Drupal site unfinished.
So my coding is not good enough to help them since I could never satisfy them.
Nevertheless, I won't be using Drupal much longer if this this doesn't happen.

I'd be happy to provide more details about my project if I got any offers for assistance. There's a public git repo (30 days behind my private one already?!? :O), and a docker image that runs the unit tests and, as of a week or so ago, even some integration tests against an FTP daemon. Help is helpful.

If they don't develop this, people like me are going to continue to drop Drupal.

@fatkinson -- Respectfully, but I don't think you have read #80 a second time. There is no "they". There is only 'we'. Please, read webchick's comment again-- then maybe even a third time. She's already said it so clearly and succinctly it doesn't need to be repeated in a different comment.

This is the type of hyperbolic threat people often make to commercial entities when they've 'paid' physical money for a product that doesn't meet their needs. In an open source community, this type of statement will very often garner nothing but negative reactions on the part of those that actually contribute to the project.

The analogy I like to use is that if one of your neighbors mowed your lawn for free, would you complain to them that they didn't edge or mow low enough for your liking (most people would consider that ungrateful or even extremely rude)? Or would you simply thank them and either finish the job yourself or hire someone to do it the way you like it?

This really is no different. You're using and benefiting from the work of others for free.

If Drupal doesn't meet your needs then by all means find a product that does. Unlike a commercial product where you very likely have made a substantial financial purchase, there's nothing to keep you using Drupal other than it does what you need it to do.

Or, even better, come aboard and help make it the product you need it to be! Everyone is welcome and everyone is able to contribute in whatever way they are able.

Am I benefitting?
It's creating too much maintenance work which should be automated.
I am sorry you took it as a threat. It was meant to let you know what I consider important in Drupal development.
You can think of it as a threat or as an opportunity.
Every time I turn around, I am getting notices that I need to install updates. When I do, it takes a great deal of my time.
I am at the point where I ignore the messages any more because it is just too much work.

A possibly completely stupid question of a non-php programmer:

Since drush also seems to be written in php, and provides ready-to-use update functionality for core, isn't it an option to copy/port "drush pm ..." functionality to drupal, and "just" build an UI for that?

I understand that there is an additional webserver in between - does this "break" drush coding for drupal?

Tobias

You obviously didn't read the comments I made about my hosting provider.
I don't get shell access as part of the package. If I want it, I have to pay extra for it for *each* Web site that I want to add it to.
Others are in the same boat.
This feature is badly needed.

Thanks for the periodic reminders that there's a need to write some more code to this end, @fatkinson. This is definitely the demographic/use case I hope to help with. But no open-source coding on my birthday is my rule I think :). I will try to channel the energy later on...

From my point of view, Drupal is segmented for mid-end user on this feature (people who can get shell access, or people with knowledge/not novice user, or people/organization with more money to develop and maintain Drupal). For newbie people or "low budget" organization, Wordpress is still more interesting than drupal since they can upgrade the site core from UI, even Wordpress can automatically upgrade itself when the admin is away.

Having a core update on the admin UI is a big advantages on Drupal site since it will attract more newbie or low-end budget organization, and with that, Drupal user will be more increasing. It will be a long term investment for Drupal.

But since Drupal is a community project, and there is no company to complain for the bug/feature request. I think it will be better if we can make a crowd funding for a feature request that people ask to the community so there will be people interest to develop the request. Either the funding is backed on Drupal Association or on the 3rd party service is no problem for me. So if people really want to add some feature while there is no progress, no people working on it, create a funding and make the feature happen.

Understand.
I think Drupal has a lot of potential.
But this is a big obstacle.
It's one I would like see Drupal overcome.

I, personally, am holding back on implementing Drupal until this is solved...

But once i get to the point i can't wait no more it won't matter how much better or secure Drupal is... i'll be heading the Joomla way just because of this feature.

This is not a critic... it's just for developers to have an idea of how much important this feature is for me... and how much important it can be for others...

Over the years, I have heard from my service oriented employers that if one customer is complaining that there are hoards more customers that are silent but just as unhappy.
When multiple customers are actually complaining, multiply that number by the number of customers complaining to realize how many unhappy customers you actually have.

Issues like this one disclosed yesterday underscore that doing this properly is difficult, and can potentially make all participating sites hackable.

The vulnerability we describe below may have allowed an attacker to use the WordPress auto-update function, which is turned on by default, to deploy malware to up to 27% of the Web at once.

— Hacking 27% of the Web via WordPress Auto-Update

But i'm not in favor of the feature being enabled by default.
I truly do think it should be off by default and warn anyone who tries to turns it on.

But it should, ultimately, be a user choice.

(and i don't see wordpress getting any less users because of this kind of disclosures. In fact, has far as i know, wordpress continuous on getting more and more users. (even bad publicity, is still publicity)

This issue does not propose adding auto-update functionality, so I think the last two comments belong on a different issue?

One-click updates do have some risk too, of course, but the reality is that updates.drupal.org is already a pretty big target since there are contrib modules (e.g. Views) that are installed on the vast majority of Drupal sites and that can already be updated via one click in the Update Manager, so adding one-click updates for core on top of that is not going to increase the risk significantly. (That is why I suggested above that #509010: Build a Package signing system for d.o. so that people can securely and simply download and verify packages is not a prerequisite for this issue.)

@David_Rothstein true: #2367319: Implement automatic background updates for highly critical security issues

@dww, cool, help would definitely be appreciated if you do feel so inspired and do not find my approach to solving this too wild and eccentric. Whether or not you decide to jump on board, perhaps this is a good time to give an update on what I'm up to.

As I've said before, my motivations for doing this really revolve around #2367319: Implement automatic background updates for highly critical security issues, so the direction I'm going is substantially informed by a desire to reach that goal. I also believe (as evidenced by the extensive dialog in that issue) that large portions of the "securely update a PHP web application" task turn out to be nontrivial but also turn out to be essentially the same thing whether one wants to update Drupal, WordPress, Joomla, MediaWiki, ... So, rather than having every project reimplement a solution, what I'm doing is more One Updater To Rule Them All than a patch that depends on Drupal. I wrote some additional details about exactly how I see that looking here.

Ultimately, the architecture I envision is my updater component -- I've taken to calling it "Curator" -- packaged as a single .phar that lives on webservers alongside the sites that wish to be updated. The meat of the .phar's functionality will be accessed via API endpoints, enabling project (e.g. Drupal) infrastructure to push automated updates out (more on why I believe pushed updates are superior to periodic polling here), but also allowing site owners to perform manually-initiated updates in browser by hitting the same endpoints from an AngularJS user interface that the .phar conveniently also serves up.

With that 10,000 ft overview in mind, here's the github: https://github.com/curator-wik. There are two main repositories:

• curator-wik/curator: The server-side code.
• curator-wik/curator-gui: The AngularJS GUI. I have in fact spent like 4 hours on this one to date, and it doesn't do anything yet.

What has been accomplished:

• A filesystem access abstraction layer a little different from Flysystem or plain-old stream wrappers in that it allows operations to be routed to different I/O mechanisms depending on whether they're reads or writes, while always giving the rest of the app a single path to refer to a file. This is basically to avoid performing all my reads through something slower than a mounted filesystem just because I may need to perform my writes through ftp/ssh/etc. That stuff lives in the curator repo under dist/src/FSAccess.
• A way to do some basic key/value persistent storage by leveraging the filesystem access and writing it out to a file -> dist/src/Persistence. I eventually do imagine a Drupal module that works with the .phar to provide a tighter integration, including defining a db schema for persisting settings that way, but priorities.
• Getting the GUI off the ground and getting it embeddable into/served out of the .phar.
• Lots of rigging up of tests and CI setups.

Immediate next priorities as I see them are:

• Implement a batch processor conceptually similar to Drupal's.
• Start defining/implementing APIs! A number of them don't need batch processing, like inputting file upload method and credentials, so this can happen now.
• Start hitting the APIs with the GUI!

I fixed a few bugs yesterday and successfully built the project from source trees into a .phar using the instructions on my README.md's, on a few different computers. Only gotcha I'm aware of is that to do all the scss compilation and other build steps for the GUI you need a newer node.js than some linux distributions' defaults. You should also be able to stick the .phar attached in the .tgz on any PHP 5.4+ server and "run" the thing, such as it currently is. If the server is not configured to interpret .phar's as php, you can use the attached .php file to get around that.

Sounds like you're trying to reimplement composer, among other things.

I hope not. Unless/until Drupal and similar software add Composer to their system requirements and make it a necessary part of creating a new installation, I don't see Composer's functionality as essential. Of course, as a developer, I recognize the benefits of building a site with Composer, but probably most of the people doing it that way are going to prefer the CLI to get stuff done anyway.

I don't think I'd be opposed to finding a way to integrate Composer's core services into Curator eventually, but it shouldn't be necessary to allow upgrading core in-browser.

I'm also not sure how push notifications from Drupal.org will work

I'll choose not to open that can of worms just now. It's a subject more appropriate for the other d.o issue anyway, and much has been said about it already over there. I spent quite a while exploring the questions you raise last July/August, and I feel there are satisfactory answers to most of your questions, for most people.

Furthermore, I'm quite skeptical that you're going to get all these different systems to support One Updater To Rule Them All(tm)

Could you elaborate on the problems you foresee? The way I see it, CMSs/wikis/forums/etc all get upgraded by

1. Cause the updated source files to be written to disk where the webserver interprets or serves them
2. Redirect the user to that app's in-browser post-update runner.

...and if they don't support getting upgraded using that procedure, then Curator probably won't work.

For #1, it can be surprisingly hard, but how is it different from one app to another?
For #2, you make map of (app => path to update.php), and write some sniffing of files/directory structure to determine which app is being updated. That sniffing ought to be the extent of the code that targets specific applications/systems.

Thanks for sharing your perspective @dww, it's important to have multiple points of view.

@dww, thanks for giving my approach more consideration! Sorry, yes I meant #2367319: Implement automatic background updates for highly critical security issues, I'm surprised it wasn't already marked related!

I am of two minds on the Composer issue. On the one hand, I think composer should be regarded as a tool that is used by developers to create releases, not as a tool for release deployment. In a monolithic system (vs. one composed of modules like Drupal), that would be the end of it in my mind -- wrong tool for the job. On the other hand, in the context of Drupal, identifying a workable set of external dependencies and their versions that satisfies all modules on a given site might really need Composer's talents, as more and more sites/modules are built that way.

Regardless, I don't think downloading composer packages should be the only way for Curator to source the updated code that it can install. In the interests of making visible progress easier to achieve, I'd like to stick to reading more straightforward upgrade packages that contain the literal code to be patched for now. All the file management and batch processing infrastructure I'm working on should be useful to any future efforts to make Curator Composer-aware.

Regarding adoption by d.o. and other projects: you'll see when you do get a chance to look over #2367319: Implement automatic background updates for highly critical security issues that I made a valiant effort to get buy-in on a specific approach before it actually existed, and that more or less got nowhere. So, I am forced to go with the "build it and (hope) they will come" approach. I think the task of writing server software to perform en-masse pushed updates to a registered set of sites is a sensible next step, though, because once that software exists, hashing out the details of its deployment at one place or another becomes much more achievable. You wouldn't argue that drupal.org and wordpress.org need to each have their own custom http server daemons because they have to serve drastically different content; both projects instead figured out how to use existing software to meet their needs. Once a pushed update server is existing software, my hope is that it would similarly be adopted.

And, I should add, while it'd obviously be my preference for projects like Drupal to eventually provide releases officially through Curator, the entire thing could be sustained without official buy-in. The software doesn't care who is running the servers. Consequently, it'd be totally reasonable for not only the in-browser core updates but also automatic updates to be fully functional contrib things for a while and just see what happens.

I may have some time to work on a new patch for this issue during the next couple weeks. It's not guaranteed (depends on what client work might come up) so if someone else wants to look into it in the meantime, don't let me stop you (just leave a quick note here saying that you're working on it). But hopefully I will be able to.

This is the type of hyperbolic threat people often make to commercial entities when they've 'paid' physical money for a product that doesn't meet their needs. In an open source community, this type of statement will very often garner nothing but negative reactions on the part of those that actually contribute to the project.

The analogy I like to use is that if one of your neighbors mowed your lawn for free, would you complain to them that they didn't edge or mow low enough for your liking (most people would consider that ungrateful or even extremely rude)? Or would you simply thank them and either finish the job yourself or hire someone to do it the way you like it?

I'm not sure that's a fair analogy. In recent years the Drupal Association (which is supposed to represent and grow the community) has solicited and spent a lot of money directly on core development. Questioning how that money was spent seems reasonable to me.

A better analogy might be if you live in a neighborhood that has a common area, and the neighborhood has a homeowners' association that helps maintains the common area but individual residents also volunteer their time to improve it. Complaining to your neighbors about their volunteer efforts is rude, but questioning how the homeowners' association spends its money isn't.

In any case, I personally agree with everyone else that this is a very important feature that Drupal needs to keep the majority of its current users happy, as well as to attract new users - so I'd like to see it happen :)

On the technical side of things:

1. I agree that what @mbaynton is working on is interesting, and also that it's beyond the scope of this issue (since it's basically about replacing the entire Update Manager, whereas the feature proposed in this issue is possible to add with the current Update Manager). So I like the idea of a separate issue and/or sandbox project.
2. Similarly, @dww:
   

   Given how likely it is that we're going to want to solve this by basically having authorize.php be able to run composer for us...

   It is not obvious to me why you see that as a possible requirement. I actually filed a related issue a long time ago (#2538090: Allow the Update Manager to automatically resolve Composer dependencies) but the goal there was to allow Update Manager to be able to install modules that have Composer dependencies (i.e. that require code beyond what is hosted on drupal.org). However for Drupal core, all the code is included in the tar/zip file that is downloaded from drupal.org, so it seems to me like we should just be able to use that file.

3. I agree with #87 that it would be useful to look at how Drush does this. I am particularly curious if they deal with the possibility (first mentioned in #4) that the new version of core could have deleted files compared to the old version, or if they just ignore that and unpack the tar file right over the existing installation.

   I agree with others above (#4, #7, etc) that if ignoring the file deletion problem winds up being the simplest technical implementation, it is fine to go ahead with that. Within Drupal 7, as well as within an individual branch of Drupal 8 (e.g. within Drupal 8.2.x) I think we can guarantee that no file deletions will happen (or at least no important file deletions will happen). That's probably not true between Drupal 8 branches (e.g. Drupal 8.2.x => 8.3.x) because of external library updates, the possibility of removing "experimental" modules, etc. However, a core update feature that can be used for everything except branch jumps within Drupal 8 (which only happen every 6 months) would still be very useful.

I agree with #87 that it would be useful to look at how Drush does this. I am particularly curious if they deal with the possibility (first mentioned in #4) that the new version of core could have deleted files compared to the old version, or if they just ignore that and unpack the tar file right over the existing installation.

I believe Drush makes a backup because I've had Core updates fail before for various reasons and, if I recall correctly, it rolled it back to the previous version. I don't think it does this once it gets to the database updates; once those have occurred, it's up to you to roll back to a database back-up and a previous commit to get things back if the database updates have screwed something up.

I've written a module (https://www.drupal.org/project/dfm_autoupdater) that implements a shell_exec() that could be used and/or modified to do this via UI though it requires SSH access on the host and the shell tools of choice for updates, e.g. drush, composer, git, etc. Not sure if this is 100% applicable here but maybe it will add to the discussion/ideas. This topic (updating Drupal), in general, has been a long-standing interest of mine (and obviously many).

I've been continuing to work on this issue, and have been focused for the past few months on a goal of getting my code to perform a complete core upgrade, at least inside an integration test by 1 month to DrupalCon. I am happy to say that with commit 9181e8 yesterday, I've succeeded in that goal.

There are still important elements to be developed, but the project has long since passed the point where there's any risk of my becoming bored and abandoning it. This work will definitely result in an avenue to performing in-browser core updates, and it will be sooner than later.

Although it's not strictly "Drupal development" in that I can't point to a component or file of Drupal's that I'm enhancing, and in that I ultimately hope it could be useful to projects that aren't Drupal as well, I hope others with an interest in this nearly 8 year old issue will be open to taking a look at the momentum I've built as a path of least resistance to getting this done. A guiding principle of the project is that it doesn't require any additional server system requirements beyond Drupal's.

In the month remaining, I'll try to make it more presentable for informal demos to any interested parties in Baltimore. If you follow this issue and would like to see it, let me know and we'll schedule a time to meet. If you won't be at DrupalCon Baltimore, I should also be able to post interesting-looking demos online by around that time as well. In particular, the complete upgrade integration test demonstrates that the server-side code required to affect an upgrade via HTTP requests is there & working, but I still need to write the JS to replicate it via in-browser UI. No UI = not much of a demo.

A few cool technical elements I've incorporated include a framework for batch processing of tasks that's built from the ground up to support concurrent execution, this is used to write out files or run ftp uploads for several changed files simultaneously. We've also made real progress on making the update package be a set of patch files plus delete and rename information, instead of shipping complete files in the update data. My colleague has developed a utility in C linked to libgit2 and some related libraries that generates the patch data between two git commits in a patch format that is more easy to work with in php code. It mostly works but some diffs still give it problems.

The user experience I'm going for is for a user to be able to see an available core update through existing Drupal administrative UI, with some added buttons/links on that page that transport them away from Drupal-handled routes to my code, which gives them basically a progress bar while it performs the code modifications. It then sends them back to Drupal's update.php.

This will be a big step in the right direction!

Finally!!! The "Final Missing Piece" to make Drupal really the best!!!
Thanks @mbaynton for this.

I have a question, however: To clarify What kind of upgrades will this be able to do? "8.x -> 8.y" or "8.x -> 9.0" ?

Will be watching this bug report closely :D

@MavMan, you've probably seen Making Drupal Upgrades Easy Forever, where it says

Drupal 9.0 should be almost identical to the last Drupal 8 release, minus the deprecated code.

Based on that, 8.x -> 9.0 sounds like it might even be doable, assuming

• My project (Curator Web Install Kit) becomes and remains enough of a concern by the time there is a 9.x for me and others to be keeping its associated D8 module up-to-date with the core APIs.
• The same is true for all the other modules on your site, and you can update all of them to ones that'll be able to handle a 9.x core first as well.
• You have a pinch of luck.

Who knows, by then, maybe Curator will even have some update rollback capability...

I noticed this issue because it has 'upgrading core' in the title. But when I skim the issue this seems to be about updating from a minor version to another minor version and not upgrading a major version. See Updating and upgrading Drupal core. I've been reviewing the upgrade documentation recently so getting the words correct is on my mind. So, is this issue about update, upgrades or both?

@mbaynton I was not aware of that blog post!!! Thanks for sharing. :D

It does look very promising, but... will all those upgrades be doable thru a native Drupal Web GUI (like in Joomla/Wordpress)??? Or to have a GUI we'll need Curator Web Install Kit???

@quietone AFAIK this issue is about upgrading (which ends up including updates. If it can do upgrades, for sure updates will be easier). So, i think the answer is actually "both". But i'm not an expert... so i could be wrong...

Any progress on this issue?

Any progress on this issue?

Usability is preferred over UX, D7UX, D8UX etc. See https://www.drupal.org/issue-tags/topic

the title was causing confusion, and didn't reflect the issue description

@MediaFormat: Really?!?! I was under the impression that "Allow - UPGRADING - core with the update manager " was what was being talked here!!! At least I was talking about Upgrading, not updating (e.g.: Drupal 8.3->9.0 migration, not bug fixing).

Maybe a bug with title "Allow upgrading core with the update manager" should be created, then...

@MavMan, feel free to open another issue for Upgrading between 8.x to 9.x. This was addressed in #62 by @jcnventura

#62 was probably accurate when it was written, but is a bit pessimistic now I think. #114 addressed it as well.

Any system that is capable of doing 8.xy to 8.xy+1 can do 8.x to 9.0 too, though maybe a separate issue would be nice just to track things like a superficial check that the installed modules claim support for 9.x.

Edit: A separate issue would also be useful to assess interest in the feature. It's unlikely to be of interest to the "ambitious digital experiences" that are D8's target audience.

Please help filling more details that might be needed and/or any corrections/clarifications to the bug report :)

Posted suggestion here -- https://www.drupal.org/node/2908394#comment-12274918 which could be used to do core updates via update manager.

However, for git users I've started using https://www.dependencies.io/ which creates a PR (and triggers CI) for any available composer updates but not only restricted to composer.

Sad that there has been no progress on this issue.

@fatkinson: since, apparently, this is getting nowhere i'm investigating Typo3. It's rumored to be at least as powerful as drupal (some say it's way more powerful and directed to very-large businesses) and has upgrade wizards to change between major versions. Also, it looks pretty easy!

It is sad. Drupal has the capability of becoming a stronger contendor in the CMS market.
But this weakness is its handicap.
As long as Drupal has been around you would think this would have already been resolved by now.
Let me know what you find out about Typo3.

In light of the recent WordPress backdoor revelations, Drupal stands very strong security-wise:

• Backdoor in Captcha Plugin Affects 300K WordPress Sites
• Three Plugins Backdoored in Supply Chain Attack

Core updates for Drupal 8 via update manager would of course be a real killer feature. Optionally automatic updates, like Dries proposed in his Vienna keynote, would be even better :-)

@fatkinson: So far it seems to serve my needs (personally and professionally)... will do more tests when i get some free time.
in the meanwhile i guess this is a good comparison to sum it up: https://www.aoe.com/en/knowledge-base/cms-vergleich-typo3-vs-drupal-vs-j...

I’m writing this comment as my FTP client is simultaneously uploading about a years-worth of core updates.

My primary website – built in Drupal 7 – has gotten behind on core updates for the sole fact that it’s a very manual process. In my case, it’s a process of 1.) download core files from drupal.org 2) extract to my computer and 3.) upload to the server.

And yes, I get that there is Drush. But let’s be clear: to even use or deploy drush requires knowledge of command lines and shell access, which a majority of everyone is not familiar with.

Now before you jump my case, let me say that I’ve built websites with Drupal for over 8 years. I have very thorough experience working with small-to-medium size businesses in deploying custom Drupal websites.

The number one problem I’ve had with Drupal for years has been the inability for my clients to do their own updates. The result is that I still have clients from 8 years ago who every now and then need me to go apply updates to their website. I hate it.

In one particularly embarrassing scenario, a client paid me close to $5,000 for a custom Drupal website. This was a top-notch, custom-built theme. Very solid back-end functionality. Arguably, would have been charged $10-$15k by a full-service Drupal company.

But one day, I went to check his website and it was gone! It had been replaced with a crappy, garbage wordpress themed website.

I called my client in disbelief and asked what happened to the beautiful website I had designed just for his company. You know what he said? "Yea...we couldn't really manage it so we decided to just use Wordpress." This client is a young lawyer who is very hands-on with his digital marketing (but not enough to be a website developer). He wanted that ability to update the website and make sure it was secure on his own merit. He needed easy integration with his business' services. Drupal wasn't delivering for the new era of digital marketers.

It was quite embarrassing for me to have invested all that time and energy into this custom solution for my client only to have it disappear roughly 1.5 years later because he decided it was too cumbersome to maintain.

This is, by all practices in 2018, and outdated way for a CMS to conduct itself.

The excuse “Drupal is a developer-first platform” no longer holds water. As indicated at the top of this thread, this has apparently been requested for over 9 years.

I was disappointed in the Dries keynote. It seems to indicate that this is still no where even close to a priority for Core.

I’ve sang Drupal’s capabilities for many years. I've fought my clients when they say it's too cumbersome. I have to step up and say "don't worry, let me just update it so you don't go somewhere else." I’ve cheered Drupal's awesome open-source platform. I love the ease at which you can create new custom content types and style those types to your needs.

But the gap is closing. I look at Wordpress – 8 years after making the cognizant choice to learn Drupal over Wordpress – and I realize I may have made a mistake. Drupal will become obsolete.

Here’s why:

I run a news publication now. It’s pretty successful. Gets lots of traffic. It needs a solid, scalable platform from which to operate. From day 1, we’ve relied on Drupal. In fact, we started in Drupal 6.

But today, I find our website being held hostage in Drupal 7 for a variety of reasons – among which is the issue of Drupal 8 not solving the update issue. I have other general reservations about Drupal 8 (a release for which I was pretty excited because of some of the front/backend changes.) But I’ve tried using Drupal 8 numerous times on projects, and just can’t bring myself to saddle a client or myself with this cumbersome CMS.

Drupal 8, in my opinion, takes a step backwards with the introduction of Composer. Being a non-command line guy, I took one look at that, laughed, and said “I simply don’t have time to learn a whole new system just to maintain my website.”

Because if I have to learn it, then I can be assured my customers will NEVER learn it.

This archaic method of maintaining a website is what has prevented me from carrying my news publication into a Drupal 8 setup.

At this point, I can say with certainty our next website overhaul will not be on Drupal. It will likely be moved to Wordpress, since Wordpress seems to understand what the market requires.

I would also like to point out the reality of Drupal’s complexity. I found it quite embarrassing when the White House decided to move from Drupal to Wordpress. I remember this was a big win for the Drupal community: the White House had adopted its website to Drupal and had paved the way.

But under the new administration, we’ve been told what’s been known for years: the maintenance of the website was too cumbersome and expensive. The result? The current administration scrapped it and went with a cheap, easy-to-maintain wordpress site at a fraction of the development cost.

Let's be clear, that was a loss for Drupal. Regardless of your politics.

Drupal is losing. It’s not winning.

What’s further concerning is that when I read other comments like mine, they are often met with fierce criticism from the Drupal vanguard. You get the usual “well Drupal is made for developers.” Or “if you don’t like it, choose something else.”

Well I can assure you that attitude is what will kill Drupal. It’s that level of close-mindedness that will be Drupal’s downfall.

It’s already happening. As I’ve indicated, and have seen many others indicate too, the acceptance of Drupal 8 has been depressing, at best.

But here’s the problem with taking an attitude like that with someone like me: I’m not a developer. I pay to have drupal modules worked on. So when I walk away, so does my money that would be going to Drupal developers. I’m sure that doesn’t mean much to most, but to a module developer who is struggling to find the time for updates, it means the world if he/she can be compensated to work on something that benefits the entire community.

We recently dumped about $1000 and are about to put more into a Drupal module. But we’re only interested in Drupal 7, because we don’t anticipate staying for Drupal 8.

Anyway, I don’t expect my input to make a difference. But it should…

What’s further concerning is that when I read other comments like mine, they are often met with fierce criticism from the Drupal vanguard. You get the usual “well Drupal is made for developers.” Or “if you don’t like it, choose something else.”
[...]
But here’s the problem with taking an attitude like that with someone like me: I’m not a developer. I pay to have drupal modules worked on. So when I walk away, so does my money that would be going to Drupal developers.

I agree with you @arlingtonvoicellc, site builders play a very important, but sort of unrecognized role in the Drupal ecosystem, which is really too bad, because Drupal 8 is a pretty damn amazing tool to build web sites with. (I do use Drush, and couldn't really live without it ...)

I have also helped clients sponsor Drupal developers to get modules ready for Drupal 8, thereby benefiting the whole Drupal community. Another benefit of having a big base of Drupal site builders, is that they can actually install and use the contrib modules, test them and give feedback for improvement, report bugs, be Drupal evangelists, and so on.

So being able to update core with the update manager would be a huge help in keeping site builders using Drupal, by not making Composer a requirement.

Also see #2940731: Automatic Updates Initiative overview and roadmap

Ressa,

Quite some time back, I also made a posting expressing my concern about this.

I was accused of 'making a threat' rather than them taking it as constructive criticism. All I said was that I would not be doing any more Drupal development until this issue is resolved. And I haven't.

That is not good for the Drupal community or for any endeavor business or otherwise.

Perhaps if more people would speak up they would realize just how important this is to Drupal users.

But if they continue to react in this way, Drupal developers are not going to feel comfortable with sharing what they see as the issues. This includes weaknesses that could be resolved. For myself, I have learned to keep speaking up and not let their adverse reactions stifle my feedback. So I do it here as well.

You are indeed correct when you say that it is entirely too much work and time to keep manually installing the core updates.

I just can't understand why they cannot come to this conclusion on their own. I see it as all too obvious.

It is just too important an issue for them to keep putting off.

I recently saw a pie chart showing percentages of the different CMSs in use on the Internet. If that one was accurate, Drupal is falling behind the others.

This is way too important a hurdle for them to keep putting it off. This should be addressed immediately.

Fred

May I suggest that anyone in need of Drupal features to take steps to support its development, such as joining the Drupal Association, attending an event, or hosting a sprint?

perhaps what's missing more directly from this issue is tagged 'needs backport to D7'. I think 7 should be dealt with more immediately toward Allow updating core with the update manager rather than as a backport issue

D8 site are more likely to be running in a 'modern server setup' (drush/composer/command-line). current plans for 8, pointing to composer use and 'Project Browser' integration are more certain.

but no one (companies) are going to update a 7 server to be modern.. Maybe assuming they outsourced the project to begin with. Companies (the site owners) are more likely to rebuild in whatever CMS flavour is popular in that moment. as the 'developer experience' in 8 has progressed, we all know.

One last manualupdate for D7 site maintainers would be awesome!!! then @arlingtonvoicellc wouldn't have time to rant with he's uploading (i kid ya mister, but completely empathize).

It really is sad. As someone who built their clients' websites in Drupal, I no longer offer it. For about the past year, I've been delivering in Wordpress.

Years ago, a political consultant paid me several thousand to build some drupal templates that we could quickly deploy for candidates every year. You can imagine that after a few years, this became quite a few websites. These were not complex websites either. Just static content; a few pages -- nothing more.

However, I was not being compensated to maintain the websites, so I didn't. The result? They became so out-of-date, that I received a call a few weeks back from this client saying a bunch of their sites had been compromised and taken offline. The server's CPU was being overloaded by a malware injection. Apparently, because I hadn't updated them in some time, there were security holes. And because my client barely even begins to understand how to add content into the site, this was a task that fell to me.

What ended up happening is I became so frustrated trying to figure out what had brought down these websites, that I scrapped them all, bought a Wordpress template, and built them over again. It was actually easier to do that than try and fix each website and spend hours updating each one.

It's just not practical.

I'm sure I'll hear "well you should have just Drushed those sites and updated."

No....I should be able to give my client a link and say "go here, click the "update" button, and you're secure!

That's how it should be, and that's what the vanguard doesn't accept.

May I suggest that anyone in need of Drupal features to take steps to support its development, such as joining the Drupal Association, attending an event, or hosting a sprint?

If you would have to choose to take a car that is ready to go or deal with a car that you have to assemble first, which one you would take?

Your answer is exactly the kind of arrogance, which will kill drupal. The Linux community is similar. And you know where it is? Its market share got so small, that it got merged into "others". Same thing for Commodore back in days. This list could be close to endless.

Drupals marketshare is decreasing since this year. The rate of decreasing is not linear. It is growing.

What do you think how many drupal sites don't got the recent critical updates, because the people don't know how to do or because the amount of work is to much?

And when their pages got hacked, do you really think they will choose drupal again?

One thing you should not forget: drupal needs its customers. But the customers don't need drupal. If they are not happy anylonger, they take something else.

I'm currently running a D7 site, because the migration to D8 need more worktime, than I would invest into it. I have better things to do than this.

When the support of D7 ends, I will decide what to do. But if I will have no other way than a massive hand work migration, it will not be drupal to drupal for sure.

Some pertinent facts:

This is a core issue. Since it was filed, major initiatives such as changing the way the updater works have a place in the Core Ideas project: https://www.drupal.org/project/ideas

In fact, as mentioned in #137, there is a Core Idea initiative to perform automatic updates: #2940731: Automatic Updates Initiative overview and roadmap These were announced as getting the support of the DA at the Nashville DrupalCon, and DA people are working on them.

Before that can happen, a number of other initiatives have to see the light of day, such as having a site builder tool: #2940733: Site Builder Tool/Project Browser initiative and needing to un-broken-ize Composer stuff: #2958021: Proposal: Composer Support in Core initiative

It's also true that there is an initiative to allow relatively easy D7->D8 migrations, which you can find here: #2906878: [Meta] Support for D7 -> D9 contrib migrate This would enable D7 sites to move to D8 and benefit from all this work.

People are working on these things and are enabling easy(ier) updates and upgrades. Whether the result meets any specific needs might be determined by whether people with those needs participate in the conversation.

At the risk of being off-topic to this issue, except to highlight the importance of it, I was intrigued by comment #142 on the dwindling market share for drupal. On doing a search I looked at:

https://www.drupal.org/project/usage/drupal

which shows total use for drupal to be almost flat at 1.1 to 1.2 million since the beginning of 2015.

and https://briteweb.com/ideas/wordpress-vs-drupal-statistics-trends-non-pro...

"The amount of people searching for Drupal on Google has dropped by 70% worldwide since 2011."
Not to overlook that briteweb promotes wordpress.

Nonetheless, having used drupal for 10 years I agree that drupal is helpful for developers and techies to learn and do more but it has it's pain points including updates. Not only is drush essential, but with the introduction of composer the layers of dependencies has grown making the process even more complicated. Is it worth it? That depends. It may be quite different for larger commercial production companies than small development shops and individuals.

Am I the only person who doesn't see the equality between an appeal for more support for initiatives and "exactly the kind of arrogance, which will kill drupal"? I'm happy to discuss elsewhere, like my contact form or Slack.

I had another old customer reach out to me late last week. What a great way to throw a wrench in your weekend. I receive an email that a website I built for a client years ago has come under attack because it is long out-of-date. I haven't been paid to maintain the website, and do not plan to. It's sad, really, that my clients are not able to do the most critical task of updating on their own. It's the MOST critical task. It should have received high precedence in developing Drupal 8 core. It wasn't. It was, unfortunately, an opportunity missed.

Because of the incredible number of clients' whose Drupal sites have come under attack, I will no longer build customer websites in drupal. I will be moving to wordpress. Until this problem has been resolved, I cannot in good faith sell a client a website that may break in 4-5 years if they don't constantly pay for maintenance updates. I can't have that kind of uncertainty.

Wordpress gives me the assurance I need that my clients will be able to maintain security updates on their own. This is HUGE for someone like me, who prefers NOT to have a roster of clients I'm doing small maintenance tasks and updates for. I don't enjoy that.

Maybe I'll revisit Drupal in several years and see if this elementary issue has been solved. Until then, I'm out!

Clients are free to choose not to pay for maintenance or support. But it's also responsible behaviour on their part, or the designer/developer, to ensure someone is taking care of their site, and to be aware of the basics of the Drupal core release cycle; i.e. how often and when software updates released, how long support will be available for a particular version, and how security announcements are made.

Expecting a site not to break over a 4-5 year period without maintenance is unrealistic for any content management system. Yes, you might get very lucky, but it will be just that, luck.

PHP, your choice of web server and the operating system will all also need to be upgraded in that timeframe. If you use shared hosting, you still need to double check the company providing it is keeping things up to date. By way of analogy, you wouldn't leave a garden untended for years.

When we eventually do have automatic core updates, that doesn't mean a Drupal site will magically be maintenance free. Contrib modules will still need security updates themselves, and if you haven't kept a module reasonably current, applying the update may also introduce non-backward compatible changes from a previous non-security related release. Those will need to be investigated to fix anything broken on your site. That's true of WordPress too. (Note WordPress' security updates, whilst very convenient, aren't immune to problems. They've had various issues such as sites becoming stuck on a particular version or plugins conflicting with the update code.)

If you're not being paid for maintenance, you could still arrange or recommend a client use a hosting provider that supports daily/weekly backups, so in the event their site is hacked (or any number of other problems befall it) it can at least be rolled back, and whoever has to fix it will have an easier time doing so.

There's also the Drupal Security mailing list, via which everyone received advance warnings (PSAs) about the critical updates. (I had to do the most recent one in the worst of circumstances, an hour after an HDD failure on my desktop computer - there were two things, besides the advance notice, that the security team did which helped me: (a) including a simple patch people could quickly apply (b) indicating in the PSA whether a database update was going to be needed.)

The Weekly Drop newsletter normally leads with any important security info.

Finally, if you'd rather not do any of that, there are private companies that will do it for you for a monthly fee.

As others have mentioned, the Automatic Updates initiative is the place to follow what is shaping up to be a pretty sensible plan.

Complains, complains, complains, no code, no patches, rarely contribution, no donations. But with the expectations to get a flexible power-framework like Drupal for free and with only 4 knobs: install, start, stop, update - obviously without even knowing basics about the easy vs flexible paradigms issue technical tools have since ages, while talking about "clients" in the same time. I really ask myself what these people sell their "clients" then? Every kid can set up a website. And then using the word "arrogant" for people who are contributing to this mammut project night and day since ages while not even knowing what they have in their hands and what they would be able to do with it ... AND then coming up with Linux (*facepalm*).

Getting a Boing free fly and complaining about that the bird has no balcony while selling service as a technical flight expert makes me ponder to open the door and invite them to join the view...

I think the users show off enough by theirself and it actually needs no further commenting on it. Especially when comparing Apples with Oranges and calling Wordpress in a security debate *cough* ...

But please, can we move this to somewhere else and can come back to topic?

Well, if you had to choose between a car that required almost no maintenance and a car that would have to be serviced every other day, which would you choose?

@fatkinson: As I already sad multiple times: wrong comparision (Apples/Oranges) and the wrong metaphor. And by the way it appears more of an own goal. The answer would be simple and would proof my points: If you need to win a race, the same question would become rather a joke since the service is part of the overall fine tuning in the race (called "box") ;-) And so is a powerful web framework. To keep your metaphor you could say the Drupal community tries to make a racing car with fewer service requirements than typical for such car and then you would maybe rather esteem it finally.

And well, I should not ask what you refer to for "almost no maintenance" (which will be not true on the end anyway). I should write your answer on a peace of paper for reference. :)

Touchy today, aren't we?

Not really. Why are you asking? Is any of my phrasing ambiguous? I am actually fine. I hope you too? Maybe a language barrier? Just to make sure: It was not meant any offensive if it felt for you like that. I just wanted to help you to "buy" the right car ;-)

Then you should consider how you are being perceived.

Hm. Funny that you say that :) -- but anyhow,

I do not want to help letting the discussion turn away again. But I am sorry for you that you "percieve" me in a way you do not feel compfortable with and I hope you are well. If you felt offended in any way please accept my apologizes and let's get back to topic.

If we want to get absorbed into completely ridiculous metaphors...

Average users don't need race cars. They need delivery vans. They may want race cars, but wouldn't have the faintest idea of how to maintain one properly. So if we're going to be building race cars for the average person to use, they'd better be race cars that fix themselves. Whether or not we should be building race cars or delivery vans is a separate, sad topic.

If we want to get absorbed into completely ridiculous metaphors...

No, we don't :) I just picked up where it was (sadly). And as I sad (in short): There is no "average" user today. There are millions of different use cases and a world split into 2; web cards and web applications. And Drupal is prepaired to build web applications and tries to make it as easy as possible for web card makers too. But it won't happen without the help of everybody who tries to get something out of Drupal. Drupal is as strong as we all are, and contributing does not mean always code. Any report, any review, any contructive(!) thought or help on the documentary would help to get more time for a better Drupal experience. And for me Drupal is almost what you call a race car that fix itself :) ... But maybe only for me knowing how hard it is to even come close to such "metaphor"...

Oh I forgot, and to put something on top: A flexible and modular set up race car is able to be a delivery van if required. It's just a fast delivery van then :D

I still don't understand why this very important issue is not being addressed.
I see this as the foremost Drupal issue and a very serious Drupal shortcoming.
I would like to develop a Website with Drupal. But it would invoke more maintenance than I would be willing/able to deal with.
It is crazy that the feature to update the add on modules nearly automatically does not include updates for the core.

The Automatic Updates Initiative has already been referenced a couple of times and has been mentioned in DrupalCon keynotes. That's a good place to follow proposed changes, express an opinion or find a way to help. There's also a link there to a video of the DrupalCon presentation about the auto update proposal.

Recent comments have become repetitive; I'm going to mark this issue as closed for the benefit of the 80+ people following it. (As ever, this doesn't prevent further discussion or somebody else changing the status.)

@wturrell++

Sounds like censorship to me!

This issue is about updating core with the Update Manager. It doesn't have anything to do with auto-updates specifically.

I wish I did have time to get back to working on the patch I started trying to figure out a year and a half ago (see also the comment below that one). I looked into it a bit at the time, but never had "real" code to share. I probably won't have more time for a while, if ever :( But there are other patches/approaches in this issue that others could still pick up and work on. There are some challenges to overcome, but it still seems like a technically achievable feature to me, for both Drupal 7 and 8.

@fatkinson: Please do not try to read too much between the lines here. "censorship?" It's all opinion here. Nothing more. The point is, there will always be this very soft line between a user only and a contributor, sometimes the same person, sometimes half the same person, who tries to use Drupal and who tries to contribute back to Drupal or tries to get something special out of Drupal. In all cases some more, some less. The momentum something gets, is based on the critical mass seeing a certain importance in it or sees any chance to fix it without loosing years in life. There is no censorship here despite of life itself, since noone belongs more to Drupal then the other here in the meaning of not being able to force something in any way. As you can see the issue reporter does not agree on the suggestion of somebody who thought it would be best to close this issue in faviour of another before the comments get cluttered. Or saying it the other way around: Drupal belongs as much to you as you decide. The more you contribute and try to get in contact with others of same interest as more you will push things forward for everyone and as part of it also for your needs. @David_Rothstein did a lot of work on this and has no time yet to go on with this. If you think it is needed than feel free to help fixing this issue. Nobody will get in the way, despite of saying an opinion, which is everybody's right to say here.

'I am going to mark this issue as closed' suggests that one person is trying to stifle the conversation.
It doesn't flow along with free speech!

'I am going to mark this issue as closed' suggests that one person is trying to stifle the conversation.

No it "suggests" to close the issue. Nothing more. And he actually sad:

I'm going to mark this issue as closed for the benefit of the 80+ people following it. (As ever, this doesn't prevent further discussion or somebody else changing the status.)

Well ... I think it makes no sense no more to hook into this here with you. Feels like thread trolling.... I already tried to explain and you should realize that you're still able to write more on this issue here. Not sure what free speech has to do with an open source code issue queue. Do what you want with it. I think this conversation leads to nothing from my side no more and I will unfollow this because it still pings me all the time...

Have a good week.

Guys & gals,
I've more or less done this now with this module!

I don't want folks to go installing it on their important production sites and finding bugs that way yet, so it's a sandbox project for the time being, but if you're experienced enough with Drupal to have places where you can test it, I really could use testers!

The existing implementation should already solve most of the hard problems outlined by @dww waaay back in #4 (9 years ago!) I won't reiterate implementation too much in this comment, but in short it doesn't depend on core while the update is in progress and the thing it downloads is more akin to a set of patch instructions than a full release archive, which also simplifies a lot of stuff.

Also I know that there's obvious missing functionality. The roadmap I've got in mind for next steps, in order, is

1. Rollback support -- in progress
   This will leverage the same basic code used to perform the update in the 1st place, by just generating a set of "patch" instructions that are the reverse of the ones it's making as it goes, and stuffing this in private files.
2. Drupal 8 integration. Shouldn't be that big of a deal since all the heavy lifting is done by code that doesn't depend on one or another core.
3. FTP / non-writable filesystem support. The important pieces are already written and tested, I just need to add UI and/or use existing Drupal UI for configuring the connection you want to use.

Well, I took a course in Wordpress and will be implemeting it shortly.
And the single reason I am making this change is that it doesn't supoort automatic core updates.
I am glad if everyone else is happy with it as it is.
But for my purposes, I am moving on.

Thanks @mbaynton! I have tried your Web Install Kit module to update Drupal 7 core in browser. It worked perfectly, and I could update from Drupal 7.40 to 7.59 via the GUI.

I can't wait to test it for Drupal 8, so don't hesitate to post updates here if you add that feature.

Also, could updates be triggered and applied via the hourly cron job? That would be a simple way to create an automatic Drupal core update, something like this under admin/reports/updates/settings?

Apply updates automatically
[ ] None
[ ] All newer versions
[x] Only security updates

PS. I had to turn off NFS in Vagrant/VirtualBox for it to work, but that's probably a quirk in my system.

@mbaynton - Interesting solution. I'm watching this one. Thanks for putting time into this.

Too bad that this issue got cluttered by complaints and off-topic. And I am sorry for having made myself getting provocated feeding this too. Still an important issue. Maybe let's wheel back to the most important phrase made in this issue by @webchick at #37 and let's start from their to plan road maps?

love to have this. A huge win for security.

I've released a beta module for Drupal 7 for this 🎉
https://www.drupal.org/project/core_update

If this feature is valuable to you please give it a try and provide feedback in the module issue queue (including if it's "it worked"). I'm really curious to see if there are hosting environments out there that break it in unexpected ways.

This is in a way the culmination of like 3 or 4 years of work for me. If there's robust demand I'll definitely support it and add features based on strong feedback, but am hoping to free up time to participate in Drupal 8, composer-aware solutions now.

@mbaynton, Nice! I like the UI integration and the fact that others are working on this. I have a solution I had offered that takes a different approach (https://www.drupal.org/project/dfm_autoupdater). It seems just about any attempt to do this requires a key factor. With yours Drupal core files must be writable by the webserver. With mine, it requires SSH access to your host. I think D8's requirements are making it less and less compatible with cheap, shared hosting, so I don't know if such requirements are things that need to be considered much for a D8 solution in the future. I'm not sure, but I think both of our approaches could probably be ported to work on D8 pretty easily and allow composer commands. But I like the ideas that have been presented here and elsewhere to make this work without needing to know composer or technical commands. Either a scheduled system or click-based system would be best for general adoption.

Thanks @DigitalFrontieresMedia.

Regarding FTP, for what it's worth, early on when I was working on this I put all kinds of effort into FTP support. The automated tests run in an environment with an ftp server, it can deal with servers that chroot the logged-in ftp user to some other path than the one Drupal sees, is smart about issuing reads to the filesystem and only writes to the FTP server, etc. But that was all premised on the Drupal community's long tradition of yelling loudly at people who configure their filesystem to be webserver-writable. It was pointed out to me that WordPress automatic updates require a webserver-writable filesystem, and everyone that wants a website with effortless maintenance wants WordPress automatic updates to work, and so the low-end webhosting industry has probably by and large gone that way. To finish off FTP support in my module would mostly just a matter of piping FTP connection parameters from the Drupal UI to the backend Curator engine, but I haven't finished it cause my hunch is the target audience probably doesn't actually care. Am open to be proved wrong.

@fatkinson, I understand you don't care much for Drupal, but there's an Unfollow Issue feature it offers that you might be a good candidate for ;)

Congratulations on releasing the Beta! I just tested the Core Update module, and it works almost perfectly. I got an error, but it didn't stop the update process from completing. I have created the issue #3038337: Database error, but update succeeds about it.

Drupal core updates via the GUI will be extremely valuable to the Drupal community. A cron triggered option is still on my wishlist (#170), to enable automatic Drupal core updates. Just as @DigitalFrontiersMedia also mentions:

Either a scheduled system or click-based system would be best for general adoption.

I am looking forward to your next endeavor with a Drupal 8 Composer solution, I see the Composer requirement as the biggest obstacle for wider Drupal 8 adoption.

"It was pointed out to me that WordPress automatic updates require a webserver-writable filesystem, and everyone that wants a website with effortless maintenance wants WordPress automatic updates to work, and so the low-end webhosting industry has probably by and large gone that way."

Didn't know that but makes sense now that I think about it.

@ressa, my approach works off of cron if that's an interest of yours. However, it does NOT have an Update Manager UI interface like mbaynton's. Feedback is through watchdog logs.

Thanks for pointing that out @DigitalFrontiersMedia. If I have SSH access (which I have to all my web sites) I back them up daily, and cron trigger drush up --security-only -y every hour with https://github.com/thenewgroup/auto-updates/. If the script updates anything, it sends me an alert email that the site has changed, and which modules or Drupal core have been changed.

These are not mission critical web sites, so if the update fails (which it hasn't for over two years, running on several Drupal 7 sites) I can always restore yesterday's back up. I did get a warning after the update to Drupal 7.63, but the updates went through nevertheless.

@ressa, cool. I see. Yes, my approach is basically the same idea. I just had different servers and this helped standardize executing a cron script across them without having to worry about changing PATH, permissions, etc. for cron directives to execute properly. Thanks for sharing the link to that script. Useful!

Sorry just found this issue

We are now handling this in the Automatic Updates initiative

The first merge request to add the MVP, patch level core updates via Composer his here #3253158: [PP-1] Add Alpha level Experimental Automatic Updates module