Plugin Manager in Core: Part 3 (integration with installation system)

First run at these wireframes:

Feedback and skitching requested

http://pajamadesign.com/Install/Extension_browser.html

Best,
J

Interesting...

FYI: http://drupal.org/node/395472#comment-1374054

Is the link to the proposal.

I'm 2 minds about it. Right now, the d.o. interface is pretty imposing way to find what you are looking for.

BUT

d.o. can be made more streamlined and more like drupalmodules.com
and
I prefer to get something up quickly, and the proposal you have put down is not bad at all. Pity I made such nice wireframes :)

So I'm willing to ditch my dream of Synaptic for Drupal and go with the original proposal. Maybe you can outline the changes on d.o. we need from a code level, and we can start messing with that.

I don't like the md5 solution though. Peter Wolanin, Joshua Rodgers (now and Acquian too, like you:) ) discussed this today on the scrum, and we're keen on using openSSL to sign packages and verify their contents.

This would mean storing a private key on d.o. and sending the signature in update.xml feeds or along with the link as you specified in the proposal so it could be verified locally.

I've been thinking about this as well. There's so many modules on drupal.org that we're best of focusing on one interface to find them and optimising that for more cases - we also need to handle theme install. Also I hate synaptic, better than downloading a tarball from somewhere, not as good as apt.

However, I think once the 'given a list of modules, ask for credentials and install' workflow is figured out, we should look at a way to browse the Drupal.org project pages from within core.

What I'd love to see is this:

You install core's default profile.

Overlay pops up with features/packages/(run-time install profiles) - you can select those and pull them down or just dismiss it. You have tabs for themes, modules, translations etc..

You can get back to this screen later to add extra packages, themes or modules. updating is handled via the modules page (or /admin/status/update).

The workflow you get is similar to what cwgordon mentioned - flag a module for download, grab them when done - but we display it from the Drupal site rather than taking people offsite and back on again. We don't want to store any uninstalled project information in Drupal itself - update status has enough issues doing that just for installed modules. We'd need instead a pretty basic webservice, or a glorifed iframe with some custom theming, and the only bit in Drupal is a way to browse this once the basics of the install interface are in place already.

While typing this I realised I'd pretty much described the firefox addons interface. Note the link to 'browse all addons' which could take you to drupal.org/project and then do the same as cwgordon's idea.

@catch:

We'd need instead a pretty basic webservice, or a glorifed iframe with some custom theming, and the only bit in Drupal is a way to browse this once the basics of the install interface are in place already.

I'm confused. Do you support browsing in d.o. and sending the user back to their site to install, or do you support building an interface based on a webservice (like the wireframes I threw up)?

If we go for browsing on d.o, I agree it isn't a great user experience to see "Install" links which leads you to a page which says "already installed".

What about if we did something like this:
http://img.skitch.com/20090617-cghfkcw4npxksbw7xtapyhswtp.jpg

Where we use d.o. for browsing, and we provide an overlay using JS when the iframe's location matches a regex. So if they are at project/(.*), we look and see if $1 is installed, and then show the overlay. If we want to be cleaner, get the d.o. team to put something in the Document that we are allowed to look at which will tell us.

Of course, there is no way to degrade this nicely. I guess we could try to support cwgordon's original idea as a fallback in case the user doesn't have JS installed. This means telling d.o. that ?site=idontusejs.com and then showing install links on all module / theme pages which take them back to their site. If they clicked an install link, and had the module/theme installed, it would just tell them.

Here's a plan going forward, please let me know if there is any objections, would love to get this rolled out pre-DC:

Critical path issues

1. Build the browsing page, and the JS callback. I think it is best to just load the list of installed modules as JS vars when the page is loaded
2. I don't think we'll be able to respond to click events in the iframe due to domain restrictions, but we can poll for iframe.location in a loop with a slight sleep. If the project/(.*) is installed, show an "already installed" overlay, if it isn't, AND the module / theme is compatible with the user's site, then show the install overlay with the link
3. Implement the "downloading" and "installing" parts of the workflow (roughly based on the wireframes above)
4. Figure out the key and package signing debacle (perhaps a separate issue?), until this is done, show a warning that installs are insecure (which we need anyway for people without OpenSSL).

Other stuff that would be nice to have, and probably needed at some point:

1. We will need collaboration with the d.o. team to provide a special stylesheet to users which is optimized for finding modules
2. The "install and go back to site" button as cwgordon described for people without JS

What do ya'll say. Should we go for this?

subscribe.

Looks cool!

We need ot have a sit down though. We're approaching the "massive duplication of effort" zone. I think we need to get the d.o. infra team together with me, you, chx and probably JoshuaRogers to get a plan in place before we go coding beyond this point.

I'll be posting a patch to part 2 today which will need a lot of love, but handles the authentication stuff.

Best,
J

Based on just the screenshots I'm not at all keen on the select lists for actions - can we leave this as just 'fetch' and then rely on a better modules/themes page for enabling stuff? I count at least 5 differently named actions between themes and modules in those drop-downs, I don't think we need to provide this in an initial patch, just select things, hit button, pull them down, job done - then we can focus on streamlining install/enable/default elsewhere.

There's plenty of situations where a theme or module will be impossible to enable - i.e. if it has unmet dependencies, that's not for this issue.

Subscribe.

In #14 "There's plenty of situations where a theme or module will be impossible to enable - i.e. if it has unmet dependencies, that's not for this issue."

leaving out the drop downs (for actions like download and enable) makes sense to me, and later if it is really needed/wanted can be added back in if appropriate.

So lets start with laying out the interface components. What components are there, which problems do you see already? What steps are absolutely necessary?

The interface as proposed, will unlikely be the one we will use though. How can you not get a Md5 confirm?

I'm really in favor of using openSSL and key signing (installed on about 85% of hosts connecting to the Acquia network). It is safer, more usable and more standards compliant (close to the gpg keys used in GNU/Linux repos). For the remaining people, just a warning message and instructions on how to get SSL setup is good enough IMO.

-J

I'm not talking about an HTTPS connection (although that is also good). I'm talking about signing and verifying packages. See:
http://in2.php.net/manual/en/function.openssl-verify.php

I'm in.. (Subscribing)

Subscribing.

We need a status for "has syntax review, but no functional review."

I just spent 15 minutes and found no spelling errors or obvious syntax errors for the patch in #17. Short, sweet, and very well written code.

Josh

Take a look at the wordpress plugin manager.

Here's how it works:
1. You search.
2. You click install
3. You get an overlay with a description of the plugin and what it does, reviews, etc
4. You hit install, and it asks for connection deets, it installs it.

One of the goals here IMO is to make Drupal less of a PITA to deal with for newbies. If we don't solve dependency checking and requirements checking here, we've missed big.

However, that being said, I'd like to get something more than nothing, and if fetching is all we can manage, that's also cool for a starting point.

I would like to get dependency checking and resolution going though, because in the updater, we will need it anyway. (If a new version of a module has a new dependency).

-J

Hey everyone. Check this out: http://drupal.org/node/509010#comment-1771058

It seems that Dries is not considering signing a prereq for this. My comments are inline, Joshua also had a valid comment.

I'm a little cautious here, but I also agree w/ Dries that it is a "nice to have". We can start with a version that just warns the hell out of people. At the end of the day, the only thing which should be able to introduce a trojan is a MiM attack or DNS poisoning, and it's just not common enough on the public internet IMO to delay getting the feature in.

For paranoid users, they have options. That being said, we should get signing in if possible.
So if we ditch the signing component, how does this change our approach?

If anyone hasn't seen it, check out WP's version:
http://skitch.com/jacobsingh/bsh6t/install-plugins-grabbing-hanumans-lon...

This is an approach where d.o. acts as a simple web service. See Joshua's request for this service: http://drupal.org/node/363466

I think browsing d.o. is a nice idea, but we need to have more practical ways of "get 'er done" IMO. If we can't read the iframe, then what? We have to tell d.o. exactly what modules we have installed and have it store it in a session? Or we have to have d.o. make callbacks to anonymous random URLs? I don't really get how we can do it yet. Not that I'm against it, as stated earlier, I like the idea, but it seems like it might end up holding us back vs implementing solr as a web service and a simple search box w/ JSON backend that might take an two afternoons to code up.

cwgordon's screenshot looks pretty close, although I agree the user doesn't need to know about fetching, etc. But everyone should take a look at the WP installer. I see two key differences:

1. No batch install. Just installs one module at a time. If we can do dependency resolution, I also favor this.
2. Rich meta data and ratings to help people choose. Nice!

Can't wait to see the next iteration!

So I know the UX is going to be a large deal breaker, but let's see if we can get a nice experience without killing to many developer minds.

1. User should only need to provide FTP/SSH information, all other information should be done programmaticly.
2. Intelligent module browsing, it makes no sense to browse d.o in an iframe - let's use the information of the system to enhance the browsing experience.
3. Our experience should exceed Wordpress. - hey, you mentioned them! :)

So I think @JacobSingh #27 is absolutely right, with the need for a lot of stuff to be refined and worked on.

So one, admittedly ugly, idea for dependencies, but which might not require huge dependency mappings on Drupal.org would be this:

Download a module.
Check the .info file - if it has a dependency check if it's installed or enabled locally.
If not, download the dependency - probably this would need a message/confirm form - 'pathauto requires token module, you will need to install this as well, [continue]".

In the case that a dependency has an unmet dependency, keep repeating until it's done - hopefully that would be quite rare, and at least the work is being done for you otherwise.

@30: IIRC, there is currently no way to find out to which project an uninstalled module belongs. So for any type of dependency checking, we'd need to parse all info files on drupal.org, build a module - project mapping table and then have a way to post a module name to drupal.org and get back the name of the project to which it belongs.

here is a new patch?

Frando, makes sense. Seems like we could still do a 'best guess' on module dependency = project dependency (for views, panels, cck, token) - and manually prompt if that fails. And/or add a 'project-dependency' key to .info files for modules to specify the package they depend on as well as individual modules from those packages. The main thing for me is what we can get into core in the next 6 weeks or so, vs. what can be tidied up from Drupal.org itself later.

I'm with up Bojhan for the UI: Browsing d.o. in an Iframe does not really help and relies on how people do their project page - oh my. Instead pulling in the relevant information gives a consistent look and lots of possibilities.

I am a big fan of the Firefox Plugin Manager

They manage to cram all the relevant information in one little popup. Note the flooded state harmaonica for Detail view per module. For us all the "see all plugins, see all results, go to project page" links have to be treated with more caution.
Because sure, Firefox has one giant advantage: No matter to which webpage you browse, you can always install a plugin right away, because you are carrying your system with you. Still: This is a sensible thing to include, especially the viewing of other search results and go to Project page.

We don't need a popup for plugin manager, going to all external pages with a target = "_blank" always keeps the plugin manager open. But they manage to fit it into so small a window and have all relevant information and controls at hand. We should mange to get to that.

It's quite similar to the mockups provied in the group discussion: http://groups.drupal.org/node/17765#comment-60950 Steve JB and Skilip were quite in the same boat on not using an Iframe, as I understand it.

Has anybody considered potential impact on the d.o infrastructure?

I'd think browsing and installing from Drupal installs is likely to involve less drupal.org requests than browsing and installing on Drupal.org itself - obviously it depends how it's implemented, but seems possible to at least allow it to have neutral impact.

Yeah, it's going to require some brains and some effort. But it's going to be awesome, and it will transform the Drupal economy IMO and I think actually provide a massive cultural change in the quality of modules.

The question should be "if" it should be "how" at this point.

I see three options:

1. We do apt-get style. User downloads a tarball of all the packages and all the info about them for their Drupal version. They update this every now and then. We build a crappy search engine to search them or try to use core search in some weird way.
- Pros:
-- No server load, can be distributed.
- Cons:
-- Not as current
-- No rich data, no user contributed data available
-- Not nearly as effective search as solr

2. We build the infra on d.o. Project Solr as a web service basically. We can make it as complex or simple as we like. We have to implement some kind of rate limiting and caching mechanisms, but nothing more than d.o. already needs.

- Pros:
-- Rocks.
-- Very relevant, up2date results.
-- Rich search exeperience

- Cons:
-- Central point of failure (d.o. infra)
-- Requires scaling out solr servers and maintaining them
-- Probably means an increase in search traffic (but who knows, maybe it will be much less).

3. We build a distributed webservice of Solr indexes. d.o. acts as a solr master, has an index just for project data. Various companies and individuals volunteer servers to run Solr and perform searches. They replicate, pass back to a group of d.o. balancers.

- Pros:
-- Everything in #2
-- No single point of failure
-- Community contribs from businesses, d.o. doesn't need to deal with it all.

- Cons:
-- Complex to setup
-- Have to get buy-in from partners
-- Will take awhile to get going.

Hope that helps.

Best,
J

I just completed a very critical review of this implementation of plugin manager which is far too in depth to post here :

please check out http://groups.drupal.org/node/24709

My take-aways from Adrians rather long post:

1. If a module breaks your site, plugin-manager does not provide a safe enough box that can "push through" the problems of your implementation (I.E. - it doesn't have the site offline, different theme, works without database feature)
2. Cannot opt-out of functionality
3. Only works with one version of a site by design, pushing people to bad practices
4. Regarding point #1, this hostile environment is not actually safe to enter credentials
5. Unsafe because passwords are not encrypted.
6. ssh2 extension is not commonly available and those that could set it up will not be using Plugin Manager
7. Not multi-site aware
8. Replacing original packages in sites/all/modules can have huge impact on all sites. Perhaps replace original with sites/[site]/modules instead
9. To deal with multi-sites, this system needs to be far more intelligent about Drupal's environment
10. Only Drupal.org sourced packages are managed, there is no way to "plug-in" another site for managing custom module releases
11. Can wipe-out any patches you made with no easy "undo" action.

How Adrian proposes we fix it:

1. Separate Plugin Manager from core in every way possible
2. This includes not fully bootstrapping Drupal
3. Get rid of false security by going to the "web writable" route or at least removing ss### extension
4. Use a web writable 'overlay' search path by implementing a new folder called contrib that mimics the sites/ folder and is seen as a staging area
5. Provide simple killswitch for Plugin Manager
6. Make an exception on code freeze for this system
7. Adrian promises he has ideas on the "many copies of the same module" problem and other issues, but this should be a good start

Adrian apparently also posted a link to his comments at #395474: Plugin Manager in Core: Part 2 (integration with update status) and there is already much discussion.

Hope that summary is true to Adrian's post and helps those of us with less time ;-)

Josh

This issue is getting a new home:

#536630: [Meta issue] Plugin manager for D7 code freeze spec.

Best,
Jacob

Adding the "Update manager" tag so this (and other parts of this "Plugin Manager in Core" issue set) can be triaged again now that #538660: Move update manager upgrade process into new authorize.php file (and make it actually work) is in.

Has this task been forgotten? I ask because it came up at #336743: Port Plugin Manager module to D7.

Setting back to active since it's been almost a year since any progress, and a lot has happened on D7 since then.

...subscribing

Sorry, way too late for this now.

This is a feature, not a task. Major tasks are now preventing development of new features per the new guidelines at http://drupal.org/node/1201874. Despite JacobSingh's post in #40, this issue is still relevant as it basically deals with direct collaboration between drupal.org and individual Drupal sites. Other issues have been marked fixed, but this issue clearly has not yet been implemented.

Sounds like this part of core is pretty much not maintained.

Should we close this in favor of Module/Theme Browser:

#1243332: Deploy Project Browser Server and drupalorg_pbs on d.o
#1248606: Include drupalorg_pbs for deployment on Drupal.org

http://drupal.org/project/project_browser_server
http://drupal.org/project/drupalorg_pbs

@klonos I think this issue is about installation profile tweaks?

Not sure. I was simply asking because I saw from this issue's summary that it aims to bring Plugin Manager in core. From the project's page:

The plugin manager is designed to provide an in browser interface to administrators to install modules and themes. It automates the process of downloading a project release, extracting files from it and installing them in their appropriate locations.

Then I also saw the screenshot for the UI suggestion back in #34 and all this made it sound very similar to the way the Project Browser demo works.

Oh dear. I'm sorry for spamming. The term "Plugin" is too overloaded. This stuff here is not related at all to the Plugin system in D8.

Let's change this implementation and also issues to Extension manager, pretty please...

YAY!!!!!

shall we mark this as duplicate and close it?

Yes. Duplicate of #1841788: Add project browser