Facilitate use of Spark components in other distros and non-distro sites

Moving to Spark; closing down Phoenix. See http://buytaert.net/announcing-spark-authoring-improvements-for-drupal-7...

Interesting. Do you know if there is any documentation about this anywhere? The only standards related to interoperability between distributions I was aware of is the http://drupal.org/project/kit specification, and http://groups.drupal.org/open-app-standard for Apps.

But we certainly would ideally love for Spark to be usable as a base installation profile, so +1 on the general concept.

Yes, it would be great to see Spark components easily integrated either as stand-alone site components or as part of other distros.

The Profiler project is an interesting attempt but hasn't gained a lot of usage. I looked at it for Open Outreach and decided against it.

There's a recent initiative to allow different distros to inherit each other, see #1356276: Allow profiles to define a base/parent profile and the blog post http://www.agileapproach.com/blog-entry/inheriting-your-drupal-profile-e... (see also #1713516: Provide Documentation on How to Use Panopoly as a Base Distribution). While that would be useful, I don't know that it's really required here.

The most straightforward way would be to package Spark components as Apps, given that (a) a growing number of distros are apps-based, see http://www.agileapproach.com/blog-entry/drupal-firms-are-making-their-di... and (b) using Apps would facilitate integrating Spark on sites not based on distros.

Yes, there are lots of remaining problems with the Apps model. I outlined a few here: http://chocolatelilyweb.ca/blog/drupal-apps-answer-plug-and-play-drupal-.... But one way or another we need to tackle those problems.

I really hate the apps model because it seems to fork the concept of modules, yet again, when we already have modules + features that do essentially the same thing. I really wish we'd just make improvements to Update Manager in core, so these nice usability enhancements could help everyone.

The way Spark has been architected, each piece is a separate module that can be added as a dependencies[] line separately for people who want or do not want that functionality. Want inline editing w/ Aloha integration? Add the http://drupal.org/project/edit module. Want a sexy responsive Panels layout builder? http://drupal.org/project/layout will get you there. We've also taken over http://drupal.org/project/dashboard since it was abandoned, and hope to make some nice improvements there as well.

Can you explain what else beyond this is required for interoperability with other distros?

Yes, definitely we need a broader discussion about apps and if/how we can get it or some equivalent set of solutions into Drupal core and drupal.org rather than e.g. pushing key schema data onto disparate branded servers. We've now got a lot of the potential building blocks on d.org, like http://drupal.org/project/project_dependency, which would be a much better way of answering the question "what dependencies are needed for this app/feature" than the makeshift apps approach.

Can you explain what else beyond this is required for interoperability with other distros?

It's great that you're pushing Spark development into standalone modules. If as a distro Spark simply enables those modules, then any other distro can easily do the same and we're done.

The main remaining questions are around integration of these modules. To work out of the box, we'll need things like roles created, permissions assigned, text formats created, and so on. Does any of that or other scaffolding go into Spark? If so, does it make sense to facilitate reusing the integration directly, or is it more of a model that other distros could choose to follow?

Can Spark help promote consistency in the way its contributions are integrated?

Ok, right, I see what you mean now.

So integrating the WYSIWYG and Layout stuff did result in a couple of things that need to happen:

1. Layout requires jQuery 1.7, so we set some variables related to jQuery Update module at http://drupalcode.org/project/spark.git/blob/refs/heads/7.x-1.x:/spark.i...

2. Below that, we needed to make some adjustments to Filtered HTML (add some additional tags to be allowed, enable+re-weight another filter).

We'll likely be doing more stuff with default configuration of Panels + Panels IPE + Panelizer and so on, which would also go into here.

I'm not sure we can really generalize this into the Edit/Layout modules themselves, because not all profiles are necessarily using the same set up as standard, so we can't make assumptions about even basic things like that an input format called 'filtered_html' even exists. Therefore, this felt more "right" at home in spark.install, and that's where this code is.

If I were another distro author, I would just copy/paste spark_install() and take a hatchet to it, since that's basically what I did to standard.install to get what's there now. So from that perspective, I think yes a distro author would have whatever they needed. But I don't claim to know anything about best practices. :D

I also made a separate module, Spark Demo (http://drupalcode.org/project/spark.git/blob/refs/heads/7.x-1.x:/modules...) which as the name implies creates example content for the Spark distro. The only generalizable thing there that people might be interested in doing would be having the ability to create content types with one of every field on it, but that generalized functionality is in http://drupal.org/project/every_field, a separate contrib module.

Is this supporting distribution re-use enough, do you think? Otherwise, is there a way I can not drink the Apps kool-aid, but still support others who want to use these modules in their own distros? A proliferation of "bootstrap_foo" modules or..?

One way to think of it is this: apps - if you choose to use it - is just a wrapper around features. I recently added an apps wrapper around all the Debut features, only because people trying to use the features on their own (not as part of a distro) were having a hard time with downloading and installing all the dependencies. In certain cases, apps makes that easier. But, unless I'm forgetting some minor detail, doing so required absolutely no changes to the existing features, which we'd already written with interoperability in mind.

So, no, you don't need to drink the cool aid!

A demo module is great--if you're a developer. You can clone the demo, strip out help text and such, make some changes, and you're away. But of course it doesn't help you if what you want is to turn it on and use it.

What I'm thinking is, rather than a demo, modules that you can just turn on and have the stuff work. They can be features modules if you want to use features. If not, they could use Drupal APIs directly.

So, maybe, spark_edit, spark_layout, etc., packaged as separate d.org projects. spark_edit would include an input format and so on.

Any distro author wanting to use spark_edit can then just edit their drupal-org.make file and add it as a dependency. And if - heaven forbid! - someone wants to sully spark_edit by including in an app, well, they're free to choose their own poison.

If you're using Features, the only place it gets really dicey is if you want to use components that many different features will need to share. Like a role name that you're assigning permissions to. If you include the role directly in your feature, no other feature will be able to do so as well (conflicts). If you instead pull it into a general feature (spark_core) that all your features depend on, no dice either, because now your feature set conflicts with any other feature set that did the same.

Last week I posted an attempt to address this issue, Apps compatible, that's half-baked and may go nowhere beyond Debut, where I've already rashly added it as a dependency for all debut features, but at least sketches in some ideas. The easy alternative is e.g. just to use user_role_save() in hook_install().

Speaking for a moment as a distro maintainer. I'm sick of having to do everything. My interest in Open Outreach is building what's most valuable to smaller nonprofits. I want to focus on that and only that. But right now I can't build my distro without tackling absolutely everything under the sun. I have to worry about input formats and Wysiwyg editors. I have to wade into responsive images for a home page slider thingy because the sky would fall if any site didn't have one that looks spanking on an ipad. I have to integrate blogging and commenting and forums and social links even though none of those are intrinsically specific to the nonprofit use case. Often, it feels like, I end up doing everything except the specific use cases I really want to focus on.

Instead, as I do with other modules, I want to leave the integration to the folks who really know what they're doing. I'd love to contribute the occasional tweak or enhancement if needed. But, please, save me the headache of having to write and maintain another feature module that's tangential to what I'm really interested in. Let me, for once, be the one to just blithely switch it on and watch it work.

Getting #1427752: Support drush make includes[] in drupal-org.make files reviewed and committed would help facilitate integrating spark components into other distros without using apps.

Also maybe useful for reference: Developing Distributions with Panopoly.

> The Profiler project is an interesting attempt but hasn't gained a lot of usage.

Curious how you're measuring usage? We did an informal survey about this, and if I recall, 50% of install profiles written were using Profiler. You can't look at Drupal Usage statistics for it, because it's not a module. :)

Wow, nedjo! Thanks so much for the very helpful write-up. I'll give this some thought.

q0rban! :D Hello!

Are there any good docs about using Profiler as the basis for inheritable installation profiles? Cos what I'm seeing here is that there are a few classes of different things:

a) The base modules (Edit, Layout, etc.), which should make no (or as little as possible) assumptions about their ultimate intended use. These are handled with normal dependencies[] = lines, and any distribution can do the same.

b) Setup code required for these modules to work properly, but that are difficult or impossible for the module to boldly make assumptions about on existing sites, so they can't really go in module_install(). A good example of this is adding <p> <br> <img> to filtered HTML allowed tags. This is easy enough for a distro to do in hook_install(), though, and generally speaking all distros will want to use this same set up code. I think this is where profiler could come in handy?

c) Demo code, which is fairly specific to a given distro (e.g. Panopoly's vegetable picture meme in all of its content, Spark's "every field" default content type + node to demonstrate inline editing). Nedjo (I think) is suggesting moving these to separate contributed modules; I've just been shoving them in the profile/spark/modules folder directly cos I don't figure anyone but us is going to want them.

Also, Nedjo, I love your vision for distro authors being able to stick to what they love. I just want to figure out a way to do it that's compatible with Drupal 8 core, since that's Spark's intended destination, which means no Apps and no Features (and no Profiler either, but I'm curious where it fits :)).

@q0rban: I stand corrected :) I'd done no more than glance through a few of the more-used distros looking for Profiler usage. I've opened #1727488: Document distros/intall profiles using Profiler.

@webchick: The key here is handling of exportables, in two places: (a) the generic modules that come out of Kit (Edit, Layouts, etc.) and (b) any code that Kit produces to integrate those generic modules.

To clarify from comment #10, it's your point (b) - not (c) - that I'm suggesting could go into separate drupal.org projects. For example, a spark_edit project might ship with an edit-specific text format (since relying on there being a filtered_html format would miss many distro-based sites that don't run the standard install profile) and accompanying role and permission handling. This way edit could be easily integrated on any site or into any distribution, whatever the content types or fields used.

Here are some relevant handbook pages. Might be good to push some of this discussion into enhancements to those pages.

• Exportable configuration.
• Installation profile utilities.

Re user roles in particular, see the handbook page on exportables and user role IDs in features.

The ideal - but probably huge - enabling work might be to first tackle the question: what will D8-style exportables look like? I sketched in some distro-related questions and issues in a g.d.o post on the configuration management initiative and Drupal distributions. The answer might include a backport of the file-based configuration that went in as part of the configuration management initiative.

A ton of work? Absolutely, and I'm not in any way suggesting you take it on. But - if you're trying to map out a D8-ready solution that's not using any of the D7-specific contrib tools (features, etc.) - figuring out what the CMI means for distros and exportables is a challenge that's staring us all in the face.

I think if a larger pattern is ever established within the distro community that's compatible with the way core does things ever emerges, we'll be happy to follow that. Barring that though, I think the way we've set up the distro reflects current standard practices and we don't intend to do much more engineering here, so we can keep focused on D8 instead.