Branching off of #903730-56: Missing drupal_alter() for text formats and filters webchick and I discussed on IRC the state of Drupal's support for Distributions. Some of the remaining issues concerning distro support are of high impact for building distros, minor impact on Drupal's code base and could be justified as critical issues for a Drupal 7 release.

However, there is an understandable concern that addressing one of them could lead to an expectation that all of them need to be addressed in a situation where our most important goal is to get Drupal 7 out the door. To get a better common understanding of what these remaining open issues are, webchick asked me to compile a prioritized list of them. So here we go.

This list contains all issues that I am aware of as a Drupal 6 distro maintainer and a Drupal 7 API user who is building sites with a Distro paradigm (installation profiles, exporables, etc).

I might have missed important issues here, you might agree with the priorities I came up with or you might not, please comment accordingly.

I have prioritized issues as Drupal 7.0 release issues following these considerations:

  1. A fix is non-invasive; i. e. it does not change or only minimally changes existing APIs.
  2. There is a significant negative impact by not fixing the issue with 7.0; i. e. a fix in 7.1 has negative repercussions.
  3. The issue is non-fixable outside of core.
  4. The issue is a regression.

Quick summary

  • For Drupal 7.0 we should make sure modules can be tested in distros (1), filter formats have machine names (2.a) and modules are notified of changes to machine names (3)
  • Machine names on roles can wait for a Drupal 7 point release (2.b)
  • A more comprehensive exportable API solution can only be subject to Drupal 8 (4)

1 Testing modules from distros

What is the problem?

#911354: Tests in profiles/[name]/modules cannot be run and cannot use a different profile for running tests

Drupal 7 introduces the ability for tests to rely on installation profiles. This causes problems when a module that should be tested sits in one installation profile directory, but requires a different installation profile to be tested with. Once Simpletest starts the test run it can't find the module it was supposed to test. Many distros place their modules in the installation profile's directory. In fact, this is a best practice as it is the only place a distro has absolute control over. We should not create a condition that discourages this practice.

Proposed solution

The larger question here is whether it is a good idea to have tests rely on profiles at all, but at least we should try to fix the fallout from it and make sure that modules can be tested from within installation profiles. It is clear that these modules won't actually use the installation profile they are sitting in. This fix is about avoiding impractical workarounds and WTF moments for people starting to work with distros.

#911354 proposes a configurable search path for drupal_system_listing(). The host site's profile search path would be added to the test site's search path and thus any module sitting in the host profile search path can be reached by the test site.

Priority

There is no way in contrib to fix this. Some have suggested using symlinks, but that raises practical problems with Drupal's registry not liking switching module locations much. We should aim for getting this into Drupal 7.0 as a) this used to work in Drupal 6 and b) not being able to test modules from within a profile directory would hamper the practice of keeping a distro's dependencies within its jurisdiction where we should be encouraging it.

How invasive is this change?

Minimal, only drupal_system_listing() and SimpleTest are affected. There needs to be more consensus around a solution though.

2 Machine names for configuration

What is the problem?

Sequential ids like role.rid or filter_format.format are not portable. They have a very high likelyhood of varying across systems. If for instance a filter format "Markdown" is created on a Drupal site and then a View using this format is created it will refer to it by its sequential id (for instance "2"). The result is configuration that is not portable. If the Markdown filter and the View are exported on one site and then used on another site, there is no way of knowing that "Markdown" will have the same id on the target system like on the source system, it will depend on which other filter formats have been created on the target system.

Proposed solution

The good news is that there are only two configuration types in Drupal that do not have machine names:

2.a Filter formats (filter_format.format field)
2.b Roles (role.rid field)

The bad news is that especially filter formats are referenced widely by contrib and are thus a major pain point for maintaining distros. The proposed solution here is to do the same thing that has been done with vocabularies: add machine names and change all portions of core that do refer to roles or filter formats to use those machine names instead of rid's and format ids. At some point we should also think about abandoning sequential ids altogether but that needs more consideration.

Priority

Unfortunately there is no good way to work around serial identifiers for configuration.

That said, by merely adding a machine name for filter formats as suggested in #903730: Missing drupal_alter() for text formats and filters we could alleviate the largest remaining obstacle for making configuration portable: contrib could start referencing filter formats by machine names. We should do this for Drupal 7.0. This change is not invasive and not rolling out the gates with it will force many module maintainers to refer to filter formats by their sequential ids.

Roles are not at all as widely referenced in contrib and are therefore not quite as critical and could be subject to Drupal 7 point release. Changing remaining core to reference filter formats and roles by machine name and ids could be subject to a Drupal 7 point release or Drupal 8 depending on how invasive it turns out to be.

How invasive is this change?

3 Changing machine names

Problem

Machine names in core like node.type or image_styles.name can change hence code that refers to these identifiers needs to be notified of changes. Change notifications are widely implemented in core in the form of hooks with the exception of taxonomy_vocabulary.machine_name:

#876762: Modules have no way of knowing if vocabulary machine names are changed

Proposed solution

Make taxonomy_vocabulary.machine_name behave like rest of core (node.type, image_styles.name). This is exactly what #876762 is about. In the context of Exportables in core for Drupal 8 we should review whether changing machine names is a good idea in the first place.

Priority

Vobulary machine names being changeable, but change not being detectable is a straight up bug. Needs to be fixed in Drupal 7.0. Making machine names non-changeable is a Drupal 8 topic because it requires aside from a consensus the ability to copy existing configurations.

How invasive is this change?

The fix to Drupal 7.0 is minimal and follows an established pattern in core.

4 Exportables API in core

What is the problem?

Drupal has many very powerful features that allow for point and click site building: Variables, Content types, Fields, Vocabularies, Views, etc. The problem is that once configured, these settings sit in the database and can't be ported between sites.

What is the proposed solution?

The Views project has established a now very widely followed pattern for controlling change. Fully configured views objects can be exported to a PHP structure and then placed in a "default" hook implementation. Views will recognize the exported configuration as the "default" for the one present in the database. Once exported, the configuration in the database can be completely removed, Views will fall back to the default in code. At any point can the configuration be modified and - if desired - exported again.

This concept has caught on under the name of "Exportables" or "Export to code". CTools has captured this approach in a library (ctools/includes/export.inc) and thus made it very easy for contrib modules to make their configuration Exportable. Features module makes extensive use of exporables to allow for bundling of configuration components into higher level features such as a Blog or an Event calendar.

Note that machine names are a prerequisite for Exportables as they solve the problem of sequential ids not being portable (2).

The concrete suggestion is to implement a CTools/export.inc style exportables API in core and to move Drupal configuration as far as possible onto that API. This will likely include some sort of exportables UI infrastructure that consolidates the typical UI patterns around exportables: display the configuration status, override/revert a default configuration, export configuration (If you are not familiar with these UI paradigms, check out Views' administrative UI).

Priority

Drupal 8. While machine names in core will allow contrib to apply the exportables pattern to core, core will be missing any native user interfaces for managing exportables. To provide a consistent user experience to site builders and site managers, we will have to provide exportables infrastructure in core simply because core will need to rely on it. Due to the invasiveness of this change we won't have a shot at implementing this before Drupal 8.

How invasive is this change?

This will touch all portions of core that manage configuration one or the other way on at least two aspects: loading configuration and the configuration UI.

Comments

DamienMcKenna’s picture

Subscribe.

mfer’s picture

sub

arianek’s picture

This all seems to outline the issues well, and I think Alex has weighed the priority for those of us who would really like these vs. invasiveness/priority for D7 really accurately.

Boris Mann’s picture

Note that for #2, this fixes a long standing way of Drupal core / default.profile doing DB queries for filter formats and roles, so this gets us CRUD. If #4 is implemente, we automatically have some API hooks for CRUD operations throughout the system.

langworthy’s picture

subscribe

Crell’s picture

Subscribing.

cdale’s picture

sub.

bonobo’s picture

subscribe, and +1

Better support for distributions in D7 will help increase Drupal adoption during the entire D7 lifecycle.

webchick’s picture

While subscribing, could you please say "yes I agree" or "no I don't", to give the core maintainers some idea of whether what Alex lays out here is the extent to which we'd have to change core in D7 to support this?

I'll respond to this in detail later, but want to get the gauge of the community first.

Dave Reid’s picture

1. I'm not sold on yet.
2. Sorry these sound like API changes. My hook_tokens_alter() patch was turned down and I have to live with it until D8.
3. Should be fixed - its a valid bug
4. I think it has to still prove itself as the defacto and mature solution. D8 material anyway.

chx’s picture

Version: 7.x-dev » 8.x-dev

This issue has a name and it's extortion. Anyone speaking against will be boo'd at for talking up against this because he is against distros and that's not nice. Bring it on.

Can be justified as Drupal 7 criticals? It can't even be justified as a Drupal 7 issue! I talked to you in person in Paris. That was more than a year ago. Drupal 7 is over.

http://drupal.org/node/927792#comment-3515044

Appeal to Dries if you think I am wrong.

Boris Mann’s picture

Version: 8.x-dev » 7.x-dev

Dave - other than sour grapes for #2, do you have technical reasons you don't like that one?

chx’s picture

Boris did you crosspost or did you just appoint yourself to Dries?

Dave Reid’s picture

#2 is too late. We've had forever to fix those kinds of things. We've lived so far in D5 and D6 with the same downsides. Shipping D7 is not going to explode if #2 stays status quo.

webchick’s picture

chx: this thread was posted at my request. It's staying 7, for now, because Dries and I want to understand the extent to which Alex, et al need Drupal 7 would need to change to support the distribution work. Then we need to figure out what of that, if anything, still is and is not on the table. But we can only do that by first seeing what the scope looks like.

sun’s picture

1) Having faced testing problems in contrib (primarily http://drupal.org/project/edge), I see a clear requirement to allow testing. A range of tests in core only work, because core is able to add testing stuff anywhere. That doesn't work from anywhere else, so this is a straight bug and entirely prevents testing. The proposed solution in that issue is over-engineered IMO, but I'm confident that we can find something more staightforward that just cuts it.

2) Can be solved. Easily. The API change is minimal. Learned something new today. Details ain't important for this issue.

3) Sounds like a straight bug to me. Whether renaming is good or bad or whatnot is for D8.

4) Definitely D8.

In short: Bug, Doable, Bug, No. Overall, basically agreeing with outlined proposal.

auzigog’s picture

Subscribe

langworthy’s picture

I'm probably missing something with #1.

I can currently test an install profile in D6. It seems I may not be able to do so in D7. Can someone explain if this is not a regression?

DamienMcKenna’s picture

I thoroughly with anything that can be done to make Drupal fully exportable at the data layer, CTools / Features et al can fill in the gap until D8 but for D7 we need the data structures in place.

q0rban’s picture

subscribe

gdd’s picture

#3 seems like a valid bug to me, although I am unsure why we allowed changing machine names in the first place that is neither here nor there. #1 probably is too. However valid bugs get pushed out in dot-releases all the time and I don't see any reason at all to hold up 7.0 for them (uness there is an implementation detail preventing it which is possible as I'm not 100% familiar with the solutions here.)

#2? Sorry time's up. Anything that was broken in D6 is not critical to fix for D7. We established this over and over, and it shouldn't change here. I can't get behind that at all personally.

#4 Completely ignores the needs of content deployment which is needed just as urgently as anything else (Dries has called it out in Drupalcon keynotes twice now.) I want to see an Export API in core as much as anyone else but I would prefer to see a framework in place that addresses the needs of exporting anything in Drupal, not just configuration, and while this may not be possible I'd like to see it at least tried or even considered. As everyone has pointed out this is a D8 issue anyways, so more discussion to happen down the road there, but I just wanted to make sure we don't appear to have unananimity on this solution.

As long as we're spouting exportable dreams for D8 we should add #5 Ability to generate UUID or other non-serial keys.

arianek’s picture

Hey - more to say (fairly much +1'ing heyrocker's comment...)

I just had these issues explained to me much more extensively by several core folks, and now seeing a bit more clearly which are truly considered "bugs", my support for #2 is waning. I am totally in the camp of exportables-palooza, as it's a far more efficient and safe way to work (and what we use at work, so I do have a vested interest), but I am also torn in two by the fact that I know D7 needs to launch and that it's very late right now (and that it's really uncool for all the people who have been working so hard on this).

Echoing Sun, I think #1 and #3 do need to be fixed, #2 is questionable, and #4 (like Alex said) is certainly a D8 issue, not D7, and certainly could use some more fleshing out if it was to contend with content as well as config.

And from what I understand bugs (1 and 3) can certainly be fixed in minor releases, and possibly also #2 (small API changes) according to http://buytaert.net/8-steps-for-drupal-8 (step 6) it looks like that might be accepted?

So to summarize, yes 1, 3 and maybe 2 should be fixed in 7, but IMO, they do not *need* to hold up the release if it's an option to fix them post-release.

eaton’s picture

I'd agree with Sun's assessments in #16 -- especially if the input formats issue doesn't require as dramatic of a shift as we had previously thought. I'm eager to hear what his thoughts are. With that in mind (ie, if input formats are a relatively easy win) I also lose my only objection to heyrocker's comments in #21.

I agree vigorously that pervasive "exportables in core" will require tearing up a lot of underpinnings, and there is no excuse not to do the heavy lifting of adding UUID support. I don't think they're in opposition to each other, as 'exportables' deal with capturing database configuration in code while UUIDs give us the fundamental prerequisites for effective content deployment. Frankly, if we can fix #1, #2, and #3, and D8 fixes nothing but #4 and heyrocker's #5, I would consider Drupal well equipped for the next five years.

dixon_’s picture

I pretty much agree with Sun's comment in #16.

As Dries mentioned several times in keynotes both test driven development and distributions are very important for the growth and adoption of Drupal! For #1, tests and distros simply doesn't work together. I think that must be fixed. For #2 I agree with Sun's comment that the API changes are minimal while the win is huge for all distros. I agree with heyrocker that #3 probably should be considered a bug. #4 is definitely for D8.

I also quote eaton:

Frankly, if we can fix #1, #2, and #3, and D8 fixes nothing but #4 and heyrocker's #5, I would consider Drupal well equipped for the next five years.

mfer’s picture

I'm in the heyrocker camp. #1 and #3 can be fixed. #2 is too late and should be in D8. #4 needs more definition to handle more of what we need to handle and I would hope will be in D8.

sun’s picture

The simple solution for #2 is: #934050: Change format into string

Getting that patch into its current form was done in 90 minutes. I'm confident that the very same change can be easily done to user roles, too. The essence is to convert the primary keys into strings, and remove the serial/auto_increment behavior. Actually, we should have done the same for vocabulary ids, as all of these things are plain configuration items and should therefore work like node types (which similarly need an explicit $type->type to be set and have no serial ID). It's unfortunate that this stupidly simple proposal comes so late, but it is a working solution that makes the impossible possible for the next ~2-4 years, and comes with a tiny API change only.

Please keep any details, comments, and feedback on that issue. Keep thoughts and opinions on the overall, macro-level problem and discussion space here.

bonobo’s picture

@eaton nails it:

Frankly, if we can fix #1, #2, and #3, and D8 fixes nothing but #4 and heyrocker's #5, I would consider Drupal well equipped for the next five years.

Getting #2 in place for D7 would (indirectly) be an enormous usability win.

Without this, install profiles and truly reusable features will be things that require more work/technical expertise than site builders/non coders have.

With #2, we will have site builders contributing back creative solutions in the form of reusable building blocks for the next several years of the D7 lifecycle.

Given that one of the explicit goals of D7 was UX, addressing #2 actually allows people to solve UX issues much more efficiently - site builders can leverage the work of other site builders.

Dave Reid’s picture

Given that one of the explicit goals of D7 was UX, addressing #2 actually allows people to solve UX issues much more efficiently - site builders can leverage the work of other site builders.

We had sufficient time to fix UX issues, and that time is over. Period.

sun’s picture

Let me clarify that I perfectly understand the objections and arguments raised by various people here. I am basically with you.

An important detail of #2, which may not be sufficiently clear, is that I see serious security issues with every existing and previously proposed attempt to make user roles and text formats exportable into code by introducing machine names for them. Both of these configuration items are the heart and core for a securely configured site.

The currently existing attempts to put them into code have one critical flaw. Exported and imported configurations still use a serial ID that is automatically incremented. The rest of the entire system uses the serial ID to reference the configuration items. It is impossible to ensure that a certain configuration item uses a certain serial ID. Since the ID can change under the hood, this immediately becomes a critical security issue. Content may be suddenly rendered in the wrong text format. Users may suddenly have permissions they should not have. This has an immediate effect on any other exportable configuration item (e.g., panels, views, wysiwyg, etc) that happens to contain a reference to those insecure IDs.

After thinking through all of this, I am considering all existing approaches to export and import these configuration items as severe security vulnerabilities. I would even go out on a limb and say that the entire related functionality in Features, Input Formats, and any other module is insecure and has to be removed. As of now, it is impossible to perform this securely. Period.

However, I think that everyone agrees that exportables have indeed become a major building block for Drupal sites and Drupal distributions during the development cycle of Drupal 7. Personally, I realized this far too late, because I was completely busy and buried with core development, so I installed and used Features module for the first time a few months ago. Only afterwards, I fully understood the giant hype and praise for the development model of using exportables.

Without any doubt, it is the future of doing Drupal development. While the exportables functionality certainly will still have to mature in contrib, it is now crystal clear to me that the two most important configuration items in Drupal core cannot be safely supported in their current state.

Drupal 7 will be around for the next 2-4 years. The stupidly simple solution proposal, which can be applied to both, user roles and text formats, removes and completely eliminates the last blocker for working with exportables. Doing so will heavily simplify the development of sites and distributions for everyone of us, especially for people that are building large sites and products based on Drupal. As far as I can see, it is the only way to make the exportables approach mature in contrib, and therefore, the only way to remotely consider to move some exportables functionality into core for Drupal 8.

To summarize: Yes, it is late. But the proposed, required changes for #2 are negligible, compared to their major impact on Drupal's success and world domination. Given a sign of agreement, I would offer to personally invest my time and money to get the required patches RTBC in the next 2-3 days.

Crell’s picture

Per webchick's request in #6, offering more than just a subscribe:

I'm honestly torn here.

On the one hand, most of these are changes I agree with. I think we're all in agreement, even Alex, that "exportables in core" is not even on the table until Drupal 8, and knowing our track record it would probably change dramatically once we get to really putting something like that in core. These are by and large good changes.

On the other... As DB maintainer I dropped improvements and cleanups I had on the table for the DB layer *over a year ago*, when code freeze happened, *like I was supposed to*. The DB queue for Drupal 8 is now larger than for D7 because we pushed a lot of stuff off, some of it esoteric, some of it much smaller in impact than the stuff in this thread (E.g., #664722: Make insert queries Countable), some of it much needed DX refactoring (#432440: Update Schema API to make it more usable). The D8 queue is already filling up with cool stuff, good stuff, stuff we all want, that didn't quite make it. That happens every release. It sucks but we have to ship sometime.

On yet another hand... it's not like we haven't let through even bigger changes and additions long past code freeze (#780154: There is no listing API for field API comes to mind), so what's one more exception between friends?

I don't envy Drieschick this question, as it's a damned-if-you-do-damned-if-you-don't moment that's largely our own making.

As to the issues at hand themselves:

1) This sounds like a minor regression and a minor fix. drupal_system_listing() is not an oft-used function so the impact should be minimal.

2) I am a firm supporter of machine names over serials for configuration objects, so +1 on both of these changes whenever they are allowed to happen. I think Alex is right that if we do it for Drupal 7 we will need to have a double ID (some int, some machine name) to avoid breaking too much but for Drupal 8 we can switch to just machine names.

3) This sounds like a hole in the API; it's an API addition more than an API change, and I generally favor all APIs having a very consistent pattern.

4) Not in Drupal 7, not worth discussing further.

chx’s picture

Now come on. EntityFieldQuery was an addiition, noone was using field_attach_query that it replaced because faq was completely unusable there were no ported modules that got broken because of it -- we did not change any schemas for example, we did not add code to critical code paths and so on. The whole field API would have significantly less usable without it. And it got committed in June. Today is October. Note that a bugfix was committed yesterday to it -- despite having over sixty tests we still had a very significant bug in it.

1) requires a lot of thought, already taking to alex over it. 2) Drupal 8 3) is the only that has a chance of getting in as it does not change an API or schema 4) D8.

irakli’s picture

My two cents:

I see a lot of disagreement on _when_ these should happen and almost none about the value of the changes, per se. Considering this and for the sake of the community's sanity I would strongly urge that we separate the question of WHAT from WHEN.

I feel like such approach will make the discussion much more productive. Also - I've had a conversation with webchick on IRC yesterday that relates to this. People who advocate that these changes HAVE to go in D7 (and therefore go in standstill with people who want to release D7 ASAP and have a good point about that) do this mostly because - nobody knows when that D8 wil be released. If somebody guaranteed (what a horrible word) that in a year we will release D8 - there would be no argument for rushing these into D7.

However, nobody can say that because we see D7 in works for almost 3 years (since the D 6.0 release) and yes, we have tests now and yes D7 was unique yadda, yadda, yadda but on everybody's mind there's this lingering thought - what if D7 was not a unique case? What if it is a necessary price you pay in the new, bigger Drupal? We can't wait another 3 years for these changes, is the problem.

But that's a completely different discussion that I have an absolute vested interest in opening and have promised to do. I know _parts_ of the question have been "discussed and beaten to death", but it's not necessarily the same reality now, anymore. In any case, I will try my best to both make the case, as well as propose solution (so it's not just b**tching about how things are). However I promised webchick that I won't let it interfere with D7 release so I am not even touching it before the beta is out.

So, going back to the original suggestion - how about we discuss technical merits of the changes here and leave when/how to a separate discussion?

I think it's pretty clear that delaying beta for this changes is both unfeasible and won't happen. I think it's also very clear that these are not the only changes we will have to roll into Drupal7, if Drupal8 ends up being 3 years from now, too. So let's accept that and work from there?

grantkruger’s picture

At a high level: Dries keeps talking about how distros are the future and how this is a huge deal that is making us stand out from the crowd. If distros are the future then you have to cater for them, plain and simple.

At a lower level: Seems like many think D7 has been too long coming. There appears to be a lot of D7-fatigue and everyone wants to get it out the door already. There is something to all this.

There's likely a balance to be found here, but I keep coming back to what Dries said about distros. They're the future. How do we not enable them?

Kevin Hankens’s picture

1. Testing has given so much reliability to core, it would be great to allow (rather, not deny) distros to leverage it fully. That will only help Drupal's growth. +1
2. A little scary, as sun points out, but if done right, this would be a great step towards exportables. +1
3. As alex_b points out, this does seem like a bug. +1
4. An export/import API in core would be awesome! Also echoing that heyrocker's suggestion of UUID support is right on! +1

eaton’s picture

One issue that's been brought up in side discussions is the potential for adding specific features or fixes in subsequent point releases of Drupal 7. For example, 7.4 might include notification of machine name changes as it would be a new event fired (presumably) rather than a change to existing contrib functionality.

My understanding is that .info files can now list specific VERSIONS of modules and core for compatibility purposes, making that a less daunting prospect than it was in D6. But is this a realistic prospect? Are there any of the above issues that drieschick would consider taking that approach with, or would anything not done right now be punted directly to D8?

mfer’s picture

I don't think anyone is against these changes. The question is, what goes into D7 and what needs to be put off to later.

@eaton - I do think adding features to D7 that are backported from D8 is possible if the API does not change. For example, adding the ability to do drupal_get_path('library', lib_name'); would be fantastic. And, you could do it without changing the API but instead add to it.

Adding a new hook to notify if a machine name changed I'd put into that category.

irakli’s picture

@eaton, there are ways to make changes (even significant ones) without the necessity for modules to indicate their compatibility with specific core version (which I think is a horrible idea, personally).

This is not a wheel that needs to be invented. There's a lot of industry experience. And, sorry, but no, Drupal is not unique. Yes, it is open-source project, but it is not the only open-source project.

Let's not try to learn on our own mistakes, shall we?

febbraro’s picture

My perspective comes as of the owner of a company that has invested *heavily* in distros.

I hear and agree with everyone's point about "lets just get the code out". Nothing would be worse than delaying D7 much longer, it's time we all start realizing the benefit of all the hard word and thank the core maintainer's for killing it over the past 2+ years.

That being said, distros have only started to proliferate within the last 12 months and that has been *since* the code freeze, not before it. The time for API changes stopped before many of us even started building or actively maintaining and upgrading distros. It is already a beast to maintain them, so things to make it easier and more secure as certainly warranted.

Again, I realize that that we need to get D7 out and that is a great goal, lets do it. However, it has been approaching 3 years since D6 was released. We can't wait for D8, especially with all the talk of gutting configuration to make it exportable, revamping blocks and layout a la Butler, etc. With all of that work waiting for D8 that might possibly be 5+ years away (if the timing of release trend continues)

My real question is why must a change wait until D8? We can implement something in core at a later point release in D7 while maintaining backwards compatibility for the entire 7 line, then rip out the backwards compat for D8? Other opensource projects (see Java) have been following this practice for over a decade. It has worked for others and it can work for us.

All of that to say, I'm for #1, #2 and #3 at *some point* before D8. Ideally before D7 goes gold, but if not at least *soon after* the gold release of D7, but please lets not wait for D8, it's too late then.

eaton’s picture

@eaton, there are ways to make changes (even significant ones) without the necessity for modules to indicate their compatibility with specific core version (which I think is a horrible idea, personally).

This is not a wheel that needs to be invented. There's a lot of industry experience. And, sorry, but no, Drupal is not unique. Yes, it is open-source project, but it is not the only open-source project.

Not wanting to derail the discussion, but I definitely agree with the idea that we should learn from others -- and most other projects allow non-breaking additions to be added during point releases. My point about the version dependencies was just to note that modules that absolutely rely on a function added in, say, Drupal 7.3, would be able to declare that dependency while the D6 dependency system doesn't allow that flexibility.

I'm not arguing in favor of it or against it, so much as I'm interested in clarifying whether that approach (adding new helper functions and hooks for notification, etc) in subsequent point releases would be acceptable to Dries and Webchick.

chx’s picture

Status: Needs review » Closed (duplicate)

This is one of the resource sucking meta issues that leads to nowhere. Who cares about +1 or -1? The decision to break APIs and change schema lies with Dries and Dries alone and is done on an issue by issue basis. To fix bugs we have invidiual issues. There is no need to duplicate four issues into this one.

ericgundersen’s picture

Status: Closed (duplicate) » Needs review

Dries mentioned a few hours ago[1] that he is going to comment on this today, lets at least keep this open till then.

[1]: http://twitter.com/#!/Dries/status/26670753571

webchick’s picture

For those who haven't read it, http://buytaert.net/8-steps-for-drupal-8 is worth a read. In it, step 6 is "Backport small changes to stable releases". Drupal 7 is the first release we've been able to entertain the idea, since we have a testing framework with 25K tests and a version-specific dependency system. What the boundaries are of such changes isn't quite defined yet, because Dries and I have kept pretty busy with, you know, getting Drupal 7 out the door. :P~

fago’s picture

1) Let's go for it
2) Yep it's important, but it's too late. There would be lots of other important stuff too, but we have to release now. Something like a Drupal 7 and a half might be worth a thought.
3) Should be on the table for d7. Nothing breaks if machine names suddenly cannot be changed any more, contrary we have no way to resolve those updates yet. So not allowing those changes, would keep everything working and we'll get a more reliable system.
Also I'm with alex here, lots of modules don't get node type re-names right and it won't be different for any other machine name.
4) Sure d8, exportables are the way to go. As one API for CRUD is enough, one should be able to make an entity type exportable. For d7 the entity API already does that.

chx’s picture

Status: Needs review » Closed (works as designed)

Note that 2) and 4) is an exportables issue and have nothing to do, whatsoever, with profiles / distros. We only got as far as we got in D7 and so distros mean you install something and if you install it then you know their ID, end of story. Did I mention that this is an extortion-like meta issue that leads to nowhere and there is no reason to keep it open? Dries can follow up on a closed issue and noone else needs to.

webchick’s picture

Status: Closed (works as designed) » Needs review

I'd also like to contextualize some of the core developers' reactions here for those who are coming in here from Twitter or whatever and going "Dude, WTF?"

this post lays out the Drupal core release cycle. People have been working on Drupal 7, unofficially, since 2007. The core development team runs almost entirely on volunteer efforts, and so is woefully understaffed at the best of times (and the pre-release bug-fix slog is definitely NOT the best of times). These are volunteers who have been sacrificing their nights, weekends, family time, etc. for the past three years. To get features in that they cared about, to help others achieve the same. To attack the critical bug count issue. by issue. by issue. pushing it down from over 300 issues to single digits over the course of the past year. This has been grueling, arduous, work, and has involved significant sacrifice; both on a personal level, but also because we've all had to let go of our own pet issues because they were not ready in time for feature freeze, despite how hard we worked, and despite how important they were.

So when core developers see a bunch of people who have been largely out of touch with core development (at least lately), who they haven't seen busting their ass on critical bugs for the past 12 months, who they haven't seen making sacrifices in terms of time and itches gone unscratched, suddenly waltz in just as we're all on the cusp of a major project milestone over a year in the making, and say "OMG! STOP! My feature is the MOST important and it MUST go in before Drupal 7 releases!" … well let's just say that it does tend to raise hackles. And I get that. Your companies may have invested in distros, but you didn't invest in core development. And so, we each individually have had to step up, and invest our lives. As a result, we feel a profound sense of ownership in the project, and in some cases it's very personal.

But the thing core developers also need to remember here is that the people "waltzing in here" suddenly, although they may not in a lot of cases been working on core (or at least recently), are not the bad guy here. There is no bad guy; there are only good guys (and girls ;)) who are prioritizing different things. While we're slaving away on establishing UX guidelines and hammering out tricky upgrade path bugs, others are building out infrastructure like exportables, features, and install profile tools that help solve really tough, intractable problems that actively hitting folks right now, in the field. A lot of those solutions (version-level dependencies, install profile refactoring, etc.) have actually made their way back into core in Drupal 7 already. And some of them are just too damn massive to make their way back into core right now -- even the most ardent proponents of exportables agree that incorporating them wholesale into core is a D8 effort -- and so are being handled in contrib, by necessity. And if each time someone working on distros comes into the core queue they get their face chewed off and spat back at them, they will stay in contrib.

So please, everyone, remember in your interactions here: there is no "us" vs. "them". There is only "we." And we all want exactly the same thing. To make Drupal *@&#ing awesome.

And thanks to those who've been giving more insightful/detailed replies lately.

rickvug’s picture

No one is against the change proposed in #2. The issue is simply when the issue should be addressed. Perhaps a list of pros vs. cons on moving forward would be help. Admittedly I'm biased towards the +1 camp...

Pros:

  1. The last remaining non-exportable configuration will be exportable. Massive win for D7 distros and site builders.
  2. As mentioned in #29, repeatability reduces the chance of insecure configuration.
  3. Core will be sending a message to contrib as to the importance of using machine names when defining configuration.
  4. We can put this issue/distraction behind us.

Cons:

  1. Potentially cause regressions and a distraction, delaying 7.0.
  2. Unless we can find a way around it this is a late API change for module authors.
  3. Hurt feelings and frustrations around why this patch was allowed and others have to wait.

Sizing up the cons, #1 and #2 aren't blockers to me. This is a common, tested pattern used throughout Drupal. This is mainly an issue of causing temporary pain. We also can not assume that working on this issue automatically slows the momentum of others. It is the community frustration that is the most worrying to me. These changes clearly break the rules. It is not fair to others who had their patches rejected. Reflecting on this, I still think we should go ahead. Drupal needs to stay nimble, which means breaking its own rules when necessary.

tsvenson’s picture

I find this topic very interesting, but I also note that it is kept on a very technical level plus with where we are with when D7 will be released.

#29, by sun, is one of the few exceptions looking into how distributions and exportables would work in D7 if these issues are left for Drupal 8.

I also think it is important to look at who distributions and exportables, using for example the Features module, are aimed at. Distributions will be great since they will save tons of time for users to get a foundation site with features that are quite close to what they are looking for. Exportables allows the creation of features that can be quickly deployed on existing sites.

Even if there will be hundreds of fantastic distributions, none of them will be a perfect match for those who opts to base their site on them. Yes, they will provide a great shortcut and require less skills of the site owner to get started with, but they will still be needing tweaks and additions to suite their needs.

Installing a simple module to add a new feature is generally not a problem. However, here we are talking about a combination of modules, often from module maintainers that only has Drupal in common, plus configurations that ties these modules together to form the distro or feature.

Drupal is known for its flexibility and that it, with contributed modules, can be used to build almost any kind of website with very little need of custom PHP coding.

But how will this work with distros? If I build a new site with version 1.0 of a distro, then make some tweaking, adding new features and so on. How easy will it then be to later upgrade to version 1.1 of that distro without any of my tweaks or added features breaking?

I love the concept of distros as well as the goal with the features module, but I am also starting to wonder (due to the current state of the code) that it might actually force the users to make a choice when they start customising a distro. Either they will have to limit their customisation to be able to (hopefully) keep upgrading it to newer releases of the distro or they simply use the distro as the start and then add all new tweaks and features on their own.

As sun wrote, there is a lot of hype around distributions and exportables within the community. Maybe we are forgetting that for a lot of people that will en up using them, which are a major target group for this, are not going to have the skills needed to sort out the underlying problems pointed out in this discussion.

They will only notice this when it start to go wrong and then it will often be to late for them change their minds.

If getting rid of things such as serial ID's instead of proper machine names will delay the release of D7 with a few weeks, or even a couple of months, I am all in favour of that. Especially if the alternative is to have a system that is almost there and then sooner or later breaks with a sour taste for the site owner.

These things simply needs to work correct or else they clearly needs to be marked as _experimental_ for anything but shortcuts the get a site foundation quickly up and then only do custom work on it from there.

irakli’s picture

@webchick,

thank you. I am grateful that you point out that while some of "these people" have not been "scratching their itches" on the core, they have indeed been busting their asses off just as much elsewhere (e.g. in contrib) and there's no shortage of sleepless nights or sacrificed time outside the core. These distros and 7,000 modules did not just emerge out of nowhere. And while core team's work is definitely very much appreciate, let's not make it sound as if Drupal is just core. Let's also appreciate all the work outside of the core!

There's only "we", indeed.

P.S. I am not in favor of delaying Drupal7 release. I actually wanted to see a discussion about the merits of proposed changes (and other required changes) without discussing when they should be adopted. The problems stated by Alex is a good list but they are not a full list by any means, by the way.

webchick’s picture

Er. You did read the rest of my post, right..?

irakli’s picture

@webchick, my apologies. I edited my previous post to make it clear that I am not arguing with you, on the contrary.

Dries’s picture

Great post with many thoughtful comments. All of the proposed changes are good changes that we want to see happen; the question is merely "when?".

I truly believe distributions are important. I've talked with Alex_b and others about exportables at DrupalCon Paris, and about distributions at DrupalCon San Francisco. It's a pity we didn't get these changes in months ago because I certainly left the door open early in the code freeze and encouraged it repeatedly ... At the same time, I realize we've learned a lot since the code freeze -- as @febbraro points out, distributions, features and exportables have only really started to proliferate within the last 12 months.

As a result, I'm quite thorn about the issue. Strict rules are important, but at the end of the day, I believe that common sense should always be able to overrule rules. Our ability to adapt to the changing world around us will define whether we'll be successful or not. Our ability to stick to rules, not so much. That said, I think we've demonstrated quite some flexibility and willingness to adapt during the code freeze ... and that's an understatement.

Here is the summary of my thinking:

  • #1 should get fixed, although I'm not too excited about the proposed solution for #1.
  • #2 is questionable -- it is important so I want to think about it some more, and discuss it some more.
  • #3 should get fixed.
  • #4 has to wait for Drupal 8.

That should allow us to make progress on #1 and #3. Start!

So what about #2? I realize that it is a pain, but I'm not sure it will prevent distributions from happening. To me, it sounds like it is a barrier but not a show-stopper for people building distributions. This seems evident given the recent uptake in distributions based on Drupal 6, which is far less exportable than Drupal 7. If you disagree with that assessment, it would be good to speak up because it is the thesis of my thinking.

First of all, people can build distributions without using features/exportables. Second of all, when using features/exportables for the purpose of a distribution, you can code your way around this limitation. The latter is obviously a pain. To understand how big of a barrier #2 is, I'm trying to understand how big of a pain it is to maintain an exportable-based distribution without #2. I'd love to get some more color around that from those who think #2 is a must fix. You gotta help all of us to better understand why #2 is critical.

Once we better understand the pain, and agree on it, I'll be in a better position to weigh the opportunity cost of delaying Drupal 7 by a bit vs having more and better distributions over the next couple of years.

irakli’s picture

@Dries,

thank you for your comments.

the biggest problem in not having #2 is that yes, you can build distros without exportables and Features UI (and we actually did, early versions of our distros) but it creates very significant roadblocks on the update path. Let me explain...

The only way to deploy changes in Drupal, other than through changed exportables is to write update hooks. This can be a problem because distributions do not have the same life as modules do. Most Drupal developers do not "hack" modules - they do not change its code. This is not true for distributions. Most distributions are expected to be customized. So if you change something in ver 6.4 of a distribution, and somebody needs to run an update on a legacy 6.3 installation - what if the change affects something that has been customized? Update hooks are very black/white: you either change or not change something.

Features UI is much more flexible in this sense. First of all, it allows you to see changes more granularly (I can see that my blog feature is now overridden because a specific view has changed and i can even see changed config code if I have diff module installed) and it also allows you to apply changes selectively. This makes big difference.

I am sure there're other issues that I can not remember off of my head, right now, but I am sure alex_b will add at least couple.

chx’s picture

Finally some sense. Meanwhile, yhahn and I have worked out a smaller impact change on #1 (i won't call it prettier. pretty solutions do not seem to be available now) and i see tons of activity on #3 -- very good.

More tech, less meh please.

jgraham’s picture

I had a much longer post I am truncating in light of @dries #51, as I agree on the assessment of 1,3,4.

To elaborate on #2:

  1. This will not prevent distributions from happening, but it presents an unnecessary technical barrier to creating and maintaining distributions. The distribution space will continue to grow and evolve separate of this issue but at a much restricted rate and slower development cycle on a per distribution basis. The fact that distributions exist in D6 is a good example.
  2. Agreed, features and exportables are not a requirement for building distributions. However, the use of features considerably lowers the technical barrier, and makes the maintainability of a distribution considerably easier. I'm not sure what a distribution without features would look like, but based on the existing documentation it likely has way too much in common with the underlying DB schema rather than the Drupal API. (see: http://drupal.org/node/67921) Again this is an unnecessary technical barrier, and presents maintainability issues.
  3. Regarding coding around the issue; this is the crux of the unnecessary technical barrier. Coding around the issue and using features from my experience results in the following
    1. Less re-usable features: This is a considerable barrier to a rich distribution space for Drupal. See http://drupal.org/project/kit and http://drupal.org/project/debut why re-invent the wheel if you could re-use a bit of functionality.
    2. Features that rely on certain details such as filter formats and roles have to build on top of a set of assumptions. Filter format 1 is "filtered HTML", etc. Currently this is solved in an install profile, by forcing the ids with direct DB manipulation. This is a hack, but the standard method of resolving this issue as there is no other way (to my knowledge) around this issue since. This creates an awkward dependency for a feature to an install profile. If #2 was addressed then an install profile dependency would be unnecessary as these would be provided as another feature.

Many thanks to webchick(re:#45), and all the hard work by all the core, and contrib developers.

bonobo’s picture

@all of the folks who have been putting countless hours into core development for the last three years - I definitely want you to know that your time, effort, and attention to detail is not taken for granted, and is not undervalued. I appreciate you making additional time to look at this. My motivation in speaking up here is not to delay the D7 release, but to see what can be done to make D7 more usable by more people for the entire D7 cycle.

RE:

it is a barrier but not a show-stopper for people building distributions.

and

First of all, people can build distributions without using features/exportables.

This is absolutely true for everyone reading this thread. For the people here, getting this change in will make our lives easier for the next three years, but we could definitely work around it, as we have been.

However, with #2 in place, building install profiles that are usable out of the box becomes accessible to non-developers. This means that an entire group of people using Drupal will be able to contribute back, and benefit from, features that are truly reusable. Right now, install profiles are the places where we code around what isn't exportable. It works for us, because we can do it, but for people who don't know how to code, much less what to code around, it's a showstopper.

RE:

To understand how big of a barrier #2 is, I'm trying to understand how big of a pain it is to maintain an exportable-based distribution without #2. I'd love to get some more color around that from those who think #2 is a must fix. You gotta help all of us to better understand why #2 is critical.

The pain is in the initial build of the install profile. Once that pain has been endured, you can build and maintain features on top of that. However, this minimizes the use of features because they are, effectively, tied to a base install profile.

Now, in building features, you can go one of two directions:

  • tie your features to a specific install profile (a Managing News feature, an Open Publish feature) - this works because your feature can assume the existence of elements present in the install profile; or
  • create general features that require additional config to actually make them do anything useful - this works if the person using the features knows how to perform the site administration, but it certainly is not an automated process, and will still be beyond the reach of many people, especially people who are unfamiliar to site building in Drupal.

With #2 in place, it starts to become possible to build features that are more like modules than install profiles, using the UI. We're not all the way there, but that's what contrib is for, and support for this in core will make it possible for the details to get hashed out in contrib during the D7 life cycle.

This has the potential to be the killer feature for the D7 release, as it brings us closer to people being able to bundle and redistribute the killer features they have built on their sites.

eaton’s picture

To understand how big of a barrier #2 is, I'm trying to understand how big of a pain it is to maintain an exportable-based distribution without #2. I'd love to get some more color around that from those who think #2 is a must fix. You gotta help all of us to better understand why #2 is critical.

I think the emphasis some of the folks have placed on features may have clouded the specific nature of the problem. I built three different profile/distro packages for large clients before starting to use Features, and the problems that ripple out from #2 aren't unique to that particular tool. As others have noted, it just make it easier for a wider range of developers to do the legwork of building install profiles and distros around exported content types, views, flags, and so on -- and thus, run into the problems with Role IDs and Filter Format IDs.

When writing a profile or an individual module from scratch, there's a reasonably workable solution: your code creates all the roles and filters you need programmatically, your store the resulting filter IDs and Role IDs in variables like mymodule_editor_role and mymodule_comment_input_format. Then, whenever you have code that refers to said roles or formats, you variable_get() and use the stored ids.

The big problem comes when you build a view with "Full HTML" in its footer text, or a flag that should only be visible to people with the "moderator" role, or a content type with a custom field that needs to be passed through an input format, or a Panel that needs to have one pane hidden from users who aren't editors, or any of the above. You can export the definitions of those elements without problem, but the definitions themselves are "dumb" regarding the role ids and filter format IDs: once you take those exports and add them to your custom module, you have to hope and pray that it will be installed in the same order, in the same environment, and so on -- nothing that would cause those exported references to filter and role IDs to fall out of sync.

One possibility -- and I've used this before in moments of desperation -- is to go in and hack the PHP code of the exported CCK types, Views, Flags, and so on. You can tweak that code and replace references to hard-coded IDs with variable_get() calls. That fundamentally breaks any tools that automate the process or help users package and distribute the tools, however, and I can say from experience that it's pretty fugly. It also means that anyone who wants to create a reusable export-driven solution for a given niche has to learn how to edit raw Views data structures by hand, and so on.

In reality, what it means is that one of two things happens:

  1. Solutions that are built to be used in distros (like views and VBO based editorial tools) are ghetto-ized, and rely completely on the configuration and setup done by the profile. Without the hard coded role IDs and filter IDs that everyone assumes will be there, everything breaks.
  2. Tools don't do any pre-configuration for the user or default to the lowest common denominator of input filters, access control, etc. This works for some solutions and not for others. It hasn't worked for the ones I've built, because the people doing the setup needed the distros to be actually usable.

Whether those problems are worth solving before D7 ships -- and we start building the next generation of user-facing solutions on top of it -- is a decision for the core team. I'm absolutely, utterly sympathetic with the push and pull of demands for assorted features. If it is possible to come up with a low-impact solution to this problem, though -- either by converting the rid and fid columns to strings and allowing ids to be passed in on creation, while defaulting to sequential numbers; by adding a well-placed drupal_alter(); or some other mechanism -- it would be a definite boon for solution sharing, distro "polish," and so on for the next two to three years.

For some Drupal developers, that's considered a show stopper. For others, it's a nice-to-have. Figuring out the balance isn't the purpose of this thread, but offering the perspective of those who consider it important so it can be weighed against the costs is hopefully beneficial.

eaton’s picture

Well, while I wrote that up it looks like bonobo already said everything I said. Feel free to ignore me. ;-)

chx’s picture

Eaton, very seriously, this whole issue is basically banging the table "but these are distributions and you want distributions, right?" I know you are picking on me but there is quite a demand in the original issue

and could be justified as critical issues for a Drupal 7 release

. But, webchick said everything needed to be said, but I am fed with eaton picking at me here and on twitter and while people here theoretise and chatter I have already worked on fixing the bug that falls under the subsystem I maintain. Now, I am going to take a break from core for a few days. I very seriously had enough. Go and do whatever you want, I am throughly fed up and disgusted from all this.

alex_b’s picture

Thank you for your responses. I know everyone is very busy and I appreciate the time put into the comments here and the ensuing work on related issues.

I think we agree pretty much unanimously on the priorities of (1), (3) and (4) as layed out by Dries (the latter needing more discussion around implementation).

Now to (2):

I'd love to get some more color around that from those who think #2 is a must fix. You gotta help all of us to better understand why #2 is critical.

From a higher level perspective, portable configuration is the prerequisite for building a Distribution. Sequential ids aren't portable. But portable configuration not just facilitates building and managing distributions, it also solves the staging and deployment problem for configuration. This is why (2) is not only a Distro issue, Distros are merely primarily affected.

Now for the lower level perspective. The current workaround for exporting configuration with sequential identifiers to code is to assume that their human readable name won't change. That's scary enough, but exportabability is only one side of the portability coin. The other side is referring to such configuration, and herein lies the real challenge.

Modules like Node, Views, Field refer to filter formats by their sequential ID. The current workaround for this problem in Drupal 6 is mostly hope. Hope that sequential ids won't change. But that's a tough bargain when the odds are that sequential IDs are different between installations. Consider this example:

A new release of Distro FooBar adds a second filter format "Awesome Markdown". Before the new release a site owner using a previous version of FooBar has created a filter format "Test - what's this?". "Awesome Markdown" will be created with a format id 3 (the test format reserves 2) while all other configuration shipping with FooBar will refer to it by format id 2. The result is that against the intentions of the Distro publisher, content is being displayed in format "Test - what's this?" instead of "Awesome Markdown". What's worse, the site owner has no straightforward way of fixing the situation: ids are sequential.

Depending on the situation, there is a little more than hope to work with: for instance, we have at points used a Drupal variable to hold a vocabulary vid and have custom Views handlers resolve that vid by that variable. We have used the vocabulary.module field to implement a pseudo machine name in Drupal 6. A hack that depends on talking other module maintainers to refer to vocabularies by their vocabulary.module field (!). These approaches are only possible in scenarios where the referring module can be patched or is flexible enough to be extended accordingly (Views). Either way, we are not talking sustainable code here.

From the get go, I only brought up machine names for filter formats in 7 under the condition that a fix would be clean, straightforward, would not break existing APIs, would have a solid upgrade path and would save us aforementioned atrocities. In short: a simple fix with a big benefit.

Still being convinced that a simple fix is possible I am planning to go ahead and work with sun and others on a patch for HEAD. We can then judge it on its merits and decide whether we want to commit it to D7 or wait to a D7.1 or D8 with it.

arianek’s picture

That sounds like a good approach Alex - definitely seems there's agreement on 1 (though maybe further need to discuss approach on that issue), 3, and 4.

From what I understand, there's no reason why those who are keen cannot work on 2, and then allow webchick and Dries to decide when and to what version it gets committed, right?

Is everyone on board with that?

catch’s picture

Looks like I missed loads of drama... just saw this.

Don't really care about #1 - simpletest changes are much more on the table than any other module in core (unless they actually break existing tests). However http://drupal.org/project/simpletest has a 7.x-2.x release that boombatower maintains so if it's in invasive change I don't see why it couldn't be put in there + D8 core.

taxonomy module not calling field_attach_rename_bundle() (#3) is a straight bug. This is a completely different class of issue from #1 and #2.

On the other hand #881530: Exportability of vocabularies is ruined by taxonomy field's 'allowed values' setting was an actual API change, that no-one flagged up as such, and just went in last week, this annoyed me as both taxonomy module maintainer and someone who has code in the wild which was broken by the change.

Every single API change that goes in at this point directly leads to additional work in documentation, head2head updates (or hopefully not those any more but you never know), and those few people who are building actual sites with D7 having to figure out what changed when they cvs up on dev and things start exploding. Not only this, but it was left for me to re-open the issue, mark it as 'needs documentation', explain what the API change was so that Randy could send it out etc. etc. That doesn't lead to a great feeling of sympathy when I see more last minute API changes being promoted loudly here.

I agree with the numerous people who already said it that #2 is too late. I personally posted, and wrote an initial (hacky) patch for #624018: Exportables and Features support for WYSIWYG 7.x in November 2009. #643566: Text formats was posted shortly afterwards.

People working on distros/features have been aware about these issues for almost a year, it is not something that has been suddenly discovered, there was zero effort to even document it as an issue in core until one month ago. This isn't just asking for an exception for something that's not quite ready yet. Saying "distros didn't really start to take off until a year ago" is no excuse when the issues were well known and left for the entire duration of that year before being brought up. That makes this issue particularly frustrating to someone who moved dozens of their own issues to Drupal 8 long before that.

merlinofchaos’s picture

For what it's worth, it was decided by several folks in Paris that we would not be working on export.inc in core for D7, based upon core's needs. A year later, that hasn't changed. For me, it is a priority in D8. I think that's well agreed by everyone here, though, so there isn't much point in discussing it, beyond letting people know my intention of getting CTools' export.inc into core, even if it is not in its current form.

sun’s picture

I guess we need more facts in order to evaluate this better.

In #29, I tried to express that everyone and the world is - already right now - trying to put configuration into exported code to develop and build sites, products, and distributions, and to keep the configuration under version control. People are doing this already. People are exporting all kind of configuration items, because it heavily simplifies and streamlines their work. However, people are entirely underestimating the actually existing security problem. Since user roles and text formats are referenced by their serial IDs, but previously exported code may be loaded/imported with a different serial ID, serious security problems can arise. So the entire current approach is flawed when it comes to exported configuration items that happen to reference user roles or text formats. Such configuration items can literally be anything; views, panels, node types, vocabularies, everything. If the exported code happens to reference a user role to limit access, or a text format to render any kind of text, then it is busted. But nevertheless, people are exporting their configurations, because the benefits outweigh the potential problems. I'm quite sure that almost everyone on this issue has already exported or is actively exporting configurations into code, and it's perfectly possible that many people don't even know of the existing security problem. The only valid way to avoid it would be to not export affected configurations into code.

Admittedly, that would be a valid answer to the problem space.

It would mean that the corresponding functionality that exports user role access conditions and text with an assigned format would have to be removed from Features, Views, Panels, and other modules, as it can too easily lead to security vulnerabilities.

Obviously, that would be a major step backwards.

Earlier in this issue, I stated that machine names for text formats can be introduced in a simple way, and that the same simple thing could potentially be done for user roles. I can only guess that core maintainers need a more precise list of issues to evaluate the way forward for #1, #2, and #3, so I'm trying to compile this here:

  1. Testing modules from distros
  2. Machine names
  3. Modules are not aware of vocabulary machine name changes.

    (Actually, we really should have done the simple approach as proposed for text formats and user roles, instead of introducing a separate machine name for vocabularies. I guess it's too late to change this once more now, but I wanted to mention it.)

fago’s picture

Ad point 2).

If we want to fix this for d7, we need to really need to minimize any API changes. So a bit more thoughts about that:
* For filters, what about adding an *additional* machine name and support both loading by id and name? If something numeric is passed, it is loaded by id, else by name. No one gets hurt by that.
* For roles, something like that won't help as there is no loading function used. However we could easily introduce a new lookup-by-name helper, once it is defined by core any module referring to roles can use it to look up the roles.

Well in most cases I think feature-like module should define their own permissions + check that anyway. Assigning roles to permissions is up to the installation profile then. Doing so, we have no problem here. However there are still some other cases left where directly referring to roles makes sense, e.g. a user reference field. For those remaining cases, the above described lookup should be sufficient.

sun’s picture

Adding a separate machine name does not help for these configuration items, because the entire rest of the system keeps on referencing the serial IDs. If a serial ID changes for any reason, your system blows up.

You can simply compare all of this with node types and/or field bundles. Those do not use serial IDs either. The configuration items are always referenced by their machine name, either by $node->type or $bundle. Unlike serial IDs, those machine names do not change under the hood, and if they happen to be changeable, then Node type API hooks as well as http://api.drupal.org/api/function/field_attach_rename_bundle/7 is invoked accordingly, to allow modules to react to the changing ID (name) in a controlled fashion.

The most important point, however, is that the entire rest of the system references the configuration item by its static machine name, not an arbitrary serial ID.

The simple change that is proposed to overcome this situation is to

- convert all 'format' columns into strings. Which simply means that $anything->format becomes a string, and instead of an arbitrary serial ID, it contains a unique machine name.

- convert all 'rid' columns into strings. Which simply means that $anything->rid becomes a string, and instead of an arbitrary serial ID, it contains a unique machine name.

Since both of these serial IDs can naturally not be changed currently (as the database simply assigns a new auto-incremented ID for $format->format and $role->rid), we do not have to implement any additional logic or APIs to allow to rename the machine names. Just record them once and be done with it.

The resulting API changes for contributed modules are fairly minimal:

1) Update your 'format' or 'rid' columns (if any) from integer to varchar. (That's it!)

2) If you happen to call filter_format_save() or user_role_save(), then you must pass a machine name in $format->format or $role->rid. (Just like you need to pass a $type->type to save a node type).

irakli’s picture

"Well in most cases I think feature-like module should define their own permissions + check that anyway"

Actually, while it makes sense from the functionality-encapsulation perspective, in practice, that is not a good pattern. At least not for distros. Jeff Miccolis and I talked about it in Copenhagen. When you build features you want to separate "very-probable-to-be-overriden" configuration from a more "durable" one. Permission settings are always a "suggestion" and you always expect site-builders using your features/distro to override those. Hence it's a bad idea to package them with [hopefully] more "durable" configuration like: content types, views etc. because a simple matter of changing permission will cause your feature to be overridden. Overridden features are the plague of distros - they create headaches (or in extreme cases - roadblocks) on the upgrade path.

Dries’s picture

Still being convinced that a simple fix is possible I am planning to go ahead and work with sun and others on a patch for HEAD. We can then judge it on its merits and decide whether we want to commit it to D7 or wait to a D7.1 or D8 with it.

I hate wasting people's time, but nothing beats being able to look at an actual patch that updates all of core. Configuration management and distributions are important so I certainly want to give it a hard look and careful consideration.

My gut tells me that the opportunity cost of not being able to properly support distributions and configuration management, is bigger than a modest last-minute API change. But that assumes that the change is manageable.

Working on a patch for others to look at, including me, is smart. It will help us identify if the change is manageable and if we can stomach it this late in the game. In the worst case, we'll get them into Drupal 8 early in the release cycle.

Based on #63, it sounds like we'd need to get at least 6 patches in. I'm going to have to spend more time on these to help me assess the complexity and impact of the changes. I hope to see the distribution builders and advocates active in these 6 issues because I'll want their input.

To be continued ...

Crell’s picture

The security issue that sun mentions is, I think, the most convincing argument to make an exception for an API change in beta. People are going to export roles and text formats. We can't really stop them. But we should try to not make it a built-in security hole if they do.

I have commented further on #934050: Change format into string about the implementation.

I would note, though, that if we're going to change this API we should do so FAST. Beta is not just a mental break, but there are multiple dead-tree books on Drupal 7 that are going to be released before 7.0 ships; while I agree that such books are at their own risk, we should be aware that we're a year past "code freeze" and "people reading dead tree books about Drupal" are another constituency that get screwed by such late API changes, even more so than module authors. Module authors can patch their code. :-) (Disclaimer: Yes, I am a co-author of one such book and I have been fighting a losing battle with the publisher to delay the book further for precisely this reason.)

Crell’s picture

Dries: See #934050: Change format into string for the filter format one.

alex_b’s picture

Status: Needs review » Closed (duplicate)

Working on a patch for others to look at, including me, is smart. It will help us identify if the change is manageable and if we can stomach it this late in the game. In the worst case, we'll get them into Drupal 8 early in the release cycle.

We'll work under this premise on the related issues that sun has listed in #63 then.

#624018: Exportables and Features support for WYSIWYG 7.x that catch mentions in #61 is actually a great example of what machine names for filter formats could unblock in contrib. The very reason why that patch is postponed is that filter formats are identified by sequential ids, see jmiccolis' comment in #624018-18.

@catch: I am sorry that #881530: Exportability of vocabularies is ruined by taxonomy field's 'allowed values' setting wasn't properly flagged as API change. Like you've pointed out in #881530-38 what was painful about the change is that vocabularies need to be identified by $vocabulary->machine_name instead of $vocabulary->vid. I think this is something that we can completely avoid by converting filte_format.format to a varchar field rather than adding a machine name. Needs further work on #934050: Change format into string.

I am setting this to duplicate because relevant issues discussed here are properly captured elsewhere.

webchick’s picture

Thanks, Alex. I think that makes sense.

I did want to say though that regardless of what is ultimately decided by Dries, something that would greatly help to smooth out tensions here is if we all saw a lot more participation by Development Seed, Phase2, FunnyMonkey, et al in the core queue from now until release. I'm not talking chx, sun, Damien, catch, webchick-level core participation, mind you. Just patching/reviewing a few core issues per week (esp. in the "major"/"critical" priority) would really help to demonstrate good faith, and to show solidarity with the folks who have been driving Drupal 7 and who were understandably put off by the timing of this initiative.

My 3 cents Canadian. ;)

alex_b’s picture

webchick: these three cents make a lot of sense - working on it.

bonobo’s picture

@webchick - we'll be there.

irakli’s picture

@webchick, you got it :)

catch’s picture

So let's document issues where this initiative is likely to push back the Drupal 7 release date, so we know what the trade-offs really are here:

#937442: Field type modules cannot maintain their field schema (field type schema change C(R)UD is needed) which is a spin-off from #934050: Change format into string, and is currently rehashing many of the arguments that bogged down #556022: When a text format is deleted, some modules (like text.module) don't update their data, even though we say they do for months. The eventual answer to #556022: When a text format is deleted, some modules (like text.module) don't update their data, even though we say they do was to punt the bulk update issue until D8/contrib and remove any pretence at bulk updates, that's mutually exclusive with the proposed text formats change, given that we need to support beta-beta upgrades. Making the change before the beta would've been quite doable though, inconvenient for head2head and mongodb_field_storage modules but wouldn't have added an intractable issue to the major queue in the process.

http://drupal.org/node/936836#comment-3562902 - looks like #881530: Exportability of vocabularies is ruined by taxonomy field's 'allowed values' setting may have broken the Views D7 port. No views issue for that yet.

moshe weitzman’s picture

@catch - our beta => beta upgrade is a 'best effort' sort of thing. we try really hard, but if we can't we can't. I don't think we need to cripple a release because of this. Also, the folks who run betas are also the folks who can run drush scripts to fix up their data. I think we should consider offerring drush scripts for long running 'data migrations'.

moshe weitzman’s picture

Or perhaps we put one script into drush core that just processes the ReliableWorker queue. I need some queue system experts to advise here.

walshtechnet’s picture

I have been holding off on commenting on this thread as I thought it was more appropriate to let the technical community members resolve the specific issue. However, now that action is being taken, I want to address the white elephant in #38 - the 5 year D8 critical uncertainty argument - as it continues to crop up in conversations with friends in the Community (particularly those related to this thread).

From my perspective, if Drupal is going to continue to attract new users while maintaining existing ones, D8 clearly cannot remain in development for five years. I therefore feel strongly that we (as a Community) should shy away from justifying positions on any core topics based upon the length of time it has taken to release D7.

In fact, the whole notion of whether 2 years is too long is itself a subjective point that too often is taken at face value in our Community. Having worked on product teams at major software publishers and in a federal public sector business development role, I have grown to appreciate the need for software vendors (and by extension open source communities) to adhere to regular release cycles in order to be successful in a competitive enterprise marketplace. For enterprise stakeholders (an important group of end-users within our own Community), continuous innovation and release of platform products can be extremely disruptive - particularly in environments where compliance standards are paramount - because it undermines the business rhythms of the organization. This is why some software vendors rely on service packs to augment individual patches - they provide an opportunity to bundle patches that substantively evolve a platform product post-release while at the same time limiting the number of major releases for each generation of the product.

I therefore would argue that the Drupal Community should prioritize moving to more consistent release cycles which include "service pack releases." This would help Drupal core, as well as downstream software publishers (who are creating distributions), be able to better sync product innovations with the realities of enterprise deployments - especially for large government organizations who face costly security accreditation requirements. I believe that if such an option existed - and could be adhered to (a separate conversation), D7 SP1 (which would be scheduled for 9-12 mo post-release) would be the perfect option for some (#2; #4) of the challenging issues confronted in this thread.

* Before anyone comments, this thread clearly is not the place to discuss the points raised in this post - they are not core to this issue. I just wanted to throw these ideas out because I think this specific issue provides a nice inflection point for such an exchange. Ultimately, it is my hope that our Community completes a thorough review of this issue as part of a postmortem on the D7 release (conducted AFTER it releases). I believe it would help us to explore whether major SDL process changes for D8 could be undertaken that provide better options for insertion of important 11th hour issue fixes than D8.1-20 or D9. Any option would be better than having such important issues like these debated at a similar stage of D8. :)

Boris Mann’s picture

@waistechnet - you're right, this isn't the right place to have this discussion, and I'm actually tempted to delete your post it's so off topic. We've had this "enterprise needs" discussion before.

Luckily, there are folks like Acquia (oh, look, you work for them!) that you can pay to have slower stuff happen. The mainline will continue operating at the speed it needs to, and I can tell you it will likely be a hell of a lot sooner than 9 months from now for (several) point releases to come out.

KingSalibah’s picture

I am like a baby in diapers, compared to the geniuses that work on this project! So many bright people. But, we really really need D7 to be the most awesome thing around! So much hard work has been done already. Somehow, if everyone increased their knowledge, increased their responsibility and increased their control, D7 will be released and it will smoke! Let's make this go right!

Boris Mann’s picture

@walshtechnet - I realize I may have come off as overly harsh. Let's have the discussion over on GDO - http://groups.drupal.org/node/100584

walshtechnet’s picture

No worries - I understand folks are passionate about this topic. I will follow-up on the other thread. For the record though, the views expressed are my own not those of my employer or colleagues. :)