And let me drive it. Visibility, acknowledgement would help me here, I would say #1366710-6: Remove chx from MAINTAINERS.txt (except migrate) is pretty typical of what I hear on IRC but people don't want to go against the flow so let this be part of the flow.

To start, I talked about separating the form rendering and data structures and #1324618-4: Implement automated config saving for simple settings forms has a good idea which I would like to adopt and also experiment with a light OOP-ification of form API to make it easier to document / integrate / understand. Also get http://drupal.org/sandbox/chx/1088508 in. This two already will effect most of core.

Meanwhile we can take a good, systematic look at the mess of a caching we added to core.

Will keep us busy for a while.

Comments

chx’s picture

Issue summary: View changes

explained a little

andypost’s picture

Great initiative! I's like to help. Also we need a tag because there's already a lot (code cleanup, cleanup adn so...)

Dries’s picture

Thanks to the great brainstorming in #1224666: [meta] Unofficial Drupal 8 Framework initiative, I've actually been thinking about an official code cleanup initiative. Happy to talk more about what that could look like. I'd like to see us agree on some more specific goals so we fix the biggest pain points.

chx’s picture

I gave you specific goals, I thought: split apart the monstrosity hook_menu became by separating router information, tasks and links (and assorted small pieces). Separate the form data and rendering structure. If performance allows use classes instead of arrays for form elements.

This is a beginning of course. We can do more but if this makes into D8 that's already a huge step ahead.

chx’s picture

Oh and then there is the single weaver issue #1331486: Move module_invoke_*() and friends to an Extensions class . I talked in London about https://docs.google.com/document/d/1FbiUha_xsXk7kyPyiCi3THY0ycX52W7sPDZK... this docs as well. Most of it is still relevant and reusable. Not everything, I will write this up more next week.

webchick’s picture

I'd be all about a "Clean up technical debt" (or similar) initiative. That's a huge long-time passion of mine, as are efforts in general that make Drupal easier to learn for new people. Woohoo! :D

But it'd be great to hash out something of an idea of what that looks like (and which has some consensus around it within the core development team), since there's literally no end to what "cleaning up code" means in an 11 year old framework. :)

Other initiative owners are working on putting a list together of issues around their initiatives categorized as "must-have" (this initiative doesn't succeed if this doesn't get done), "should have" (we should really, really try and get this done), "could have" (would be nice if this gets done, but not critical), and "won't have" (out of scope). This is useful for explaining to the larger community what the initiative actually entails, and it also helps focus efforts of people who want to see that initiative happen on the most important stuff.

I dunno if you want to get that fancy or not, but spending some time fleshing out with other folks on what would be encompassed in a "Code Cleanup Initiative" I think would be really valuable. It sounds like it might include some of the Framework initiative, but also some other things too.

willvincent’s picture

I'm in, let me know how I can help.

Xano’s picture

Three short words: DRUPAL NEEDS THIS. I'd be more than happy to support this initiative with chx leading it.

valthebald’s picture

I heartily support and would be glad to join this initiative. especially if lead by chx

Anonymous’s picture

yeah, sign me up. will put some effort into getting a list together.

neclimdul’s picture

Being as this is such a general and far reaching concept I'd be a little concerned of it stepping on the toes of other initiatives which have more defined goals of clean up X, Y and Z. I say this because working on WSCCI its already hard to keep up sometimes with catch trying to cleanup caching and the core move and all sorts of other work going on that is touching thing being targeted by WSCCI.

So concern voiced, there is plenty of code to cleanup needed in core and a Cleanup Initiative could very supplement existing initiatives, spearheading cleanup patches and issues that block other initiatives but are less core to their goals. e.g. I'm working on WSCCI and identify that drupal_form_submit() is the devil and is adding huge levels of complexity to X which is relevant to WSCCI. I post an issue, and tag it to the clean up queue and we discuss and resolve it with a extended group of developers.

Dave Reid’s picture

I support this Cleanup Initiative 599%. I really worry that for every two steps that we are taking forward with our new features and initiatives that we are taking two steps backwards with code complexity and lack of unification. With Drupal 7 I really think we've hit a critical point in we need to do something about it. I would greatly prefer that we have every other core release cycle have a much more limited feature list and also focus on cleaning up, consistency, unification, and eating our own dog food. The Entity API and Field API in Drupal 7 are big shining examples of this; core doesn't even freaking use entity_label() nor entity_uri() most of the time with its own entities.

We see this cycle already in other software like Mac OS X Snow Leopard. Having a "cleanup" cycle also allows us in contrib to catch up. I speak saying that having a tremendous amount of contrib experience. We have seen with both Drupal 6 and Drupal 7 that contrib is lagging with both releases; I do not consider D7CX to have been a success nor is it sustainable.

I am not against new features at all. In face we need them. We need CMI, HTML5, better mobile support, etc. But we can't just keep throwing everything against the wall. A Drupal 8 with HTML5 support but focused on cleaning up our new entity and field APIs would have been what I consider an ideal release cycle after Drupal 7. Adding new features generally contradicts with the goal of cleaning things up, so I don't think we can focus on having ten billion awesome new initiatives at the same time as cleaning things up. I think we are seeing initiatives like CMI and WSCCI, which encompass large sweeping changes, need more time to develop and "get them right." Alternating cycles of cleanups and major wide-sweeping features I think would give us the best path forward. During the cleanup cycle the next feature initiative should be in planning stages so that when the new feature cycle comes out they are ready to hit the ground running, unlike how most of the D8 initiatives have started.

By having a shorter core release focused on unification, I don't think we sacrifice our ability to add new features. By having a more unified core, it should actually be easier to add new features as it is less mess to wade through. Right now we are in fact sacrificing our ability to retain maintainers and keep development momentum. Which is worse? We can't do both and this requires someone to make an executive decision on cycle strategy.

chx’s picture

I would be in love with a Drupal 8 cycle that is just what Dave suggests perhaps with a few of my fixes (most importantly the menu tear apart) added in. Then we can hit the ground running with Drupal 9 with CMI and WSCCI being almost ready as testing and DBTNG was for Drupal 7. They are not ready, not even close to be ready and saps everyone's energy and attention.

Move those to Drupal 9, freeze D8 when our agreed-upon list is done tentatively Denver and release Munich-ish.

Xano’s picture

This may be a bit our of scope here, but earlier this evening I was thinking about new features etc for even version numbers (Drupal 8, 10...), while odd version numbers (9, 11...) do not introduce new systems (or the other way around), but rather improve existing code in terms of simplicity, integration, documentation, DX, and performance. Dave Reid's mention of Snow Leopard in #11 and a "cleanup cycle" illustrates the same idea.

Neclimdul in #10 notes that other initiatives have more clearly defined goals. As chx points out in this original post in this issue, there is a number of subsystems that are in need of a cleanup, such as FAPI. We can set several such goals.

casey’s picture

Maybe Drupal 8 should be a cleanup release (unifying api's but no new functionality) which could be released earlier (lets say end of next summer), while all new features go into Drupal 9. Stuff like CRUD via the entity api could be considered a unification.

edit, we seem to think alike.

Dave Reid’s picture

Yep, the main thing with the major feature initiatives is that it's painful to watch them need so much planning (which they should need), knowing that we're eating time in D8 development, that these major features will land later in the cycle, we will find major bugs that we did not anticipate even later in the cycle, have to just eventually call core "good enough" and release, repeat ad nauseum. Having an in-between cycle allows time for those major initiatives to be announced and start planning while we can still help more core better with the cleanup cycle. I know that initiatives are a new concept for us with D8, but they way they work needs to change sooner rather than later. It may hurt feelings but I think we honestly need to take the drastic step of pushing back some of our current initiatives to D9.

Things that we would need to discuss:
- Changing the policy about only having two core versions being "officially" maintained. If D8 is a cleanup cycle, then we can't and shouldn't really EOL D6 until D9 is released.

agentrickard’s picture

I support the cleanup release concept. Granted, almost all of my core work is cleanup work, but let's look a bit at why.

What I need from core is stability and predictability. Anything else I can write (or find) in contrib. I also firmly believe that we're going to spend half of the release cycle -- no matter how long it is -- just uncovering bugs and edge cases that we failed to consider. Allowing a cleanup cycle to directly address that -- while maintaining the kind of D8 -> D7 backports we have for such fixes -- seems ideal to me, and to our clients.

If I had to make a hit list for D8 right now, it would be:

* Consistently use core api functions in all of core.
* Decouple all CRUD functions from the form system.
* Native support for HTML5.
* Clean up FormAPI
* Finish the EntityAPI
* Improve the maintainability of the menu system -- either through simplification or by attracting new contributors.
* Fix the node access system (ok, that's my pet issue)

andyceo’s picture

I think it would be nice if Drupal 8 release cycle will concentrate on code cleanup and clarification. The core become a real strong and fast basement for all developers and future releases.

I think using php namespaces (#867772: Use PHP 5.3 namespaces) in modules and hook system will give us more clean and clear code, so it may be the thing we must to do in Drupal 8.

sun’s picture

In order to do a "clean-up release", the release must actually contain clean-ups.

However, close to no one is working on those clean-ups, despite even having a list of badly necessary clean-ups and an even longer on more diverse topics than the base system. (same applies to entity CRUD & co patches)

The official, currently communicated stance on this is: (note, totally not my stance)

There's nothing that prevents anyone from working on these issues.

Aside from a /core directory change, D7 and D8 are otherwise mostly the same at this point in time. Kinda needless to say that especially clean-up issues take lots of time and energy to get right and done. And way more often than intended they have similar consequences like major API changes; things will unintentionally break, hard. Thus it would be false to assume that focusing on clean-ups effectively allows us to release sooner.

What I'm able to extract and read between the lines in the follow-ups on this issue is that many would like to decrease the amount of goals and topics being attacked in parallel for the release cycle. And after (heavily) reducing, they'd like to add back a new goal to focus on cleaning up the system to make it more sane and sustainable.

I can get behind that idea, but want to amend that #1224666: [meta] Unofficial Drupal 8 Framework initiative deliberately had "unofficial" in the title, on purpose, right from the start.

Everett Zufelt’s picture

I can get behind the proposal described in #18. I wouldn't care if 0 more accessibility improvements made it into the next release, if the actual point of the next release was to cleanup what currently needs cleaning up. I would happily commit my time, where I can be useful, to move this along.

webchick’s picture

I think the challenge with ditching any and all features for D8 in favour of a clean-up-only release is the following:

  • In order to do this, we need to break APIs.
  • Breaking APIs means forcing work on module/theme authors to upgrade their stuff.
  • Modules/Themes don't get upgraded unless there is critical mass around the community to make the move to a new major version of Drupal.
  • You need compelling new features/functionality in order to build that critical mass.

You're not going to convince end users of Drupal to invest however many thousands of dollars to port their D6/D7 sites to D8 because the menu system now has a cleaner architecture. It's just not going to happen. You can convince them to move to Drupal 8 because you cleaned up the menu system as a result of making Drupal 8 a robust web services capable framework, however. That's why I think we need both. Features can drive API improvements, if done right, and they're a lot more compelling and fun to work on for most contributors, which helps with the lack of manpower issue.

Speaking of that. I see a lot of focus in this issue around the lack of manpower within the core development team, which I agree is a challenge. But I do not understand how locking down the patches allowed into Drupal 8 to only API clean-ups is supposed to help with that.

Clean-up work is not sexy, and it's difficult to rally a massive amount of people to work on that stuff, as evidenced by where the development energy around D8 has been directed so far. When we introduced issue thresholds, we effectively said "Code freeze is back on for another 6 months while we clean up our stuff," and the rest of the community who wasn't already working on nothing but core bug fixes for the past 18 months effectively said "Great, I'll go back to contrib again then; let me know when you're ready." We only recently started recovering from this situation post-DrupalCon London when the thresholds were finally met. Saying "We decided to cancel progress on solving major Drupal pain points until Drupal 9 while we work on even more clean-ups" is only bound to make the problem worse by 1000-fold, IMO. :(

Finally, two things:

1) The initiatives have actually been a great way to bring in some new blood to the core development team who were never involved in Drupal 7, and would never get involved in something as vast as "all of core." They allow people interested in a particular topic to easily find someone very knowledgeable about it who can give them advice about how to be effective and focus on important things. This is why I think a "Code Cleanup Initiative" is a great idea, because I imagine it will have the same "lightning rod" effect around cleaning up technical debt as we've had with things like HTML 5 and Multilingual, for example.

2) Remember that now that we have an automated testing framework, we can be a bit more liberal in the patches we support for backport in Drupal 7. I've committed several patches that add missing hooks, optional parameters, etc. to help contrib module authors today that don't require us going through an entire release cycle. This buys us some time to make Drupal 8 something we can all be really freaking proud of.

Dries’s picture

I think we're selling ourselves short as we implemented a good amount of clean-ups in Drupal 8 already; from removing unnecessary functionality, to improving our APIs and documentation.

Either way, I wholeheartedly agree with the fact that we have a lot more clean-up work to do. As a software application evolves, its complexity increases unless work is done to maintain or reduce it.

However, like webchick, I disagree with the notion of making Drupal 8 a clean-up release. We have to find a good balance between new features development and clean-up work. Like it or not, but both are necessary for Drupal to stay relevant.

chx -- your suggestions are all good ones as far as I'm concerned. I'd like to see if/how our thinking evolves over the next week or so before making this an official initiative. Lookin' good though.

tsvenson’s picture

Dries:

We have to find a good balance between new features development and clean-up work. Like it or not, but both are necessary for Drupal to stay relevant.

Fully agree, but we also need to ask if the current organizational structure are able to support that. Judging by the growing frustration and heated discussions I'm afraid we are far from that.

However, I believe the challenge is much bigger. When I joined the community some 3+ years ago I had not heard of Drupal before. Since then a lot has happened, such as:

  • Drupal has gone from a rather unknown project to a well respected leader and benchmark talked about almost everywhere
  • The community has grown exponentially
  • The ecosystem around Drupal has exploded in growth
  • Drupal is increasingly attracting new kinds of users that are less code savvy

At the same time we have seen a lot happening in the way the internet is used. The introduction of the iPhone, cloud services and the push for HTML5 has turned things practically upside down.

At the same time, the time between major Drupal releases has gone from under a year to multi year:

  • Drupal 4.7 - May 1, 2006
  • Drupal 5 - January 15, 2007 (8 months)
  • Drupal 6 - February 13, 2008 (13 months)
  • Drupal 7 - January 5, 2011 (35 months)
  • Drupal 8 - ? (XX months)

Personally I don't predict Drupal 8 to have a shorter release cycle time than Drupal 7, we are already 11 months into it...

All this is creating frustrations amongst, and between, those who see signs of things spiraling out of control and those who are afraid that we will fall behind unless we add needed new features. This discussion is yet another evidence of that.

Worse is that we see long time contributors losing their passion for the project.

I don't believe there is any simple remedy for this by trying to patch how things have worked before. In the same way as the community realized that DrupalCon grew to big for any one Drupal shop to manage, and take responsibility for, we need to admit the same has happened to both Drupal Core and the d.o infrastructure.

webchick:

Clean-up work is not sexy, and it's difficult to rally a massive amount of people to work on that stuff, as evidenced by where the development energy around D8 has been directed so far.

Then how do we do it? Expecting that it will happen on a volunteering basis will clearly not work. Thus we must look at what other alternatives there are.

Is it time for the community to take the next step and have community funded teams that work full time with Core and d.o?

I believe it not only is time for it, but also that the community will be able to find the right balance and transparency to make the transition as smooth as possible.

We have a solid base to build upon and we can also look at organizations such as Apache and Mozilla for inspiration.

valthebald’s picture

I think that clean-up work can be sexy, if we look at it as a real challenge.
May be I'm wrong, but one of the central reasons of constantly growing difficulties of the core maintenance is too tight coupling of core components (is that issue kind of duplicate of another chx's issue #1255674: [meta] Make core maintainable?).
Not only tight coupling of core components makes it harder for any sane person to keep track of core architecture and possible side-effects of individual commits, but also hardens API upgrades (for both core and contrib code), and prevents proper separation of clean and tiny "platform" code from accompanying, appealing to less geeky people "product" code.
In contrary, loosely coupled code will not require fool knowledge of the big picture to make individual contributions.
So, if we understand clean-up as not only removing unused or ineffective pieces of code, but restructuring it to the platform/product parts, this sounds not just sexy, but VERY sexy.

xjm’s picture

I'm of two minds in this discussion... On the one hand, webchick's points about the reality of upgrades are something I think it's easy to forget for developers--upgrades are hard, time-consuming, and expensive. @acouch also pointed out to me that an earlier D8 would also mean EOL for D6--which most developers might find a relief, but would be very hard on a lot of sites.

On the other hand, though, the release described in #15 - #18 is what I've really wanted for the past six months, and when Dave Reid described that version of D8 yesterday I was very relieved to see I'm not the only one who feels that way. The API changes in that D8 (do we even call it D8?) would be nowhere near as skull-fracturing for contrib authors as the D6-D7 ones.

For something other than the full D8-with-initiatives, I think we might consider the upgrade/migrate work I keep hearing about peripherally. Because if we can un-f*** our update system, we take a lot of the agony out of upgrading, now and in the future.

I definitely do not think an earlier release would or should include a stop on initiative work. It would just be nice to release a manageable Dsomething as a prelude. Edit: D6 to D7 is a scary-hard upgrade, and it would be nice to have an upgrade that is not the even scarier-harder D7-D8 upgrade I see on the current path.

Edited for less rambling. :P

xjm’s picture

Also, I guess I don't have a pulse on everyone, but for me personally, as a contrib developer, I'm more interested in working on core cleanups/APIs/fixes than new core features. "Needs backport" is my favorite flavor of ice cream. Did that many developers really check out because of our code freeze, or was it more just recovery from the D7 cycle generally? Just a thought.

pounard’s picture

Yes, clean-up can totally be sexy! I vastly agree with #11 suggestion, even if I'm working on WSCCI and even I really like this and I really would like to see it on D8, I have to agree that it is non essential for Drupal 8; code cleanup really is.

D8 can be a code cleanup release, the cost of this, in order to avoid a feature freeze, would be that D9 branch could be created right now with new features, while D8 could be clean-up (plus maybe entities API finalization) only, but with a shorter release cycle in order to get on feature track as soon as possible once done.

valthebald’s picture

#25: oh yeah! You're not alone :)
Any chance this initiative can give some push to backwards compatibility as well?

markabur’s picture

I love the idea of a cleanup initiative and love the idea of a cleanup release even more. Personally I'm more interested in 7.twentysomething then 8.anything right now.

I agree that if the changes in a release cause it to be a difficult upgrade, then it ought to have compelling new features to justify the effort. Which means that if there's a cleanup, performance, and dogfood-eating release, then it should be an easy upgrade from the previous version. The feature releases would be harder. I imagine that the #DxCX pledges would be high for the cleanup releases and low for the feature releases -- which is fine, really.

wizonesolutions’s picture

I resonate with #1366940-11: Start a Code Cleanup Initative and #1366940-18: Start a Code Cleanup Initative but can also see the real-world realities in #1366940-20: Start a Code Cleanup Initative and #1366940-21: Start a Code Cleanup Initative.

People care about what Drupal can do and about tangible things. To many people, code cleanup is intangible; they don't even understand much about code to begin with. So they probably won't do the upgrade and will wait for what they'll undoubtedly call the "real" release.

So it makes me wonder what the best approach is. I almost think that cleanup should be a priority, but there should at least be one killer new feature that sites have to have (like CMI or HTML 5) for Drupal 8 that'll make people want to upgrade or build new sites on it.

So I suppose this is most similar to #18...drop some initiatives, keep some that are really awesome (you can tell I favor CMI personally), and prepend the cleanup. I don't think this will shorten the release cycle, really, but it'll enrich it.

cweagans’s picture

@valthebald, Backwards compatibility would be extremely detrimental to code cleanliness.

The changes that chx suggested sound awesome. I'd also like to see some focus on performance. Our memory footprint is quite large.

pounard’s picture

"quite large" => "hopelessly huge" you mean?

Crell’s picture

We should be careful what "code cleanup" means. To be sure, there are a lot of small to moderate things we can do to D7/D8 that would only be minor API changes, or even have no BC break at all. #977052: Implement hook_hook_info() for Field API hooks comes to mind as something that should totally be doable as such a cleanup, for example.

However, there's also a lot of cleanup that cannot be done without major, epic-sized breakage. Unravelling the spaghetti that is render API / template files / theme functions / preprocess functions is critical cleanup work, but that is decidedly a major, probably epic, API break that will take a lot of planning, work, and coordination. Really, the way to do that is to take a step back and rethink how that system *should* work, and redesign it around that model rather than just throwing more small patches at it. So... is that "code cleanup"?

I'm not yet for or against a "cleanup only" release. I'm undecided. But a lot of the reason some of the initiatives are so big is because, well, they have to be. "Cleanup" and "small, manageable changes" are not always synonymous.

Jacine’s picture

I totally support and welcome this initiative 100%. If cleanup doesn't happen, Drupal will suffer, so I am very to happy to see people excited about it.

However, there's also a lot of cleanup that cannot be done without major, epic-sized breakage. Unravelling the spaghetti that is render API / template files / theme functions / preprocess functions is critical cleanup work, but that is decidedly a major, probably epic, API break that will take a lot of planning, work, and coordination. Really, the way to do that is to take a step back and rethink how that system *should* work, and redesign it around that model rather than just throwing more small patches at it. So... is that "code cleanup"?

I'm wondering about this too and couldn't agree more with taking a step back and rethinking how it should work. Lots of people care about this and have no idea where to begin to get it under control, myself included. So, I hope that there is room in this initiative to take this on, because it needs the smartest developers we have on it, or there is little hope IMO.

catch’s picture

I agree with Crell that we need to clarify what clean-up means, at least for me 'code clean-up' here means 'refactoring', and I'd like to see Dries' list of what we got in already, because off the top of my head I was only able to come up with a very short list.

I think what we are really talking about here is not a bit of a spring clean, but more like underpinning a house.

For me the entity system API completion work is a vital part of that underpinning, because what we basically did in Drupal 7 was add an extension (Field API), then bits of the ceiling started collapsing in on our heads when it interacted with mostly untouched 5+ year old APIs like tracking of node type status, disabled modules and the major version upgrade system. The entity API in D7 is a few bits of Field API factored out where they didn't make sense, and multiple loading, and that's it - then contrib modules have taken that to try to make it something useful for them. It is only clean-up in the sense that if we complete the entity API, then we'll be able to reduce the maintenance overhead of the various entity providing modules (both core and contrib) because they'll all be built from a single solid foundation. Right now it is jammed under, between, and over, existing legacy code which wasn't built for that purpose, but there's also a decent plan for fixing that that's been worked on for some time now.

There are other areas that are seriously overburdened that currently don't have any active work on them at all - the module system, FAPI, rendering, theme system (and as or more importantly the interactions and interdependencies between all of those) are all in desperate need of attention, and they are causing pain both for contrib authors and for those of us trying to deal with the various race conditions, performance issues and bizarre bugs that are entangled within in the core bug queues.

Both the CMI and WSCCI initiatives are likely to include a lot of refactoring of existing code as well as adding new APIs, but I am already concerned about WSCCI's interaction with the current state of the module system in terms of both performance and fragility with the current patch on that issue. To go back to the house analogy, HTML5 is replacing our rotten windows (and fortunately replacing rather than re-painting since there is genuine refactoring of markup and CSS going on as well as introducing HTML5 stuff), WSCCI feels like building an extension and moving some cramped bits of the existing house into it. But there is also subsidence and termites in the basement.

arianek’s picture

Another +1 to the suggestions of chx, davereid and others to do a code cleanup initiative, if not release cycle.

Particularly supporting xjm's comment #24 and pounard's comment #26 - I think a major cleanup could be exciting and re-motivating for many people who've lost their Drupal mojo.

I know there are many large refactorings and dumb small issues that have languished for far too long, and it would be amazing to clean them up and close the issues! This would help get Drupal closer to a more "release ready at all times" point, and decrease the substantial amount of technical debt that has been accumulated over the years.

From my side, it would also significantly help us to do a Documentation and Help System "cleanup" and refactoring, and give time to work on two issues of focus #1031972: Discussion: What would a better help system for D8 be? and #1291058: Discussion: Make a curated docs section/system (once #1278256: Develop a plan to make it more clear that the current Documentation on drupal.org is community maintained. rolls out). The documentation for Drupal 7 was never completely finished or brought up to snuff, and it would be a great opportunity to re-energize people to work on building the "Curated Docs" infra and then updating and writing the content for the Drupal 7 Core "curated docs". Launching D8 in a place where the online docs are up to date seems almost impossible right now, and this might give the opportunity to catch up before huge amounts of new features need to be documented.

The in-code/API documentation may be in good shape, but the front line online docs for many new and existing Drupal community members (and potential members) has a ton of room for improvement, so I think this would serve not only core but the documentation for core very well.

valthebald’s picture

@cweagans: Of course, backwards compatibility is a separate task, but code cleanliness could ease it a lot (http://drupal.org/node/1248802#comment-4862786)

chx’s picture

Very briefly: I will work on scope this week. I seriously doubt a major refactoring of the render API will be in it, however, because I have no clue what that would be. Also, I somewhat think it's beyond scope. Finishing entity API, absolutely. That's basically what this initiative wants to be: take the crufty pieces and smooth them out and not fundamentally changing how things are done.

We might change drupal_render to loop instead of recurse to improve performance if that's possible, for example.

Dries’s picture

I agree with Crell that some of the "clean-up" tasks are everything but minor. That is perfectly fine though. Take i18n/l10n as an example. Gabor has been doing a lot of clean up on i18n/l10n; he has been doing that in small incremental steps. I think we committed at least twenty patches that clean up various aspects related to i18n/l10n. I personally like these small intermediate wins and I'm quite comfortable with them as I know Gabor has a vision for where he wants to end up with the i18n/l10n work. In other words, I think we could "evolve" towards a Better World, as long we can paint a relatively clear picture of where we want to get to.

neclimdul’s picture

I'm very happy to see chx's clean up cruft not rewrite API's approach to this and think its firmly in the sort of initiative we could still support during this release cycle. We've got a lot of cooks in the kitchen at this point integrating major changes so I see it as something we need to be careful about. As far as performance improvements to rendering in drupal, I know dstrauss is struggling with this right now as something to do in D7 Pressflow. It might be worth talking to him about inside or outside of this initiative.

@catch I actually see the current initiatives as the places where we're designing and building a new modernized extension to replace a bunch of temporary buildings and aging structures. That's the point since those tasks are larger overall designs not individual patches which is why we need the conceptual grouping and teams. I'd see this new initiative as the exception in this metaphor and focusing on fixing existing structures that are weak but not necessarily outdated. Replacing support beams and rotten columns with more sturdy versions that work in more or less the same way. Or just ripping out rotten terribleness that we really have no use for I guess. :)

My concern as stated earlier is that when you start shoring up one of those buildings other initiatives are working on and then we have a nice shiny replacement and a sturdy old building and a lot of wasted effort and some even tougher decisions. Even worse and more likely, cleanup starts on the basement and then we drop a new building on that doesn't fit the new support structure. Git merging is great but design changes still require a lot of human work to fix subtle design bugs in the integration. I don't think chx's plan will do this but I think its still a valid concern and, going out on a limb, something I'm sure he's willing to take on as part of his initiative.

catch’s picture

@neclimdul: I agree on this with the initiatives, d8mi is mostly clean-up (at least so far). CMI is building a direct replacement for a very poorly subsystem (variables) and one off storage in others, then refactoring others to use it. wscci has blocks/regions/rendering pipeline in its sights which are also in desperate need of refactoring. However at least with wscci it is not clear to me at least exactly what is and isn't out of scope for that initiative in terms of subsystems.

For example in the back of my mind I think there will probably need to be work done on the router system - for handling responses other than HTML for example so we don't have to have menu callbacks that call exit(), but that's not really been discussed yet, or how far it might go, or how it might interact with other plans to refactor the router (such as decoupling it from links).

xjm’s picture

xjm’s picture

During the "Fireside Chat" this morning and the subsequent IRC discussion, I tried to sort out in my head the common threads in concerns raised by people in this issue and in IRC.

Points of concern:

  1. Any new features or significant architectural changes will always, inevitably, introduce technical debt.
  2. D7's 18-month code freeze was an especially painful example of such technical debt.
  3. We spent the first 8-9 months of 2011 working primarily on the backportable technical debt from D7, and we're just now patching the first non-backportable technical debt (entity cleanup).
  4. We are nearly a year into D7 release and no code freeze date for D8 has been remotely considered.
  5. Most of the initiatives are just landing their first patch or two.
  6. It can seem like the core initiatives are the main focus of D8, and that D8 will not ship without them.
  7. All the points above lead to the fact that D8 seems a very long ways off.
  8. I think a lot of contrib maintainers, myself included, are very concerned at the thought of having to wait for years in order for the D7 technical debt to be paid off at D8 release.
  9. Initiative owners are also affected by this, as many of the initiatives rely to some extent on the entity cleanups, and furthermore a lack of a timeline makes it harder for initiative leads to determine the proper scope.
  10. Unpredictable release cycles are harder on everyone.

However:

  1. Dries, webchick, and others reassured us today that it is not required for initiatives to be "done" for D8 to ship, and that they are not intended to trump other work.
  2. The initiatives also actually include some of the refactoring and cleanup that is needed.
  3. The introduction of issue thresholds helps ensure that our technical debt will never get so out of control again.
  4. To some extent, different sorts of people work on maintenance/cleanups than work on initiatives.
  5. Our test suite and testing policies make it safer to backport more, as webchick mentioned above.
  6. The sky is not falling. :)

My proposed solutions:

  1. This initiative.
  2. Set a code freeze date for D8 as soon as possible, and make it earlier rather than later. Let the code freeze date help determine the feature set, rather than vice versa.
tsvenson’s picture

xjm: Looked over the clean up core action list and it looks great.

One thing I don't really find in it is core features such as #967566: Several important Core User settings need to implement hook_field_extra_fields() and the mess the Update Manager is in to take two examples. Making it update Drupal Core would of course be fantastic, but at least fixing it so contribs and themes can be updated without any issues would be a great start.

For me as a site builder those things are in many ways even more important.

Also, what about things such as making the node title a proper field/entity, plus others that didn't make it, and maybe even get the File Entity module into core to complete it?

While I fully understand, and agree with, the needs of getting APIs and other dev stuff up to par, I also see many benefits of completing site builder features such as the above.

Would be great to know these things is also covered in the cleanup.

damien_vancouver’s picture

[ EDIT: This post has been severely edited for relevance and brevity. The full op-ed version can be found here: http://groups.drupal.org/node/211383 ]

Further to the ideas in #14 - 20, etc. about maintenance / bugfix releases, I'd like to suggest we consider end users and their reliance on stable contrib as priority #1, and introduce a "Testing" phase into the release cycle, borrowing from the Debian Linux model.

On the planned Drupal 8 release date, it would become "testing". This would signal the start of the efforts for module maintainers to port their modules to the new version, and early adopters to kick the tires. Brave people could even use it to build sites for their customers. Freeze restrictions are relaxed as suggested at http://groups.drupal.org/node/210973, so we can actually fix things effectively during that time it's in testing.

Core work can forge ahead with Drupal 9 as "unstable", with no contrib available, much like D8 is right now.

Drupal 7 meanwhile would remain as "stable" for another year (or however long it is). Drupal 7 contrib would get a chance to catch up, customers get more value and time to live out of their Drupal investment. It would keep all the restrictions it now has, which are safe and will not cause breakage.

When Drupal 8 it is declared actually stable, then it is released for a second time as Stable, Drupal 6 goes EOL, and Drupal 7 becomes "oldstable". The default version that users get when they visit Drupal.org and download becomes Drupal 8. More rigid API freeze restrictions go into place.

xjm’s picture

Edit: Discussed in email, so not needed as a public post here. :) Carry on.

Anonymous’s picture

I'm interested in this code cleanup discussion. I am of the philosophy that Drupal should be a small but effective comprehensive development system. I have run into some of the issues mentioned in the leadup to this thread and I hope that Drupal remains something one developer can be an expert in. Glad to see people doing stuff about this.

I will be at DrupalCon Denver if there are any code cleanup events or discussions.

arianek’s picture

@xjm - was just wondering if you can clarify what you said on the other thread ([#5348380]) about damien's post not being relevant to the cleanup initiative. there was a lot of discussion about where to expend core energy and what to prioritize on this thread, so it seemed relevant to me, and i'd actually like to hear that as well.

For those playing at home, the conversation was moved again over to http://groups.drupal.org/node/210973#comment-695708

xjm’s picture

@arianek -- The current post is relevant, if tangential to the title topic. :) I was referring to the previous variant.

@damien_vancouver -- If I understand correctly, the release cycle you suggest sounds fairly similar to what we have now, except with stronger support (and hopefully more contrib work) for the "testing" phase. I guess the key things we'd need for that would be:

  1. Complete, stable, well-tested upgrade path prior to that phase. I think it's clear in the thread above that we need to make that a priority.
  2. More effort to get contrib developers involved. (But how? I personally am just starting to port my sites to D7, so as a contrib maintainer, there was little incentive for me personally to work on my modules' ports before now, much less before D7 release. This probably merits a separate discussion somewhere on g.d.o or such.)
  3. And then the point I'm not sure about, which is the point closest to the topic of this thread... API stability. On the one hand, as a contrib maintainer, I find it hard to justify working on a port if the API is going to change and break stuff. On the other hand, as the backport policy thread and your post suggest, not being able to fix things in the API and the DX also hinders contrib maintainers. I find myself wanting to both have and eat the cake. =/

An aside, for anyone following this issue who might not have seen this post:
http://buytaert.net/drupal-8-feature-freeze-december-1st-2012
Maybe that Dec. 1 2012 date corresponds to the start of @damien_vancouver's testing phase?

neclimdul’s picture

Doesn't @catch have an issue open about this? Is all this really related to a code clean-up initiative?

xjm’s picture

Well, the point about how and when we freeze APIs is, sorta, like I said. :) However, the rest of the discussion is more appropriate to g.d.o.

arianek’s picture

Thanks for the response xjm, I was like huh?! Will keep following this over on g.d.o...

sun’s picture

Not sure what the status and proper status of this discussion and issue is, but want to point out that

  • many long-standing clean-up ideas are "unintentionally" happening through parallel initiatives and efforts already.
  • entity CRUD operations and feature completion is actively happening through the Entity system effort.
  • router system improvements are happening through the Web services initiative.
  • massive clean-ups and decoupling/untangling of subsystems is implicitly happening through the Testing system effort.
  • further clean-ups and streamlining is happening through the Framework initiative and general API clean-up tasks.

Get involved:

» Entity system
» Web services
» Testing system
» Framework initiative
» API clean-up

andypost’s picture

Is this still major?

chx’s picture

Status: Active » Closed (won't fix)

No. We are not interested in clean code any more.

chx’s picture

Issue summary: View changes

fixed typo