Problem

  • Too many interdependencies between the base system and the module/hook system, especially in the System module:
    • installer
    • bootstrap levels prior to DRUPAL_BOOTSTRAP_FULL
    • registry (class autoloader)
    • cache system
    • most other subsystems
  • Too many interdependencies between modules.
  • Some base system features are not really required to run Drupal, but there is no way to remove them from loading without hacks (altering them out etc.)
  • Some "required" modules are actually not required to run Drupal (e.g., Node), but there is no way to remove them without hacking core.
  • Overloaded System module.

Symptoms

  • Low-level subsystems cannot be fixed or refactored without simultaneously refactoring others.
  • New dependencies are added without proper documentation or considerations about consequences.
  • Many chicken and egg situations where dependencies are recursive/mutual.
  • Large runtime code-weight for rarely or never used features.
  • Inability to unit-test most subsystem code, since dependencies cannot be mocked.

Definitions

  • Core include files: Files in /includes/*.
  • Module system: the ability to load modules and invoke/define hooks.

Principles

  • There should be a clear dependency chain without recursion or mutual dependencies (file A and file B should not both depend on each other, nor module A and module B).
  • Functionality that is early in the Drupal bootstrap process should not depend on functionality that comes later in the process.
  • Functionality in core include files should never depend on functionality in modules or on the module system.
  • Functions related to one purpose should be grouped/isolated/encapsulated into one file.
  • Files that contain functionality that is not really required to run Drupal should not be automatically loaded on every site, but should be in optional modules or classes that can be auto-loaded/enabled when needed.
  • All files should use a consistent, unique namespace for their functions (even include files).

Proposed Solution

(many of these are closely related, there are a lot of ways to cut spaghetti)

See also issue tag queue: http://drupal.org/project/issues/search?issue_tags=Framework%20Initiative

Challenges

  • We may add new modules with no administration screens; the modules screen and help system are not well suited for this yet.
  • [possibly more to come]
  • This is a lot of work, we need help!

What this is not

We are NOT talking about the following:

  • removing modules from core
  • having a separate 'framework' download from the Drupal CMS.

However if you are interested in these topics, then you want to help this effort, because it will make those topics possible, whereas they are currently not.

Benefits

  • Proper unit-tests for subsystems in core can be written.
  • Faster resolution times for critical bugs: they can be fixed in an isolated way.
  • Simpler to backport functionality to previous versions, since it's better encapsulated.
  • Ability to use parts of core as components in other applications.
  • Ability to incorporate components from other libraries and PECL more easily (e.g., Zend framework/Symfony3).
  • Ability to omit unneeded functionality instead of altering away or hacking it out.
  • Ability to build our own PECL extensions for low-level subsystems to replace user-space code with C (optionally).

How does this fit with other initiatives?

The WSCCI and CMI initiatives aim to modernize outdated parts of core. They are mainly aiming to do so 'from the ground up'. However there is a lot of low hanging, large, and very smelly fruit between the ground and the sky.

How can I help?

If you are subscribing to this issue and want to help, note that there is a very long list of issues linked from the opening post that are dying out for reviews, re-rolls or initial patches so it should be easy enough to find somewhere to start.

If after looking at that list you don't know where to start, grab catch on IRC (#drupal-contribute), or just ask around, and we'll try to help you find something.

Comments

Issue summary:View changes

Updated issue summary.

#679112: Time for system.module and most of includes to commit seppuku marked as duplicate, we need to move the individual issues from there to the summary here, but this is the synopsis of that issue and more.

lot of low hanging, large, and very smelly fruit between the ground and the sky.

then you should codename this Durian.

Issue summary:View changes

Updated issue summary.

The Blender project already has a project called Durian. (http://sintel.org).

In keeping with common naming exceptions, I propose better_durian. =P

All joking aside, I really like this idea, but I think that it needs to be divided into smaller actionable tasks before it will take off. A good starting point would be the list of sub-issues that were created off of #679112: Time for system.module and most of includes to commit seppuku.

Here's the list:

#1008166: Actions should be a module
#1018602: Move entity system to a module
#1033410: Move token.inc to a module
#1033818: Move xmlrpc includes into xmlrpc.module
#1120928: Move "history" table into separate History module
#1161486: Move IP blocking into its own module
#1167144: Make cache backends responsible for their own storage
#1033818: Move xmlrpc includes into xmlrpc.module

The issue summary looks great but after reading it there is one thing that I'd like us to consider, quoting from a g.d.o. discussion I had with Gabor some days ago:

In a broader scope however, the discussion above seems to outline that we lack a reliable way to split and/or modularize our code without resorting to modules (pun not intended :), which IMHO should be user-oriented groups of features, not developer-oriented portions of code. These two often overlap, but here we have an example where they don't and IMHO users should take precedence in this case.

Does this make any sense? IMO presenting users a very very long list of API/helper modules might be overwhelming. Can't we find a way to separate the concept of module as an extension enabling features the user can directly take advantage of, from the one of module as a way to avoid loading unneeded code? Maybe plugins are the answer?

If / when the current entity code gets moved to a module, entity.module and field.module will be cross-dependent : field.module heavily relies on entity_get_info(), and some generic entity ops (but not all, that's a known discrepancy that was pushed for d8 to polish) call the corresponding field_attach_[op]() function.

Not the 1st time I wonder whether Field API should become a mere sub-part of the entity API. I.e, can/should the entity API ignore that (some) entities have fields ?

I think we could fix that by having field module implement entity hooks and add a dependency (except that at the moment they're both required). When field api initially went in there was this desire to have it not know about nodes, users etc. but IMO the constraints the field api imposes are partly what led to an entity api, and like you said we already made that switch partly.

Issue summary:View changes

Formatting fix.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

updating summary.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

re @catch #6 : lol @ perfect circle back to CCK and hook_nodeapi(). Yes, I guess that's how it should be.

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

Ready to help, starting from end of August.
I would propose this to add in proposed solutions: remove file generation from image module and create a common independent layer for static file generation / cache (would benefit for both Image and CSS/JS aggregation): http://drupal.org/node/1014086 (that may be need to be renamed)

Subscribing, and also adding a link to another related issue:
#782672: Loosen the coupling between the user module and the installer/maintenance theme

Our best guess with catch is that the reason field API did not react to entity hooks because there were no entity hooks (chicken, meet egg). So yeah, full circle but it makes sense why.

Issue summary:View changes

Updated issue summary.

I would like to help, subscribing.

Issue tags:+Durian

I think it's time to officially christen it to Durian, way catchier than "API clean-up" .

I think we could fix that by having field module implement entity hooks and add a dependency (except that at the moment they're both required).

Yep, I agree that field module should depend on entity module and just implement entity hooks. Though it's going to be more complicated for the form and display stuff...

This initiative is an excellent idea -- lots of code debt to pay off here :D

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Updated issue summary.

removing modules from core (#smallcore)

For the record, this is a blatant and frequently-repeated mischaracterization of the stated purpose of #smallcore. It was repeatedly stated that "Could you separate the framework from the modules?" was a useful litmus test for functional separation between project components, but it was also repeatedly stated that splitting modules from core and removing "crufty" modules like Blog were premature goals that would harm the project.

I wish your initiative the best, and perhaps after a couple of years the people who were fundamentally opposed to it have reconsidered. But you'll have to figure out another way to distinguish it from Smallcore - what you're describing is what the smallcore folks were talking about.

EDIT: The original compare/contrast section that defined this initiative vs. smallcore was removed. The existing clarifications in the above document are useful ones, and I'd like to note that i do support the proposal's goals. Removing the tight coupling between Drupal's assorted subsystems, to the point that 'every module could be disabled,' is definitely a worthy goal.

Issue summary:View changes

Updated issue summary.

For the record, this is a blatant and frequently-repeated mischaracterization of the stated purpose of #smallcore. It was repeatedly stated that "Could you separate the framework from the modules?" was a useful litmus test for functional separation between project components, but it was also repeatedly stated that splitting modules from core and removing "crufty" modules like Blog were premature goals that would harm the project.

It was a mis-characterization that was repeated by both supporters and detractors of #smallcore. Lots of people posted patches for Drupal 7 (including me) with the title "Remove X from core", then lots of people would subscribe to the issue with something along the lines of "#smallcore Yay". While there may have been a much more thoughtful subtext to some of this, it is how most people thought about it, and I don't think you can mischaracterise a hash tag. I don't remember anyone arguing against decoupling core at any point, lots of people disagreed with the silly hashtag and fanboyism though.

However, I've updated the summary to remove the explicit reference.

I don't remember anyone arguing against decoupling core at any point, lots of people disagreed with the silly hashtag and fanboyism though.

There's a big difference between "Decoupling" and "unbundling." Ensuring that it's possible to disable system.module without breaking all of Drupal is decoupling, while removing turning blog.module into a contrib project is something separate entirely. While I think an eventual long-term goal of distribution support and a smaller, lighter core are desirable goals, keeping that distinction clear is probably an important one for the initiative outlined in the original post.

Issue summary:View changes

Updated issue summary.

Right, blog module is pretty self-contained and with clear dependencies, so the question of where it lives is mostly social. Code maintenance (whether extra code to carry along between core versions or finding people to look after retired modules in contrib), user expectations, packaging. Those discussions are fine but in terms of patches it is mainly git rm modules/stuff-we-don't-like and/or git add modules/cool-way-to-install-stuff (i.e. http://drupal.org/project/project_browser).

However up until 2008 (and apparently since Drupal 3 according to agentrickard's presentation), blog and aggregator modules, while both optional, had strange inter-dependent code between the two, see #233407-11: Aggregator module hard coded references to blog module for the code that was removed.

Until that patch got in, git rm modules/blog or git rm modules/aggregator would have to be accompanied by a patch to the other module (since core modules can't rely on contrib ones).

Other than ripping it out, 'blog-it' could also have been made into an extremely tiny core module, depending on both blog and aggregator - which again if either got removed would not have required any surgery removing individual dependent lines of code, grepping around just in case etc.

Keeping the responsibility for a particular feature self-contained (or with clear dependencies) helps a lot of things other than ripping it out later - like refactoring, testing, or just understanding where the hell things are coming from.

So regardless of what people think about packaging/bundling issues, we can work on the actual separation of concerns with a lot of agreement (judging by the responses to this issue anyway, yay!), and that will take us a long way towards a lot of different things.

For the "move stuff into classes" and "convert include files into classes" goals, we could create a whole bunch of Novice issues, once we've defined their architecture.

Very infrequently used stuff like Flood, Locking, etc are trivial targets. Simplest architecture would be a class for each with static functions. Then, convert all callers from whatever_*() to Whatever::*(). Part of the architecture discussion should be whether there should be a namespace prefix for the classes; e.g., CoreWhatever.

Might make sense to open an architectural discussion issue for that?

Assigned:Unassigned» Dries

Adding this to Dries's queue.

There's discussion over at #495608: Move parts of robotstxt module into core. about whether robots.txt should be generated from code in a robotstxt.module, so that would mean it potentially falls under this initiative.

I want to help, bring those Novice issues =)

In a broader scope however, the discussion above seems to outline that we lack a reliable way to split and/or modularize our code without resorting to modules (pun not intended :), which IMHO should be user-oriented groups of features, not developer-oriented portions of code. These two often overlap, but here we have an example where they don't and IMHO users should take precedence in this case.

Does this make any sense? IMO presenting users a very very long list of API/helper modules might be overwhelming. Can't we find a way to separate the concept of module as an extension enabling features the user can directly take advantage of, from the one of module as a way to avoid loading unneeded code? Maybe plugins are the answer?

This is the choke point that quite a few Drupal architectural discussions have run into over the past several years. One possible solution is separating different types of modules in the UI; behind the scenes, install profiles are effectively modules at this point. In the future it might be worth splitting off another type of module that is "pure API" and has no behavioral impact on the site or its operation -- essentially a wrapper for Drupal Plugins or helper functions or what not. Right now we're stuck with the 'hidden' flag.

One of the biggest lines in the sand that separates includes from modules at this point is that includes are passive: they do not implement hooks or participate as "responders" in any event driven Drupal operations. That distinction is pretty important, I think.

I had been thinking about dependencies on the module system (i.e. invoking hooks) as a reason to move something to a module, but discussions over the past 2-3 days in #drupal-contribute and eaton's post here and the token issue suggest we could shift that a bit.

Currently the module system depends on the module system - since absolutely required infrastructure for it is required by system module, but this is a mess and one that CMI should help to clean up.

However, on irc we were discussing what the minimum definition of a working Drupal installation would be, and came up with installing a module.

This is pretty rough, but could look like this:

- download Drupal
- create files directories
- set up settings.php for database connection etc.

But then if you have drush installed, you never actually need to go through the installer (no drush si) - you can just immediately start installing modules.

All systems lower level than modules use CMI for storage or otherwise use pluggable back ends (and we ship with file implementations for many of these).

This also means it ought to be possible to have the project browser available very early on, possibly early enough to browse install profiles on Drupal.org and download their dependencies.

I've always been a big fan of the "final success metric" being, "Can you install Drupal with none of its bundled modules turned on, have it successfully generate a 404 error, then turn on your one custom module and have it generate a page successfully?" There's a lot of stuff in between here and there, but most of it is covered in the proposed patches above.

At least in my experience, a lot of devs who are frustrated with core bloat are less frustrated with /includes than they are the growing list of required and tightly coupled modules. The amount of emphasis on teasing apart those couplings and requirements in the patch list above is IMO at least at really big win.

Overall I like this plan, though I'd quibble with some details.

For example, #1033818: Move xmlrpc includes into xmlrpc.module makes no real sense - xml-rpc is already an isolated subsystem. Or is making it a module a way to disable it without using e.g. .htaccess?

I too am interested and would like to help.

If you are subscribing to this issue and want to help, note that there is a very long list of issues linked from the opening post that are dying out for reviews, re-rolls or initial patches so it should be easy enough to find somewhere to start.

If after looking at that list you don't know where to start, grab me on irc (#drupal-contribute) and I'll try to help you find something :)

Issue summary:View changes

Add path vs. locale decouple link

I agree with the document and I fully support it. Last year development of Drupal core project was really frustrating for me and I must admit that despite my nomination as Drupal core maintainer for the Openid module and despite my 5-6 years experience and friendships in the community I have been actively searching for some CMS/framework alternative which could replace Drupal - at least partially - because it seems to me that Drupal will implode with its growing complexity sooner or later ...

This "Durian" (?) initiative gave me new hope :-)

+10^6

I also think this is the way to go with Drupal. Have a solid, and small framework to handle just the necessary stuff and create a standard distribution project to do everything else. We could have a solid small framework D8 version in a third of the time and then have a reasonable time to update modules before jumping into D9.

This is actually ridiculous but no matter how much modules we keep adding to core, Drupal doesn't do anything cool out-of-the-box, I still have to download a ton of modules. And Drupal is so flexible that core cannot be a solution for everything, not even to have a small blog, and I think that's the way we're going.

I don't want to say the magic word, but if we keep building a bigger, slower and really complicated product, someone will create a fork eventually.

Have a solid, and small framework to handle just the necessary stuff and create a standard distribution project to do everything else.

Just to reiterate again, this issue is not about having a framework + a distribution product or anything like that.

Before you can separate anything out, or begin to properly define what should and should not be in a core offering (with some exceptions in the definite not category that haven't made sense for years), all those different core components need to be decoupled architecturally so it's actually possible to see what's in there and what the dependencies are.

This issue is about that task, not about packaging etc. which are all social/management/packaging issues (which are fine to discuss, but there's other places to do that). It will undoubtedly be easier to do some of this refactoring if we drop some legacy cruft from core (like profile module), but that's also a different discussion.

OK, so who's preventing the decoupling from being done? I take it, before anyone spends many hours preparing patches to accomplish these tasks (and possibly some patches or calls for patches are already out there in various stages of completion?), core developers would want to know that their patches would in fact be committed? So, is it a matter of convincing certain people with commit authority for Drupal 8? or is it more a matter of getting other core developers to commit to cooperating on planning how to go about doing the decoupling and then working on it together?

Can we (by we and mean you guys who are core developers since I'm mostly an end-user) just start already? If you state clearly who we need to get on board, maybe we can focus on convincing them directly.

Because if Sun, and Chx, and Jeff Eaton, and Dave, etc. are all more or less for this but none of them can make it happen, and you are all among the most inside people other than Dries himself, then it's really confusing for the rest of us to figure out how ANYTHING core-related actually gets decided. Is this a community meritocracy or isn't it? It's really mysterious to me.

Just to reiterate again, this issue is not about having a framework + a distribution product or anything like that.

Nthing Catch's comment. Separating product and framework components of Drupal into separate projects or distros is a separate issue: that goal would certainly benefit from all the work being done here, but we gain significant benefits from the patches outlined in this meta-issue even with a monolithic Core Download.

subscribing. Very cool initiative.

Issue tags:-Framework Initiative

Subscribing.

Issue tags:+Framework Initiative

Oops, saved too late, accidentally restored tags. Fixing.

I'm very much in favor of this, hopefully the "un" will really be dropped soon (or things will be committed anyways).

Issue summary:View changes

Add a how to help message.

Issue summary:View changes

fixed the url to issue list

As part of this might it also be worth considering: #1263478: Identify pieces of Drupal that could be replaced by Symfony code ?

I don't want to say the magic word, but if we keep building a bigger, slower and really complicated product, someone will create a fork eventually.

Sometimes we need a crisis and a movement from someone to see a big new step, and a fork can be.

Oskar

This issue is just plain awesome IMHO. The Drupal core is overly complex, but this work will make it much easier to recruit more hands on deck for later core initiatives!

menu system, anyone?
Menu links should be optional, and navigation menu should be too, and with it menu_navigation_links_rebuild.
(my favorite topic, but fits here quite well I would say)
#1170278: Introduce hook_menu_router_update()
Following that, breadcrumbs need to be decoupled from menu_links and navigation menu.

@sin, please don't comment to subscribe anymore. Click the follow button at the top of the issue instead. thanks :)

I just came up with a possible approach for simplifying and non-special-casing the installer (also maintenance etc) at #1226380: Generate minimal mocked or SQLite environment and use to install Drupal in full environment that has a lot of crossover with the tasks here. I have a patch over there that identified the areas of core that we need to make DB independent to be able to bootstrap (a very minimal) Drupal without a database:

- variables: we need to ensure they all have working defaults, and skip trying to load them if we have no DB available
- registry: we need some way to collect defaults and use those if the DB is not available (may be easier when CMI lands)
- lock: we need to make this pluggable, and make a blackhole version (same as cache-install.inc)
- menu: WSCCI should be pluggable enough we can replace with a trivial router that just says "no db? call install_page function".
- system: we can rejig the logic so that if no database is available it rebuilds the system array from the filesystem on each request.
- node & user: ideally these dependencies can become optional (in the meantime, the node query should go away if we set to something else, IIRC).

@nielsvm, see #110: please don't comment to subscribe anymore. Click the follow button at the top of the issue instead. thanks :)

Is there any movement on the first bullet point of fixing dependencies on our includes? Also, does that include providing more sane, base framework of includes/functionality especially in early bootstrap. I keep running into weird wrapper functions and self contained functions like drupal_system_listing that for no apparent reason are only available in a full bootstrap and would like to follow any sort of discussion around fixing that.

Assigned:Dries» Unassigned

I just re-read this issue and there is a lot to like. The issue is assigned to me but I'm not sure what feedback is expected from me.

Generally speaking, I'm fully in support of "decoupling core" (better separations), but not necessarily of "unbundling core" (moving things to contributed modules).

If a specific question needs answering, I'd be happy to try and answer them and to help us make progress. Just re-assign the issue to me along with your specific question(s).

Issue summary:View changes

Updated issue summary.

Minor status update to break the silence:

We're apparently making pretty good progress already:
http://drupal.org/project/issues/search/drupal?status[]=Open&issue_tags=Framework+Initiative

We could use some help with re-rolling against HEAD -- the essential refactoring is prepared/done for many issues already.

OMG, I forgot this list of tasks existed. THANK YOU SUN!

Following, all the issues....

Issue summary:View changes

Swtiched to the new issue about moving the lock backend to PSR-0.

Issue summary:View changes
Status:Active» Fixed

Looking at Drupal 8 today, and seeing how much we've achieved - hitting the mark here and even going beyond that - I believe it is safe to call this "Frameworkification" meta issue as "Done." :)

The original goals are (obviously more than) met and we're looking at a much better and more maintainable architecture for the future.

That is a stellar achievement! It goes way beyond of what everyone was able to imagine, back in the "old" days.

As a community around a product, we performed extraordinarily well. I'd like to thank everyone who made this possible. It is a huge success! :)


Of course, that does not imply that all of these major changes would be "fully done", too. With massive architectural changes, there's always lots of clean-up work in order.

In fact, when diving into the new world order of D8, you'll notice that there's hardly a class that does not call into global-state/procedural functions. So our overall accomplishment of the goal has heavily advanced in the meantime, but if you look at individual implementations, you'll notice very quickly that we're still looking at a big sad mess. ;)

I don't know the stance of core maintainers with regard to that, but after personally diving into D8 with a fresh and unbiased pair of eyes, all of the global-state/procedural function calls in classes as well as @deprecated wrappers are incredibly disturbing in terms of architecture and DX. Personally, I'm not able to see how D8 could be reasonably released with these things in place.

I'd love to point everyone to a dedicated initiative/effort to eliminate all of that, but unless I missed something big, there does not appear to be such a coordinated effort - yet.

We need such an effort. And we need to come up with scripts/tools that allow us to identify the non-obvious offenders. E.g., by introducing \Drupal (with good intentions), we implicitly introduced more code debt that (also) needs to be addressed by this effort.

The vast majority of these issues will present API changes (since interfaces need to be adjusted), so it would be best if we'd attack them as soon as possible.

Any takers? :)


In any case, the previous chapter definitely isn't meant to downplay the enormous success we've achieved by now. It's incredible. It's a whole New World Order™. We managed to progress in a pragmatic way, rewriting and revamping the entire application framework. Doing so involved a lot of pain and tension, but after all, we're able to present a much more maintainable and sustainable Drupal to the world.

Good job, everyone.

Status:Fixed» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.