Allow modules to register links for menus, breadcrumbs and tabs (if not with hook_menu)

I would say,
the spaghetti in menu.inc is hiding the potential of our traditional system.

Conceptually it is not so bad, it is just
- poorly implemented, with a lot of global state, poorly managed dependencies, and a lot of procedural spaghetti.
- responsibilities are not clearly separated, or this separation is effectively hidden in spaghetti.

The solution would be a clearly designed food chain, from the module that provides stuff, to the systems that consume it.
A lot of this could have been achieved with just some internal cleaning-up, and with very little change to the API.

-------------

The role of Hooks (shop metaphor)
=====================
(Ok, we have many different types of them, let's just ignore that for a moment)

A hook, in my philosophy, is like a shop, in the most general sense. Modules can go to that shop and buy products and services. They can buy low-level stuff like routes, input filters, or they can buy higher-level stuff like entire views. They can also go to the shop to have something modified. Or they can go to the shop to hire a hitman that will go somewhere and change world history.

A shop has a front door / public area for customers, and a back door / workshop for product delivery or manufacturing. Between that, it can have any kind of management and further processing going on.
In the public area, the customer might buy a custom-made bicycle. In the workshop, they will get the pieces together from different providers, and do some handcrafting, to finally have the product together.
The front area process is designed for the comfort of the customer. The customer only needs to answer questions that are relevant to his end user experience. And we don't let him go into the workshop, that's not his territory.

There can also be other shops, that provide just the pieces for the bike, not the entire bike. One shop that specializes on tires, another that specializes on frames. And another that is just sells stolen bikes (ouch). The bike shop might use those lower-level shops as providers. But customers who care can also go to the pieces shops directly.

Separation:
The bike shop, the tires provider, the frame provider, they are all separate, and connected with some flexible "glue" (free market). All elements of this system are individually replaceable.

"Packaged" reselling:
The bike shop does combine pieces that are unrelated in how they are produced (chemistry): Frames, tires, other stuff.
But 90% of customers want this stuff in a combo package, called "bicycle".
Still, this does not hurt the separation of the pieces providers any bit.

hook_menu() as a package reseller
========================
We have different concerns that should be treated separately (quote #1):
- Routing
- Declarative links
- Local tasks (aka tabs)
- Local actions (aka operations on a page, which is just weird)
- Contextual links (wtf?)
- Access control
(I would add breadcrumbs maybe)
I perfectly agree these should be separate subsystems, each of them replaceable and independent.

However, from a module's point of view, in 90% you want these in a combo pack:
"I want a link to show up here, and if you click that, you see the following (e.g. a configuration form). It should naturally integrate in the hierarchy, have a breadcrumb, tabs, just like all the others."

If we understand hook_menu() as a reseller of those distinct services, it will now pass this info onto the various components that provide those services.
E.g. the menu links component will say, "Ok I already have a number of menu links here, but I will try to squeeze those in, as I see fit."
The breadcrumbs component will say "I already have organic groups and taxonomy and menu links telling me about breadcrumbs, but I will also consider your stuff."
The tabs (local tasks) component might say: "I would like to help, but I'm disabled. Sorry."

hook_menu() itself does not even need to know all the services that are available.
E.g. the module might say "I want a contextual link on that one", hook_menu() will just say, "I have no idea what that is, I write it down and see if we have it." Then, as hook_menu() is going to the various providers, one of them will say, "ah, they ordered a contextual link, I will take care of that".

---------------

I think the above is a pretty sane concept.
Unfortunately, we were never really clear that this is how it is meant to work. Instead, we have a lot of hardcoded stuff and a big procedural spaghetti mess.

Still, the API of hook_menu() itself is the least to blame for that.

Shortcomings of hook_menu(): API
=======================
This being said, hook_menu() does have some issues that I agree with:
- It is procedural, so it adds up to a .module file that is already too big.
- Implementations are not DIC-powered, but gets all their data from global space.
- Some module implementations have had circular dependencies with other info hooks.
- Some module implementations have huge memory issues, because they load a lot of stuff that they don't need.
- menu_router items are keyed by path, not by identifier (this has never been a problem to me in real-world development, though)
- you always need a full rebuild, if something changes

Shortcomings of hook_menu(): Implementation
==================================
The above was for the API aspect.
For internal implementation (in menu.inc), there is
- procedural spaghetti
- global state
- hardcoded _menu_navigation_links_rebuild(), instead of having that as a separate subsystem
- some inefficient algorithms (I think)

Oh dear.. how can I get out of this?
I would have a number of things to say, but they are all horribly distracting.

The role of being a core maintainer should be justified by the trust of the community.
I don't see this being the case with me.
And if part of the task is to not hurt anyone's feelings, I am even less qualified, apparently.

Besides of that, I see the ball in the court of those who want to kill hook_menu() in the first place, as a side effect of Symfony routing.
I think my personal opinion on hook_menu() is clear, but the goal is to find a common ground.

@rcross, this does make sense.
If we move from hooks towards event subscribers, this problem kind of answers itself.
Otherwise I would say it depends on how big the difference we end up with - so the naming is the thing we should decide last. But definitely sth to keep in mind.

Correct issue for menu links to entities

I wanted to do this with a PM, but I'm afraid I have to do it here.
I think one major difficulty is that I am talking about the code we have now, while you talk about the process how it was made.
In fact a piece of code can be perfectly carefully architected at some point in time, this doesn't mean we can't be unhappy with it today.
This says nothing about the original developer, it just means that now we look for something else.
In fact this is the most normal and natural flow in software development. We build something in a specific way for good reasons (or because noone came to help), and years later we decide that parts of it stink.
And sometimes we even know that something stinks the day we build it, and still we build it exactly this way, because there is no other choice. This does not make us poor developers, but the product stinks nevertheless.

So, I sympathize with chx in #15 and #16, and I admire the engineering of specific parts of the menu system, but I still want to have the right to say that I don't like specific parts.
My language to express my unhappiness with specific parts might be exaggerated, and it also was quite unspecific. The goal of that was not to insult the people who put a lot of effort into building this stuff, so I apologize if it had this effect.
It would be easier and probably more productive to talk about existing code as an unpersonal matter. I realize this is an illusion.

-------

To be more specific, here are some things I like and dislike:
(note: this is all pre-symfony talk)
First, there is a lot of "carefully engineered" stuff in there. Inventive and admirable logic and APIs (e.g. the wildcard loader system, and the entire hook_menu, imo).
But there is also ugly stuff. (and I don't even think you disagree)
- Access checking and localization of menu links are somewhat intermixed. There might be a reason for it, but it feels confusing. And this makes it hard to reuse parts of it.
- Access checking and localization logic for menu links and router items are sharing the same code.
- More general, there's a number of functions where menu links and router items are interchangeable.
- Menu links (esp., finding the "active trail" of the current page) are hardcoded into the page routing.
- Associative arrays being passed around and receiving additional keys like a snowball, so you always need to be careful about the order in which things happen. A key might not be there yet when you expect it. This is how most of Drupal works, so I can't really blame the developer of the menu system - but still it is ugly.
- The bulk access check for links with node/% paths - this is really cool, but it's all hardcoded and not properly encapsulated. Also, it bakes an application detail (the existence and popularity of the node/% path) into the framework. Again this was a right decision at the given time, but that does not mean we have to like it.
- The logic to build a menu links tree from router items is fragile (I am trying to rewrite this but it's quite a lot of work).
- Menu tree loading does too much stuff in one function, so it is hard to reuse parts of it.
- Sometimes we load a complete tree and then discard parts of that, because a function to only load a part does not exist (that was last time I looked, might be different today).
- most of menu links is in includes/menu.inc, when it rather should be in a module (and optional).
- all those files are generally too big.
- a lot of functions are too big. this is due to having a limited global namespace, and the normal thing in drupal, but this doesn't make it any better.

The thing is, none of these problems is caused by hook_menu().
The dual nature of hook_menu() in itself is not an architectural problem or obstacle.
The meta information (not routing related) in the router table is not an architectural problem or obstacle.
This could all remain unchanged, it would not stop us from cleaning up the architecture behind it.
(and again, this is pre-symfony talk)

So,

I called for the removal of hook_menu and the decoupling of the roles of it on the summer of 2009

my alternative was this, #1170278: Introduce hook_menu_router_update(), but it is coming too late.

Noone ever could agree of what breadcrumbs/active foo should be

I maintain a breadcrumb module that I am quite happy with.
I also maintain a menu tree loading module which needs a major cleanup/rewrite, but basically has the right ideas.
I don't have a clear idea for local tasks. But probably it would good enough to just decouple this a bit.
I even proposed a session on DrupalCon Munich, where I wanted to explain how I see this stuff.

------

We now have to rethink a lot of this from scratch, because the traditional routing is dead.
This is not necessarily a problem, but now I would like to see some ideas from the WSCCI people.

Updated issue summary.