Download & Extend

New menu system: faster and less memory

Project:Drupal core
Version:6.x-dev
Component:menu system
Category:task
Priority:normal
Assigned:chx
Status:closed (fixed)

Issue Summary

When I enable all modules in core, the menu array eats over 800K of RAM. This indicates that we are bound to have serious problems when lots of nodes are added to the menu tree via menu OTF. We need to have an SQL table against which we can load only those that are visible. I think with proper indexing we could even have some performance win.

Comments

#1

Version:x.y.z» 6.x-dev

Roadmap of the new menu system:

  1. New menu system to map Drupal paths to their callbacks with appropriate access checking.
  2. Drawing visual aids like navigation block, primary links, tabs, breadcrumbs.
  3. Making the menu editable
  4. Providing an upgrade path.

This will let us focus on the task at any time. The point of this post is to stress if you apply the patch to be submitted then your Drupal will not have anything listed in 2,3,4: navigation block, primary links, tabs, breadcrumbs, editable menu and you must start with a fresh database and be ready to loose it ongoing.

Of course, I have at least the ideas (well, some code too) to get all four so Drupal 6 won't be crippled for long.

#2

Title:Proper menu storage» New menu system: faster and less memory

Nice roadmap. Looking forward to this.

#3

Subscribing.

Of course, I have at least the ideas (well, some code too) to get all four

Could you post some of the code, even if not in the form of a patch? I'm curious as to how this will work.

#4

Preliminary code is in my sandbox for two months now (menu_create.php , newmenu.patch and test.php).

Patch is coming this weekend: I have most modules converted and menu.inc working. I just want to finish converting and do some basic testing & benchmarking.

#5

Documentation is at http://drupal.org/node/102338 .

#6

Status:active» needs review
AttachmentSizeStatusTest resultOperations
newmenu.patch190.65 KBIgnored: Check issue status.NoneNone

#7

hook_start was not called.

AttachmentSizeStatusTest resultOperations
newmenu_0.patch191.42 KBIgnored: Check issue status.NoneNone

#8

A copy-paste error caused map_arguments to be filled with access arguments...

AttachmentSizeStatusTest resultOperations
newmenu_1.patch191.19 KBIgnored: Check issue status.NoneNone

#9

I needed to make an exception to the inheritance rules so that node/add/story does not inherit from node/%/% (defined in comment), the rule is that if you have two percent signs then on build time this ancestor is not taken into consideration for inheritance purposes. I hope I will get people to review and discuss such decisions.

If you want to benchmark this then remove the menu_rebuild() command from index.php. I kept it there because it's so useful when hacking menu.

AttachmentSizeStatusTest resultOperations
newmenu_2.patch192.41 KBIgnored: Check issue status.NoneNone

#10

I discussed the above problem with Steven and now we inherit by path and not ancestor. This was used in the skelton version but I thought ancestors woudl be better. I was wrong. (When custom menu items will come, they will be treated as if run time)

AttachmentSizeStatusTest resultOperations
newmenu_3.patch191.97 KBIgnored: Check issue status.NoneNone

#11

we have hook_boot and hook_init as discussed on #drupal . This means that if you used hook_init, now you need to upgrade because bootstrap.inc calls hook_boot.

menu_map now assists in arriving to a drupal_not_found .

AttachmentSizeStatusTest resultOperations
newmenu_4.patch198.08 KBIgnored: Check issue status.NoneNone

#12

It's still difficult to grok the semantic difference between _boot() and _init(). Not sure it can be made clearer but I can see this being a source of confusion.

<?php
+ * To generate these, we will use binary numbers where bit 1 represents
+ * original value and bit 0 means wildcard. If the path is node/12/edit/foo
+ * then the 110 bitstring represents node/%/edit where % means that any
+ * argument matches that part.
?>

I'd write 'value 1' and 'value 0'. When you say 'bit 1', it makes me think of the bit at position 1.

In the example, what happens with 'foo'? The original path consists out of 4 parts, but the pattern only has 3. This needs to be clarified in the code comments. No need to clarify it in the issue.

There are some typos in the following snippet (i.e. 'will is an integer). Also, it is not quite clear what you're trying to say. The documentation describes the implementation details but not the bigger picture? I.e. why do we want to unserialize an array in an awkward way? Needs to be documented in the code. :-)

<?php
+ * Unserializes an array. Integer values are mapped according to the $map
+ * parameter. For example, if unserialize($data) is array('node_load', 1)
+ * and
$map is array('node', '12345') then 'node_load' will not be changed
+ * because it is not an integer, but 1 will is an integer. As $map[1] is
+ * '12345', 1 will be replaced with '12345'. So the result will be
+ * array('node_load', '12345').
?>

#13

I renamed the hook invokes but not the implementations. Fixed.

AttachmentSizeStatusTest resultOperations
newmenu_5.patch197.92 KBIgnored: Check issue status.NoneNone

#14

Subscribing...

#15

Rerolled and adjusted the comments per Dries' advice.

AttachmentSizeStatusTest resultOperations
newmenu_6.patch198.35 KBIgnored: Check issue status.NoneNone

#16

Subscribing (I think its time we have a subscribe feature in D6)

#17

subscribing

#18

This breaks the system_get_module_admin_tasks function which still expects the old ['path index'] key from menu_get_menu. It will need to be rewritten. The effect is that the page q=admin/by-module has no module specific links.

#19

Another problem in system.module, system_main_admin_page seems to return halfway through, causing q=admin to white-screen.

Perhaps this is intentional and both the admin and admin/by-module above are waiting for the core of the menu system to stabalize - I'm unsure.

#20

dflechter on IRC added after finding the admin pages not working:

nice work by the way this has been really solid no matter what I throw at it :)

The admin pages are navigational aids -- phase 2. I added return; // TODO: this needs to be rewritten for the new menu system. to the beginning of affected functions in system.module. They are now empty as they should be but do not emit error messages. In the previous patches I started converting one of these functions but no, this is clearly phase 2.

AttachmentSizeStatusTest resultOperations
newmenu_7.patch198.26 KBIgnored: Check issue status.NoneNone

#21

I really need to provide more context in IRC.

I was talking about my little test module which was thoroughly tested. That is the part I found to be solid, obviously not the admin stuff ;)

#22

The earlier patch contained an experiment from another patch. And more improvements.

AttachmentSizeStatusTest resultOperations
newmenu_8.patch198.93 KBIgnored: Check issue status.NoneNone

#23

If I would be particularly bold I would mark this RTBC because dfletcher, dvessel, bdragon, chrisxj have visited every single defined path (more than 200) and quite some that are not specifically defined. Thanks guys. I just needed to do 20 or so instead of 260+ and fix the occassional error. All this is documented http://groups.drupal.org/node/2354 . Unlike in the earlier patches, the menu_get_active_help now works, its conversion was so trivial.

AttachmentSizeStatusTest resultOperations
newmenu_9.patch207.32 KBIgnored: Check issue status.NoneNone

#24

you included your settings.php

#25

Sigh. This happens when our bzr mirror breaks.

AttachmentSizeStatusTest resultOperations
newmenu_10.patch206.09 KBIgnored: Check issue status.NoneNone

#26

New day, new patch-- HEAD broke search and node.

AttachmentSizeStatusTest resultOperations
newmenu_11.patch206.09 KBIgnored: Check issue status.NoneNone

#27

That was still yesterday's patch :(

AttachmentSizeStatusTest resultOperations
newmenu_12.patch206.3 KBIgnored: Check issue status.NoneNone

#28

node/add/page and node/add/story both return "page not found" errors for me.

#29

I see them on 4si.hostignition.com/d, the public demo and developer site of this patch. Register a user, because anonymous has no rights but authenticated users have every right.

I typed 4si.hostignition.com/d/node/add/story and I see 'Submit story'. Want a screenshot?

#30

Hmm, I logged out and logged back in and it worked. Maybe it was a caching issue.

I did notice that if you go to /user when logged out it doesn't display the login form, but instead tries to display a user profile ("member for 37 years 3 weeks").

#31

Now if you visit 'user' as anonymous you get access denied -- as I intended. The login page is user/login.

AttachmentSizeStatusTest resultOperations
newmenu_13.patch206.32 KBIgnored: Check issue status.NoneNone

#32

I keep closely with HEAD because I hope that very soon after branching this gets in and branching is soon...

AttachmentSizeStatusTest resultOperations
newmenu_14.patch206.85 KBIgnored: Check issue status.NoneNone

#33

Removed debug code (the menu_rebuild in index.php makes debugging easier).

AttachmentSizeStatusTest resultOperations
newmenu_15.patch206.29 KBIgnored: Check issue status.NoneNone

#34

I have removed the install.php patch as well, that was not intended...

AttachmentSizeStatusTest resultOperations
newmenu_16.patch206.85 KBIgnored: Check issue status.NoneNone

#35

watchdog_help was lost due to collateral damage.

AttachmentSizeStatusTest resultOperations
newmenu_17.patch204.41 KBIgnored: Check issue status.NoneNone

#36

A basic benchmark. prepatch has all blocks disabled.

Only default core modules enabled.

node/!/edit is a non existing node (a different 404).

time (ms)
            pre     post    +/-
node/1      230.72  179.25  51.47
nopage      221.50  165.63  55.88
node/1/edit 225.00  173.66  51.34
node/!/edit 236.50  169.31  67.19


# req/s
            pre   post  +/-
node/1      4.33  5.58  1.25
nopage      4.51  6.04  1.53
node/1/edit 4.44  5.76  1.32
node/!/edit 4.23  5.91  1.68


All core modules - statistics, menu

time (ms)
            pre     post    +/-
node/1      305.28  248.84  56.44
nopage      283.50  230.56  52.94
node/1/edit 290.22  241.69  48.53
node/!/edit 289.38  234.53  54.84

# req/s
             pre   post  +/-
node/1       3.28  4.02  0.74
nopage       3.53  4.34  0.81
node/1/edit  3.45  4.14  0.69
node/!/edit  3.46  4.26  0.80

#37

Heine, how do I need to read/interprete these results? What format are they in?

#38

I gave this patch a decent workout tonight. My first impression is bliss - these are the fastest pages i've ever served on my dev box. It is magical to see a single simple query retrieve the active callback. For node/1, that query is SELECT * FROM menu WHERE path IN ('node/1','node/%','node') ORDER BY fit DESC LIMIT 0, 1

the menu cache is gone. hook_menu_alter() is introduced. there is a lot to love here.

code-wise, looks good. i think modules will easily convert to this scheme. there is some binary magic which i don't understand but chx and dries do, so thats all good.

there is some debugging in the patch, and some pages which await rewrite. i think thats OK, but i'll wait and see where chx intends to leave off for this phase 1. from my perspective, this is close to rtbc.

#39

I think maybe Heine meant:

<?php
           
(ms)                       (requests/second)
           
pre    post   change       pre  post change
node
/1      230.72 179.25 -51.47 (22%) 4.33 5.58 +1.25 (29%)
nopage      221.50 165.63 -55.88 (25%) 4.51 6.04 +1.53 (34%)
node/1/edit 225.00 173.66 -51.34 (23%) 4.44 5.76 +1.32 (30%)
node/!/edit 236.50 169.31 -67.19 (28%) 4.23 5.91 +1.68 (40%)
?>

and
<?php
           
(ms)                       (requests/second)
           
pre    post   change       pre  post change
node
/1      305.28 248.84 -56.44 (18%) 3.28 4.02 +0.74 (23%)
nopage      283.50 230.56 -52.94 (19%) 3.53 4.34 +0.81 (23%)
node/1/edit 290.22 241.69 -48.53 (17%) 3.45 4.14 +0.69 (20%)
node/!/edit 289.38 234.53 -54.84 (19%) 3.46 4.26 +0.80 (23%)
?>

... where 'pre' is 'pre-patch', 'post' is 'post-patch', and the first dataset is with only the default enabled core modules.

Please correct me if I'm wrong! I calculated percentages to compare—if it's really at least 20% decrease in page time / request rate, that's very snappy.

#40

Moshe, thanks. The pages that await rewrite are navigational aids, hence phase 2, they do not belong to this patch.

The known_keys debugging in menu.inc is something to be only removed before RC, I think. It's of great help when converting a module. Of course, that's only my opinion.

#41

And Paul's interpretation is quite right.

#42

Wow, impressive benchmark results :)

#43

I haven't tested this patch yet, but I've read the whole thing, and it looks really seriously amazing. You're a genius, chx! (but hey, we already knew that).

My biggest concern is that there doesn't seem to be proper support for non-integer wildcards in menu paths. E.g. in the patched version of node_menu(), there should just be one path for all the custom node type editing pages: admin/content/types/%. But instead, there's a foreach loop that's defining a path for each node type. This is not good: we need to be able to use strings as wildcards as well! I'm sure there will be plenty of other places, in core and contrib, where strings will need to be used as wildcards.

The binary tricks in menu_get_ancestors() are all voodoo to me, but if they work as advertised, then I say bring it on!!

#44

Good work chx.

Subscribing.

#45

There is nothing in there that would force % to be numeric. It can be whatever you want. However, in *most* cases it is a database ID and then you call map callback upon it and the default (but only the default!) map callback indeed checks for is_numeric. But noone forces you to define a map callback or even if you do, feel free to use your own.

Some suggested earlier using %d and %s but this could lead to huge database queries, with 728 maximal possibilites in the menu query instead of the 63 we currently have.

#46

BTW. the admin/content/types/% I actually inherited from the old menu system and it is certainly possible to simplify -- later. We enough changes already, I believe.

#47

The maths is not voodoo by far (just wait until the hook registry/ordering patch hits with DFS of graph theory inside). http://en.wikipedia.org/wiki/Bitwise_operation will explain. It's rather trivial, IMO.

#48

OK, I have spoken to chx on IRC, and in case anyone else reads the patch and worries like I did, please rest assured:

The is_int() check in menu_unserialize() has nothing to do with wildcards. It is perfectly possible to have non-integer wildcards with this patch.

BTW, I did actually read http://en.wikipedia.org/wiki/Bitwise_operation while I was looking at the patch, and it helped a lot. The operators themselves are not voodoo, it's just applying them in code that seems like voodoo to a lot of people (including myself).

#49

I think I should give credit before I get all. The patch was inspired by a mail from JonBob, I got great ideas in Brussels from Merlin, Adrian, Moshe and Dries (I hope I have not left out any of you guys), and later Steven helped straight out ancestors/parents. Only the binary maths and map callback are my ideas. dfletcher, dvessel, bdragon, chrisxj and Heine took great pains to test it througly.

#50

subscribing

#51

As per Dries' request so that Drupal remains usable after this is commited: we now have the navigation block. I am extremely unwilling to bring more into this patch.

AttachmentSizeStatusTest resultOperations
newmenu_18.patch210.86 KBIgnored: Check issue status.NoneNone

#52

I upped system.install with the many new fields and also this version has an index on fit which will give you a minor boost, especially on long paths.

AttachmentSizeStatusTest resultOperations
newmenu_19.patch209.5 KBIgnored: Check issue status.NoneNone

#53

Heh! I forgot out access checking (thanks ChrisKennedy and Heine).

AttachmentSizeStatusTest resultOperations
newmenu_20.patch212.43 KBIgnored: Check issue status.NoneNone

#54

Fixed a few minor coding style errors, added a few basic doxygen comments, changed home to Home in the breadcrumb placeholder.

The issue with "create content" (a MENU_GROUP_ITEM) showing when no items are available can be tackled post stage 3.

From #drupal:

<chx> with the latest code reorganization there could be something
<chx> ChrisKennedy: this needs to wait until post phase 3. MENU_GROUP_ITEM needs to be handled almost like an expanded item just that the expanded items are not rendered but each and every needs to be inspected. "
AttachmentSizeStatusTest resultOperations
newmenu_21.patch212.86 KBIgnored: Check issue status.NoneNone

#55

menu.inc uses int2vancode which resided in comment.module. No more.

AttachmentSizeStatusTest resultOperations
newmenu_22.patch214.5 KBIgnored: Check issue status.NoneNone

#56

Status:needs review» reviewed & tested by the community

verified that the most recent patch works as advertised. RTBC. later phases will happen later.

#57

Reroll.

AttachmentSizeStatusTest resultOperations
newmenu_23.patch214.51 KBIgnored: Check issue status.NoneNone

#58

Status:reviewed & tested by the community» fixed

Dries committed it.

#59

Status:fixed» closed (fixed)
nobody click here