Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
By pacoit on
In reviewing drupal 4.6, I noticed that it seems to load most or all the core code and all the modules for all requests, regardless of whether the code is needed. Perhaps the code could be structured to load includes conditionally? Here's my debug list of includes:
DEBUG:
1,056 - /src/drupal-4.6.0/index.php
17,650 - /src/drupal-4.6.0/includes/bootstrap.inc
4,323 - /src/drupal-4.6.0/sites/default/settings.php
9,253 - /src/drupal-4.6.0/includes/database.inc
6,611 - /src/drupal-4.6.0/includes/database.mysql.inc
2,098 - /src/drupal-4.6.0/includes/session.inc
6,097 - /src/drupal-4.6.0/includes/module.inc
60,883 - /src/drupal-4.6.0/includes/common.inc
26,858 - /src/drupal-4.6.0/includes/theme.inc
14,433 - /src/drupal-4.6.0/includes/pager.inc
32,970 - /src/drupal-4.6.0/includes/menu.inc
6,239 - /src/drupal-4.6.0/includes/tablesort.inc
16,937 - /src/drupal-4.6.0/includes/file.inc
30,783 - /src/drupal-4.6.0/includes/xmlrpc.inc
8,394 - /src/drupal-4.6.0/includes/image.inc
49,549 - /src/drupal-4.6.0/modules/aggregator.module
20,653 - /src/drupal-4.6.0/modules/block.module
10,504 - /src/drupal-4.6.0/modules/blog.module
32,514 - /src/drupal-4.6.0/modules/book.module
76,258 - /src/drupal-4.6.0/modules/comment.module
62,976 - /src/drupal-4.6.0/modules/event/event.module
10,496 - /src/drupal-4.6.0/modules/event/event.theme
38,441 - /src/drupal-4.6.0/modules/filter.module
34,137 - /src/drupal-4.6.0/modules/forum.module
4,236 - /src/drupal-4.6.0/modules/help.module
57,333 - /src/drupal-4.6.0/modules/libdb/libdb.module
17,350 - /src/drupal-4.6.0/modules/menu.module
69,318 - /src/drupal-4.6.0/modules/node.module
1,663 - /src/drupal-4.6.0/modules/page.module
13,307 - /src/drupal-4.6.0/modules/path.module
14,633 - /src/drupal-4.6.0/modules/poll.module
23,039 - /src/drupal-4.6.0/modules/profile.module
31,518 - /src/drupal-4.6.0/modules/search.module
21,918 - /src/drupal-4.6.0/modules/statistics.module
1,813 - /src/drupal-4.6.0/modules/story.module
35,829 - /src/drupal-4.6.0/modules/system.module
43,591 - /src/drupal-4.6.0/modules/taxonomy.module
7,275 - /src/drupal-4.6.0/modules/throttle.module
15,226 - /src/drupal-4.6.0/modules/upload.module
83,331 - /src/drupal-4.6.0/modules/user.module
6,037 - /src/drupal-4.6.0/modules/watchdog.module
7,966 - /src/drupal-4.6.0/themes/engines/xtemplate/xtemplate.engine
15,270 - /src/drupal-4.6.0/themes/engines/xtemplate/xtemplate.inc
1,050,766 bytes in 43 included files
Comments
a few notes
indeed, this is true for registered users or for anon users when caching is disabled. we have optimized the page cache for the case of anonymous users and caching enabled. in that case we load only critical modules and includes.
the trick to fixing this is implementing the conditional include in a clean way. we are not inclined to add lots of code to make this happen.
if your site is not running with an opcode cache (i.e. a php acccelerator) then parsing sall these files is the biggest performance bottleneck in current drupal. so you have highlighted an important issue. yet if you get smart and run drupal with an opcode cacce (e.g. mmcache, eaccelerator, etc.), then this becomes almost a non issue. so the fact that it can be easily solved diminishes the priority of the problem, IMO
on priorities
It's good to know there is some relief from the problem. However, I am surprised to hear that the the priority (or interest) is apparently low. Member oriented sites will benefit little from caching; and caching is compromized (or anonymous users' experience is diminished) if dynamic content is needed (or is eliminated). I would think far greater speed and far lower memory usage are important to most people (or hosting services).
I have also taken note of
I have also taken note of the fact that a typical page request includes all enabled modules. I suppose lots of sites would run faster if that wasn't the case. However, I believe the current drupal architecture makes it difficult to implement conditional includes, some "lazy loading" system. If that were to be implemented in a nice way, I suppose that would be with the help of a "global register" of events and interested modules? Also, most function calls modules make to other modules would have needed to go through a "lazy loader"?
I've myself done some
I've myself done some research on doing conditional includes. The memory consumption gies down and the execution speed goes up as predicted. But since I am runnign my own server now, I have lost interest. I simply run a php cache which does the same.
You can look at what I did here:
http://lists.drupal.org/archives/drupal-devel/2005-02/msg00082.html
http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/killes/speed-...
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Support for conditional inclusion of modules.
I've written a version of module.inc which conditionally includes the module files. I think the code is more for discussion than anything else (it's far from ready to be deployed). It breaks Drupal several places when other bits of code expect modules to be loaded (module_load($modulename) includes modules in case you wish to put it in somewhere).
The important thing in this code is use of a persistent variable realized with variable_get and variable_set. It's called $module_hook_implementors. It's structure could be something like:
array (
'myhook' => array('modules', 'which', 'implements', 'myhook'),
'otherhook' => array('other', 'modules')
)
First, in module_init(), all "bootstrap modules" are included. Others are included on invokasion of hooks which they implement. When the code is told to invoke a hook it hasn't yet registered, it includes all modules, goes through all of them to check if they implement the hook in question, and in that case adds them to the $module_hook_implementors which as mentioned is used in the process of hook invokasion.
The code may speak for itself. I've made changes to most of the original module.inc. The new function module_register_hook might be of special interest.
The code is pasted below with the hope of getting som comments and discussion going. As far as I can tell, this might work out fine. Modules checking for the existence of other modules by using something like: function_exists('module_hookname') could be considered buggy after introduction of something like this. But I'm far from on top of the Drupal code, so important things might be too difficult to implement (lots of modules may need change) to make this worth the effort. Somewhere it might be possible to get by problems by extending the register established here, or using other "registers".
You may very well test it by replacing it with your module.inc. But put back the original version after! ;-) And only "development installations" should be used.
Menu system a problem
It seems this isn't much worth as long as the menu system is built as it is...
The menu hook is invoked, which causes most modules to be loaded. I'm not sure how much of the menu system may be cached, the menu hook is also invoked in this _menu_append_contextual_items() function in menu.inc. Does anybody have some knowledge of this system? How could one conditionally include modules in a good way?
Congrats, you discovered the
Congrats, you discovered the reason, why i didn't further pursue the matter. ;)
The problem is indeed that soem hooks (not only _menu, but also _help) are called on each page load and will lead to inclusion of all modules. The only real solution is to split up the module files and put those hooks into for example. modulename.menu. We are a bit reluctant to do so since it would make programming a bit less nice. if you look at the notify.module you can see how I splir that one up into a .inc and a .module file. most of the code is in .inc but _menu and _help are in the main module file.
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Hmm... All hooks are called
Hmm... All hooks are called through module_invoke (all) so it might be possible to cache the return value of certain hooks there. I think this certainly is true for hook_help (as you've mentioned in your post linked to in your previous comment). This could be done "dynamically" like in my version of module.inc, by checking for cache, or getting the original on demand and caching that. I'm not sure about the menu though. Language, role and module and possibly location may be used to compose a cache id. Is it at all possible to determine when menu items needs to be refreshed given only the cache (should of course be rock solid)? I believe the majority of menu hooks may be cached, node_menu is an interesting exception. Possibly something like somemodule_menu(false) == somemodule_menu(true) is a useful check. A somewhat sophisticated cache system may be used to cache different versions of the menu. When an item of type MENU_DYNAMIC_ITEM is encountered, special action may be taken.
I was a bit puzzled to find that the docs said the menu hook was a nice place to put code that only needed to be run once per request, isn't that a counter intuitive approach? Maybe better with hook_init? Furthermore the way the menu is currently cached is a bit strange, no? It calls all hooks anyway?
Not all menu entries can be
Not all menu entries can be cached.
And I agree abotu the _init hook. We should have one for the purpose you describe. (we already have a hook called _init for other purposes).
--
Drupal services
My Drupal services
--
Drupal services
My Drupal services
Right. If it isn't possible
Right. If it isn't possible to determine when to refresh the cache or don't cache at all, I suppose another solution could be to parse out the function code (like documentors does), save it in db and eval that for some performance gain. However, this would lead to strange results in a scenario where the menu hook calls other module functions (don't know if this happens). The same is possible for instance for help hooks. However, I sense there isn't that much enthusiasm for something like this, and if that is the case, I will not persue this any further it at this time as I don't have much use for speed improvement.