In reviewing drupal 4.6, I noticed that it seems to load most or all the core code and all the modules for all requests, regardless of whether the code is needed. Perhaps the code could be structured to load includes conditionally? Here's my debug list of includes:

DEBUG:
    1,056 - /src/drupal-4.6.0/index.php
   17,650 - /src/drupal-4.6.0/includes/bootstrap.inc
    4,323 - /src/drupal-4.6.0/sites/default/settings.php
    9,253 - /src/drupal-4.6.0/includes/database.inc
    6,611 - /src/drupal-4.6.0/includes/database.mysql.inc
    2,098 - /src/drupal-4.6.0/includes/session.inc
    6,097 - /src/drupal-4.6.0/includes/module.inc
   60,883 - /src/drupal-4.6.0/includes/common.inc
   26,858 - /src/drupal-4.6.0/includes/theme.inc
   14,433 - /src/drupal-4.6.0/includes/pager.inc
   32,970 - /src/drupal-4.6.0/includes/menu.inc
    6,239 - /src/drupal-4.6.0/includes/tablesort.inc
   16,937 - /src/drupal-4.6.0/includes/file.inc
   30,783 - /src/drupal-4.6.0/includes/xmlrpc.inc
    8,394 - /src/drupal-4.6.0/includes/image.inc
   49,549 - /src/drupal-4.6.0/modules/aggregator.module
   20,653 - /src/drupal-4.6.0/modules/block.module
   10,504 - /src/drupal-4.6.0/modules/blog.module
   32,514 - /src/drupal-4.6.0/modules/book.module
   76,258 - /src/drupal-4.6.0/modules/comment.module
   62,976 - /src/drupal-4.6.0/modules/event/event.module
   10,496 - /src/drupal-4.6.0/modules/event/event.theme
   38,441 - /src/drupal-4.6.0/modules/filter.module
   34,137 - /src/drupal-4.6.0/modules/forum.module
    4,236 - /src/drupal-4.6.0/modules/help.module
   57,333 - /src/drupal-4.6.0/modules/libdb/libdb.module
   17,350 - /src/drupal-4.6.0/modules/menu.module
   69,318 - /src/drupal-4.6.0/modules/node.module
    1,663 - /src/drupal-4.6.0/modules/page.module
   13,307 - /src/drupal-4.6.0/modules/path.module
   14,633 - /src/drupal-4.6.0/modules/poll.module
   23,039 - /src/drupal-4.6.0/modules/profile.module
   31,518 - /src/drupal-4.6.0/modules/search.module
   21,918 - /src/drupal-4.6.0/modules/statistics.module
    1,813 - /src/drupal-4.6.0/modules/story.module
   35,829 - /src/drupal-4.6.0/modules/system.module
   43,591 - /src/drupal-4.6.0/modules/taxonomy.module
    7,275 - /src/drupal-4.6.0/modules/throttle.module
   15,226 - /src/drupal-4.6.0/modules/upload.module
   83,331 - /src/drupal-4.6.0/modules/user.module
    6,037 - /src/drupal-4.6.0/modules/watchdog.module
    7,966 - /src/drupal-4.6.0/themes/engines/xtemplate/xtemplate.engine
   15,270 - /src/drupal-4.6.0/themes/engines/xtemplate/xtemplate.inc

1,050,766 bytes in 43 included files

Comments

moshe weitzman’s picture

indeed, this is true for registered users or for anon users when caching is disabled. we have optimized the page cache for the case of anonymous users and caching enabled. in that case we load only critical modules and includes.

the trick to fixing this is implementing the conditional include in a clean way. we are not inclined to add lots of code to make this happen.

if your site is not running with an opcode cache (i.e. a php acccelerator) then parsing sall these files is the biggest performance bottleneck in current drupal. so you have highlighted an important issue. yet if you get smart and run drupal with an opcode cacce (e.g. mmcache, eaccelerator, etc.), then this becomes almost a non issue. so the fact that it can be easily solved diminishes the priority of the problem, IMO

pacoit’s picture

It's good to know there is some relief from the problem. However, I am surprised to hear that the the priority (or interest) is apparently low. Member oriented sites will benefit little from caching; and caching is compromized (or anonymous users' experience is diminished) if dynamic content is needed (or is eliminated). I would think far greater speed and far lower memory usage are important to most people (or hosting services).

Adagio’s picture

I have also taken note of the fact that a typical page request includes all enabled modules. I suppose lots of sites would run faster if that wasn't the case. However, I believe the current drupal architecture makes it difficult to implement conditional includes, some "lazy loading" system. If that were to be implemented in a nice way, I suppose that would be with the help of a "global register" of events and interested modules? Also, most function calls modules make to other modules would have needed to go through a "lazy loader"?

killes@www.drop.org’s picture

I've myself done some research on doing conditional includes. The memory consumption gies down and the execution speed goes up as predicted. But since I am runnign my own server now, I have lost interest. I simply run a php cache which does the same.

You can look at what I did here:

http://lists.drupal.org/archives/drupal-devel/2005-02/msg00082.html
http://cvs.drupal.org/viewcvs/drupal/contributions/sandbox/killes/speed-...

--
Drupal services
My Drupal services

Adagio’s picture

I've written a version of module.inc which conditionally includes the module files. I think the code is more for discussion than anything else (it's far from ready to be deployed). It breaks Drupal several places when other bits of code expect modules to be loaded (module_load($modulename) includes modules in case you wish to put it in somewhere).

The important thing in this code is use of a persistent variable realized with variable_get and variable_set. It's called $module_hook_implementors. It's structure could be something like:
array (
'myhook' => array('modules', 'which', 'implements', 'myhook'),
'otherhook' => array('other', 'modules')
)

First, in module_init(), all "bootstrap modules" are included. Others are included on invokasion of hooks which they implement. When the code is told to invoke a hook it hasn't yet registered, it includes all modules, goes through all of them to check if they implement the hook in question, and in that case adds them to the $module_hook_implementors which as mentioned is used in the process of hook invokasion.

The code may speak for itself. I've made changes to most of the original module.inc. The new function module_register_hook might be of special interest.

The code is pasted below with the hope of getting som comments and discussion going. As far as I can tell, this might work out fine. Modules checking for the existence of other modules by using something like: function_exists('module_hookname') could be considered buggy after introduction of something like this. But I'm far from on top of the Drupal code, so important things might be too difficult to implement (lots of modules may need change) to make this worth the effort. Somewhere it might be possible to get by problems by extending the register established here, or using other "registers".

You may very well test it by replacing it with your module.inc. But put back the original version after! ;-) And only "development installations" should be used.

// $Id: module.inc,v 1.67 2005/04/03 08:03:18 dries Exp $

/**
 * @file
 * API for loading and interacting with Drupal modules.
 */

/**
 * Initialize all modules.
 *
 */
function module_init() {
	global $module_hook_implementors; // this is used fairly massively, so keep it here.
	$module_hook_implementors = variable_get('module_hook_implementors', array());
	//module_load_all();
	module_mass_load(true); // loading bootstrap modules
	module_invoke_all('init');
}

/**
 * Call a function repeatedly with each module in turn as an argument.
 */
function module_iterate($function, $argument = '') {
	// Not sure how to handle this... Possibly smart to include all as done for now
	// where is it used?
	module_mass_load(false);
	
  foreach (module_list() as $name) { // with normal module_list, this will only include bootstrap
    $function($name, $argument);
  }
}

/**
 * Collect a list of all loaded modules. During the bootstrap, return only
 * vital modules. See bootstrap.inc
 *
 * @param $refresh
 *   Whether to force the module list to be regenerated (such as after the
 *   administrator has changed the system settings).
 * @param $bootstrap
 *   Whether to return the reduced set of modules loaded in "bootstrap mode"
 *   for cached pages. See bootstrap.inc.
 * @return
 *   An associative array whose keys and values are the names of all loaded
 *   modules.
 */
function module_list($refresh = FALSE, $bootstrap = TRUE, $throttle = true) {
	static $list = array();
	static $bootstrap_list = array();
	static $throttled; // has the lists been "cut" because of throttling?
	if (!count($list) or $refresh or (!$throttle and $throttled)) {
		foreach (array('filter', 'system', 'user', 'watchdog') as $modulename) {
			// Absolute bootstrap
			$list[$modulename] = $modulename;
			$bootstrap_list[$modulename] = $modulename;
		}
		
		// Query db for enabled modules
		$modules_rs = db_query("
			SELECT name, filename, throttle, bootstrap
			FROM {system}
			WHERE type = 'module' AND status = 1"
		);
		
		$throttled = false;
		while ($record = db_fetch_object($modules_rs)) {
			// Could check for file here, what to do with the check? drupal_load also checks
			// Omitting records may lead to strange stuff, maybe better with error from drupal_load (if any).
			$throttle = ($module->throttle && variable_get('throttle_level', 0) > 0);
			if (!$throttle or ($throttle == false)) {
				drupal_get_filename('module', $record->name, $record->filename);
				$list[$record->name] = $record->name;
				// Also add to bootstrap list if appropriate
				if ($record->bootstrap) $bootstrap_list[$record->name] = $record->name;
			} else {
				$throttled = true;
			}
			// Why asort?
			/*asort($list);
			asort($bootstrap_list);*/
		}
	}
	return $bootstrap ? $bootstrap_list : $list;
}

/**
 * Load all the modules that have been enabled in the system table.
 *
 * @return
 *   TRUE if all modules were loaded successfully.
 */
function module_load_all() {
	// This will hopefully not be used much
	module_mass_load(false);
}


/**
 * Load bootstrap modules as defined in the system table.
 *
 * @return
 *   TRUE if the modules were loaded successfully.
 */
function module_mass_load($bootstrap = true, $enable_throttle = true) {
	$list = module_list(false, $bootstrap, $enable_throttle); // no need to force refresh
	$status = true;
	foreach ($list as $module) {
		$status = (drupal_load('module', $module) && $status);
	}
	return $status;
}

/**
 * Load a module
 * Probably not needed, just use drupal_load
 * @return
 * true/false depending on success

function module_load($module) {
	static $loaded_modules = array(); // is this needed?
	if ($loaded_modules[$module]) return true;
	if (!drupal_load('module', $module)) return false;
	$loaded_modules[$module] = true;
	return true;
}
*/

/**
 * Determine whether a given module exists.
 *
 * @param $module
 *   The name of the module (without the .module extension).
 * @return
 *   TRUE if the module is both installed and enabled.
 */
function module_exist($module) {
  return array_key_exists($module, module_list(false, false));
}

/**
 * @defgroup hooks Hooks
 * @{
 * Allow modules to interact with the Drupal core.
 *
 * Drupal's module system is based on the concept of "hooks". A hook is a PHP
 * function that is named foo_bar(), where "foo" is the name of the module (whose
 * filename is thus foo.module) and "bar" is the name of the hook. Each hook has
 * a defined set of parameters and a specified result type.
 *
 * To extend Drupal, a module need simply implement a hook. When Drupal wishes to
 * allow intervention from modules, it determines which modules implement a hook
 * and call that hook in all enabled modules that implement it.
 *
 * The available hooks to implement are explained here in the Hooks section of
 * the developer documentation. The string "hook" is used as a placeholder for
 * the module name is the hook definitions. For example, if the module file is
 * called example.module, then hook_help() as implemented by that module would be
 * defined as example_help().
 */

/**
 * Determine whether a module implements a hook.
 *
 * @param $module
 *   The name of the module (without the .module extension).
 * @param $hook
 *   The name of the hook (e.g. "help" or "menu").
 * @return
 *   TRUE if the module is both installed and enabled, and the hook is
 *   implemented in that module.
 */
function module_hook($module, $hook) {
	global $module_hook_implementors;
	
	// Check if the hook is register in the "registry"
	if (!isset($module_hook_implementors[$hook])) module_register_hook($hook);
	return isset($module_hook_implementors[$hook][$module]);
}

function module_register_hook($hook) {
	// Note: The register should be flushed (maybe by calling this) when enabled modules changes.
	// When modules are disabled, their hook entries should be deleted/unset
	global $module_hook_implementors; // used around here
	$module_hook_implementors[$hook] = array();
	// Make sure all modules are loaded (no throttle)
	module_mass_load(false, false, false);
	// Traverse through the entire list of modules and register those who implements
	// the hook in question.
	foreach (module_list(false, false, false) as $module) {
		if (function_exists($module.'_'.$hook)) {
			// This module implements $hook, register it in our persistent variable.
			$module_hook_implementors[$hook][$module] = $module;
		}
	}
	// Make variable persist.
	variable_set('module_hook_implementors', $module_hook_implementors);
}

/**
 * Determine which modules are implementing a hook.
 *
 * @param $hook
 *   The name of the hook (e.g. "help" or "menu").
 * @return
 *   An array with the names of the modules which are implementing this hook.
 */
function module_implements($hook) {
	global $module_hook_implementors;
	// Register the hooks when it isn't already.
	if (!isset($module_hook_implementors[$hook])) module_register_hook($hook);
	return $module_hook_implementors[$hook];
}

/**
 * Invoke a hook in a particular module.
 *
 * @param $module
 *   The name of the module (without the .module extension).
 * @param $hook
 *   The name of the hook to invoke.
 * @param ...
 *   Arguments to pass to the hook implementation.
 * @return
 *   The return value of the hook implementation.
 */
function module_invoke() {
  global $module_hook_implementors;
  if (!isset($module_hook_implementors[$hook])) module_register_hook($hook); // reg hook when isn't
  $args = func_get_args();
  $module = array_shift($args);
  $hook = array_shift($args);
  // Make sure module is loaded. Maybe 
  drupal_load('module', $module); // maybe check here if loaded... Probably not much if anything to gain
  $function = $module .'_'. $hook;
  if (function_exists($function)) {
    return call_user_func_array($function, $args);
  }
}
/**
 * Invoke a hook in all enabled modules that implement it.
 *
 * @param $hook
 *   The name of the hook to invoke.
 * @param ...
 *   Arguments to pass to the hook.
 * @return
 *   An array of return values of the hook implementations. If modules return
 *   arrays from their implementations, those are merged into one array.
 */
function module_invoke_all() {
  $args = func_get_args();
  $hook = array_shift($args);
  $return = array();
  foreach (module_implements($hook) as $module) {
	drupal_load('module', $module);
    $function = $module .'_'. $hook;
    $result = call_user_func_array($function, $args);
    if (is_array($result)) {
      $return = array_merge($return, $result);
    }
    else if (isset($result)) {
      $return[] = $result;
    }
  }

  return $return;
}

/**
 * @} End of "defgroup hooks".
 */

Adagio’s picture

It seems this isn't much worth as long as the menu system is built as it is...

The menu hook is invoked, which causes most modules to be loaded. I'm not sure how much of the menu system may be cached, the menu hook is also invoked in this _menu_append_contextual_items() function in menu.inc. Does anybody have some knowledge of this system? How could one conditionally include modules in a good way?

killes@www.drop.org’s picture

Congrats, you discovered the reason, why i didn't further pursue the matter. ;)
The problem is indeed that soem hooks (not only _menu, but also _help) are called on each page load and will lead to inclusion of all modules. The only real solution is to split up the module files and put those hooks into for example. modulename.menu. We are a bit reluctant to do so since it would make programming a bit less nice. if you look at the notify.module you can see how I splir that one up into a .inc and a .module file. most of the code is in .inc but _menu and _help are in the main module file.
--
Drupal services
My Drupal services

Adagio’s picture

Hmm... All hooks are called through module_invoke (all) so it might be possible to cache the return value of certain hooks there. I think this certainly is true for hook_help (as you've mentioned in your post linked to in your previous comment). This could be done "dynamically" like in my version of module.inc, by checking for cache, or getting the original on demand and caching that. I'm not sure about the menu though. Language, role and module and possibly location may be used to compose a cache id. Is it at all possible to determine when menu items needs to be refreshed given only the cache (should of course be rock solid)? I believe the majority of menu hooks may be cached, node_menu is an interesting exception. Possibly something like somemodule_menu(false) == somemodule_menu(true) is a useful check. A somewhat sophisticated cache system may be used to cache different versions of the menu. When an item of type MENU_DYNAMIC_ITEM is encountered, special action may be taken.

I was a bit puzzled to find that the docs said the menu hook was a nice place to put code that only needed to be run once per request, isn't that a counter intuitive approach? Maybe better with hook_init? Furthermore the way the menu is currently cached is a bit strange, no? It calls all hooks anyway?

killes@www.drop.org’s picture

Not all menu entries can be cached.

And I agree abotu the _init hook. We should have one for the purpose you describe. (we already have a hook called _init for other purposes).
--
Drupal services
My Drupal services

Adagio’s picture

Right. If it isn't possible to determine when to refresh the cache or don't cache at all, I suppose another solution could be to parse out the function code (like documentors does), save it in db and eval that for some performance gain. However, this would lead to strange results in a scenario where the menu hook calls other module functions (don't know if this happens). The same is possible for instance for help hooks. However, I sense there isn't that much enthusiasm for something like this, and if that is the case, I will not persue this any further it at this time as I don't have much use for speed improvement.