Admin Performance + Site memory consumption

Zvi,

Please note that the devel project now contains the "performance" module written by me, the original author of the memory patches. It provides a much more accurate measurement than the above patches, and require no patching of core. It also allows measurement of different URLs on the site.

Use that instead.

Zvi,

Please note that the devel project now contains the "performance" module written by me, the original author of the memory patches. It provides a much more accurate measurement than the above patches, and require no patching of core. It also allows measurement of different URLs on the site.

Use that instead.

subscribing, I'm also starting to explore performance issues and interested in following this.

@kbahey: As of Devel 6.x-1.20, there is no "performance" module. There is only a "Performance Logging" module, which does not do what this patch accomplishes--display per-module memory usage reports on the page.

Please let us know where this "performance" module really is.

I've opened #874124: Where are per-module memory statistics? to see if the Devel guys can clarify #3.

When I said "performance module" I meant "performance.module" which is called the "performance logging" module in the .info file.

As for per module memory, the patch that is on 2bits.com, will give you an idea of the initial memory a module uses (code size, initial data structure if any). It will not give the memory that the module uses when doing certain tasks, e.g. allocating a large array, processing a large data set.

The performance logging module will measure total actual peak memory a page took, and will tell you that on a per URL basis. It is accurate, although it does not give per module data.

Thanks for the clarification. So sounds like this patch is next to worthless except for specialized cases?

The patch used to give us an idea of how much of a memory hog the code in the module will take.

That had some value when we had only one file to load, in the days when Drupal modules were only one file (e.g. example.module).

Nowadays, not even that can be deduced accurately, because the menu system in Drupal 6 and later can dynamically load other files (e.g. example.inc, example.pages.inc, example.admin.inc, ..etc.) on demand.

The patch cannot measure memory use by module in cases where the module allocates large data sets in a certain paths and not others.

The performance logging module will measure this accurately (by path), but as an aggregate for all loaded module.

I'm going to be bold and marked this as "closed" since it appears the patch has no value.

Is this an adequate place to discuss some alternatives?

I think the goal is to identify potentially problematic modules that have either excessive memory, CPU or query consumption.

Perhaps there is an alternative metric that we could investigate .. even if it is just based on a specific page being rendered, it would be helpful to know things like:

- how many nodes are in the static cache at exit (other static cache stats could be useful .. taxonomy or path related)
- which module or function triggered which nodes to be loaded into the static cache
- CPU time spent per module(?) -- this could be better as per-function
- % of query time per module or function (total all query time and then possibly display as a pie chart with the most database-hungry modules at top)
- cache friendliness of a module? this combined with the previous metric would be ideal -- maybe a module has an extremely high query load, but is also extremely cacheable.

Just some thoughts .. as a developer that likes to use Drupal on data-heavy projects (collaborative data analysis/research), the ability to open a dialog on any page to get a performance snapshot of that page would be extremely useful.

I may argue that these statistics are actually more valuable now that a module only loads what is 'used' on a given page .. :)

Dumping more notes here .. perhaps this can require apd, and a method like this should be considered --

rename_function('hook_load', '_perfmod_default_hook_load');
override_function('hook_load', '$args', 'return _perfmod_monitor_hook_load($args);');

function _perfmod_monitor_hook_load($args){
        // get memory usage
        // start Timer
        return _perfmod_default_hook_load($args);  
        // stop Timer
        // get memory usage
}

Override a set of core PHP functions and API hooks/drupal functions and wrap with monitoring points.. apd requirement is probably more reasonable than core patches.

If anyone is interested in knowing their per-module memory usage on load (which is what this patch does) I have a 6.19 version of it that includes the memory.inc file. I built this patch for pressflow, so I am not 100% certain as to how well it will apply to vanilla Drupal.

I do agree with the folks above, however, that this is really just a rough baseline measurement. The desired stats listed above would be ideal. I wonder if it would also be useful to track hooks and see what modules are running hooks and how memory usage looks after they execute their hooks (though that could be so large a list that it would be useless).

any version of this patch for drupal vanilla 6.20 ?

I finally succeeded to apply the last patch with small modifications.

After using this patch I have strange results on my dev compared to my production website.
On my dev with acquia installer on windows i have around 5 times less memory usage(around 12MB) than on dreamhost shared (around 60MB).
it's exactly the same modules used on the 2 environments, the only difference is that on dev my drupal is a fresh acquia 6.20, and on my production it's successive upgrade of acquia drupal with svn up.

Does the usage memory is different according to the platform or it should be mostly the same ?

You could be looking at a lot of differences between your dev env and dreamhost. Different apache modules and different php modules, different versions of apache, (or even IIS if you are running that on your windows box).

One thing that I would strongly recommend is to get Pressflow Drupal (www.pressflow.org) and drop that on top of your current Drupal installs. It is an optimized version of Drupal for PHP 5.2.x and MySQL 5.1.x--It removes a lot of extra code that is used for compatibility with PHP 4.x and Postgres SQL, etc...

I also see *a lot* of people complain about Dreamhost. You should really look at cloud hosting with a place like rackspace. They have really good prices (starting around $20/month) and they can be elastic to handle more traffic when you need it.

I finally switch my website to the smallest vps on advomatic, with pressflow by the way :-)
But the memory usage is exactly the same than dreamhost, even little bit more.

Does APC or any other solution could reduce my memory usage on views/panels.

I will have a look to rackspace.

Your memory usage will likely not change no matter where you are, but your performance should increase by using something like APC as it caches memory usage.

I found that when using APC in combination with the CacheRouter module and setting Drupal's caching to 'Normal' my admin page loads were in the 300ms range and anonymous loads were sometimes as low as 9ms.

Advomatic installed me APC, but after reading your message i wanted to give a try to cacherouter also :-)
What a big mistake...my site is now done, the system is swapping extremly...But I don't undestand i still could use drush. But unfortunatly I have no root access to restart Access and the opeing time for service are only for US times...So my 2 sites on this VPS will be DOWN till tonight...

Hey Heyyo;

I'm not sure how you did the install for cacherouter, but the default install information is not correct--that may have caused your issue. There are two things that you'd want to change:

1. The include location in the default examples is wrong (you need to remove contrib/)
2. The default cacherouter setting is to use 'db' you need to change that to 'apc'

Yes I did that, but i think it was more a memory problem. Never mind for the moment the website runs quite fast. If needed I switch to another offer on Advomatic with varnish and memcache. For the moment there is no reason to break the website again, i have untill 8 clients in parallel. not a lot but quite enough for the moment.

If you can swing it, Devel + XHProf has proven useful for this. You do not see it split by modules per-say, but it is CPU time + memory used by functions and their children.