Tracing large memory usage and slow response time - looking for additional hints

rotax42 - January 16, 2009 - 19:32

Hello

I have following problem:
Something (Maybe some module ?) on my portal is consuming large amount of memory
and for some reasons the response times for simple pages are very slow.

I tried to narrow it down using devel and trace modules, but it didn't help.

Devel output shows:

Page execution time was 2021.95 ms. Executed 591 queries in 93.71 milliseconds.
Memory usage:

* Memory used at devel_init(): 79.83 MB
* Memory used at devel_shutdown(): 97.55 MB

Nearly 80 MB of memory were used by something before the request has been started at all.
How can I identify what is it ?

On one other very simply Drupal installation I get following output:

Page execution time was 549.31 ms. Executed 117 queries in 94.14 milliseconds.
Memory usage:
Memory used at devel_init(): 0.7 MB
Memory used at devel_shutdown(): 1.37 MB

So only 0.7 MB

Similiar thing when I am using trace.

On the complex installation:

#210cc3df T=0.000000 [REQUEST] 2009-01-16 18:21:20.837796 GET /community HTTP/1.1
#210cc3df T+0.315741 [NOTICE ] E_NOTICE: Constant PREG_CLASS_PUNCTUATION already defined in `modules/search/search.module':83

The "NOTICE" is the very first entry in trace log after the "REQUEST"
How can find out what happens in the time period of 0.315 sec inbetween ?

In the simply installation in the same time I (0.315 sec ) get already the first "RESPONSE"

Has somebody any hints how can find out what consumes the time and memory
in the complex installation ?.

Details:

Complex installation:

Drupal 5.8
206 ative modules
Server has two CPUs and 4 GB RAM

CAPTCHA Already 0 blocked form submissions
CURL INSTALLED
Configuration file Protected
Database schema Up to date

Mime type detection Browser & Extension
MimeDetect is using the browser supplied mime type or file extension lookups. It is strongly recommended that you install and configure the PHP Fileinfo Extension or the UNIX 'file' command to provide more accurate severside mime type detection.

MySQL database 5.0.32 MySQL database for event module 5.0.32
PHP 5.2.0-8+etch13
PHP 5.2.0-8+etch13
PHP register globals Disabled
Unicode library PHP Mbstring Extension
Web server Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13
jQuery Update Installed correctly
The current installed version of jQuery is 1.2.6

Cachinng is enabled in normal mode
CSS aggregation is enabled

Simply Installation:

Drupal 5.10

few Modules active (core, devel, tracing and few others)
Server has one CPU and 256 MB RAM

Database schema Up to date
GD-Bibliothek 2.0 or higher
Configuration file Protected
MySQL-Datenbank 4.1.10a
PHP 4.3.10
PHP Register Globals Disabled
Unicode-Bibliothek PHP mbstring-Erweiterung
Webserver Apache/2.0.53 (Linux/SUSE)

Try this:

gpk - January 16, 2009 - 21:13

Try this: http://2bits.com/articles/measuring-memory-consumption-by-drupal-bootstr...

The problem is likely to be "206 ative modules".

gpk
----
www.alexoria.co.uk

Yes I agree :( But I hope i

rotax42 - January 17, 2009 - 00:13

Yes I agree :(
But I hope i can identify some "heavy weights" and reduce the memory load.

The link is very good thanks :)
Regards

This patch brings it realy

rotax42 - January 17, 2009 - 23:01

This is a part of my results:

komplex installation:

Page execution time was 1907.66 ms. Executed 591 queries in 129 milliseconds.

Memory usage:

Memory used at devel_init(): 78.6 MB KB MB
Memory used at devel_shutdown(): 96.72 MB

Start 115224 113 0
End 101293328 98919 97

Modules 73645368 71919 70
Bootstrap 27532736 26887 26
Total 101178104 98807 96

event 1903016 1858 2
user 1901880 1857 2
views_filterblock 1731720 1691 2
node 1666728 1628 2
cck_address 1663504 1625 2
daterange 1519304 1484 1
simplenews 1446440 1413 1
buddylist 1408080 1375 1
privatemsg 1390720 1358 1
system 1328072 1297 1
webform 1292144 1262 1
cck_fullname 1223896 1195 1
fckeditor 1210504 1182 1
comment 1199368 1171 1
views_ui 1197728 1170 1
smtp 1147808 1121 1
eventrepeat 1115936 1090 1
taxonomy_manager 959728 937 1
views_theme_wizard 913824 892 1

.
.
.
Long list follows

Now I can look more systematic for modules which should be deactivated or replaced by something else.

Do you have a PHP cache?

Keyz - January 18, 2009 - 00:07

Do you have a PHP op cache on the big server? e.g. APC, eAccellerator, or Xcache, etc? I find that makes a big impact on memory use. I just tested on an almost-no-modules Drupal 6 install to refresh my own memory on the memory use with and without a PHP cache. This demo site has very few modules, but has 100,000 nodes, test from loading the standard front page.

Without APC:
Executed 155 queries in 647.01 milliseconds. Page execution time was 833.96 ms.
Memory used at: devel_init()=0.94 MB, devel_shutdown()=7.77 MB.

With APC:
Executed 155 queries in 635.52 milliseconds. Page execution time was 693.68 ms.
Memory used at: devel_init()=0.18 MB, devel_shutdown()=1.93 MB.

I had more extreme results (both in memory use and execution time) on a production D5 site with a lot of modules in the past before/after using a PHP op cache (not sure the module count but could have been about 100).

Also on a separate note... "Page execution time was 1907.66 ms. Executed 591 queries in 129 milliseconds." ... looks like you have some slow/buggy code somewhere, not a slow query. Usually (in my experience at least) the page execution is only a bit higher than the query execution, the queries usually being the majority of the total execution time.

-- David
davidnewkerk.com | absolutecross.com
View my Drupal lessons & guides

Is there any good APC tutorial on the web ?

rotax42 - January 18, 2009 - 00:37

Thanks for Your hint,

I've been digging furhter on the 2bits site and read also some articles about op cache.

http://2bits.com/articles/benchmarking-drupal-with-php-op-code-caches-ap...

On the small server with the basic Drupal installation I have the APC extension installed.
I found it using phpinfo output:

apc
APC Support enabled
Version 3.0.15
MMAP Support Enabled
MMAP File Mask no value
Locking type File Locks
Revision $Revision: 3.151 $
Build Date Dec 31 2007 16:17:37

But I don't know how to find it out on in the complexe installation.
in phpinfo output of this installation i couldn't find any hint about ACP.

But I am not sure if this is the proof, that APC is not enabled ?

Is there any good APC tutorial on the web ?

I had already a look on the pecl site but there was nothing I could use ( on my level )

Regards

I believe in my case (CentOS

Keyz - January 18, 2009 - 02:54

I believe in my case (CentOS 5) I installed APC with yum (if I recall correctly). If I view my php info, it shows /etc/php.d/apc.ini under "additional .ini files parsed". Also if I run apc.php (comes with ACP files I believe) I can see the cache status. If your php info shows no "apc" whatsoever, then I'm pretty sure it's not installed. If you have eAccellerator, I believe it usually appends its name near the bottom.

I'd recommend looking up how to "install APC your-linux-version" and there's very likely a variety of ways.

-- David
davidnewkerk.com | absolutecross.com
View my Drupal lessons & guides

High memory usage

Gomez_in_the_South - April 5, 2009 - 04:31

In my case it was the macro module that is included in the devel project package. I had used it for some troubleshooting before and had neglected to clear the information that it gathered. Even with the devel module disabled it was loading more than 50MB of data before each request.

I identified this by going to the Variable Editor in the devel menu and sorting the variables by size. After realising it was the devel macro module that was causing the problem it was a simple matter to go into the macro editor and clear the variables it had stored. This dropped by memory usage shown by devel_init to 1MB from 58MB.

Huge memory usage on Dreamhost VPS, how to find which module

wwwoliondorcom - July 25, 2009 - 21:00

Huge memory usage on Dreamhost VPS, how to find which module causes troubles ?

Hi,

I have the same problem, how can i find what is the module that uses too much memory on my dreamhost virtual private server ?

Website has a lot of traffic but no more than before and suddenly thousands of errors everyday when i haven't changed a single module, so what's wrong ?

Thanks.

 
 

Drupal is a registered trademark of Dries Buytaert.