Steps required to reproduce bug: build website in Drupal 7 using Zen-based subtheme.

Expected behavior: fast website.

Actual behavior: abysmally slow website.

The priority is major because many people seem to be having similar issues which make Drupal 7 entirely unusable.

The information below is a reproduction of the original forum post at http://drupal.org/node/1080330#comment-4174588.

I chose Drupal over Wordpress for its functionality and used it to develop a webpage (visible in its nascent form at http://chriswalkerimages.com).

As you can see, the website is unusably slow, both on the production server (GoDaddy shared "deluxe" linux) and localhost (XAMPP, Windows 7, x64, no performance issues).

I am using page caching for anonymous users as well as browser caching, so while the first page request takes > 30 seconds frequently, subsequent requests are completed in an almost-reasonable timeframe. Of course, this does not help load times for registered users.

During development with the Zen theme, I thought the slow speeds were due to the theme registry being rebuilt on every page load. This is not the case: theme registry rebuilding has been disabled in every possibly related .conf file and double-checked in the MySQL tables.

I enabled Xdebug on my localhost and examined the results using kcachegrind: it appears that almost all of the loading time is repeatably due to these php functions:

drupal_load and system_list call drupal_get_filename (~58 times combined).
drupal_get_filename calls drupal_system_listing (~16 times).
drupal_system_listing calls file_scan_directory (~32 times).
Most of the time is spent in file_scan_directory, which is called (including self-calls) ~1184 times.

As far as I can tell, Drupal is scanning many directories every time a page is requested, looking for enabled and bootstrap modules. I believe the module list ought to be cached in an SQL table, but it doesn't appear that this table is being used.

I've deleted all modules I added and disabled all unnecessary modules. This temporarily brought the number of file_scan_directory calls to about ~600, and now it is back up to double that for an unknown reason...

The site is entirely unusable in its current form, but I know this is a bug that can be fixed: there is no good reason to search all the site directories (multiple times) for every page request!

I have attached a couple of representative kcachegrind images to illustrate.

The first cachegrind file was created right after getting Xdebug working, the second file is the most recent, after I deleted all installed modules and disabled all unused core modules. There was a speed improvement on the localhost due to this change, but the problem persists.

Thanks,

Chris

CommentFileSizeAuthor
#4 new_install_uncached_anonymous.jpg111.6 KBWalker
#4 new_install_uncached_authenticated.jpg118.26 KBWalker
callgrind1.jpg88.56 KBWalker
callgrind2.jpg92.05 KBWalker

Comments

artatac’s picture

artatac commented

Sub

bfroehle’s picture

bfroehle commented

Perhaps related to or even a duplicate of #1081266: Avoid re-scanning module directory when a filename or a module is missing or #993334: Core slows down when few modules are missing.

cdu’s picture

cdu commented
Version: 7.0-rc4 » 7.0

I'm new to Drupal, and installed version 7. I am also experiencing very slow performance on both, my localhost, and on a the server.

Can you tell me if version 6 is any faster, or is Drupal just very demanding on resources?

Thanks,
Carl

Walker’s picture

Walker commented
StatusFileSize
new new_install_uncached_authenticated.jpg118.26 KB
new new_install_uncached_anonymous.jpg111.6 KB

Thanks for the links- based on the second bug report I checked out the system table and there were indeed entries for modules that I had deleted. (Apparently one can install a module by moving it to the right directory but uninstalling it is not so simple.) I haven't had a chance to try the patch from the first report yet.

After removing these entries, the load times did not improve noticeably. However, the other bug reports seem so similar that I think we are fighting the same problem.

On a whim, I downloaded Drupal-7.0 again and moved my drupal/sites folder to the new install. For an unknown reason, the load times have now decreased significantly.

The delay before the XAMPP server starts sending page data on the localhost is about 2.5-5 seconds for authenticated users, 2-2.4 seconds for unauthenticated users looking at uncached pages, and about 50 ms for unauthenticated viewers looking at Drupal-cached pages. The unauthenticated/uncached time is probably still going to be too slow when I move the code to a shared server: I am about to upload the new version to my shared hosting server to see the speeds there.

After looking at cachegrind files from the new version, it appears that file_scan_directory is still responsible for a significant delay when serving uncached pages, as the attached images indicate.

Request: intelligent error message. Just as Drupal notifies the site builder of the performance hit caused by rebuilding the theme registry on every page load, the user could be warned of this issue (assuming this problem gets nailed down exactly). For instance, Drupal could point out the module in the system table that was not found and explain that it was taking a performance hit by searching through directories to find it. The faster solution would be for Drupal to ignore the module and warn the user rather than searching for it. I don't have enough experience to know if this would frequently put Drupal in a situation where it would break before being able to report this error or if this contradicts some Drupal design philosophy.

yakker’s picture

yakker commented

subscribing

glacialheart’s picture

glacialheart commented

subscribing

chichiMi5’s picture

chichiMi5 commented

sub

Thorsten’s picture

Thorsten commented

Is anyone taking care of this? This seems to be a very good candidate for the serious landing page hit regressions I and many other shared-server-user experience (up to 10 sec with D7 whereas loading in a snap with D6) ... please, don't force me to switch to wordpress ...

catch’s picture

catch
he/him
Primary language English
commented

drupal_get_filename() should never be calling drupal_system_listing() during normal operation.

Try saving the admin/modules page and see if this continues.

If that doesn't help, you'll need to do the following:

1. Pick a module that is being passed into drupal_system_listing() - any of the 58 or whatever it is modules will do.

2. Post the results of SELECT * FROM system WHERE name = 'module_name'; to this issue.

3. Post where the module is actually on your file system (if it's there at all).

catch’s picture

catch
he/him
Primary language English
commented

@Thorsten: passive aggressive comments do not help motivate people to work on issues. If you can't post useful information to help people track down bugs, then don't post anything at all.

catch’s picture

catch
he/him
Primary language English
commented
Component: file system » base system
Issue tags: +Performance
damien tournoud’s picture

damien tournoud commented

@catch: this is a confirmed bug. I saw that happening a few times already. Just remove a module from an install, and Drupal starts scanning the whole filesystem a few hundred times per request.

Thorsten’s picture

Thorsten commented

@catch: please don't get me wrong, I didn't mean to be aggressive ... I really wanted to know whether there is any covert activity on the issue and maybe wanted to raise some awareness for that. Especially the first-landing-page-hit problem can turn out to be a project-stopper for shared-server-projects - or at least, to be frank, for me. I'm not so sure whether this is tied to module removal, but need to look into that further by testing.

catch’s picture

catch
he/him
Primary language English
commented
Status: Active » Closed (duplicate)

So this is a duplicate of the issues in #2 then. See people over there.

Thorsten’s picture

Thorsten commented

After reinstalling from scratch, it shows that this (compared to D6) unusual shared-server behaviour is unrelated to module removal - no module/theme has been added, nothing removed. Standard Cache settings all switched on. After not-using the site fo a couple of hours, the fist page hit takes about 10 seconds before the site starts loading. The more I'm looking at it, the less I'm sure this belongs to this issue ...

catch’s picture

catch
he/him
Primary language English
commented

If you're leaving the site and coming back, this is almost definitely a cron run.

Thorsten’s picture

Thorsten commented

This could be it, I just had a 30sec-wait, checked cron run time afterwards and it fits ... I'll set the cron runs up to once day and see whether that solves the issue. Bad me, I should have seen this. Thanks so much! Never mind my desperate comment above, very sorry for misposting on the wrong issue!

cdu’s picture

cdu commented

Not sure why, but D7 seem to be working fine now. It might have improved after clearing all caches.

veganjosh’s picture

veganjosh commented

Just my own experience with this issue, for anyone else that finds their way here trying to fix this problem: for me, it wasn't a missing module, it was that I'd deleted a script file I'd added with drupal_add_Js() without removing the call to add it. I removed that call and 60+ second page loads dropped to reasonable 0-3 second loads. Thanks to the above posters for turning me onto the right line of thought.

SandPond’s picture

SandPond commented

I made 4 installs of Drupal 7.16 (updated to 7.19). My Internet connection is 5Mb/s. Two of the installs were DrupalWiki distro. Using Firefox 18.01 I experienced average front page load times of 14 to 30 seconds for the first page, then all was well with the rest of application naviation (about 1--2 sec each, except for admin pages). Caching modules were no help. One site uses Godaddy SSL. There was no difference in speed (although one might expect a much slower load time). Then I discovered that the first page load times using MSIE 7 were 1.5 to 2 seconds. Wow! I was careful to clear caches while testing this issue. I had several Drupal installations at Godaddy and a CentOS CloudServer that we manage at Rackpace. Next I tested the page load times using Safari on a brand new Mac. The Mac page load time was 60 sec to "NEVER". I verified the speed of my connection prior to each test. It would appear to me that at least some versions of FireFox and Safari are unfriendly to Drupal, or vise-versa. I have not found much about this. My brother had exactly the same results! I can not understand why there is such a tremendous lag, and so much client/server negotiation prior to first page load on these browsers. I would use reverse proxy server to mitigate this problem, but first I need to understand what Drupal, the webserver (Apache2 in all cases), and the client (FireFox or Safari) are struggling to negotiate while attempting to connect. I do not have the resources and technical skills to analyze this. I don't think this issue is dead by a longshot. Currently at least some of our FireFox and Safari clients can not use our Drupal installations. Is there anything further on this problem ??? Sorry about the long post, but I needed to articulate my particular experience (as best I can).