system_requirements() calls install_profile_info() when $runtime. This means that on admin/config it does a file_scan_directory(), which can take around 78% of the page request.

Patch adds an optional third parameter to install_profile_info(), which skips searching for and loading dependency information when set to FALSE. This speeds up the initial request to install.php by 50% (according to firebug) along with the admin/config issue.

carlos8f mentioned using system_get_info() instead of install_profile_info() in system_requirements(), that might be a less intrusive option.

Marking this major because it's going to completely kill admin performance for upgraded D6 sites and any D7 site not using 'standard'.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

catch’s picture

FileSize
4.04 KB

Without xhprof cruft in .htaccess

bfroehle’s picture

+++ .htaccess	4 Jan 2011 06:18:20 -0000
@@ -2,6 +2,10 @@
+php_value auto_prepend_file /usr/share/php/xhprof_includes/header.php

Well, leave this out, of course.

Edit: Nevermind... you beat me to it.

carlos8f’s picture

This one's a bit simpler :)

I don't think micro optimization during the installer is super important. Also, doesn't require an API change.

carlos8f’s picture

Component: base system » system.module
FileSize
1.74 KB

Let's try this one then, here we also optimize system_modules().

catch’s picture

Status: Needs review » Reviewed & tested by the community

Discussed this with Carlos in irc, I'm happy leaving the installer alone and just optimizing these two places given where we are in the cycle. Marking RTBC, please wait for the test bot prior to commit, however this works just clicking around.

The system_modules() fix removes a full second from that page in webgrind.

carlos8f’s picture

Status: Reviewed & tested by the community » Needs review
FileSize
1.67 KB

This patch gets more to the source of the problem: drupal_required_modules() does a full .info scan, which trickles down to any function that uses install_profile_info(). This patch is more API friendly and less hacky.

catch’s picture

This will also fix the issue in contrib, mollom modules calls drupal_profile_info() from _mollom_get_version() which also triggers this full system scan, just to get the distribution name.

webchick wanted a third reviewer here since Carlos and I have been working on this together in irc, so hopefully one will appear.

Status: Needs review » Needs work

The last submitted patch, 1006714-6-file-scans-are-bad.patch, failed testing.

catch’s picture

So the reason this fails is because module.test actually uses the $info['dependencies'] bit of install_profile_info() during runtime - where this would now be taken from enabled modules rather than the full list from file_scan_directory(). While it's likely to only ever occur in this test, that's a sign that we shouldn't change behaviour here during the same week that 7.0 gets released.

So... I'm re-uploading #4 for the bot, let's open a new issue to clean this up more drastically for Drupal 8, but until then we have a four line patch that removes 78% of the time spent building admin/config.

Only change in this patch is I added a code comment to install_profile_info() documenting the performance issue.

catch’s picture

Status: Needs work » Needs review

Forgot the status change.

catch’s picture

David_Rothstein’s picture

The test failure in @carlos8f's patch looks like it's actually just due to a simple bug in the patch, which is easy to fix. The previous code guaranteed that drupal_required_modules() always returned the install profile as its first array element, but the new code doesn't. So when install_profile_info() calls array_shift() on it, it winds up removing the wrong element from the array. We could definitely fix the code in the patch so as to keep that assumption valid.

However, I guess there is still one API-affecting thing in the patch; by getting the info file dependencies from the database rather than always scanning the filesystem, it means that if you use hook_system_info_alter() to affect the 'required' status, that will now start affecting the return value of drupal_required_modules() too, but only post-installation. That's not necessarily wrong (in fact, it might really be a bugfix on its own), but it is a bit of a change.

So overall I think @carlos8f's approach is preferable, but if we're worried about that for D7 then the patch in #9 is good too. However, with this code:

+  $profile_info = system_get_info('module', drupal_get_profile()) + array('distribution_name' => 'Drupal');

I think it's really ugly to have the calling code responsible for that. Could we add some code to _system_rebuild_module_data() to ensure that the default 'distribution_name' = 'Drupal' winds up stored in the {system} table for profiles that don't specify their own? (OK, one could argue that's an API change too, but I think the argument would be pushing it.) There's already lots of special-case code in _system_rebuild_module_data() to handle the install profile's .info file, so adding one more there seems much better than having the calling code do it.

catch’s picture

Hmm, I'm split between the two approaches, for now I re-rolled #9 with the suggested change from #12, that is indeed a lot cleaner.

David_Rothstein’s picture

Let's go with this one for now, since it's safest. We could always do the other approach in #1029390: No API function for getting install profile info without a full file directory scan and possibly backport it.

The new code comments in #13 had some typos, so I rerolled to fix those.

I also realized that in the case of drupal_install_profile_distribution_name(), it already has separate code paths for the installer and everything else, so why not just fix the performance issue internally to that function and not have to change the calling code? That seemed better to me, so I made that change in the attached patch also.

This looked fine to me otherwise and testing it out with custom distribution names everything seemed to work fine. So feel free to RTBC if the new changes look good to you.

catch’s picture

Status: Needs review » Reviewed & tested by the community

This version looks great. I agree it'd be good to get #1029390: No API function for getting install profile info without a full file directory scan fixed and backported as well.

Also a note I keep seeing people popping up in #drupal/the forums complaining of extremely slow loading times for admin pages with D7. I haven't been able to actually get enough data out of them to figure out if this issue is the actual problem, there's certainly other possible candidates, but I'd really like to see this one get in as soon as possible. We're talking about a full second or more of admin page load times.

andypost’s picture

Status: Reviewed & tested by the community » Fixed
mfb’s picture

BTW, found a related issue, #1081266: Avoid re-scanning module directory when a filename or a module is missing: file_scan_directory() is called for each missing module or install profile, this was continuing to slow down admin/config for me.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

q0rban’s picture

Status: Closed (fixed) » Needs work

This has introduced PHP Notices, as there is data being added to install_profile_info() that doesn't get added in system_get_info.

Notice: Undefined index: distribution_name in drupal_install_profile_distribution_name() (line 202 of includes/install.inc).
catch’s picture

q0rban’s picture

Hmm, the patch itself looks like it accounts for this though. I'm going to put back to fixed, as this was actually something rfay ran into, so I can't replicate it.