This patch: http://drupal.org/node/176003 (Contrib module's hook_requirements are not called when installing) causes this error (as seen in Drupal 6) to appear for us: http://drupal.org/node/176003 (Installation failed w/ max execution time of 30 seconds)

What seems to happen is that we have a large number of default installed modules, These modules, on install, have each of their folders checked recursively for .module and .install files. These file existence checks are done about 38 times various ways in install.php.

A simple solution would be to add a max_depth parameter to file_scan_directory() in includes/file.inc since we can be fairly sure that there are no modules more than 3 folders deep inside of modules, sites/all/modules, or sites/default/modules. This solution would still allow modules that contain sub modules to function correctly.

I'll attach my simple patch, but it might require more in depth analysis by someone involved with the installer.

Normally, this isn't a problem on hosts with fast file systems, but as speed decreases, these file look-ups build up and cause the execution time to run out. Another solution would be to cache the look-ups after they are performed.

Comments

harking’s picture

Status: Active » Needs review
drumm’s picture

Version: 5.12 » 7.x-dev
Status: Needs review » Needs work

Interesting idea.. this is probably an issue in all versions of Drupal, so it is best to start with the development version, where more new code will get reviewed, and backport from there.

Are any of the files being checked twice?

dave reid’s picture

Title: hook_requirements patch in 5.11 causes timeout issues on install. » Add $max_depth parameter tofile_scan_directory()
Component: install system » file system

Patch needs work, the extra parameter needs to be second to last, since the last parameter is used internally in the function to determine it's current recursion depth. There's also a call to file_scan_directory inside the function that needs to be changed as well.

drewish’s picture

it'd be great to postpone this until after #255551: DX: Array-itize file_scan_directory()'s parameters gets committed.

codecowboy’s picture

Status: Needs work » Needs review
StatusFileSize
new820 bytes

The parameterizing is complete. Here's a new patch to get the ball rolling.

drewish’s picture

Status: Needs review » Needs work

need to document the new parameter and add unit tests.

superspring’s picture

Version: 7.x-dev » 8.x-dev
Category: bug » feature
Status: Needs work » Needs review
StatusFileSize
new3.5 KB

I have rewritten this patch, including documentation and a test.

Status: Needs review » Needs work

The last submitted patch, file_scan_directory_max_depth-343444-7.patch, failed testing.

superspring’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, file_scan_directory_max_depth-343444-7.patch, failed testing.

superspring’s picture

Status: Needs work » Needs review
StatusFileSize
new4.2 KB

Same patch as above for the new 8.x-dev.

tim.plunkett’s picture

Title: Add $max_depth parameter tofile_scan_directory() » Add $max_depth parameter to file_scan_directory()
Issue tags: +API addition
+++ b/core/includes/file.incundefined
@@ -1421,15 +1421,23 @@ function file_download() {
+  // Check if this function should run.
...
+    // No? Then immediately return.

I would just have one comment above the function, and make it less conversational :) Though it does sound friendlier

+++ b/core/modules/system/lib/Drupal/system/Tests/File/ScanDirectoryTest.phpundefined
@@ -70,7 +70,7 @@ function testOptionCallback() {
+    // Grab a listing of all the Javascript files and check that they're

This should still be JavaScript.

+++ b/core/modules/system/lib/Drupal/system/Tests/File/ScanDirectoryTest.phpundefined
@@ -132,10 +132,8 @@ function testOptionRecurse() {
+   * Check that the min_depth options lets us ignore files in the starting dir.

Start with "Checks"

----

The functional changes of this make sense.

superspring’s picture

This new patch adds in Tim's review.

sun’s picture

+++ b/core/includes/file.inc
@@ -1421,15 +1421,22 @@ function file_download() {
+ * @param number $depth

s/number/integer/

This is RTBC after adjusting that.

sun’s picture

Status: Needs review » Needs work

That said. On a second look:

+++ b/core/includes/file.inc
@@ -1421,15 +1421,22 @@ function file_download() {
+ *   - 'max_depth': Maximum depth of directories to return files form. Defaults
+ *     to unlimited.
...
+  if (isset($options['max_depth']) && $options['max_depth'] < $depth) {

+++ b/core/modules/system/lib/Drupal/system/Tests/File/ScanDirectoryTest.php
@@ -144,4 +142,14 @@ function testOptionMinDepth() {
+  function testOptionMaxDepth() {
+    // There are only CSS files in the 'css_test_files' directory. A max depth
+    // of zero (zero children down) will not show any of these.
+    $files = file_scan_directory($this->path, '/css/', array('max_depth' => 0));
+    $this->assertEqual(0, count($files), 'Maximum depth limits files found');
+  }

Can we flip the second condition, so that it becomes $depth > 'max_depth' ?

This inherently leads me to the question whether the condition shouldn't be >= or not?

Because it is not really clear for potential users of this function whether max_depth is inclusive or exclusive (unlike the min_depth argument, which is relatively clear), we should add an example to the phpDoc for the option that explains the effect on a hypothetical directory scan.

Additionally, the needs some more love, since it only verifies the special condition of 0, which of course won't ever return anything (but is still good to test, since that is a possible value, so let's keep that).

The test should additionally verify that scan that has files in depth = 0, depth = 1, as well as depth = 2, does not return files from depth = 2 if max_depth = 1.

E.g., you can scan /core/modules/node with the above parameters and you should only get files from:

/core/modules/node/*
/core/modules/node/templates/*
/core/modules/node/...

but not from:

/core/modules/node/lib/Drupal/...

superspring’s picture

Status: Needs work » Needs review
StatusFileSize
new6.19 KB

Same as the patch above with the improved comments and additional tests.

sun’s picture

+++ b/core/includes/file.inc
@@ -1421,15 +1421,23 @@ function file_download() {
+ *     to unlimited. Depth is measured based on number of children, so depth as
+ *     0 would be the given directory only.
...
 function file_scan_directory($dir, $mask, $options = array(), $depth = 0) {
+  // Check if this function should run, if not immediately return.
+  if (isset($options['max_depth']) && $depth > $options['max_depth']) {

1) This still does not clarify whether max_depth is inclusive or exclusive.

2) The particular case being described in the phpDoc is actually identical to the 'recurse' => FALSE option, so I don't think it makes sense to describe that case.

3) Given the above, I wonder whether it would even make sense to merge 'recurse' into 'max_depth'. We can potentially keep 'recurse' as a simple convenience helper option, but its only effect would be that 'max_depth' is set to 0 (zero).

+++ b/core/modules/system/lib/Drupal/system/Tests/File/ScanDirectoryTest.php
@@ -144,4 +142,23 @@ function testOptionMinDepth() {
+    // When the depth is increased to include the CSS directory this should
+    // show all 11 CSS files.
+    $files = file_scan_directory($this->path, '/css/', array('max_depth' => 1));
+    $this->assertEqual(11, count($files), 'Maximum depth limits files found - Max 1');
+
+    // Same as above going up another directory. This adds another CSS file.
+    $files = file_scan_directory($this->path . '/../', '/css/', array('max_depth' => 2));
+    $this->assertEqual(12, count($files), 'Maximum depth limits files found - Max 2');

Mmm... this will cause test failures for many other innocent patches in the queue ;)

Instead of testing the count, we should verify that the path names of the returned files are limited in the expected way; i.e., as outlined above, verifying that files up to a certain depth appear, but no files beyond that depth.

superspring’s picture

So I've added some more to the documentation, merged the recurse option and changed the tests.

Status: Needs review » Needs work

The last submitted patch, 18: file_scan_directory_max_depth-343444-18.patch, failed testing.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

kim.pepper’s picture

Issue summary: View changes
Status: Needs work » Closed (outdated)

Closing as outdated. Please re-open if you feel this is still relevant.