Modules or themes with too many files kill drupal_system_listing performances

civicrm shows we need a combination of whitelist and blacklist.
But how should the system decide which rule has priority?

[drupal_system_listing]

; don't look into subfolders of civicrm
scan[] = "- civicrm/*"

; except for some-subfolder/module-1
scan[] = "+ civicrm/some-subfolder/module-1"

; don't scan further down in some-subfolder/module-1
scan[] = "- civicrm/some-subfolder/module-1/*"

Like this, the priority would be clear: Later rules override previous rules.

Alternative to the array notation (faster to type, but doesn't allow comments):

scan = "
  - folder1/*
  + folder1/subfolder1
  - folder1/subfolder1/*
"

Or, we could target specific files, so we don't need additional exclusion rules.

scan = "
  - civicrm/*
  + civicrm/module1/module1.info
  + civicrm/module1/module1a/module1a.info
  + civicrm/module2/module2.info
"

EDIT:
Removing my own comment for being off-topic.

Here is a draft for a possible solution. The function file_scan_system_directory is meant as a replacement for file_scan_directory in drupal_system_listing. It's not meant as a general replacement for file_scan_directory!!!

Warning: This is draft code, I didn't even test it on my localhost.

The scandir.ini files can be placed anywhere in any module directory, or even the modules root dir. Paths in the scandir.ini files are relative to the location of the scandir.ini file.

I am not yet sure how the scandir.ini files should look like, but the result should look like this:
(assuming the file is "$path/scandir.ini")

<?php
array(
  // exclude "$path/dir1", except for ..
  'dir1' => array(
    'allowed_subdir_1',  // .. "$path/dir1/allowed_subdir_1", and all contained files and folders
    'allowed_subdir_2',  // .. "$path/dir1/allowed_subdir_2", and all contained files and folders
  ),
  // exclude dir2/aaa/bbb, except for ..
  'dir2/aaa/bbb' => array(
    'allowed_file.info',  // .. this info file
    'ccc/another_file.info',  // .. this info file
    'ccc/ddd/another_allowed_subdir'  // .. all files and folders in this subdirectory
  ),
  // exclude everything from dir3
  'dir3' => array(),
)
?>

I hope the rest is obvious from the attached file..

The idea:
- Modules can define their own scandir.ini files
- Users can have their own settings for modules that don't provide scandir.ini files.

A patch for Drupal 6.x dev.
I don't have one for D7, I hope someone else can do that.

Attached is the patch, and an example scandir.ini file that is right now sitting in my sites/all/modules/contrib. Obviously, the scandir.ini makes sense for the modules that I currently have installed on the test site (a lot), but you would have to adjust it to your needs.
My tests show that the patched drupal_system_listing produces the same $files array as the old implementation: Same keys, same order, same values. And it's a lot faster, with the scandir.ini in place.

Theoretically the scandir.ini could be auto-generated. But, that would also mean something has to take care that it gets rebuilt from time to time - we would be back to square one, because rebuilding is just as slow as it always has been.

The only sustainable solution is having scandir.ini files shipped with contributed modules. Until then, the hand-made scandir.ini can be used as a temporary solution.

A simple standalone script to automate the creation of scandir.ini files in sites/all/modules/contrib, if you want to use the above patch. If your modules are somewhere else, you need to adapt the script. You can run it for core modules, but they are usually not the big performance killer. You can run it for custom modules, but then you need to re-run it every time you change the folder structure or add new sub-modules. You don't want that.

Some instructions:
- You need to remove outdated scandir.ini files when you update a module. An easy way to do this is to delete the folder of the old module version, before you upload the new version. This is why I made separate scandir.ini files for each module.
- It is a good idea to re-run scandir.php every time you install or update or patch a module, but this is not required. The system will do its job without any scandir.ini files, it will just be a bit slower (about as slow as it is without the patch).
- You do NOT need to run scandir.php when you just enable or disable existing modules in admin/build/modules, as this does not change the file structure in any way.

Note: This is not the intended use of this mechanism. The original idea was that modules define their own scandir.ini files, which is especially useful for big ones like civicrm.

I would not suggest to make the automation script part of Drupal: It is very likely that Drupal does not have write access to the modules folder.

How is this helpful?
A typical use case: I made a custom module and want to test the auto-installer. This means, I have to repeatedly enable and disable this module. Each time I have to wait for drupal_system_listing, which can be quite slow with big contrib modules. The patch removes this one bottleneck.

@alexanderpas:
Sure, please invent a better syntax that fits in the *.info files.
But please consider that some module might not have a modulename.info in the root folder..

I see that style is a big issue here, we don't want ugly files lying around. For now I just wanted to have something that works, as a proof of concept and as a quick fix for people who don't want to wait for something pretty.

Maybe you can even come up with a convincing database-driven solution.
The point of having it in files was that these files get automatically deleted or updated when you update the module.

With the proposed patch my sites/all/modules/contrib/cck/scandir.ini file looks like this:

.[] = 'content'
.[] = 'modules/content_copy/content_copy'
.[] = 'modules/content_permissions/content_permissions'
.[] = 'modules/fieldgroup/fieldgroup'
.[] = 'modules/nodereference/nodereference'
.[] = 'modules/number/number'
.[] = 'modules/optionwidgets/optionwidgets'
.[] = 'modules/text/text'
.[] = 'modules/userreference/userreference'

which means, inside this folder please look for
content.info, content.module,
modules/content_copy/content_copy.info, modules/content_copy/content_copy.module
etc.
(dot means the current directory)

The idea was that the following should also be allowed:

.[] = 'content'
.[] = 'modules'

Now this would mean please look for
content.info, content.module,
and everything inside modules/

unfortunately, I think the current implementation does not look like this will work - but hey, it's the idea that counts.

The third notation this syntax is capable of (for the patch I'm not sure if this works):

css = '#'
js = '#'

which means, scan everything except the css and js subfolder. Or in other words: For the current folder there are no restrictions. For the css subfolder, please only look for css/#.info, css/#.module, css/#/* - which does not exist, obviously. For the js subfolder, please only look for js/#.info, js/#.module, js/#/*.

In the real world I think the first solution is totally sufficient.

@alexanderpas:
Another problem I see with putting it in modulename.info files is that any auto-generation would have to mess with the original contents of the info file, and that you can no longer simply delete a file to force a re-scan.

effectively, if i'm reading the API docs correctly, we're left to only using this to scan for the .info & .tpl.php files effectively (as all the other files are declared in the .info)

I did not even think about tpl.php files! Thanks for reminding me.
I think it would make sense to separate hints for *.tpl.php files and *.module / *.info files. Usually x.module will be in the same folder as x.info. Not so for *.tpl.php.

So what about

; .info and .module files (suffix is auto-appended)
modules[] = 'content'
modules[] = 'modules/content_copy/content_copy'
modules[] = 'modules/content_permissions/content_permissions'
modules[] = 'modules/fieldgroup/fieldgroup'
; template folders
templates[] = 'templates'
templates[] = 'modules/fieldgroup/templates'
; individual template files
templates[] = 'modules/fieldgroup/something.tpl.php'

I am not sure if the auto-appended .info and .module suffix is a good idea, but it helps that you don't need two separate definitions for each.

Folder hints vs file hints:
In 3rd party modules the scandir.ini (or the information in modulename.info) can be auto-generated before the module goes to CVS, so there is little extra cost in providing file hints instead of just folder hints. Folder hints make sense during development and for template files - if one folder contains nothing but template files.

we've already determined that we don't want the requirement to declare the template files inside themes.

Makes sense. Usually themes are not that big, and only one is active at a time. On the other hand, if there is a mechanism for modules, it's probably not such a big step to do the same for themes. It would be all optional, so a theme developer would only use this if the theme contains some big 3rd party lib or tons of images.

What if we would introduce naming conventions?
Any directory name starting with underscore would not be scanned.
Or, if we already found a *.info file in a directory, then we only scan subdirectories named "modules".
Maybe a bit too simple..

just to make things clear: if a module directory doesn't contain an info file, we should scan it's sub-directories recursively until we have found those .info files.

For civicrm this would mean (afaik) that we scan everything.
Maybe this should be fixed in civicrm then? Let it provide the "Drupal" part via drush dl into sites/*/modules, and move the "civicrm" part into sites/*/libraries. This would be a big change for civicrm, but imo a good one. Not sure if we could produce an upgrade path..

---------

If there are other projects where a heavy subdir (such as, a big css or js or img subdir) is not blocked by a modulename.info, then my idea would be a projectname.project.info (to not be mistaken for a module info file). And if we already consider doing that, then maybe put everything into *.project.info files, and never into modulename.info? I think it is more logical to think of this as per-project, not as per-module.

The other question is whether we should auto-generate these files.

Drupal would think that civicrm_project is a module then. Which it is not. Is this a problem?
Otherwise, I would say "civicrm.project.info" instead.