Consider introducing a native ClassLoader for performance and PSR-4

Let's see if this works.

Forgot the rebase with the last two in #1, so ignore them pls.

Some interesting benchmarks with xdebug.
For simplicity, I only give the file size of the cachegrind file.
This turns out being quite representative of the duration, and it is almost 100% reproducible between requests!
(mostly +/- 500 bytes, sometimes +/- 2000 bytes)

8.x, as it is now (Symfony ClassLoader): 12,171,524 bytes
"piecemeal" with composer autoload.php: 11,414,884 bytes
custom classloader (direct copy of Composer's): 11,435,139 bytes
custom with PSR-4: 12,243,687 bytes
custom with PSR-4 and additional predictor character: 11,274,134 bytes
custom with PSR-4, two additional predictor characters: 11,259,079 bytes (not currently posted here as a patch)

Conclusion:

• Switching to Composer from Symfony does speed us up.
• Having a custom loader that is a plain copy of Composer's has no significant performance impact, compared to using Composer's loader directly. This is good to know.
• Introducing PSR-4 does slow us down, if we are not careful.
  I think the reason is that Core and Component are still registered as PSR-0, and thus get a lower priority behind modules. If we care about performance, we should register Core and Component as PSR-4 before modules.
• The performance tweak helps, but we have to assume this is still suboptimal due to Core and Component not having top priority.

(all those have been on admin/config/performance)
And now, what happens if we apply the predictor logic to PSR-0, starting from the "custom-classloader" patch, without any PSR-4 ?

Custom classloader, direct copy of Composer's: 11,435,139 bytes
Custom classloader with predictor logic: 11,290,428 bytes

That's an improvement, but not that much!

Now, what if instead we try admin/structure/views/view/content?
This should have more module classes involved.

Custom classloader, direct copy of Composer's: 19,269,512 bytes
Custom classloader with predictor logic: 18,843,899 bytes

I am going to attach this PSR-0 + predictor logic.

EDIT: I suspect the fail is bogus.

How does it compare to classmap?
(now with real durations!)

I suppose the difference will be greater if more modules are installed, and more module classes are used.

And more benchmarks.
This time with PSR-4.
One time we add only modules as PSR-4, while core is added as PSR-0. Another time we add both core and modules as PSR-4.
EDIT: All from admin/structure/views/view/content

Explanations:
"full classmap": Add all enabled modules added to composer.json, then run composer dump-autoload --optimize.
"core/vendor classmap": core and vendor classes as classmap with --optimize. module classes as psr0 / psr4.
"psr0": using the composer-style psr0 loader, no psr4.
"psr4 (modules)": use the psr4 loader, register module namespaces.
"psr4 core/modules": register also core classes as psr-4.
"predictor": using the loader with PREDICTOR_INDEX as in the patches from above.

EDIT: Added Symfony loader

Conclusion:
- Composer is clearly faster than Symfony. The reason is probably that it uses the first character as a predictor.
- Moving to PSR-4 has a clear benefit, if done correctly. Even if we don't move our class files around, PSR-4 is easier and thus faster to compute. However, we need to register modules and core.
- The predictor index pays off big time.
- Class maps are obviously the fastest solution. But they might not always be available. Especially for modules.

Caveats:
- Maintaining our own native loader instead of using the default Composer one does have a price.

Disclaimer:
- Adding more modules will change the numbers.
- Measuring on a different page will change the numbers.

Combination of different patches from #1 which promise to bring the best result:
- The "piecemeal" patch from #2038135-44: Use the Composer autoloader to make everything simpler
- Introduce a custom core/autoload.php and a custom Drupal\Core\Autoload\ClassLoader.
- Make this loader capable of PSR-4.
- Add predictor logic to shorten the foreach() loop.
- Register core and modules as PSR-4.

This patch does not introduce anything classmap-related, because we can still do that any time in the future.

If we actually commit this, I recommend to do it step by step as in #1, for the sake of a nice git history.

Follow-up:
Whenever Composer changes its class loader, we should adapt:
- When Composer introduces PSR-4, we can register core namespaces directly via composer.json, instead of the ugly hack in the patch.
- When Composer introduces autoload_files.php, we no longer need to explicitly include anything.
- If Composer actually introduces the suggested performance improvements, we can switch back to Composer completely! We may keep the custom autoload.php, but it will only be an adapter for the Composer autoload.php.

Interdiff #9 vs #2.

EDIT: We can see, this is a bit ugly. But it will only last as long until Composer introduces PSR-4 support.

Yep, it needs a re-roll after the Composer autoload patch.

Here is the same on github, so I don't have to upload all the history as intermediate patches..
https://github.com/donquixote/drupal/compare/8.x-psr4-x-predictor

I really tried making this a useful history.

https://github.com/donquixote/drupal/compare/8.x-psr4-x-predictor
changed history again.

EDIT:
The modified comment compared to #13 is
https://github.com/donquixote/drupal/commit/6018fba444ed762fd51d7f51890f...
I added getModuleNamespacesPsr4() and registerNamespacesPsr4().

-      $classloader->addPsr4('Drupal\\' . $name . '\\Tests', $dirs);
+      $classloader->addPsr4('Drupal\\' . $name . '\\Tests\\', $dirs);

because the class loader wants to have namespaces ending with '\\'.

History:
https://github.com/donquixote/drupal/compare/8.x-psr4-x-predictor

add links to related issues about PSR-4

Yes. All that we want to keep from this issue is already part of #2083547: PSR-4: Putting it all together.

Whether we want some additional performance tweaks for the class loader is a different question and I don't think we should waste any time on that in the near future.

Recent benchmarks I did myself did not show such a big performance difference anymore.