Magic Cache, APC, Memcache, Varnish, etc.

This module become quite complex, I need to rewrite it from scratch probably using classes and implement some new ideas.
Hopefully I'll find some time soon.

just out of curiosity, isn't APC kind of doing the same thing?

well, I mean this APC (http://php.net/manual/de/book.apc.php), doesn't have anything to do with Drupal

APC does work, but it cache only file requests. You can play with the settings (apc.ini):
My are:

apc.shm_size = 256
apc.stat = 1
;apc.include_once_override = 1 ; more aggressive cache
apc.optimization = 1
apc.num_files_hint = 2000

Memcache caches resources, but firstly module have to use the cache_get and cache_set solutions. Secondly 'clear cache' removes all cached data, which happen very often.
If you have more than 1 web server, Drupal have to have couple of memcache instances run separately, because if you have one memcache server, it could slow the performance, rather than improve. If you have large tables, you could run couple of memcache processes on different ports specifically for selected tables.

Varnish caches static pages, it's suitable for load balancers (only for anonymous). You can set different kind of complex rules in the configuration.
See:
http://groups.drupal.org/node/25425
http://drupal.org/project/varnish
Using Varnish you could have some session problems. Sometimes you end up with hacking the Drupal core.

Some Drupal caches just combine and compress CSS & JS files or other static files.
Boost module create static pages (works only for anonymous).

Alternatively you can use as well Pressflow, instead of Drupal. Which could additionally boost your environment.

Magic Cache trying to cache the code which is executed thousands of times per one load. Drupal 6.x doesn't use cache_get much, especially in some node_load, user_load,
Views doesn't use cache much as well, see: #479886: Cache support for views_include_handler(), etc.

In the other hand, non-advanced users (like testers) or on Windows machines, they don't care about cache engines, because they don't know how to install and how to test it if it does work better or not.
Another case: Live servers could have plenty of cache solutions (load balancers with memcache+varnish), so they don't care about other cache solutions. Developers and testers using the environment on their local machines, which doesn't have any cache solutions and it could be very, very slow. Especially if you have environment set up on Virtual Machine.
In my current environment I've got 413 .module files (488 entries in system table, >10k nodes, >50k node revisions, >10k term_data) and the website is just half way. If you go to admin/build/modules, it's enough to go to make a good Coffee. And the most of the people test this website using VM.

By the way, somebody purposed me different approach to Magic Cache, to make it more object oriented (and to use maybe __autoload) and convert RegEx functions which are parsing the PHP code, into lexical scanner (token_get_all function).
Currently I'm on contract position till March, so probably after that time I could find some time.
But it's better to wait for Drupal 7, otherwise backport could be not possible.

The code have to be very stable and each change have to be tested by profilers.
Probably I need to build some time statistics mechanism to not run each time IDE Profiler. It's important to test, because each change could make the performance worst and after some time if performance is worst you don't know why. Even when you have PHP Notice messages, drupal error handler is called and it's slowing down everything, especially if some function could be called hundred of thousands times.
So for this you have to have special debug environment set up (I used before PhpED IDE with build-in Profiller), but currently I don't have environment set up to do any code changes. Unfortunately PhpED doesn't work in Linux environment.
Currently there are some bugs, because it's using too much memory and bigger mysql packets that are in default settings, so it could break your Drupal environment totally. It's doing some magic things with the core behavior, so currently it's not suitable for non-advanced users, who don't want to screw up their Drupal environment. Basically it's trying to limit the calls of function which are called thousands of times, but caching results directly from the functions!
See some recent tests here: http://drupal.org/node/471322#comment-1663432

So it just matter of time, I think.

400+ Modules.... you better have some serious CPU power on your productive machine, or your magic cache really does some magic. I'm definitely going to test it these days.

Closed because Drupal 6 is no longer supported. If the issue verifiably applies to later versions, please reopen with details and update the version.