Initial test

Yes, I forgot about one option to mention. If you can, please modify following line:

function magic_cache_replace_file($module_names, $files = array(), $new_filepath = NULL, $replace_mode = 'file', $init_code = '') {

to: $replace_mode = 'eval'

I'm not sure if this will work, because I'm still working on it.
But with 'eval' it will work much faster.
I've commited with 'file' for testing purposes (it's better to debug it in this way).
You can also turn standard Drupal cache, it will go faster as well.
Sorry for that.

In test plan should include some operation testing when user is logged in as well, when the most of caches doesn't work then.

Basically the idea of this module is to cache some function which normally are impossible to cache and limit some calls will take a lot of time (like require_once, node_get_types, recursive theming elements - like dates in calendar, etc.) and to limit number of queries to db of course.
This of course will work on heavy website, when functions are repeatedly called hundreds of times without any cache for authorized users.

Test with current version on single refresh in attachment (committed today as next working version).
Green - with Drupal Cache
Red - with Magic Cache

Not a big improvement, but:
bootstrap.inc, commons.inc have better execution time
database.mysql.inc have less queries as well
And it's only part of it what it can be done to be more faster.

Environment:
OS: Windows7 on WAMP
AMD X2 Dual Core 6000+
Memory: 4GB RAM
So 2 seconds on single refreshment with standard Cache it's terrific.
I've got some websites, where execution time is around 5-10 seconds on some pages, but I'll make some other tests later.

UPDATE:
Commenting line 176 boost additional 0,2sec what makes 1,8 instead of 2,1sec

           //require_once $new_filepath;

How many contributed modules you have installed? I've got 132

SELECT COUNT(*) FROM system WHERE type = 'module' AND status = 1;

I think on this stage of development you need to have minimum 100-150 activated modules to see some improvements.

In example I've got around 190 .module files in my contributed dir (when Drupal making recursive scanning on admin/build/modules or loading half of them on each refreshment, it takes ages sometimes).

Do you have some additional cache engine activated for anonymous? If yes, please disable any memory caches.
Which paths are you testing? Different? Or test one which is quite complicated to render, like calendar block rendering (where all dates are rendered separately) and on /admin path, where all modules are loaded within the same time (with cache activated for system module, currently disabled as default).
Can you create one calendar block on every page where each date element is rendered? Later I'll test on some simple pages.

But at the moment the heavies and problematic websites are my goal:)
Another extremely stat:
I've got website which execute 17,3sec on my fronpage (with MC it's 16,4sec)
(
8530 calls to _db_query I've reduced to: 8350
875 file_scan_directory calls to 858
78839 (LOL!) _db_query_callback() to 78482
)
See attachment.
Definitely there is something wrong with the code workflow of this website (chart showing standard Drupal single refreshment). When you are log in as admin, none of modules could cache it.

UPDATE: Disabled devel, bench_chart, wsod, PHP4, back to 1,9sec:)

Something about caching methods.
There are 3 caching methods available:

FILE:
all core modules are loaded from separately files with cache callbacks
it's a little bit slower than eval, but it's good for diagnostics purposes

EVAL:
All core modules are loaded from memory through session (later you can choose from which source you want to load them, so it will be faster). In this case files are not included each refreshment. Application will spend less time making include_one call to hundreds of files and interpreting each file separately per each refreshment.
Requirements to see some improvements: probably around more than 100 activated modules

DYNAMIC (commited today):
All available functions (handled by cache engine) are declared on start, but body of functions are created on the fly when it's called (eval).
It's good method, when you've got active hundreds of modules which are loaded file by file each refreshment (include_one) and the definition of all functions takes a lot of time to load them in one time (even from memory).
Requirements to see some improvements: probably around more than 150 activated modules

I'll make some tests with Devel as well, thanks, I didn't know that it have this functionality.
How to dump stats from node_counter as you did?

Thank you very much for your statistics.
Yes, you have right, it will not boost a lot when you have a lot of batch operations (yet).

Test by Devel Generate module with following settings (see statistics available in attachment):
Nodes to generate: 100 (around 12 different content types)
Comments per node: 5
Option enabled: Delete all content before generating new content
Disabled option: node_counter table (external statistics used).
Hardware Environment: AMD X2 Dual Core 6000+ with 4GB RAM
OS: Windows7 on WAMP

Log output:
# Deleted 100 posts
# 100 nodes created

Times:
Drupal Cache: 51sec
Magic Cache (dynamic): 1min 6sec
Magic Cache (eval): 1min 10sec
See attachment.

In this case comparison is bad, because in batch processing there is no any theme rendering (you see only one page) and you are not using the same nodes in memory, you just creating them and deleting for one time.
In real usage you using the same nodes all the time and each time when page is fully rendered, Drupal is loading all activated module files each time and executing the all hooks from hook_init to hook_exit.
That's why in batch operations, this cache will not work good.
Yes, it'll be possible to improve the times for batch processing by declaring interior cache hooks with specified some rules which could boost it, but not on this stage.

For the best result, test plan should simulate some real usage of the website by users.
The best method to do that, is to execute JMeter test with some real data:
http://jakarta.apache.org/jmeter/
In JMeter there is functionality to load Apache Access Log and then check average execution of websites and check the execution times by adding i.e. 'Graph Result' object
One test with Magic Cache, and one without it.

With Devel Generate you can't accurately simulate usage of the website by real users.
Standard test should include: login process (showing the front page with all rendered items), and time of all rendered pages which are used the most.
Standard user on the website should spent at least around few minutes.

Committed today latest version.

You can download here:
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/magic_cache...
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/magic_cache...

From the beginning I thought that session will be a good place for cached data, but Drupal store it in database as well so it cause problems, if the session become too big. Other thing that cache data stored in session is only available for specified user.
So later I'll add new storage method as standard cache_get & cache_set

Thank you for testing.

Next improvements.
Today's version with the same configuration as in #9
# Deleted 100 posts
# 100 nodes created
+ 5 comments
Is 17 seconds less than it was before in previous test with the same configuration.

In comparison, it's only 7 seconds difference with Drupal in batch processing. Not tested yet with some other plans.

Features:
- added support for tpl templates
- improved installation process, added cache clearing and theme rebuilding, but it still can show some errors when enabling the module, but after refreshment or deletion files/magic_cache directory it should help

Improvements:
- moved storage method from sessions to standard cache table, so it could be combined later with some memory cache as well
- conversion some arrays into cache object
- what's the most important, reduced notice messages handled by drupal_error_handler in module to 0 in magic_cache_generic_call() and it should stay like that to keep performance
Probably that was the main reason that it was slowing down (if there was were around 20 notices, so 30k calls to generic_call() cause more than half a million calls to Drupal handlers!)

I still have some problems how to create new object to empty object and increase it without error:
I've tried:

$hits = $func_obj->stats->hits ? $func_obj->stats->hits : 0;
$magic_cache_data->$cache_key->miss++; 
even: $magic_cache_data->$cache_key->miss = 1; 

If it's possible, I want to not use if's as many as it's possible.
It's still showing the Notice error

Direct links to update:
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/magic_cache...
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/magic_cache...

Next version from today:
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/magic_cache...

TEST 1: 100 nodes
Environment: #9
Configuration for 100 node test the same as in #9

Improvements from version before:
Now is 56sec instead of 1,10min (now 14sec faster)
Only 6sec after Drupal

TEST 2: Front Page (single page refreshment)
Drupal: 2,2sec
MC: 2,1sec
Basic difference:
- _db_query calls reduced from 316 to 184
- module_load_include() reduced from 150 to 30

TEST 3: Admin Page (single page refreshment)
Drupal: 5,1sec
MC: 3,3sec!
Basic difference:
drupal_get_schema_versions() reduced from 60 to 40
module_load_include() reduced from 1,8sec to 148,8ms even it's still 177 calls, Why? Filesize of included files has been reduced to minimum
drupal_get_filename() 624 calls - it's a number of files? LOL;/
db_query() only reduced few times

CHANGES:
- added cache engine rules to design code workflow and set callbacks
- replaced error_handler to ignore NOTICEs totally and log errors to database on shutdown
- added autolearn mode as default, which reducing side effects to minimum
- removed MC_CACHE_GLOBALS and MC_CACHE_SESSION methods, not necessary anymore
- now cache sub-directories are removed correctly
- other bug fixes

Do you have APC installed? (or some other PHP cache?). If I understand correctly what your module does, I would not expect it to make a huge improvement when APC is working.

Currently I'm on Linux environment, but on Windows I never couldn't set up any of cache including APC (WAMP/XAMPP/clean inst.). Even when could run it, there was some crashing and unexpected behaviour. On more complex projects I couldn't develop on Windows at all, sometimes loading was more than 1 minute.
I'm planning to rewrite this module from scratch, because now it become too complex. Based on classes, and to add some sql caching.
But currently I'm out of time for the next months.

Closed because Drupal 6 is no longer supported. If the issue verifiably applies to later versions, please reopen with details and update the version.