Block caching disabled by permission changing modules

Hello

We are building a complex website with a bigger community related to gaming & game development (daimonin).
I have nearly all solved, the site is working and the flow established.

Our problem is caching for registered/logged-in users.
Some permission changing modules, which are important to organize the access rights on a web 2.0 site,
will disable the block caching which disables the only useful caching mechanism from drupal for logged-in users.

Let me first describe in a deeper way the basic problem.
I read alot threads around this problem and i had the feeling that many people had noticed the effect
but not understand the reason.

Itself, we are a combination of web 2.0, project development and "account managment". Account managment
in the terms like for example world of warcraft is managing their game accounts over a website.

You HAVE in the future to register yourself to our website to manage the accounts of our mmorpg.

We talk about alot registered users. I reseted our old website with 150k and a 2nd with 70k registered users.
After major releases we had over 300 new users every day - without the need on the old website to register!
So, i can think about szenarios where we have over 1k new users per day when our new game client is
coming and site registering is needed.

As i love drupal i found that drupal is not a good web 2.0 system from nature.
Why?
Because for a social website you need a senseful caching system for registered users and the possibility to include dynamic content. In both areas drupal is weak compared to other CMS. Drupal is more a news & document related system, where a smaller team with access rights is generating content for guests.

That was really true for drupal 4 and 5 and a reason i used postnuke/joomla.

With drupal 6 and native block caching, which work also for roles & logged-in users, drupal was
able to close that gap. Its still not perfect but with a good flow & site design drupal is fast enough
with the base web 2.0 modules installed.

And here comes the problems:

A handful of the core web 2.0 modules comes with permission changing modules to manage their content.

Mainly Organic Group with its default permission module but also the related modules which handles group access
or group member managment. The node gallery permission part and many more. Every time you see after module
installation the "the permission must be rebuild" your system lost all the logged-in users caching mechanism.

Thats true even you have all the caching modules installed. We have memcached API, block caching setting, patch cache and part of advanced cache installed. The problem is not the cache - its the fact its works to 90% only for guests except the block cache.

In real numbers: If you enable that not unimportant modules, your web 2.0 site will drop in performance for a medium sized web 2.0 site (Organic groups, views 2 (no cck & panels), voting, comments, a gallery, events, notificications, messages,
privatemsg, subscription... thats all nothing special, i let out userpoints, active listing and such, but without this base you should not call your drupal site a web 2.0 site) to page generating around 0.5 seconds in the middle.

For logged-in users of course - but we talk about web 2.0 (or social websites), right? Many web 2.0 site have zero guest content, accept from a "log in" page. So, logged-in users is what counts for a social website.

The 0.5 seconds (thats 500ms) is true for every normal root server you can rent. If you are able to rent a high performance
cluster giving significant better numbers, then you are also able to hire 5 web programmer coding a native system for you without drupal and your name is facebook, myspace or you have a good money source for your new company.

The bad page loading time is natural for a LAMP/php system doing all the stuff described above and NOT A SPECIFIC DRUPAL PROBLEM. Also joomla and Zikula will drop to that numbers when you use them uncached with all that features (and even worser). That btw even true for a drupal core installation.

Compare the drupal forum access to a good native forum like SMF. Try to include 200k forum nodes, 50k users and let 20 logged-in browsing the forum. Thats normal numbers for a smaller mid sized gaming community. Drupal will work (as for example drupal.org) but compare it to SMF. SMF can cache all kind of users by nature, its noticable faster as drupal in this szenario when you setup it right with memcache. You can host that kind of website with SMF with good speed even on lower servers. Its all about caching - not the used cms/forum. On the other hand: IF your forum is accessed by more guest and only a small community builds it up, then drupal can head up - by using their aggressive caching.

But back to joomla & Zikula: There you have alot ways to cache things also for logged-in users.

I know that people will start to argue when it comes to real numbers, but i have 4 years experience with that kind of sites and i must say that the results was pretty similiar and repeatable.

The rough numbers: Without block/logged-in caching drupal is 2 times slower as a comparable CMS (which can cache logged-in user content, at last parts).

With smart block caching drupal is on level and up to 50% faster, simply because the drupal core itself is pretty fast.
Note: Also in joomla & Zikula you get the main cache speed by put for example in the user profiles stuff in blocks and let
that stuff with short time settings (1 to 3 minute is absolut enough) in the cache.

So, the conclusion is that: Drupal without logged-in caching is a bad choice for a social website if you ever expect more as 2-3 logged-in user at the same time ever. And even then you should be prepared to spikes on non root servers.
If you monitor older web 2.0 sites build out of < drupal 6 you will experience often that behaviour.

I know that now some drupalists will start to argue but again - its not a drupal specific problem. You WILL experience
that problem with every php system under every LAMP installation using a normal $50 root server.

So, the important question is: How to overcome the fact that permission changing modules will disable the block cache?
What must be done to fix that?
Are there solutions?

Thanks for your patients.