Caching in token - it is necessary? (D7)

By discussing #262360: Expose control of token caching to functions that call token_replace with greggles and eaton in IRC, we recognized that further discussion on the caching procedure is necessary for future development.

Because this may lead to some major rewrite it's best to only do this for Drupal 7 branch. D5 & D6 have to deal with the $flush-option (eased in issue #262360).

Some major questions could be:

• Is caching necessary?
• When is caching necessary?
• What data is cached?
• How is it stored?
• How could it be flushed?

* In D6 implementation cache can now only be flushed as complete. There is no way to only flush a single element like node:15.
* It is not possible to cache two or more variants of one object - like "node:15 after load" and "node:15 after edit", as it is needed in rules.module - an approch to this was (#9 #10 in issue #262360: Expose control of token caching to functions that call token_replace)

There are cases token does not have to be cached at all, becaus token replacement will only be applied once or the object is allways changing. In operations, like bulk, there could be cases where caching would speed up the process, but I don't know a special case, maybe greggles could figure some out?

As discussed on IRC introducing a caching hook might me a sledgehammer solution.

So, I hope this issue will find a useful, performant way to improve token's caching, or maybe simply switch it off!?

btw: had to mark version 6dev, because there does not eem to be a D7 branch