Deep code - How to get something like module_invoke_all that works on pass-by-reference ?

OK, So I'm trying to develop a useful, general-purpose hook (to allow modules to modify a $term object whenever they want - testing Catch's Term Hooks patch on a D6 backport thing)
bangpound attempted this in a D6 bridging module last month at the EOL/taxonomy code sprint called taxidermy . I saw that that module needed to remake a module_invoke_all-alike to work as intended, and it seemed odd.

module_invoke_all() currently runs a bunch of operations on a given set of arguments - and returns an array of data which is usually glued into the original subject to add info to it.
OK. That's sorta useful, but not as useful as letting the invoked hooks add their data to the object themselves as needed.
I think.

So OK, we can just pass-by-reference and it'll work. It works well for hook_form_alter, doesn't it?
No. A few experiments with module_invoke_all and mods thereof prove that although call_user_func_array() does refs, we cannot use func_get_args() and still get the info back to the caller.

Hm. OK, so how does hook_form_alter() do it so well?

Oh Noes! hook_form_alter() (or actually drupal_alter()) has to do creepy things to make this seem to work. Namely - it passes a list of elements that have to be handled as handles as a value hidden inside the first parameter - ... or something like that. Anyway, it's voodoo. And documented as such.

I want:
- A hook
- That I can invoke, by passing in an object
- Have other modules play with that object
- And get it back changed
- Possibly with values modified or even removed from it.

//////////////////
// Experimental

/**
 * Run module_invoke_all and support pass-by-ref.
 * 
 * I can't find the magic way to do this intelligently. Brute it instead
 */
function module_invoke_all_by_ref($hook, &$arg1 = NULL, &$arg2 = NULL, &$arg3 = NULL, &$arg4 = NULL, &$arg5 = NULL) {
  foreach (module_implements($hook) as $module) {
    $function = $module .'_'. $hook;
    if (module_hook($module, $hook)) {
      call_user_func_array($function, array(&$arg1, &$arg2, &$arg3, &$arg4, &$arg5));
    }
  }
}

NOW - I know that brute-setting 5 magic-number args is not the most elegant code.
But IT WORKS.
and if you want more than 5 parameters passed by ref, you are madder than me.

The current approach used in drupal_alter to address this PHP problem is freaky and I don't really think this easily-understandable way is worse than the arcane __drupal_alter_by_ref hack.

I can copy & paste this magic inside my own place where it gets used - but I'm trying to solve the general case here. I've seen a bunch of modules which have to do their own thing because module_invoke_all isn't good enough. I'd guess that some modules that use module_implements do so inside an identical foreach loop, and do so because module_invoke_all() isn't good enough.

I'm not yet suggesting that this hack of mine should be tried in real code, but I am asking if any brighter PHP-minds out there can suggest a workable solution to my problem statement above.

Feedback? Suggestions?
.dan.