I'm pretty sure this is a bug, but I may be mis-understanding how Drupal's preprocess/process/rendering pipeline should work.

Simply put, preprocess overrides to modify the book navigation menu can't really do much, since book_children() does a manual call to drupal_render(). Shouldn't a render array be created so that THEME_preprocess_book_navigation() or THEME_preprocess_block() can actually manipulate the child items of the menu before they are rendered?

I don't know the module/rendering pipeline well enough to suggest a fix, but this seems to be a bit of a broken design point.