I think you can speed up your form altering by pre-filtering:

foreach ($form as $key => $item) {
becomes
foreach (element_children($form) as $key => $item) {
This has the advantage of skipping all the elements that start with #