Files are removed from node if FileField is unset in form_alter

I using a module that puts node forms in modal dialog pop-ups. I'm using a hook_form_alter function to change some node forms by unsetting some form items. This seems to work well with most field types, with the desired outcome of no actual changes to the database for the unset fields when the node is saved. However with filefield if I unset the form item, when I save the node it deletes all of the data in the filefields for that node. It would be great if there was a work around for something like this for filefield, so I don't have to just css hide it.

Thanks!