Conditionally add/remove handlers

I'm in a situation where my requirement for the data changes, this means that sometimes I want to store a simple address and sometimes I want to store a more complex address.

The thing is that right now the handlers used is fixed, meaning that I would have to use form alter which usually is complicated or I would have to create my own widget. It would save me a lot of hazzle if we could do add an alter before addressfield_generate is called in hook_field_widget_form and hook_field_formatter_view.