Handler validation: validate defaulted handlers too

Here's a use case:
I have an argument that changes a field.
So the argument's validate() should check that the field is actually present, like this:

  function validate() {
    if (!isset($this->view->display_handler->handlers['field']['MYFIELD'])) {
      // Where's my field?
    }
  }

This is great!
However, here's where it breaks down:

Suppose I add the arg and the field to my default display: that's ok. Validation is checked and passes.
Now I add a page view, and there I override the fields and remove the field.
The page display is now broken, because it inherits the argument but doesn't have the field.
But validation passes, because the argument it has (inherited from default display) isn't getting checked.