Exposed filters are broken

Exposed filters are generally broken in Views 3 (both 6.x and 7.x branches).
I'm using the latest stable D6, and the (almost)latest D7 HEAD.
Also using the latest dev releases of views-6.x-3.x and views-7.x-3.x.

My test is a node view with a few fields added (nid, title, type), and an exposed filter (tried "nid", and "posted date"). So, all pretty vanilla.

First encountered this in D7 while writing a custom exposed filter.

1) I add a filter and expose it, without changing anything (the value stays empty, operator not changed, etc).

Going to the view (from the frontend), I see this notice:

Notice: Undefined property: view::$exposed_data in views_plugin_exposed_form->query() (line 135 of views/plugins/views_plugin_exposed_form.inc).

Returning to the view, I edit the filter and set a value. When I return, I get:

Notice: Undefined index: identifier in views_handler_filter->exposed_info() (line 396 of views/handlers/views_handler_filter.inc).
Notice: Undefined index: in template_preprocess_views_exposed_form() (line 604 of views/theme/theme.inc).
Notice: Uninitialized string offset: 0 in element_children() (line 5484 of includes/common.inc).
Notice: Undefined property: view::$exposed_data in views_plugin_exposed_form->query() (line 135 of views/plugins/views_plugin_exposed_form.inc).

And the filter is not properly rendered. The operator select box is displayed even though I haven't checked the "unlock operator" option, the value box itself is missing (even when no value is declared in the admin). It's just broken.
In my custom filter, when the above errors occur, the value_form doesn't get rendered at all....

Also, when changing the filter options, sometimes the whole top part of views "crashes" and I get left with an incomplete admin page (only from preview to bellow, and no Save button). And of course, the notices above are shown.

In Drupal 6, the same behavior occurs, except that I get the full batch of errors from step 1, and that preview shows the filter correctly, while it's still broken when viewing the view itself from the frontend.

I wanted to look into it, but don't know from where to start, if the fix is non-trivial, point me and I can help with the patch.