Support non-entity objects

The initial consideration behind 7.x-3.x was explicitly to always use entities, because that's what D6's hook_views_bulk_operations_object_info() is, a hook_entity_info() before its time. We gain nothing by introducing an abstraction on top of an abstraction.

Anyone can implement hook_entity_info() for a data structure, it's just one hook (Entity API provides the default controller and the related functions), but also one hook that the D7 contrib world knows (unlike VBO's which is / would be completely custom).
Plus, exposing Entity Metadata allows us to automatically build the "modify entity values" form for an entity, which we'd lose if we went with a custom abstraction.
All this also gives people automatic Views integration, the ability to use Search API and efq_views, and so on, so the benefits are numerous.
There is no data structure that shouldn't be an entity type, D8 proved that.

So, I'm very strongly opposed to this.

I needed to do the same and ended up defining the new tables as entities and I could get my new actions magically to the View!
For future users, here are some points to get started.

First, in your views.module.inc file, add the following:
$data['MY_NEW_TABLE']['entity type'] = 'MY_NEW_ENTITY_TYPE';
This will make the View aware that you are querying a View.

Then, add a new entity. See this guide: http://drupal.org/node/1026420

function module_entity_info() {
  return array(
    'MY_NEW_ENTITY_TYPE' => array(
      'label' => t('MY_NEW_ENTITY_TYPE'),
      'base table' => 'MY_NEW_TABLE',
      'entity keys' => array(
        'id' => 'your_tables_primary_key',
      ),
    ),
  );
}

and then, define your actions like this:

function module_action_info() {
  return array(
    'module_remove_something' => array(
      'type' => 'MY_NEW_ENTITY_TYPE', 
      'label' => 'Remove URL', 
      'configurable' => FALSE,
      'behavior' => array('deletes_property'),
    ), 
  );
}

Note that above is not a very smart use of entity module. It's really great but in my case, it was all tiny data that I don't need a form, URL or any other powered-by-entity feature so this was a straight forward integration. Always try to make best use of Entity module if possible.
Hope someone will find it useful :)
Thanks everyone for your hard work!