hook_ENTITY_TYPE_delete() not firing

I have a module that provides his own entity - and only one entity type should be possible. When I enable the module I create a field and a field instance and attach it to the entity type.

My goal is to delete the field instance and the field itself when I uninstall the module (thus remove that entity type).

I thought hook_ENTITY_TYPE_delete() should do the job. But it never gets called. Did I miss something?

/**
 * Implements hook_entity_info().
 */
function my_entity_entity_info() {
  $return = array(
    'my_entity' => array(
      'label' => t('My Entity'),
      'plural label' => t('My Entities'),
      'base table' => 'my_entity',
      'fieldable' => TRUE,
      'entity keys' => array(
        'id' => 'pid',
      ),
      'bundles' => array(
        'my_entity' => array(
          'label' => t('My Entity'),
          'admin' => array(
            'path' => 'admin/my_entity/settings/entities',
            'access arguments' => array('administer my_entities'),
          ),
        ),
      ),
      'module' => 'my_entity',
    ),
  );
  return $return;
}

function my_entity_my_entity_delete($entity) {
  dpm('I never get called.');
}

Is it because I didn't create an entity so far? If that's the case: how can I hook into this process then?