In Drupal 7, field_widget_behavior() was a helper function to provide quick access to "the properties in the 'behavior' entry in the definition info of the widget currently configured by a field $instance".
Changes in Drupal 8 have made this function irrelevant:
- There is no 'behavior' entry in widget plugin definitions anymore
- The introduction of "form modes" means there is no single widget associated to a field $instance anymore.
The function has thus been removed from Drupal 8.
Code that might need to access the plugin definition of a widget used in a form mode can do:
$options = entity_get_form_display($entity_type, $bundle, $form_mode)->getComponent($field_name);
$plugin_definition = \Drupal::service('plugin.manager.field.widget')->getDefinition($options['type']);