How to tweak the FCKEditor plugin EnterMode

We'll have a GUI for this when #313497: Allow configuration of advanced editor settings gets fixed.
Until then, you can use an implementation of hook_wysiwyg_editor_settings_alter() to override these settings in a small module of your own.
Otherwise, the settings made in the config file are overridden by Wysiwyg module. (In the end, we want to avoid having to modify files when setting up editors.)

There are some examples of using the hook in the issue mentioned above, and some more in the issue queue.

Something like this should work (where 'custom' is your module name):

function custom_wysiwyg_editor_settings_alter(&$settings, $context) {
  if($context['profile']->editor == 'fckeditor') {
    $settings['EnterMode'] = 'br';
    $settings['ShiftEnterMode'] = 'br';
  }
}

EDIT: Fixed typo.

Fyi, for FCKeditor, the settings keys need to be capitalized.

$settings['EnterMode'] rather than $settings['enterMode'] (will not work!)