Here's how to add 'separator' and 'nolink' classes to the <LI> element

Add the following function to your theme's template.php file:

function MYTHEME_process_menu_link(&$variables, $hook) {
  if ($variables['element']['#href'] == '<separator>') {
    array_unshift($variables['element']['#attributes']['class'], 'menu-item--separator');
  }
  if ($variables['element']['#href'] == '<nolink>') {
    array_unshift($variables['element']['#attributes']['class'], 'menu-item--nolink');
  } 
}