If you are using form_panel_div, you will most likely want to create a class for your inner divs that sets them to "display: inline". Otherwise, they will not appear on lines by themselves.

The best way to do this using the default definitions is to include these lines in your theme's CSS file:

  div.form-panel-div {
    display: inline;
  }

Another way to accomplish the same thing, entirely in PHP code, is to add the style to #form_panel_div_attributes. For example:

  $form = array(
    '#theme' => 'form_panel_div',
    '#form_panel_div_attributes' => array(
      'style' => 'display: inline'),
    );

This latter method is not generally considered good Web design, however.