Dynamically alter pane settings form

I'm trying to create a pane that uses AJAX to dynamically add additional fields to its settings forms. For example, when you open the pane settings, there's a single text box and an add more button, and clicking add more keeps adding additional text boxes, similar to how lists of items with FIELD_CARDINALITY_UNLIMITED in a node edit form work.

The way the node edit form does it is to create the form with the right initial number of fields, then on ajax submit, load the form from cache and in the submit handler increment the number of fields and ask for a rebuild, which rebuilds the form with the new number of fields and recaches the result. This doesn't seem to work in the ctools world: the rebuild and caching cause problems because ctools is doing its own caching.

I'm wondering if anyone knows the correct ctools way to dynamically alter the form structure within the ctools wizard form.

Here's a hacky way I got it working, but it involves parsing the form input within the form creation step which seems to be a no-no. Many thanks for any pointers, and many thanks for all the great work on ctools/panels!

$plugin = array(
  'title' => t('Dynamic form test'),
  'category' => t('Test'),
);

function formtest_pane_content_type_render($subtype, $conf, $panel_args, $context) {  
  $block = new stdClass();
	$block->title = "Values:";
	$block->content = theme('item_list', array('items' => $conf['values']));
  return $block;
}

function formtest_pane_content_type_edit_form($form, &$form_state) {
  $conf = $form_state['conf'];
  
  // Default num fields is the number currently stored.
  $numFields = isset($conf['values']) ? count($conf['values']) : 1;
  
  // Override if we've added some.
  if (isset($form_state['input']['numFields'])) {
    $numFields = $form_state['input']['numFields'];
  }
  
  // Button to add a new field.
  $form['add_more'] = array(
    '#value' => 'Add one',
    '#type' => 'submit'
  );
  
  // Was the add more button clicked?
  if ($form_state['input']['op'] == $form['add_more']['#value']) {
    $numFields++;
  }
  
  // Create the fields.
  for ($i=0; $i<$numFields; $i++) {
     $form["field_$i"] = array(
       '#title' => "Field $i",
       '#type' => 'textfield',
       '#default_value' => isset($conf['values'][$i]) ? $conf['values'][$i] : '',
     );
  }
  
  // Hidden field to store the current number of fields.
  $form["numFields"] = array(
    '#type' => 'hidden',
    '#value' => $numFields,
  );
  
	return $form;
}

function formtest_pane_content_type_edit_form_submit($form, &$form_state) {
  $conf = $form_state['conf'];
  
  // Copy all the values from the form into conf.
  $form_state['conf']['values'] = array();
  
  $numFields = $form_state['values']['numFields'];
  for ($i=0; $i<$numFields; $i++) {
    $form_state['conf']['values'][] = $form_state['values']["field_$i"];
  }
}