How do I integrate custom layout built in panels with theme layout and general settings in the theme?

Did you find the documentation for this?

It depends what type of plugin you are talking about -Page Layout or Panels Layout.

This is from the AT Subthemed info file, it has pretty good docs on this:

;----------// Custom Layout Plugins

  ; Adaptivetheme supports both Page Layout plugins (it's own native layout
  ; plugin type) and Panels layout plugins. Page layouts control the sidebars
  ; and main content column layout - these are controlled via theme settings for
  ; Page layouts. The Panels layouts are for Panels module and are controlled via
  ; the theme settings for Panels layouts.
  
  ; There are seperate directories for each type of layout and an example
  ; plugin for each. These don't do very much, they are just examples, you can
  ; use the core themes layout plugins as a guide. Please see the _README in the
  ; layouts directory of this subtheme.
  
  ; Remove the semicolon to umcomment and clear the sites cache.

  ; Page layout plugins
  ; plugins[page_layout][layouts] = layouts/page

  ; Panels layout plugins
  ; plugins[panels][layouts] = layouts/panels

Ha, I just read that and realized I never added the example Panels layout plugin, ops, better do that... :))