This is a problem that we've run into on an occasional basis (anywhere from every day to once every few weeks). We're not quite sure how it happens, but the end result is that CTools can't find the plugin file for our layout and it results in a WSOD on the site. Worse, CTools doesn't report any error to watchdog and even though it returns a completely white page, it's still an HTTP 200 OK response header, so many site-monitoring tools don't pick it up right away.

Here's the rundown on how this happens:

  1. ctools_plugin_load_includes() is called, given the $info parameter that is returned (essentially) by panels_ctools_plugin_layout(). This includes the $info['load themes'] property, saying that themes can provide layout plugins.
  2. The list of themes is loaded in ctools_plugin_get_directories(), which in turn calls _ctools_list_themes().
  3. Inside _ctools_list_themes(), this bit of code finds layouts provided by themes:
        $current = variable_get('theme_default', FALSE);
        $themes = $active = array();
        $all_themes = list_themes();
        foreach ($all_themes as $name => $theme) {
          // Only search from active themes
          if (empty($theme->status) && $theme->name != $current) {
            continue;
          }

    This is where the problem occurs. The $all_themes array provided by list_themes() either does not have a $theme->status property (in the case of D6), or the $theme->status property is set to 0 directly if its not populated (D7) for the themes that are returned. Since $theme->status is missing or set to 0, the IF statement evaluates FALSE and the themes' layouts aren't loaded.

  4. The problem with list_themes() is caused either when MAINTENANCE_MODE constant is defined or in the case of Drupal 6, when db_is_active() returns FALSE.
  5. All of this wouldn't really be a problem if it only happened while in maintenance mode, but the problem is that caches are then set with the cache key "ctools_plugin_files:panels:layouts", so a white page is shown on the site until the next cache clear.

This experience has left me with the conclusion, "you really shouldn't put layouts in your themes", since it seems to be more fragile than layouts provided by modules. It should be possible to solve this problem for themes, but we'll cause problems for sites that have multiple themes that provide layouts with the same name. Hopefully there aren't any themes actually doing that (they should be namespaced), but if we load all the layouts from all themes, it could cause conflicts between those themes.

Files: 
CommentFileSizeAuthor
#2 ctools_test._php.txt690 bytesquicksketch
#2 ctools_load_theme_plugins-1949400.patch793 bytesquicksketch
PASSED: [[SimpleTest]]: [MySQL] 68 pass(es).
[ View ]

Comments

The $all_themes array provided by list_themes() either does not have a $theme->status property (in the case of D6), or the $theme->status property is set to 0 directly if its not populated (D7) for the themes that are returned.

I'm incorrect, $theme->status is set to 0 in both D6 and D7 in list_themes() directly if it's not provided:

      // Status is normally retrieved from the database. Add zero values when
      // read from the installation directory to prevent notices.
      if (!isset($theme->status)) {
        $theme->status = 0;
      }

This means that CTools will skip all themes except the default theme when MAINTENANCE_MODE is defined.

Status:Active» Needs review
StatusFileSize
new793 bytes
PASSED: [[SimpleTest]]: [MySQL] 68 pass(es).
[ View ]
new690 bytes

Here is a patch and a demonstration PHP file that can replicate the problem. Because this problem only occurs in maintenance mode (e.g. when running update.php or a fatal error is encountered), reproducing the problem is a little tricky. I thought this sample file would help demonstrate it clearly. However in order to test it you'll need to provide a layout in a theme, as documented at http://drupal.org/node/495654

The test file basically emulates the conditions that cause the problem: Being in maintenance mode, caches cleared, and either the layout being in a non-default theme, or if the default theme cannot be determined (such as when the database isn't available). The test file needs to be placed in your site root, in the same place as index.php.

Result before patch (incorrect, layout not found):

array
  empty

Result after the patch (correct, layout found):

array
  'example_layout' =>
    array
      'title' => string 'Example layout' (length=19)
      'category' => string 'example_layout' (length=13)
      'icon' => string 'example_layout.png' (length=17)
      'theme' => string 'example_layout' (length=13)
      'css' => string 'example_layout.css' (length=17)
      'regions' =>
        array
          'top' => string 'Top' (length=3)
          'right' => string 'Right side' (length=10)
          'left' => string 'Left side' (length=9)
      'module' => string 'bartik' (length=5)
      'name' => string 'example_layout' (length=14)
      'path' => string 'sites/all/themes/bartik/layouts' (length=30)
      'file' => string 'example_layout.inc' (length=18)
      'plugin module' => string 'panels' (length=6)
      'plugin type' => string 'layouts' (length=7)
      'description' => string '' (length=0)

Issue summary:View changes

Clarification.

Wow. quicksketch, thank you so much -- this bug has been bothering us for years and we've never quite understood what was going on. Your research has finally nailed down what's happening.

Question: Does that accidentally cache disabled theme's plugins during maintenance mode? Could this have the opposite problem.

Let's say I have theme A and B, and both provide a plugin. A is the active theme, B is on the site but disabled. During maintenance mode, from your excellent research, it looks like we may not be able to tell.

If this is the case, then maybe a better solution would be: If in maintenance mode, refuse to write the cache? Then, the first hit after maintenance mode will force this to run in a proper environment?

If this is the case, then maybe a better solution would be: If in maintenance mode, refuse to write the cache? Then, the first hit after maintenance mode will force this to run in a proper environment?

I think you're correct. I was thinking more on this last night and for sites that have multiple themes that provide layouts (e.g. in a multisite scenario) you could end up with cache entries that are pointing to the wrong themes. I think amending this approach to prevent caching would be the best solution. We already have a flag for that in ctools_get_plugins(), but we may need to add a similar flag in ctools_plugin_load_includes().

If you were running a large number of updates, it seems possible that CTools could end up spending a lot of cycles on reading through the file system for all CTools plugins without a cache. Maybe instead of disabling the cache we could append :maintenance_mode to the cache key? That would just keep maintenance caches separate from normal ones when the DB is available.

Status:Needs review» Needs work

That is a reasonable and easy to implement approach, I believe.

This could be considered a duplicate of https://drupal.org/node/1032936 (Drupal core: Maintenance mode should neither create nor retrieve cache)

Issue summary:View changes

Reducing redundancy.