Slow saving of views and site lockups

I am seeing another issue related to the block of code you pasted ..

/**
 * Build and set the views data cache if empty.
 */
function _views_fetch_data_build() {
  views_include_handlers();
  $cache = module_invoke_all('views_data');
  foreach (module_implements('views_data_alter') as $module) {
    $function = $module . '_views_data_alter';
    $function($cache);
  }
  _views_data_process_entity_types($cache);

  // Keep a record with all data.
  views_cache_set('views_data', $cache, TRUE);
  // Save data in seperate cache entries.
  foreach ($cache as $key => $data) {
    $cid = 'views_data:' . $key;
    views_cache_set($cid, $data, TRUE);
  }
  return $cache;
}

I have a very field-heavy system (~2000 fields) and this is giving me a views_data cache blob that is 13MB in size. The per-key loop above was added to 'split' the large views_data cache into multiple caches, creating a set of smaller caches that total 13MB in size. Storing the large blob means we now have extraneous traffic on rebuilds. My SELECT SUM(LENGTH(data)) FROM cache_views is ~26MB.

Instead of storing the data twice, why not leave the large views_data:en blob out completely and if needed, rebuild it from: SELECT * FROM cache_views WHERE cid LIKE 'views_data:%:en'?

Testing a solution that just stores keys in views_data cid and then retrieves them from the individual records. A DB SELECT couldn't work because of the multiple cache engine need.

Two functions that change in views/includes/cache.inc:

/**
 * Build and set the views data cache if empty.
 */
function _views_fetch_data_build() {
  views_include_handlers();
  $cache = module_invoke_all('views_data');
  foreach (module_implements('views_data_alter') as $module) {
    $function = $module . '_views_data_alter';
    $function($cache);
  }
  _views_data_process_entity_types($cache);
        
  // Keep a record with all views_data cache keys
  views_cache_set('views_data', array_keys($cache), TRUE);
  // Save data in seperate cache entries.
  foreach ($cache as $key => $data) {
    $cid = 'views_data:' . $key;
    views_cache_set($cid, $data, TRUE);
  }
  return $cache;
}

function views_cache_get($cid, $use_language = FALSE) {
  global $language;

  if (variable_get('views_skip_cache', FALSE)) {
    return FALSE;
  }
  if ($use_language) {
    $cid .= ':' . $language->language;
  }
 
  if ($cid == 'views_data:' . $language->language) {
    // views_data cache is now just a reference to views_data cache entries, so let's combine them all into one giant array for legacy cache access
    $cache = cache_get($cid, 'cache_views');
    $views_cache_keys =	$cache->data;
    $cache->data = array();
    foreach($views_cache_keys as $views_cache_key) {
      $single_cache = cache_get('views_data:' . $views_cache_key . ($use_language ? (':' . $language->language) : ''), 'cache_views');
      if (isset($single_cache->data)) $cache->data[$views_cache_key] = $single_cache->data;
      unset($single_cache);
    }
    unset($views_cache_key);
    return $cache;
  } else {
    return cache_get($cid, 'cache_views');
  }
}

Wasn't worthwhile. Maybe if we had a cache_get_multiple function .. I am going to look into stripping the help text from fields instead?

First things first .. I had a view using a non-existent field that was causing a cache_views views_data rebuild on that page load.

I was able to debug that with a drupal_set_message here:

function _views_fetch_data($table = NULL, $move = TRUE, $reset = FALSE) {
  $cache = &drupal_static(__FUNCTION__ . '_cache');
  $recursion_protection = &drupal_static(__FUNCTION__ . '_recursion_protected');
  $fully_loaded = &drupal_static(__FUNCTION__ . '_fully_loaded');
  if ($reset) {
    $cache = NULL;
    $fully_loaded = FALSE;
  }
  if ($table) {
    if (!isset($cache[$table])) {
      $cid = 'views_data:' . $table;
      $data = views_cache_get($cid, TRUE);
      if (!empty($data->data)) {
        $cache[$table] = $data->data;
      }
      else {
	// No cache entry, rebuild.
       	drupal_set_message('No entry for ' . $cid);
        $cache = _views_fetch_data_build();
        $fully_loaded = TRUE;
      }
    }

Great tip. A reduction of 1700 entries, with peak memory usage from 170MB to 132MB and page load from 1.7s to 990ms.

If views_ui is disabled, we could cross-reference against admin/reports/fields/views-fields to remove all unused fields?

We probably need two separate cache pools for front-end vs back-end? Front-end with the bare minimum to render views? Back-end with everything as cache_views_ui?

Tested briefly with cache_get_multiple. It seems to be ~0.3s for cache_get_multiple vs 0.18s for the single blob in my case with ~2200 cache_views/views_data:% records, which is probably worthwhile if it avoids max_packet_size or memcache insert size issues.

For reference, the updated functions that I am testing with .. needs some tweaking (will warn on flush/initial rebuild), but was enough to benchmark.

function _views_fetch_data_build() {
  views_include_handlers();
  $cache = module_invoke_all('views_data');
  foreach (module_implements('views_data_alter') as $module) {
    $function = $module . '_views_data_alter';
    $function($cache);
  }
  _views_data_process_entity_types($cache);

  // Keep a record with all data keys
  views_cache_set('views_data', array_keys($cache), TRUE);
  // views_cache_set('views_data', $cache, TRUE);
  // Save data in seperate cache entries.
  foreach ($cache as $key => $data) {
    $cid = 'views_data:' . $key;
    views_cache_set($cid, $data, TRUE);
  }
  return $cache;
}

function views_cache_get($cid, $use_language = FALSE) {
  global $language;

  if (variable_get('views_skip_cache', FALSE)) {
    return FALSE;
  }
  if ($use_language) {
    $cid .= ':' . $language->language;
  }

  if (1 && $cid == 'views_data:' . $language->language) {
    // views_data cache is now just a reference to views_data cache entries, so let's combine them all into one giant array for legacy cache access
    $cache = cache_get($cid, 'cache_views');
    $time_start = microtime(true);
    $views_data_keys = array();
    foreach($cache->data as $views_cache_key) {
      $views_data_keys[] = 'views_data:' . $views_cache_key . ($use_language ? (':' . $language->language) : '');
    }
    $full_cache = cache_get_multiple($views_data_keys, 'cache_views');
    foreach($full_cache as $key => $value) {
      $cache->data[$key] = $value->data;
    }
    $time_end = microtime(true);
    drupal_set_message(sizeof($cache->data) . ' entries retrieved in ' . ($time_end - $time_start) . ' seconds');
    return $cache;
  } else {
    return cache_get($cid, 'cache_views');
  }

}

agree, also works here -- this is a patch that I am constantly re-applying.