The hook_flush_caches only works if 'Flush page cache on cron' is enabled

I agree: drupal_flush_all_caches() should clear varnish every time.

However, this is hard to implement while respecting the "flush on cron" option: both system_cron() and drupal_flush_all_caches() invoke hook_flush_caches().

Since there's no obvious way to detect whether we're "inside" cron or not, I would say, do a backtrace in hook_flush_caches() and search for the function "system_cron", if found respect the "clear on cron" option, otherwise just clear.

This is not quite correct because 'cron_semaphore' doesn't indicate that we're inside a cron request, just that cron is currently running (or hanging, whichever the case may be).

Specifically, we only need to respect the cron option if hook_flush_caches() is being invoked by system_cron(). A call to drupal_flush_all_caches() in a different hook_cron() should be able to flush varnish, and the patch prevents that. So that's why I suggested a backtrace, since we have no other way of detecting the caller.

Example:

waiting_for_frank();
frank_calling();

function waiting_for_frank() {
  print frank_called() ? 'omg frank called me!' : 'i miss frank...';
  print '<br />';
}

function frank_calling() {
  waiting_for_frank();
}

function frank_called() {
  foreach (debug_backtrace(FALSE) as $level) {
    if ($level['function'] == 'frank_calling') {
      return TRUE;
    }
  }

  return FALSE;
}

Backtraces used in core:

DatabaseLog::findCaller()
_drupal_error_handler_real() / _drupal_get_last_caller()
DrupalTestCase::getAssertionCall()
DrupalTestCase::errorHandler()

Using debug_backtrace seems like an abuse of that function. If we need to detect if we are in a cron run or not, can't you test if request_uri() == '/cron.php' ?

Patch implements alternate detection of cron runs

Cron can be invoked from ?q=admin/reports/status/run-cron, poormanscron, drush cron, etc. so you can't just check the URI. #8 looks OK, but haven't tested it.

You can detect the first case by inspecting $_GET['q'],
I doubt anyone limited to using poormanscron would also have drupal+varnish integration
You can detect drush by inspecting $_SERVER['argv'][0] for drush.php, but i'm not sure how cross-platform friendly that is.

I agree that neither solution (8 or 10) seems ideal. I'm going to propose a wild alternative here- what if we simply tell people to use a module such as Elysia Cron to prevent system cron from running more than a certain frequency, even if cron.php is called very frequently?

here is a real world issue to consider.

I have a site that has a few hundred thousand nodes that are indexed with search_api_solr.

If there is a schema change to the solr server I would want to run cron about 2,000 times in over the next half day trying to reindex the nodes with the new schema.

Obviously elisa cron is the best approach for me at the moment, but it is a data point.

I think I found our culprit after all this time...

Now I think we had this all backwards... hook_flush_caches is supposed to return an array of caches tables, it was never made to be a trigger for flushing caches. We were not even using the right hook.

So here's a patch using hook_cron, that respects varnish_flush_cron, cache_lifetime and supports Pressflow's page_cache_max_age.