Serving unstyled pages while bundles are initially generating

I too have observed this 1st 'bundle creation lag'.
So the idea of

serve unprocessed CSS files so visitor at least gets styled page

sounds reasonable. However, that could raise the issue of needing to destroy these temp. (unprocessed) cached CSS files. Surely advagg could 'sync wait' for the CSS asset bundle completion?

Perhaps the 'answer' could be to have a distinct daemon process (and probably not a PHP concept)?
I am reminded of Win Leers CDN project - with 'File Conveyor mode' => python daemon.
see -> http://fileconveyor.org/

Here, the daemon is designed to discover new, changed and deleted files via the operating system's built-in file system monitor (via Linux 'pyinotify'). OS 'inotify' is used to detect changes on the file system (assets) as opposed to polling.

The 'known' drawback with this project (fileconveyor) is current poor handling of CSS image references.
We are hoping someone will write a 'improved python CSS engine' for this project.

wimleers commented

February 01, 2011

Hi Peter,

I know that File Conveyor still needs work. And the CSS URL updater is the most problematic one, performance-wise, but now apparently also feature-wise.

That being said, it needs more contributors than just me. If multiple people are contributing to it, it can be sustainable. If I'm the only one working on it, it can't be. That's why not much has been happening with it. Nobody in the Drupal world seems to be paying attention to it. Which is very unfortunate. I think I'm going to start to try and rally people around this :

@crea
Adding ?advagg-debug=1 to the page that is causing issues and then going to watchdog and attaching the contents as a file should help to speed up bug reports.
I need this info from you.

If async is disabled, page generation will wait for the locks to be released before continuing. So if subsequent requests get unstyled pages then something is not working with locking... are you using Drupal 6.16+? The logic for this is in advagg_css_js_file_builder. You can try the attached patch, which will increase the lock time from 30 seconds to 90 seconds.

If async is enabled then the aggregates needed for that page will be pinged, and the html will continue to be generated. This means that async mode is multi-processed. Anyway while this is going on it also uses locking to prevent multiple processes from generating the same file. If your getting the error from #3 then try searching for that MD5 value in the database like so

SELECT *
FROM advagg_bundles
WHERE bundle_md5 = '74e9b8d252b17e1f477bdeb85e93c845'

If its not in the database then I need to work my advagg_insert_bundle_db function; but this also uses locking so we need some sort of way to verify that locking is working... sounds like yet another thing to add into the status report page to test for.

In terms of serving temp files I opted against this for JS #1160890: Create uncompressed aggregate and then overwrite and save the compressed version. and CSS compress is SLOW and doesn't gain much. I include CSS compression because people will always ask for it but IMHO it's not worth it. I should change the module description page to let people know that it's not the best solution; there is no good solution to CSS file size.

This could be related.
#1194226-13: Warning: stream_select() [<a href='function.stream-select'>function.stream-select</a>]: No stream arrays were passed in advagg

The patch from #6 makes this timeout after 90 seconds. If you haven't done so already, disable CSS/JS compression and see if you can repo the bug. See if CSS or JS compression might be the issue.

I do see the main issue for this after reading #8 multiple times

      // Only generate once.
      if (function_exists('lock_acquire')) {
        $lock_name = 'advagg_' . $filename;
        if (!lock_acquire($lock_name)) {
          $locks[] = $lock_name;
          $output[$filepath] = array('prefix' => $prefix, 'suffix' => $suffix, 'files' => array_map('advagg_return_true', array_flip($files)));
          continue;
        }
      }
...
      // Write file. default function called: advagg_file_saver
      $function = variable_get('advagg_file_save_function', ADVAGG_FILE_SAVE_FUNCTION);
      $good = $function($data, $filepath, $force, $type);

      // Release lock.
      if (function_exists('lock_release')) {
        lock_release($lock_name);
      }

      // If file save was not good then downgrade to non aggregated mode.
      if (!$good) {
        $output[$filepath] = FALSE;
        $cacheable = FALSE;
        continue;
      }
...
  // Wait for all locks before returning.
  if (!empty($locks) && function_exists('lock_wait')) {
    foreach ($locks as $lock_name) {
      lock_wait($lock_name);
    }
  }

Testing that the file is good in // Wait for all locks before returning like I do in advagg_file_saver needs to occur; and if it doesn't work out then I need to be able to set $output[$filepath] = FALSE from // Only generate once.

Mind testing this out?

memcache has a progressive lock_wait implementation. What I could do is port this over to advagg_lock_wait and call the new version of it if variable_get('lock_inc', './includes/lock.inc'); equals the default. This should speedup the 1 second delay to something smaller.
http://api.drupal.org/api/drupal/includes--bootstrap.inc/function/_drupa...
http://api.drupal.org/api/drupal/includes--lock.inc/function/lock_wait/6

Hitting the 30 second mark should never happen. If it does then something is wrong most likely. Sending back un-aggregated output when a lock happens could work. Have 4 levels.
- Only serve aggregated files if they are already built [page cache enabled] (this is only available if async is enabled)
- Only serve aggregated files if they are already built [page cache disabled] (this is only available if async is enabled)
- Do not wait for locks
- Wait for locks (current method)

Refer #14

(Thoughts:)

1) Could we not make use of PHP's PECL inotify() -> http://pecl.php.net/package/inotify
and have advagg process aggregate build's on 'change'?

2) Have a PHP CLI advagg aggregate 'build command' to add to cron?

We could use inotify, but IMHO it really isn't that important. Usually one pushes changes out to production and then you flush the caches. With inotify as each core css/js file is updated it will generate a new bundle that will only be used for a very short period of time (until the next inotify runs). The reason inotify is useful for file conveyor is it's watching the directory looking for created/updated files that need to be pushed out or compressed. AdvAgg current way of looking at the mtime or md5 works great from my point of view.

I've committed #12.

@crea
In terms of the loop errors; are the advagg_(js/css) directories writable from php? Does this query return anything?

SELECT *
FROM advagg_bundles
WHERE bundle_md5 = '74e9b8d252b17e1f477bdeb85e93c845'

This has a bunch of changes that are fairly big. Works on my end. Please let me know how this works on your side.

Note: There is now a $conf['advagg_bundle_built_mode'] setting. This controls if one should use file_exists or cache_get to see if a file exists. cache_get is helpful if the files dir is mounted on something like NFS. If files are shared across boxes then it can be harmful to performance. Default is to use file_exists ($conf['advagg_bundle_built_mode'] = FALSE).

multibyte issue is with core
#837322: htmlspecialchars - Invalid multibyte sequence in argument
I'll see if I can work around it when outputting the debug info.

This patch has been committed. This should take care of the issue of slow bundles getting built. I think I've made the defaults fairly sane, if not please let me know.

In my tests, the slowest section of bundling' is waiting for JSMin+ (C version PHP PECL) to compress .JS.
Typically, for me this is about 8-9 seconds. CSS Compress build (in parallel with JS) is about 3-4 seconds.
http://drupal.org/files/issues/advagg-building.jpg

There is lower fruit to pick I think at the moment. I'm open to a patch for this; just make the wait time variable with the understanding that on windows with php4 it will be at least 1 second. Patch to get you close #802856-30: Make lock_wait() wait less

Committed this patch. Fixes a rare bug with IE.