First steps have been done.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/f51e166d...

New file format uses 3 hashes instead of 1.

CommentFileSizeAuthor
#2 advagg_d7_sandbox-1927578-2-fix-spelling.patch10.18 KBmikeytown2

Comments

mikeytown2’s picture

mikeytown2 commented

http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/9a7bae90...

Steps before official dev release
- More work in the readme file.
- Change atime cache so it's an array.
- Profile code.
- Convert db_query to db_select in advagg_variable_get_db().
- Force change counter/button for CDNs.
- Test disable/uninstall.
- Spell check code.
- Better variable names.

Steps before alpha release
- Get sub modules working.
- Get drush working.
- Get httprl integration and fallback working.
- CDN testing.
- Get requirements tests working.
- Use new formapi JS settings.

mikeytown2’s picture

mikeytown2 commented
StatusFileSize
new advagg_d7_sandbox-1927578-2-fix-spelling.patch10.18 KB

Following patch has been committed. Fixes spelling errors.

mikeytown2’s picture

mikeytown2 commented

atime cache is now an array
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/0c11a153...

mikeytown2’s picture

mikeytown2 commented

all db_query() calls have been upgraded to db_select http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/c1cba1f8...

mikeytown2’s picture

mikeytown2 commented

Some variables have been renamed
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/84f04ddd...

mikeytown2’s picture

mikeytown2 commented

Admin page has had some tweaks done. Added in counter as well.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/f63df5c3...
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/f339394b...

mikeytown2’s picture

mikeytown2 commented

Now using new formapi JS settings (#attached)
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/9d0bc733...

mikeytown2’s picture

mikeytown2 commented

Whats left to do before work on submodules can begin.
- Drush (alternatives to admin button functions).
- Requirements tests (make sure thing are as we expect).
- Optional HTTPRL integration and fallback.
- Remove stale files on cron.
- Remove dead aggregates/files from database.

Can be done at the same time as submodule development.
- Profile code.
- CDN testing.

Get the bundler submodule working first as I went from 2 db tables to 3 thus the magic query will need some work... other modules should be super easy; if not refactor hooks so they become super easy. Look into adding a way to inline all css/js on landing pages as a new submodule.

mikeytown2’s picture

mikeytown2 commented

New table layout should work with the bundler. The unoptimized query is below

// Create join query for the advagg_files table.
$subquery_files = db_select('advagg_files', 'af')
  ->fields('af');
// Create join query for the advagg_aggregates table.
$subquery_aggregates = db_select('advagg_aggregates', 'aa')
  ->fields('aa');
// Create main query for the advagg_aggregates_versions table.
$query = db_select('advagg_aggregates_versions', 'aav');
$query->join($subquery_aggregates, 'aa', 'aa.aggregate_filenames_hash=aav.aggregate_filenames_hash');
$query->join($subquery_files, 'af', 'af.filename_hash=aa.filename_hash');
$query->addExpression("LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0')", 'counter');
$query = $query->fields('af', array('filename'))
  ->fields('aav', array('aggregate_filenames_hash'))
  ->condition('aav.root', 1)
  ->orderBy('counter', 'DESC')
  ->orderBy('aav.aggregate_filenames_hash', 'ASC')
  ->orderBy('aa.porder', 'ASC')
  ->groupBy('af.filename_hash');
$results = $query->execute();

$analysis = array();
foreach ($results as $row) {
  $analysis[$row->filename] = $row->counter . ' ' . $row->aggregate_filenames_hash;
}
echo $query->__toString() . "<br>\n";
echo '<pre>' . print_r($analysis, TRUE) . '</pre>';

Will be working on the optimized version.

mikeytown2’s picture

mikeytown2 commented

Optimized version below. Query is at least 10x faster with the same output

// Create join query for the advagg_aggregates_versions table.
$subquery_aggregates_versions = db_select('advagg_aggregates_versions', 'aav')
  ->fields('aav')
  ->condition('aav.root', 1);

// Create join query for the advagg_aggregates table.
$subquery_aggregates = db_select('advagg_aggregates', 'aa');
$subquery_aggregates->join($subquery_aggregates_versions, 'aav', 'aav.aggregate_filenames_hash=aa.aggregate_filenames_hash');
$subquery_aggregates->addExpression("LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0')", 'counter');
$subquery_aggregates = $subquery_aggregates->fields('aav', array('aggregate_filenames_hash'))
  ->fields('aa', array('filename_hash', 'porder'))
  ->groupBy('aa.filename_hash');

// Create main query for the advagg_files table.
$query = db_select('advagg_files', 'af');
$query->join($subquery_aggregates, 'aa', 'af.filename_hash=aa.filename_hash');
$query = $query->fields('af', array('filename'))
  ->fields('aa', array('counter', 'aggregate_filenames_hash'))
  ->orderBy('aa.counter', 'DESC')
  ->orderBy('aa.aggregate_filenames_hash', 'ASC')
  ->orderBy('aa.porder', 'ASC');
$results = $query->execute();

$analysis = array();
foreach ($results as $row) {
  $analysis[$row->filename] = $row->counter . ' ' . $row->aggregate_filenames_hash;
}
echo $query->__toString() . "<br>\n";
echo '<pre>' . print_r($analysis, TRUE) . '</pre>';
mikeytown2’s picture

mikeytown2 commented

Added in atime and linecount. http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/8ed6ceb5...

// Create join query for the advagg_aggregates_versions table.
// 1209600 = 2 weeks.
$subquery_aggregates_versions = db_select('advagg_aggregates_versions', 'aav')
  ->fields('aav')
  ->condition('aav.root', 1)
  ->condition('aav.atime', REQUEST_TIME - 1209600, '>');

// Create join query for the advagg_aggregates table.
$subquery_aggregates = db_select('advagg_aggregates', 'aa');
$subquery_aggregates->join($subquery_aggregates_versions, 'aav', 'aav.aggregate_filenames_hash=aa.aggregate_filenames_hash');
$subquery_aggregates->addExpression("LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0')", 'counter');
$subquery_aggregates = $subquery_aggregates->fields('aav', array('aggregate_filenames_hash'))
  ->fields('aa', array('filename_hash', 'porder'))
  ->groupBy('aa.filename_hash');

// Create main query for the advagg_files table.
$query = db_select('advagg_files', 'af');
$query->join($subquery_aggregates, 'aa', 'af.filename_hash=aa.filename_hash');
$query = $query->fields('af', array('filename', 'filesize', 'mtime', 'changes', 'linecount'))
  ->fields('aa', array('counter', 'aggregate_filenames_hash'))
  ->orderBy('aa.counter', 'DESC')
  ->orderBy('aa.aggregate_filenames_hash', 'ASC')
  ->orderBy('aa.porder', 'ASC');
$results = $query->execute();

$analysis = array();
foreach ($results as $row) {
  $analysis[$row->filename] = array(
    'group_hash' => $row->counter . ' ' . $row->aggregate_filenames_hash,
    'mtime' => $row->mtime,
    'filesize' => $row->filesize,
    'linecount' => $row->linecount,
    'changes' => $row->changes,
  );
}
echo $query->__toString() . "<br>\n";
echo '<pre>' . print_r($analysis, TRUE) . '</pre>';
mikeytown2’s picture

mikeytown2 commented

added new file for cache flush operations.
created 2 more functions to be used in cron.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/df106de1...

mikeytown2’s picture

mikeytown2 commented

Whats left to do before work on submodules can begin.
- Drush (alternatives to admin button functions).
- Requirements tests (make sure things are as we expect).
- Optional HTTPRL integration and fallback.

Can be done at the same time as submodule development.
- Profile code.
- CDN testing.

Note that the bundler logic will work with the new table layout.

mikeytown2’s picture

mikeytown2 commented

Drush commands added in
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/0a4af94e...

mikeytown2’s picture

mikeytown2 commented

Moved buttons to the operations tab on admin page
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/b6b5b9b1...

mikeytown2’s picture

mikeytown2 commented

Requirement checks are now in place
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/3fde15d8...

mikeytown2’s picture

mikeytown2 commented

Remove debug line
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/54dd4afb...
Fix coder warning and gzip test
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/05e895fb...

mikeytown2’s picture

mikeytown2 commented

httprl done. no fallback coded.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/34391a8e...

mikeytown2’s picture

mikeytown2 commented

fallback coded
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/b09cc6a7...

mikeytown2’s picture

mikeytown2 commented

Submodules to create:
- bundler
- css compression
- js compression
- css cdn
- js cdn
- js footer (move js from header to footer)
- landing page inliner (inline all CSS and JS for faster loading of that one page)

mikeytown2’s picture

mikeytown2 commented

js cdn
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/5b38556c...
css cdn
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/e9c48bf9...

mikeytown2’s picture

mikeytown2 commented

fix cache bug and work around core bug.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/e8944118...

Link to core bug
http://drupal.org/node/1937860
Documented core issue
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/1099d2fb...

mikeytown2’s picture

mikeytown2 commented

CSS/JS compression modules have been added
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/909bdcb9...

They will need httprl in order to do compression testing (not written). Also need to cache compressed js in cache. Will need to think about inline compression as well...

mikeytown2’s picture

mikeytown2 commented

Submodules to create:
- bundler (almost done on local box)
- js footer (move js from header to footer)
- landing page inliner/optimizer (inline all CSS and JS for faster loading of that one page AND/OR create a big bundle)

Will need to test:
http://drupal.org/project/jquery_update
http://drupal.org/project/css_emimage
http://drupal.org/project/cdn
http://drupal.org/project/labjs
http://drupal.org/project/headjs

mikeytown2’s picture

mikeytown2 commented

Bundler added.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/86b00a87...

mikeytown2’s picture

mikeytown2 commented

Will replace current 2.x branch with what I have here.

mikeytown2’s picture

mikeytown2 commented
Status: Active » Fixed

Merge is done. Closing this issue.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.