Posted by mikeytown2 on February 26, 2013 at 3:36am
3 followers
| Project: | AdvAgg D7 Sandbox |
| Component: | Code |
| Category: | task |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | closed (fixed) |
Issue Summary
First steps have been done.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/f51e166d...
New file format uses 3 hashes instead of 1.
Comments
#1
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/9a7bae90...
Steps before official dev release
- More work in the readme file.
- Change atime cache so it's an array.
- Profile code.
- Convert db_query to db_select in advagg_variable_get_db().
- Force change counter/button for CDNs.
- Test disable/uninstall.
- Spell check code.
- Better variable names.
Steps before alpha release
- Get sub modules working.
- Get drush working.
- Get httprl integration and fallback working.
- CDN testing.
- Get requirements tests working.
- Use new formapi JS settings.
#2
Following patch has been committed. Fixes spelling errors.
#3
atime cache is now an array
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/0c11a153...
#4
all db_query() calls have been upgraded to db_select http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/c1cba1f8...
#5
Some variables have been renamed
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/84f04ddd...
#6
Admin page has had some tweaks done. Added in counter as well.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/f63df5c3...
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/f339394b...
#7
Now using new formapi JS settings (#attached)
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/9d0bc733...
#8
Whats left to do before work on submodules can begin.
- Drush (alternatives to admin button functions).
- Requirements tests (make sure thing are as we expect).
- Optional HTTPRL integration and fallback.
- Remove stale files on cron.
- Remove dead aggregates/files from database.
Can be done at the same time as submodule development.
- Profile code.
- CDN testing.
Get the bundler submodule working first as I went from 2 db tables to 3 thus the magic query will need some work... other modules should be super easy; if not refactor hooks so they become super easy. Look into adding a way to inline all css/js on landing pages as a new submodule.
#9
New table layout should work with the bundler. The unoptimized query is below
<?php
// Create join query for the advagg_files table.
$subquery_files = db_select('advagg_files', 'af')
->fields('af');
// Create join query for the advagg_aggregates table.
$subquery_aggregates = db_select('advagg_aggregates', 'aa')
->fields('aa');
// Create main query for the advagg_aggregates_versions table.
$query = db_select('advagg_aggregates_versions', 'aav');
$query->join($subquery_aggregates, 'aa', 'aa.aggregate_filenames_hash=aav.aggregate_filenames_hash');
$query->join($subquery_files, 'af', 'af.filename_hash=aa.filename_hash');
$query->addExpression("LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0')", 'counter');
$query = $query->fields('af', array('filename'))
->fields('aav', array('aggregate_filenames_hash'))
->condition('aav.root', 1)
->orderBy('counter', 'DESC')
->orderBy('aav.aggregate_filenames_hash', 'ASC')
->orderBy('aa.porder', 'ASC')
->groupBy('af.filename_hash');
$results = $query->execute();
$analysis = array();
foreach ($results as $row) {
$analysis[$row->filename] = $row->counter . ' ' . $row->aggregate_filenames_hash;
}
echo $query->__toString() . "<br>\n";
echo '<pre>' . print_r($analysis, TRUE) . '</pre>';
?>
Will be working on the optimized version.
#10
Optimized version below. Query is at least 10x faster with the same output
<?php
// Create join query for the advagg_aggregates_versions table.
$subquery_aggregates_versions = db_select('advagg_aggregates_versions', 'aav')
->fields('aav')
->condition('aav.root', 1);
// Create join query for the advagg_aggregates table.
$subquery_aggregates = db_select('advagg_aggregates', 'aa');
$subquery_aggregates->join($subquery_aggregates_versions, 'aav', 'aav.aggregate_filenames_hash=aa.aggregate_filenames_hash');
$subquery_aggregates->addExpression("LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0')", 'counter');
$subquery_aggregates = $subquery_aggregates->fields('aav', array('aggregate_filenames_hash'))
->fields('aa', array('filename_hash', 'porder'))
->groupBy('aa.filename_hash');
// Create main query for the advagg_files table.
$query = db_select('advagg_files', 'af');
$query->join($subquery_aggregates, 'aa', 'af.filename_hash=aa.filename_hash');
$query = $query->fields('af', array('filename'))
->fields('aa', array('counter', 'aggregate_filenames_hash'))
->orderBy('aa.counter', 'DESC')
->orderBy('aa.aggregate_filenames_hash', 'ASC')
->orderBy('aa.porder', 'ASC');
$results = $query->execute();
$analysis = array();
foreach ($results as $row) {
$analysis[$row->filename] = $row->counter . ' ' . $row->aggregate_filenames_hash;
}
echo $query->__toString() . "<br>\n";
echo '<pre>' . print_r($analysis, TRUE) . '</pre>';
?>
#11
Added in atime and linecount. http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/8ed6ceb5...
<?php
// Create join query for the advagg_aggregates_versions table.
// 1209600 = 2 weeks.
$subquery_aggregates_versions = db_select('advagg_aggregates_versions', 'aav')
->fields('aav')
->condition('aav.root', 1)
->condition('aav.atime', REQUEST_TIME - 1209600, '>');
// Create join query for the advagg_aggregates table.
$subquery_aggregates = db_select('advagg_aggregates', 'aa');
$subquery_aggregates->join($subquery_aggregates_versions, 'aav', 'aav.aggregate_filenames_hash=aa.aggregate_filenames_hash');
$subquery_aggregates->addExpression("LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0')", 'counter');
$subquery_aggregates = $subquery_aggregates->fields('aav', array('aggregate_filenames_hash'))
->fields('aa', array('filename_hash', 'porder'))
->groupBy('aa.filename_hash');
// Create main query for the advagg_files table.
$query = db_select('advagg_files', 'af');
$query->join($subquery_aggregates, 'aa', 'af.filename_hash=aa.filename_hash');
$query = $query->fields('af', array('filename', 'filesize', 'mtime', 'changes', 'linecount'))
->fields('aa', array('counter', 'aggregate_filenames_hash'))
->orderBy('aa.counter', 'DESC')
->orderBy('aa.aggregate_filenames_hash', 'ASC')
->orderBy('aa.porder', 'ASC');
$results = $query->execute();
$analysis = array();
foreach ($results as $row) {
$analysis[$row->filename] = array(
'group_hash' => $row->counter . ' ' . $row->aggregate_filenames_hash,
'mtime' => $row->mtime,
'filesize' => $row->filesize,
'linecount' => $row->linecount,
'changes' => $row->changes,
);
}
echo $query->__toString() . "<br>\n";
echo '<pre>' . print_r($analysis, TRUE) . '</pre>';
?>
#12
added new file for cache flush operations.
created 2 more functions to be used in cron.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/df106de1...
#13
Whats left to do before work on submodules can begin.
- Drush (alternatives to admin button functions).
- Requirements tests (make sure things are as we expect).
- Optional HTTPRL integration and fallback.
Can be done at the same time as submodule development.
- Profile code.
- CDN testing.
Note that the bundler logic will work with the new table layout.
#14
Drush commands added in
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/0a4af94e...
#15
Moved buttons to the operations tab on admin page
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/b6b5b9b1...
#16
Requirement checks are now in place
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/3fde15d8...
#17
Remove debug line
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/54dd4afb...
Fix coder warning and gzip test
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/05e895fb...
#18
httprl done. no fallback coded.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/34391a8e...
#19
fallback coded
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/b09cc6a7...
#20
Submodules to create:
- bundler
- css compression
- js compression
- css cdn
- js cdn
- js footer (move js from header to footer)
- landing page inliner (inline all CSS and JS for faster loading of that one page)
#21
js cdn
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/5b38556c...
css cdn
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/e9c48bf9...
#22
fix cache bug and work around core bug.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/e8944118...
Link to core bug
http://drupal.org/node/1937860
Documented core issue
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/1099d2fb...
#23
CSS/JS compression modules have been added
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/909bdcb9...
They will need httprl in order to do compression testing (not written). Also need to cache compressed js in cache. Will need to think about inline compression as well...
#24
Submodules to create:
- bundler (almost done on local box)
- js footer (move js from header to footer)
- landing page inliner/optimizer (inline all CSS and JS for faster loading of that one page AND/OR create a big bundle)
Will need to test:
http://drupal.org/project/jquery_update
http://drupal.org/project/css_emimage
http://drupal.org/project/cdn
http://drupal.org/project/labjs
http://drupal.org/project/headjs
#25
Bundler added.
http://drupalcode.org/sandbox/mikeytown2/1917800.git/commitdiff/86b00a87...
#26
Will replace current 2.x branch with what I have here.
#27
Merge is done. Closing this issue.
#28
Automatically closed -- issue fixed for 2 weeks with no activity.