AdvAgg - D7 Port/Re-write

Once advagg hits a full 1.0 release; I will start to work on the 7.x-2.x series.

Hard to say; if I didn't have work + my other modules fighting for my time it would be done sooner.

List of modules that I'm directly responsible for that need more love:
http://drupal.org/project/views_javascript_random
http://drupal.org/project/views404
http://drupal.org/project/location_display
http://drupal.org/project/imageinfo_cache
http://drupal.org/project/imagefield_zip
http://drupal.org/project/dbtuner
http://drupal.org/project/expire
http://drupal.org/project/boost

http://drupal.org/project/css_gzip isn't in that list because this project will replace it.

In terms of my todo list:
#1060358: CDN and SEO in harmony with my files proxy module.
I need to fix some issues for boost 7.x
Then I'll come back to advagg 7.x most likely.

Going to mark this as postponed until I get some things done.

Subscribing to this issue.

As we quickly discussed on #1014086: Stampedes and cold cache performance issues with css/js aggregation, I'd like to merge some effort. I wouldn't like to intervene on heavy CSS/JS aggregation stuff, you do it well. I'd like to see some of this code (dynamic and on-demand file generation based on the Stream Wrapper core API) being generic, modeled depending on a real practical use case (AdvAgg module seems a good one). I could also make Core Library original module being lightweight and usable in duo with AdvAgg.

Open to suggestions, discussions, and ready to help. I could even try provide a PoC patch for your module's base (not all features) working over my code base ported to D7 if I find some time, that's not a promise though.

@pounard
Still working through creating an api for this issue #1174908: Option to output JS/CSS as a files array. Once advagg is 1.0 final I'll be more vocal about how we can work together.

Ok, I'll wait for it! I saw that you need a lot of work if you want to get this working into D7, you are using huge array data structures that doesn't always fit with hook_(css|js)_alter() D7 hooks, and for an easy porting you'll need this I think.

Subscribe

#want. I might get the ball rolling with a quick through Coder, see what happens :)

I ran the latest AdvAgg d6 code through the latest version of Coder and ended up with a large TODO list to work through. I'll see if I can get time to do it in the next day or two.

A straight conversion over can't be done easily. Certain advagg hooks are very similar to D7 core hooks; thus we can use core versions for these hooks.
drupal_get_css
hook_css_alter

drupal_get_js
hook_js_alter

Other things need to be considered as well. I plan on calling the 7.x version of advagg 2.x because it will be quite different from 1.x; but it will offer the same functionality & the sub modules shouldn't have to be changed that much.

Subscribe, sounds like an interesting module.

@mickeytown2 would you consider using my work in 7.x-2.x branch of Core Library module?

@pounard
Once I start to look at creating a D7 branch, I'll be looking at both projects; Core Library & Aggregate cache. Thats the "magic" of the GPL #1066672: Duplicate module with 2 others, merge would be a good thing #1066818: Unneccessary Branching? #1063316-1: hook_init() necessity. As I've stated above I need to work on the 7.x version of boost before taking on this issue.

Yes, thanks for answering. My answer is still I'm ready, not for merging necessarily, but at least to adapt as much as it could be useful. Module duplication is not always bad, it depends on the needs, mine's goal is still to remain simple, but its core (file generation) is quite generic and can be exported as a module on its own, and easily shared thus avoiding code duplication. It's up to you!

Subscribing...

subscribe

subs

subscribing

For comparison, there seems to be similar work happening for wordpress.
http://wordpress.org/extend/plugins/w3-total-cache/

Subscribble

subbing

sub. And willing to contribute patches :).

I'm very interested in the w3 total cache implementation.

subscribe

subscribe

subscribing

+1

subscribe

subscribe

+1

follow

#36, you don't need to post comments to follow topics anymore. You can simply hit the 'follow' button next to the topic start. See also this post:
http://drupal.org/node/1306444

Suppose project page should be updated to point Aggregate cache for D7

Subscribing.

@Wim Leers:

you don't need to post comments to follow topics anymore. You can simply hit the 'follow' button next to the topic start. See also this post:
http://drupal.org/node/1306444

@JamesOakley: LOL! I apologize! I knew about that, but it's been a habit for almost 5 years now. Thanks, and my apologies! :)

For things that are feature requests, I actually like seeing a "Subscribing" message; lets me know who's wanting it. And in this case AdvAgg is now tied to CDN so thats more more fire under my rear to get this going.

Once we get the D7 site for work live, AdvAgg is the first module to be ported. ETA for a beta is late February.

Great :) Expect a new, very solid release of the CDN module for both Drupal 6 and 7. Feature parity. Advanced Help support. Various of your patches. Etc. :) I'm actually already referring to the AdvAgg module in the Drupal 7 module's documentation in the same way as the Drupal 6 module. I was already assuming it'd get ported :)

The name of the followers :)

RE-SUBSCRIBING

Ah... well... this would be great for some big projects I have on my plans.

This is really good news, I can't wait to help with testing the D7 port. (re-subscribing)

subscribe

Subscribing with pleasure :-)

subscribe

You don't need to add comment just to subscribe only - instead use the big Follow button at the top right ;)

omega: http://drupal.org/node/1171546#comment-5500878

@Kiphaas7: Ah! I missed that comment, thanks.

So +1 from me :)

@MikeyTown2: Subscribing ... as requested: http://drupal.org/node/1171546#comment-5500878

This is off topic, but since it's been brought up I thought I should provide the correct issues:

cmseasy: #1304550: Display count of issue followers when viewing an issue

pounard: #1304558: Provide a page showing all the users following a given issue

subs

Also subscribing.

+1

Stop subscribing, start following -> http://drupal.org/node/1306444

This will be worked on at the Drupalcon code sprint on Tuesday and Friday. Come by to help out!
http://denver2012.drupal.org/sprints

The need to write the subscribe message on this particular issue is no longer needed as the number of followers is displayed just under the "Follow" button now. New feature on d.o implemented this week! So, the new subscribers need not post the requested message anymore.

@gynekolog @sprice please take the time to read this thread before commenting. Especially #42 is for your attention, folks!

@melon mikeytown2 now knows how many people want this module. Number of followers is displayed under "follow" button now. Read more carefully!

@gynekolog — Exactly.

@melon — My comment was in reply to #42 in regards to the new feature implemented on d.o this week that takes care of the number. If @mikeytown2 wants to see the names of the people subscribing, then new people should still subscribe. If it's just the numbers that matter, then look under the "follow" button, and, as of this writing, you can see that 96 people are following this thread (by far more than most other threads that I've seen!).

@gynekolog @sprice Sorry I didn't notice that the number of subscribers is now available.

Anyhow, I'm looking forward to the outcome of the sprint at Drupalcon!

Initial deep dive into the APIs.

Unfortunately drupal_get_css() isn't nice. drupal_pre_render_styles() looks like the thing I wish to modify, but I can't do it. I might be able to get in by changing system_element_info() via hook_element_info_alter(). This is what pounard's Core Library does in resource_element_info_alter(), switching drupal so it uses resource_aggregation_aggregate_callback() instead of drupal_aggregate_css(). Catch's argcache also uses hook_element_info_alter and it seems to be a simpler way to do it as it doesn't implement its own stream wrapper.

Moving forward I will try my best to use hook_element_info_alter() to accomplish what I need... but for JS I will still need hook_theme_registry_alter. The other nice thing is for some of the advagg sub modules is core's hook_js_alter() and hook_css_alter().

I think I have a good way to hijack css & js processing. I should have a start on this hopefully by the end of friday; I now know what needs to happen to make this fly in D7. No code from yesterdays sprint, just investigating how some D7 modules do what I want to do.

so exciting!

Some progress. Applied the coder module upgrade changes and created a branch to get this process going.
3cc1801afd490c6e9fa4a2a8cb36e204cd639a85

This is untested and just a working start. Do not use without intent to help debug.

One of the first issues to deal with will be to remove the public file system logic, found discoverable near FILE_DOWNLOADS_PUBLIC.

This patch has been committed. Fixes public downloads and advagg now uses the correct hooks for D7. For now I have copies of core's functionality. Code is still not production ready.

How about a 7.x-dev please?

@klonos
For now git is the way to test & fix the 7.x branch. http://drupal.org/project/advagg/git-instructions. Once it actually works I'll make a 7.x dev. Core had some big changes in how CSS & JS files work; this is a big project.

Fair enough. Just make sure you update this issue when a 7.x-dev is available please.

Subscribing

@dave_______1 stop subscribing and start following FFS. see top of every issue queue.

likewhoa, you should have read the topic.
http://drupal.org/node/1171546#comment-5500878

Kiphaas7 - http://drupal.org/node/1171546#comment-5747832

@Kiphaas7 ^

+1

subscribe

subscribe

+1

Subscribing, would like to see this for D7 or know what alternatives are out there to fix the broken aggregated CSS/JS issue.

Can we please get the 7.x branch on d.o rolling instead of commenting on subscribes? Also, even on the git repo, there hasn't been work done on the 7.x branch in months.

What can I do to help?

I will gladly spend time on this as it is greatly needed. There will never be a stable release until at least a dev is rolled out. Not to mention, creating a d.o dev branch will allow the needed version tag for the issue queue. This way we can start managing what needs to get done.

I'll open another issue to become a co-maintainer.

ATTENTION: The 7.x-1.x-dev snapshot release and been created.

Please file any bugs with the 7.x branch as a NEW issue so they can be properly tracked. DO NOT post bugs in this issue and please no more subscribes. If you would like to be notified, click the big green "Follow" button at the top. This issue will be marked as fixed once there is a 7.x-1.0-rc1 release.

NOTE: Currently the 7.x-1.x-dev snapshot is no where near ready for production use and is only for those who wish to test and debug the 7.x branch.

http://drupalcode.org/project/advagg.git/commit/fb93315

I went ahead and made AdvAgg at least installable and "loadable". Please keep in mind that there are a TON of API changes from D6 to D7 and it will take time for everything to be fixed.

I get errors and can't uninstall it.

This is the error message: Warning: rtrim() expects parameter 1 to be string, array given in file_prepare_directory() (line 437 of /home/www/hariskar/mikroviologos.gr/www/includes/file.inc).
If I disable it in the modules list and click save an empty page opens. If I refresh it is still enabled. So how can I uninstall it?
Thank you!

Do you have access to drush on that server? If so, you could disable the module and then flush all the caches from within drush.

drush dis advagg
drush pm-uninstall advagg
drush cc all

Thank you, but I don't have access to the server. Any other solution?

Here is another error: Warning: strpos() expects parameter 1 to be string, array given in file_uri_scheme() (line 200 of /home/www/hariskar/mikroviologos.gr/www/includes/file.inc).

Does this help you?

http://drupal.org/node/157632

Note the bit about clearing the bootstrap cache - actually, IIWY, I'd empty all the table that start with "cache".

Disclaimer: If I were you, I'd take a backup of the database before doing any of that!

@hariskar: I agree with James, BACKUP first! If you don't have access to the server, it sounds like it's a production site. I would STRONGLY recommend against installing the dev version as this probably WILL break your site.

@EVERYONE: For those who are not familiar with module development or install a fresh local D7 site to test this module, this is NOT for you.

I restored my site from yesterdays backup. Thank you!

The following patch fixes the display of the tab "JS Compression". Indeed, only the string "Array" is currently displayed.
See #1669952: [META] AdvAgg D7 - JS Compression not working.

Sorry for the precedent bug report and status updates. I will immediately create a new ticket.
See #1669952: [META] AdvAgg D7 - JS Compression not working.

Did you create the new ticket?

What is the status of the patch?

Some similar errors reported in the forums as here:
http://drupal.org/node/1445568

Errors I'm getting are:

    Warning: strpos() expects parameter 1 to be string, array given in file_uri_scheme() (line 200 of /DRUPAL7/includes/file.inc).
    Warning: mb_substr() expects parameter 1 to be string, array given in drupal_substr() (line 518 of /DRUPAL7/includes/unicode.inc).
    Warning: rawurlencode() expects parameter 1 to be string, array given in drupal_encode_path() (line 640 of /DRUPAL7/includes/common.inc).

Is it #1669952: [META] AdvAgg D7 - JS Compression not working?

Hi,

Is this 7.1-dev version useable for a live site yet?

on BOA Nginx it is not.

I don't think it works for anyone on any web server. Just note that the last 7.x commit says just "Made module at least installable and loadable." It is not expected to work, yet.

Also, I don't think it is a task, but feature requests, and not critical, since it is *not* expected to work at this stage.

PLEASE read #87 above and DON'T post issues in *this* thread.

just a note pointing to comment #87

http://drupalcode.org/project/advagg.git/commit/7c53d59

• Fixed hook_process_html() so it actually calls the CSS and JS alter functions now (although they are currently not doing anything because they need to be rebuilt for D7).
• Initial separation into multiple files for easier management.
• Initial rework of Admin UI (now located at: /admin/config/development/performance).

Also updated issue summary, please read and add/change/remove anything that still needs to be done. Please feel free to create meta issues as well.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Per #13:

Other things need to be considered as well. I plan on calling the 7.x version of advagg 2.x because it will be quite different from 1.x; but it will offer the same functionality & the sub modules shouldn't have to be changed that much.

I created the 7.x-2.x branch and will be working on that from now on. 7.x-1.x will never have a stable release or a direct port from 6.x, as Mikey said it's quite different [code wise].

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

7.x-2.x dev has been pushed from my test sandbox #1917802: Drupal 7 Sandbox. As of right now the dev link is from January; wait for the one from March.

Notes:
- Upgrade path will not be provided but patches welcome (nuke all old data is the correct upgrade path). Uninstall of old module (1.x & 2.x) before using the latest version is highly recommended.
- Remove old advagg directory before installing the latest 7.x-2.x code.
- Latest code works 100% on my local test box, but there will be bugs. You have been warned.

Big Changes:
- CSS/JS CDN module uses core's hooks and as a result does not require advagg.
- CSS/JS CDN module is disabled if jquery_update is installed as it has the same functionality.
- 3 DB tables instead of 2.
- 3 hashes instead of 1 for filenames (readme.txt explains this change).
- A lot of advagg hooks can be placed in *.advagg.inc files instead of the *.module.
- Can use the httprl module but it is not required.
- Bundler logic has changed slightly; it should pick slightly better merges now.

Things to do:
- Add back in Admin Menu integration #1943218: Admin Menu Integration

CSS CDN module:
- Discovered a core bug #1937860: seven_css_alter does not set the type when developing it. Getting this fixed would be nice.

CSS/JS Compression Submodule:
- Will need httprl in order to do compression testing (not written).
- Cache compressed js. Provide a way to flush that cache.
- Inline CSS/JS compression.

Submodules to create:
- js footer (move js from header to footer via admin page) #1743910: Move JS to footer
- landing page inliner/optimizer (inline all CSS and JS AND/OR create a big bundle for faster loading of selected pages) #1311574: Disable bundler and swap to inline styles (for single use type pages)
- Unified Aggregation Directory #1082928: Unified Aggregation Directory

Will need to test:
http://drupal.org/project/jquery_update
http://drupal.org/project/css_emimage
http://drupal.org/project/cdn
http://drupal.org/project/labjs
http://drupal.org/project/headjs

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Updated issue summary.

Thanx for the update Mike!

Are these "CSS/JS CDN modules" really necessary? It looks (I looked at their code) like they don't really belong to be part of the advagg module. But then again, they do leverage advagg-provided hooks, which I guess is why you need them to live in the advagg module?

Let's make sure that advagg+cdn work well together again :)
The D7 CDN module comes with quite a bit of test coverage.
I think that in the D6 advagg era, one had to use advagg to have CDN'd background images in aggregated CSS files etc? That's no longer the case in D7: the D7 CDN module overrides Drupal core's CSS aggregation, just to wrap each external file in a file_create_url() call (so that the CDN module's hook_file_url_alter() implementation gets called). I plan on getting that in a D7 point release, because it would move quite a bit of code out of the CDN module.

Also please make sure that you check the CDN module's new features, so that you don't provide the same functionality again :)

Advagg cdn sub modules load jQuery & jQuery UI from the Google cdn. They do nothing if jQuery update module is installed. Will be testing with cdn module soon.

The 'broken CSS after cache flush when CSS aggregation is enabled' issue that was present in Drupal 6 has been fixed in Drupal 7 core, correct? Just want to double check since we were interested in using Advagg solely for this reason.

I see there is now a drupal_delete_file_if_stale() callback used in drupal_clear_css_cache() that leaves aggregated CSS around for 30 days in case Drupal page cache or external caches are still serving pages with references to old aggregated CSS. This was a real pain in D6.

@vinmassaro
I'm pretty sure D7 core has that fixed to a better extent, but it is not 100%. drupal_delete_file_if_stale() will delete the file if it hasn't been modified in the last 30 days. The issue is, what if that aggregate has not been modified in the last 30 days but is still being used. drupal_build_css_cache() does a simple file_exists check which does not update the mtime. It is still possible to get a 404 with css files using D7 core, it's just a lot harder to now.

AdvAgg makes 404s even harder to encounter because it uses an imagecache naming convention, meaning that based off the name it knows how to create that file. The other thing is AdvAgg uses a cache & database to keeps track of atime ( updated once every 12 hours ), and if the file hasn't been accessed in the last 30 days it will remove that file from the file system (vs core using the last modified time). Even with that file deleted, it can still be generated though thanks to the imagecache naming convention. As it stands the only way to get a 404 using AdvAgg is if an aggregate has not been used in the last 6 weeks as that is how long it takes to clean up old entries in the database (advagg_cron() calls advagg_remove_old_unused_aggregates()); so you might get a 404 for css/js when accessing the page via google cache or archive.org (totally acceptable in my book).

@Wim Leers
#1942230: CDN module AdvAgg 7.x-2.x Integration

The first alpha version of AdvAgg 7.x has been released.

Summary:
- Readme.txt is not updated.
- Everything is working except for CDN's farfuture option.
- If using jQuery Update the advagg_css_cdn and advagg_js_cdn sub modules should be disabled.
- Uses httprl for aggregate building if module is installed.
- New sub module called advagg_mod; allows for tweaks to the css/js like put all JS in the footer.
- Need css_emimage 7.x-1.3 or higher; Older versions of module will not work.

Hi,

I have installed and am running this on two sites (based from a single multisite install) on Drupal 7.22.

All I can say it, it works! It works well, I haven't noticed any problems or issues since installing it. I'm using it in conjunction with CSS Embed images as well, it's functioning correctly too.

I'm also using the jsmin.so module compiled into PHP.

So I just wanted to provide some feedback that it's working well, and to say thank you!

Updated issue summary.

CSS aggregation not happening through advagg when CDN module is enabled. aggregated CSS being served through sites/default/files/public//cdn/css/http/[aggregated_name]css

@vinaykprao1987
The patch #1942230-6: CDN module AdvAgg 7.x-2.x Integration for the CDN module states that farfuture is not working. If you go to the status report admin/reports/status it should also state that farfuture doesn't work and recommends that you disable it. Also please open up new issues or comment on the existing relevant issue instead of posting in here :)

Sorry about that did not check the Status Report. Had almost given up on the module. The patch worked. Thanks for all the help.

PS: I was not trying with far future option of CDN, without the patch CSS aggregated URL's was that of CDN rather than advagg.

Anybody notice a performance hit from advagg_bundler_analysis? I even disabled the bundler to see if I could get rid of the performance hit but in "most" cases, it still runs and takes up almost a half second worth of queries... :-/

Here's my Devel output:

406.56
1
advagg_bundler_analysis
P A E
SELECT af.filename AS filename, af.filesize AS filesize, af.mtime AS mtime, af.changes AS changes, af.linecount AS linecount, aa.counter AS counter, aa.aggregate_filenames_hash AS aggregate_filenames_hash FROM advagg_files af INNER JOIN (SELECT aav.aggregate_filenames_hash AS aggregate_filenames_hash, aa.filename_hash AS filename_hash, aa.porder AS porder, LPAD(CAST(COUNT(aav.aggregate_filenames_hash) AS char(8)), 8, '0') AS counter FROM advagg_aggregates aa INNER JOIN (SELECT aav.* FROM advagg_aggregates_versions aav WHERE (aav.root = :db_condition_placeholder_0) AND (aav.atime > :db_condition_placeholder_1) ) aav ON aav.aggregate_filenames_hash=aa.aggregate_filenames_hash GROUP BY aa.filename_hash) aa ON af.filename_hash=aa.filename_hash ORDER BY aa.counter DESC, aa.aggregate_filenames_hash ASC, aa.porder ASC

I don't know enough about how the module works to know what's going on, just doing some initial investigation into the matter. If it's indeed a bug, I'll get a new issue going.

Thoughts?

@philsward
Go ahead and open up a new issue. I can't repo what you're describing but something similar was an issue with the 6.x module #1283522: Slow query / number of bundles in advagg_bundles. Noted that the 7.x code has had some optimizations done to it #1927578-9: Get end to end prototype working. I've opened up a new issue #2006468: Slow Query in advagg_bundler_analysis. Run query in the background (via httprl) if at all possible.

Thanks @mikeytown2

I'll follow the other issue and help out with testing where I can :-)

rc1 of AdvAgg is now out. Marking this issue as fixed.

Thanx Mike! ;)