After discovering a performance decrease in mysql I checked all databases and tables, found that cache_swftools had grown to 4.1 gb. The swftools cache has been cleared several times allthough not recently. Site is drushed at least once a week, all caches are cleared upon finish.

Drupal version is 6.19 and is in line to be upgraded to 6.20

Comments

cato’s picture

cato CreditAttribution: cato commented

Nothing?

Oh well, I might add that I ran truncate cache_swftools before the weekend. Today the table had grown to 5+Gb. Switched off the swftools cache but would really appreciate some input here.

Regards,
Christopher Cato

Stuart Greenfield’s picture

Stuart Greenfield CreditAttribution: Stuart Greenfield commented
Status: Active » Postponed (maintainer needs more info)

The swftools cache is a new feature. The cache mechanism should be hooking in to the standard Drupal cache mechanism.

This means it will flush whenever a site wide flush is done, for example, by triggering it from Administer > Site configuration > Performance and clicking Clear cached data.

It should also flush when things like new modules are enabled.

If the cache has expanded to this size then I wonder if there is something unusual about the way your site is working? A new cache entry is made each time a specific piece of content is requested.

As a stop gap the SWF Tools cache can be disabled by turning it off at Administer > Site configuration > Performance and setting SWF Tools cache to disabled.

However, I'd like to understand what type of site you are running to know why the cache has grown this large. Are any other cache tables growing to this sort of size??

cato’s picture

cato CreditAttribution: cato commented

The site I'm running is a news site with a lot of videos and slideshows, both done with swftools. The site serves does something like 200K - 300K pageviews per week.

The funny thing is, the cache_swftools table is still growing even though the swftools cache is turned off. The main cache has been flushed a number of times but it doesn't seem to affect cache_swftools.

Today it was up to 5.9Gb again, slowing down mysql immensely, load avg was up to 7.5 from the usual 1.2-1.8

cato’s picture

cato CreditAttribution: cato commented

I think I found the source of my problems:

Line 481-482 of swftools.module reads 

// Cache the result
cache_set('swf:'.$options['othervars']['cid'], $output, 'cache_swftools', CACHE_PERMANENT, $script);

Unless the intention is to always cache the output of print swf() regardless of the users cache setting I think there needs to be a conditional check for cache setting here.

Oh, and I forgot to mention, the site I'm running uses exactly that statement to display flash players.

cato’s picture

cato CreditAttribution: cato commented

What I've found is that cache_get misses for playlists since it only looks up entries starting with swf:
Playlists start with xml: so there's a bit of a gotcha.

In swftools.module:

2027:    // If this is in the cache then return its data, attaching JavaScript if necessary
2028:    if ($ret = cache_get('swf:'.$cid, 'cache_swftools')) {
2029:    return $ret;

Most calls to cache_set() do not check cache settings before committing the cache. Maybe they should?

Stuart Greenfield’s picture

Stuart Greenfield CreditAttribution: Stuart Greenfield commented
Status: Postponed (maintainer needs more info) » Needs review

I revisited the cache mechanism in light of the above and have made some changes.

First, it was caching using the CACHE_PERMANENT flag, which meant the cache was rarely flushed in "normal" use. I've changed it so it now uses CACHE_TEMPORARY. This means cache entries expire when things like cron run (as per regular pages in the Drupal page cache).

Secondly, swf and xml were cached separately, but they are now cached as a single entity so we can be sure they are either both available, or not available at all.

I thought about the "optional" caching. SWF Tools no longer writes physical files for xml playlists but actually uses the cache to serve the data. Therefore when the cache is disabled SWF Tools simply bypasses trying to serve from the cache, but it will always store. But we have to cache on every call to swf() in case there is some xml we want to keep.

With the periodic purge now there shouldn't be a real drawback to always caching, but if this is still problematic I can re-examine this.

This should, I think, solve the issue you had as the cache will now flush periodically and expire the oldest items.

I will commit shortly, and then it should generate on the next dev package.

cato’s picture

cato CreditAttribution: cato commented

>I revisited the cache mechanism in light of the above and have made some changes.

>First, it was caching using the CACHE_PERMANENT flag, which meant the cache was rarely flushed in
>"normal" use. I've changed it so it now uses CACHE_TEMPORARY. This means cache entries expire
>when things like cron run (as per regular pages in the Drupal page cache).

This will probably solve the cache_swftools obesity issue.

>Secondly, swf and xml were cached separately, but they are now cached as a single entity so we can
>be sure they are either both available, or not available at all.

>I thought about the "optional" caching. SWF Tools no longer writes physical files for xml playlists but
>actually uses the cache to serve the data. Therefore when the cache is disabled SWF Tools simply
>bypasses trying to serve from the cache, but it will always store. But we have to cache on every call to
>swf() in case there is some xml we want to keep.

>With the periodic purge now there shouldn't be a real drawback to always caching, but if this is still
>problematic I can re-examine this.

I'm going to test it the coming week, I'll keep you updated on the results.

>This should, I think, solve the issue you had as the cache will now flush periodically and expire the
>oldest items.

>I will commit shortly, and then it should generate on the next dev package.

Greatness. Thanks a million Stuart!

//Christopher

Milosavlevski’s picture

Milosavlevski CreditAttribution: Milosavlevski commented

Using CACHE_TEMPORARY on a high-traffic website, the cache_swftools table gets is cleared before cache_page (I believe this is normal behavior), but usually it doesn't get regenerated on time, which causes 404 errors in JW Player. I have two instances of JW (playlists) on the homepage (one generated with Views and one with swf()).

I reverted back to CACHE_PERMANENT for now, but this is a real issue and should be fixed.

cato’s picture

cato CreditAttribution: cato commented

I never could make this cache obesity issue go away completely so we had to abandon the module on our sites and resort to a in-template replacement. As this issue is quite old by now it could probably be closed?