Performance issues with large number imagecache presets and content types
| Project: | Lightbox2 |
| Version: | 6.x-1.x-dev |
| Component: | Code |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Jump to:
There are really bad performance issues that occur when mixing large numbers of imagecache presets and lightbox2
This relates to the storage of settings in the theme registry cache.
For example, with 8 content types, no lightbox2, here are the sizes in the cache table entry "theme_registry:garland". Only one filefield image was enabled:
# imagecache presets Size (KiB)
0 52.4
1 54.1
16 80.1
24 93.9
With lightbox:
# imagecache presets Size (KiB)
0 58.1 ( + 6 KiB )
1 62.8 ( + 8 KiB )
2 69.4
4 88.2
8 147.9
16 354.8 ( + 274 KiB! )
24 678.6 ( + 575 KiB! )
After the cache setting gets to around 1MB, things become really unstable and MySQL really starts taking a hit.
The numbers above were generated from a cache flush on the "Display settings" CCK page, followed by a save. Then the values were pulled from the db.
Discovered after enabling on a site with 18 content types and 38 presets. It died almost straight away.
#1
Sorry, the table was not wrapped with a code tag:
# presets Size (KiB)
0 52.4
1 54.1
16 80.1
24 93.9
With lightbox:
# presets Size (KiB)
0 58.1 ( + 6 KiB )
1 62.8 ( + 8 KiB )
2 69.4
4 88.2
8 147.9
16 354.8 ( + 274 KiB! )
24 678.6 ( + 575 KiB! )
#2
Bumping to critical since this could kill a system during the crash with corrupted data (menu router incomplete, etc).
Should this be moved to CCK issue queue about storing redundant info in the cache table from the display options?
#3
I'm not sure what lightbox2 can do about this. Yes there are a lot of formatters created, which combine every possible usage of imagecache presets. So every time a new preset is added, the number of lightbox2 cck formatters grows exponentially. At present cck formatters have no configuration associated with them, so I don't see how I can shorten the list. I don't believe the number of content types would have an effect on this however.
#4
I think that this could be best addressed in the CCK issue queue.
The cache size appears to be the [number types] x [number presets] x overheads + non-related cache data. The cache is storing all display options, not just the selected option, which just seems redundant IMHO. We've hit this issue on three separate sites now, so I thought that it was worth reporting. We had to resort to backups twice. Note that this is < 2% of Drupal sites we've developed, not widespread but fairly drastic when it happens. It probably explains performance issues common on complex CCK based sites (with and without this particular issue instance).
Windows is more prone to this, as the max packet size on MySQL is only 1M. Our UNIX boxes have significantly higher settings. I'm not a CCK expert, so I haven't attempted to look into this issue any deeper.
#5
#6
Well, I can see the problem, but formatter settings will definitely not happen in CCK D6.
1M is an insufficient max packet size to run drupal anyway.
As far as CCK is concerned, this is a won't fix
#7
"1M is an insufficient max packet size to run drupal anyway. "
Mmmm.. 100+ sites and no problems with [mysqld] max_allowed_packet = 1M. Caching problem sites can all be traced back with heavy use of CCK/lightbox. (Note our production server settings are always 8M+ and it is very rare to have these sorts of issues once deployed, by rare I think we have only had a single WSOD that has resulted in a broken Drupal install when in production and that was due to a php max memory limit. Luckily we never have to use third party hosting providers.)
"Well, I can see the problem, but formatter settings will definitely not happen in CCK D6."
Is this a push to D7 version? I have not yet had a play with fields in D7. Have the display settings been refactored? Having large cache loads/writes is a bad performance hit, and I know of a couple of high profile local drupal sites that have this issue, resulting in fairly poor load times.
Dropping to minor as it is a fairly rare event in the grand scheme of the CCK family.
#8
D7 Field API already has formatter settings. The UI implications, though, are not resolved yet.
#9
Here's an idea that maybe lightbox2 could implement. I'm not sure how lightbox2 works, though.
In the field settings panel there are global settings, and widget settings. Maybe lightbox2 could implement widget settings so that users can customize defaults for the lightbox2 formatters. That way, maybe just one formatter could be needed.
Note that widget settings are not shared, even if you use the same field shared with different content types. So you can implement different formatter settings per widget here.
I used this technique for the Money CCK field ("Currency display mode" and "Decimals display mode" really are formatter options). :)
#10
Yes, in the absence of formatter settings in D6, some modules put their display settings in 'widget settings', which in D6 are the only 'instance specific' settings.
Problems with this approach, though :
- conceptually awful (but well, CCK D6 has nothing better to propose...)
- you need to own the widget to add widget settings :-(, which is not the case of lightbox2 - #417122: Allow drupal_alter() on Field and Widget Settings should work around this
- you cannot specify different settings for different build modes (teaser, full...) - unless you add one set of settings per build mode in the widget settings form, basically duplicating the 'Display fields' page. Yuck.
#11
I'm not sure how well a lightbox2 widget would work, considering lightbox2 has to be able to handle images, pdfs, files, videos, web page links, etc. It's possibly worth considering, but I think I'd prefer to add support for the #417122: Allow drupal_alter() on Field and Widget Settings approach.
#12
Ah, that issue is in a state that needs review. I think it would be nice to avail this opportunity to join forces with the people working on that patch so maybe it can be included in CCK as soon as it can be assured it works for more that one use-case (filefield and lightbox2 related extensions).
#13
@stella: The previously mentioned patch had been committed to CVS, so you can now embed your own settings to any field/widget. Hence this is now doable in contrib. I'm afraid CCK for D6 cannot do more.
#14
Re-assigning to lightbox2 for further review / development
#15
I'd like to emphasize the negative impact of this default behavior on large Drupal setups.
As explained in this frontpage case study http://drupal.org/node/614014, Lightbox2 alone (with its automatically-generated formatter and theme declarations that are stored in cache and fetched on every page) was responsible of 80% of the SQL traffic, i.e. 1Gbit/s out of 1.25Gbit between apache and mysql.
It would be really great to have these formatters generated only if we want them, e.g. with a config page.