Add a compressing serializer decorator

This feature should allow user to disable it by put $config['cache_compress'] = FALSE in settings.php
hook_update_N() needed to flush existing cache.

Thanks for the patch! Interesting proposal. Here's some feedback on the patch itself:

+++ b/includes/cache.incundefined
@@ -401,7 +401,14 @@ class DrupalDatabaseCache implements DrupalCacheInterface {
+        // @todo: log message?

This could be a lot of messages, so let's remove this @todo.

Also, I love bitwise logic as much as the next gal, but let's rework the patch to be a little more legible. At the least let's add some inline comments explaining what's going on. :)

Compress data = more processing, I think we still need allow user to disable this feature. And compress level, it should be also configurable.

Nice patch, but I agree with @thehong. This should be optional, configurable (might be only via settings.php) and disabled by default.

+      if (!function_exists('gzuncompress')) {
+        // @todo: log message?
+        return FALSE;
+      }

You can check this in the hook_requirements. E.g. if (variable_get('cache_compress', 0) && !function_exists('gzuncompress')) { ...

Interesting feature.

"Performance" is a pro? I think this needs more benchmarking.

Note that we have /admin/config/development/performance form and site cache is not always DrupalDatabaseCache.

We developed a module that does this without having to change core. Our cache_form table was growing by gigabytes per day, and our solution was to compress the cache data going into cache_form.

The beautiful thing about drupal's caching desgin is that it's object oriented and easy to extend and create different implementations, because of this I think that this doesn't belong in core, but in an extension of the default implementation. That's exactly what we did.

You can find the module here: http://drupal.org/sandbox/jared_sprague/1545886

This module is more flexible than the above patch because you can specify specific bins that you want to add compression too, unlike putting this in core where it's compress all or nothing.

Here are the features:
- Extends default database cache class, doesn't change core
- Can configure which cache bins you want to compress via settings.php
- Can configure the gzip level via settings.php

Currently this module is being run on the Red Hat Knowlegebase here: https://access.redhat.com/knowledge

And just compressing cache_form reduced our binary log file sizes by 80%. After it's been production tested for a few weeks we're going to apply to get it promoted to a full project.

Similar functionality would be good to have in the keyValueExpirable (and possibly even keyValue) storage, because the former stores serialized form data which can become very big when editing nodes with paragraphs.

Adding a compression step to the PhpSerializer service (maybe optional, with a configurable size threshold) would fix this problem everywhere serialized data is stored in the database.

I would go with a compression service to be generally available in core (even if fake one - no compression).
On top of that - have decorators for each sub-system that will need compression available.
This way existing code should not be changed, but everything is just tweaks to the container services definitions and pluming.

Personally I think that only cache entries should be stored in compressed manner. Raw data should be kept raw for many reasons (maintainability, portability, durability). Drupal (8) is very thorough about what to cache (almost everything), so origin data store is rarely the bottleneck when there are warm caches.

Crosslinking working code and profiling:
Redis #2826332-7: Option to compress data

Note that in D7 the https://www.drupal.org/project/apdqc module can do this with the default cache schema.

Per bin and global: Can pick to compress just serialized data or all data and the compression level. It uses the compressed data only if it's smaller (sometimes compression will bloat the size). Below is the relevant code from the module.

function apdqc_inflate_unserialize(&$cache) {
  $converted = FALSE;
  if (is_array($cache)) {
    $cache = (object) $cache;
    $converted = TRUE;
  }
  // If the data is compressed, uncompress it.
  if ($cache->serialized > 1) {
    $inflate = @gzinflate($cache->data);
    if ($inflate !== FALSE && $cache->data !== FALSE) {
      $cache->data = $inflate;
    }
    $cache->serialized -= 2;
  }
  // If the data is permanent or not subject to a minimum cache lifetime,
  // unserialize and return the cached data.
  if ($cache->serialized) {
    $data = @unserialize($cache->data);
    if ($cache->data === 'b:0;' || $data !== FALSE) {
      $cache->data = $data;
    }
  }
  if ($converted) {
    $cache = (array) $cache;
  }
}


  /**
   * Constructs a DrupalDatabaseCache object.
   *
   * @param string $bin
   *   The cache bin for which the object is created.
   */
  public function __construct($bin) {
...
    $this->compressed = $this->apdqcCacheCompressBin($bin);
    if ($this->compressed) {
      $this->compression_level = $this->apdqcCacheCompressionLevelBin($bin);
    }
...
  }

  /**
   * Checks if this cache bin is compressed or not.
   *
   * @param string $bin
   *   The cache bin name.
   *
   * @return bool
   *   TRUE if this cache bin is compressed.
   */
  protected function apdqcCacheCompressBin($bin) {
    $compress = variable_get('apdqc_cache_compress_' . $bin);
    if (!isset($compress)) {
      $compress = variable_get('apdqc_cache_default_compress', APDQC_CACHE_DEFAULT_COMPRESS);
    }
    return $compress;
  }

  /**
   * Checks if this cache bin is compressed or not.
   *
   * @param string $bin
   *   The cache bin name.
   *
   * @return bool
   *   TRUE if this cache bin is compressed.
   */
  protected function apdqcCacheCompressionLevelBin($bin) {
    $compress = variable_get('apdqc_cache_compression_level_' . $bin);
    if (!isset($compress)) {
      $compress = variable_get('apdqc_cache_default_compression_level', APDQC_CACHE_DEFAULT_COMPRESSION_LEVEL);
    }
    return $compress;
  }

  /**
   * Implements DrupalCacheInterface::set().
   */
  public function set($cid, $data, $expire = CACHE_PERMANENT) {
...
    $fields = array(
      'serialized' => 0,
      'created' => REQUEST_TIME,
      'expire' => $expire,
    );
    if (!is_string($data)) {
      $fields['data'] = serialize($data);
      $fields['serialized'] = 1;
    }
    else {
      $fields['data'] = $data;
      $fields['serialized'] = 0;
    }
    if ($this->compressed > 1
      || ($this->compressed && $fields['serialized'])
    ) {
      $deflate = gzdeflate($fields['data'], $this->compression_level);
      if (strlen($deflate) < strlen($fields['data'])) {
        $fields['data'] = $deflate;
        $fields['serialized'] += 2;
      }
    }
...
  }

this issue is gold - really looks like something to help us a lot. I did some investigation and it seems like there are currently two approaches being worked on to get the possibility / something like this in core:

1) change database cache backend.
which is this. but the patch did not get much love in the past. related: https://www.drupal.org/project/drupal/issues/2886405

2) use serializer service in database cache backend in order to be able to replace that serializer service by one which adds compression.

related issues:
https://www.drupal.org/project/drupal/issues/839444
https://www.drupal.org/project/igbinary
https://www.drupal.org/project/drupal/issues/2886405

I sincerely believe 2) is the way to go for core. But to get the compression quickly done without too many side effects I am going to either reroll this patch, or create a different database backend which extends the current database backend and adds compression.

Feature requests should be against 8.6.x. Patch still applies.

Here's a contrib module prototype (Different Cache Backend implementation): https://github.com/AndreBaumeier/module-compressed_cache

After some further testing with real world data this will become a proper module. Feedback is always appreciated.

module namespace: https://www.drupal.org/project/compressed_cache

Original issue summary

....shared hosting situation that had max_allowed_packet set to a value of around 2MB and the contents of the serialized admin menu turned out to be just a bit larger than 2MB. The hoster refuses to change this value,so I had to come up with a different solution: cache the data before storing it.

It that situation still exist and still needs compressing cache in DB?
I mean, with current SSD(most of the hosting service providing this), higher network band width etc. could we store it in flat file(only cache) avoiding DB hit completely.

1. +++ b/core/lib/Drupal/Core/Cache/DatabaseBackend.php
   @@ -162,8 +172,14 @@ protected function prepareItem($cache, $allow_invalid) {
   +      if (!function_exists('gzuncompress')) {
   +        // We cannot decompress the compressed cache data, ignore it.
   +        return FALSE;
   +      }
   +      $cache->data = gzuncompress($cache->data);
   

   Have this check on the constructor and prepare a NULL / FALSE callback as a place-holder in case gzuncompress is missing

   This way we will always call the callback and we spare the function exists call.

2. +++ b/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php
   @@ -503,6 +504,9 @@ public function testGet() {
   +      if (($cache_item->serialized & DatabaseBackend::CACHE_COMPRESSED) !== 0) {
   +        $cache_item->data = gzuncompress($cache_item->data);
   +      }
   

   What if the function is missing...

3. +++ b/core/lib/Drupal/Core/Cache/DatabaseBackend.php
   @@ -234,15 +250,21 @@ protected function doSetMultiple(array $items) {
   +      if (strlen($fields['data']) > 1000 && function_exists('gzcompress')) {
   

   Have this as a configurable thing. Avoid magic constants in code.

It that situation still exist and still needs compressing cache in DB?
I mean, with current SSD(most of the hosting service providing this), higher network band width etc. could we store it in flat file(only cache) avoiding DB hit completely.

these are still true:

Advantages:
- Performance.
- Less database storage.
- Less network traffic.

having something like boost for Drupal 8 might be worth looking at for some though.

Have this check on the constructor and prepare a NULL / FALSE callback as a place-holder in case gzuncompress is missing

function_exists calls are not cached. probably better to check that once in constructor and store a boolean value to look for within cache set/get. there are plenty of cache calls happening.

some "benchmark" values:

on a page with 100k+ nodes, prefetching the entire page took the same amount of time with the alternative cache backend of compressed_cache module as with the standard database backend. Application itself has a HA-Setup with database server and web server being on two different servers. Server A has db + web, Server B has db + web, one of them is flagged as master server for all requests coming in. master-master replication happening.

So that's a good find - it's not slower. But here's whats more important in my opinion:

Based on the results above I am adding the scalability tag.
This will drop data storage and RAM requirements for DB on any D8 installation.

Just my 2c:

Compression is always useful when there is more CPU time available than DB time, e.g. when the scalability is DB-bound and not CPU bound.

D8 was for a long time CPU bound and DB played only an insignificant role. This has changed thanks to PHP 7.2.

That said:

This should probably be based upon #839444: Make serializer customizable for Cache\DatabaseBackend instead of hardcoded ...

You can also turn on mysql compression per table:

alter table cache_page ROW_FORMAT=COMPRESSED;

I haven't tested this out, but I assume mysql would be more efficient than PHP?

This would solve the disk usage on the db instance, however, this doesn't reduce the size of the payload between php and db.

#38
this moves the burden of compression and decompression on the database server, which is harder to scale than web servers.

I am +1 to implement that as an compressing serializer decorator.

For concerns that it's not worth to compress little things and there should be a decision:

- The serializer can do:

public function encode($data) {
$data = $this->serializer->encode($data);
  if (strlen($data) > $this->threshold) {
    $data = '1' . 'gzcompress($data, $this->compression_level);
  }
  else {
    $data = '0' . $data;
  }

  return $data;
}

public function decode($data) {
  $encoded = substr($data, 0, 1);
  $data = substr($data, 1);

  if ($encoded === "1") {
    $data = gzuncompress($data);
  }

  return $this->serializer->decode($data);
}

and that's it already.

Of course in a next step we could also abstract out a CompressorInterface, but that's not needed for a simple first start.

#42: Totally agree. We had exactly this in redis module now for some time: #2826332: Option to compress data