Not all stream wrappers are backed by the filesystem. These streams sometimes fail with gzip compression and decompression because the gz* IO functions cause errors.

Additionally, performance of the compression and decompression filter functions are suboptimal because they are done manually in while loops instead of being done by low-level PHP functions.

The solution: PHP stream wrappers and stream filters should be used instead of the gz* bz* family of file functions. PHP has a built-in filtering system that handles precisely the use case envisioned here.

CommentFileSizeAuthor
#1 1452462-1-stream-filters.patch3.86 KBmbutcher

Comments

mbutcher’s picture

mbutcher commented
StatusFileSize
new 1452462-1-stream-filters.patch3.86 KB

This patch provides the following:

  • ZLib inflating and deflating through filters.
  • BZip2 compressing and decompressing through filters.
  • Efficient stream copying using stream_copy_to_stream() instead of fread/fwrite loops.
ronan’s picture

ronan commented
Status: Active » Fixed

Beautiful, thanks! Committed to dev.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

ronan’s picture

ronan commented
Status: Closed (fixed) » Needs work

This patch seems to be causing problems for some users: #1564408: Gzip backups are compressed twice when downloaded

It works fine for me so I'm not sure what's going on in those cases or how to reproduce the problem.

ropaolle’s picture

ropaolle commented

I am still not sure fopen/fclose should be used when gzip or bzip files are created, maybe somebody can confirm that. I tested this by compressing the db with Backup and migrate and the trying to open it with WinRar. Gzip files can't be open at all. Bzip files can be open but WinRar don't know the size of the content so it may have some problem to read something in the header. The only way I can make gzip/bzip work is to use corresponding open/close function.

  function _backup_migrate_filter($source, $dest, $filter, $mode = STREAM_FILTER_READ) {
    $in = fopen($source, 'rb');  
-    $out = fopen($dest, 'wb');  
+    if ($filter = 'zlib.deflate') {
+      $out = gzopen($dest, 'wb9');
+    } else if ($filter = 'bzip2.compress') {
+      $out = bzopen($dest, 'w');
+    } else {
+      $out = fopen($dest, 'wb');
+    }

..

    fclose($in);
-    fclose($out);
+    if ($filter = 'zlib.deflate') {
+      gzclose($out);
+    } else if ($filter = 'bzip2.compress') {
+      bzclose($out);
+    } else {
+      fclose($out);
+    }
ronan’s picture

ronan commented
Status: Needs work » Closed (won't fix)

I have rolled this back because it was causing problems for a lot of people.

I'm not sure how necessary this change is since compression is done on temporary files and I imagine most site setups have use a local directory for temporary files. Please reopen if I'm wrong on this assumption and we can maybe find a way around the issue.

Thanks