Gzip backups are compressed twice when downloaded

I found the same.

Same isse. Tried to uninstall and then reinstall, but it did not help. BZip also works, it is only a problem with GZip.

Can people having this problem weigh in on what server/os this is happening on? There's some indication it might be a XAMPP/WinXP problem. Also, was gzip working before the upgrade?

I am using Fedora 12 and am having the same problem with gzip, zip appears to work fine....and yes, gzip was working beforehand....

GZip not working for me as a phpMyAdmin import on both XAMPP/WinXP and CloudLinux systems.

As with #5, GZip works ok for me in B&M 2.2

I have confirmed it on Ubuntu 11.10 and my test pc with XAMPP/Win7. Can it be related to the zlib version? On Ubuntu I have zlib 1.2.1.1 and on XAMPP I have 1.2.5.

I would guess the problem is with this patch here : #1452462: Some streams cannot be correctly compressed or decompressed

This code works for me so I'm not really sure what I can do to debug and fix this. I could go back to the old code but that breaks some stream wrappers.

I'm gonna need help debugging this from somebody who is able to reproduce the problem.

Sorry I do not know how to debug this, but I have done some trial and error and found that it works if I use gzopen/gzclose insted of fopen/fclose. This will probobly break other compression filter is just an example when it works.

function _backup_migrate_filter($source, $dest, $filter, $mode = STREAM_FILTER_READ) {
-    $out = fopen($dest, 'w+');
+   $out = gzopen($dest, 'wb9');

-    fclose($out);
+   gzclose($out);

In my case, I can restore from within the module using either format. But I get the attached error with .gz format when trying to import in PHPMyAdmin. (The .zip format is not supported on the hosting server.)
Apache version 2.0.64
PHP version 5.3.6
MySQL version 5.0.95-community
Architecture x86_64
Operating system linux

found the same problem here too

I have the same problem as well with Gzip compression selected. It started happening a few days ago.

I'm pretty sure the Gzip problem started happening with the new update for Migrate and Backup was published

Same here:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

I also have corrupted gzip and noticed a strange symptom of a gzip.info file getting created at the same time. I've attached it for reference. (These files definitely started appearing after the recent update).

PHP 5.3.1 running on XAMPP on Mac 10.6.x
Zlib 1.2.3

I confirm this bug.

When trying to make backup by drush, I got 2 files in /files/private/backup_migrate/manual -- corrupted XXX.mysql.gz and unexpected XXX.mysql.gz.info. Both files are attached as manual.zip

System -- Debian Squeeze Linux 2.6.32-5-686 #1 SMP i686 GNU/Linux
Drupal -- 7.14
Backup and Migrate -- 7.x-2.3
Other system installed packages are listed in attached installed_packages.txt.gz

Does not work on restoring GZip on MAMP.

I can confirm the issue on:
OS X 10.7.4
PHP 5.3.10
Apache/2.2.21
mysql Ver 14.14 Distrib 5.5.22, for osx10.7 (i386) using readline 5.1
Drupal 7.14
backup_migrate 7.x-2.3

gzip worked with B&M 7.x-2.2/Drupal 7.12 prior to upgrading to Drupal 7.14/B&M 7.x-2.3.
zip works with Drupal 7.14/B&M 7.x-2.3.
bzip works with Drupal 7.14/B&M 7.x-2.3.
XAMPP is not installed

Confirming issue.
gz backups made with Backup & Migrate 2.3 running on Drupal 7.14 are not readable. They aren't readable by the server or by my desktop Mac running Lion 10.7.3.

My server is running:
CENTOS 5.8 x86_64
Apache version 2.2.22
PHP version 5.2.17
MySQL version 5.1.62-cll

@ronan or someone else,

If you could outline some steps for me to perform that would help debug this, I'd be happy to perform them and report results.

I'll repeat my set-up from the previous post:

My server is running:
CENTOS 5.8 x86_64
Apache version 2.2.22
PHP version 5.2.17
MySQL version 5.1.62-cll

Shai

To help us find something in common of all instances that are failing, let me add to my previous description that my server runs cPanel/WHM.

Shai

updated module 2.2 to 2.3, B&M creates corrupted .gz files that can not be opened or used to restore. But when we taking Back up in Zip format it working.

I should add to my post #11 that I'm using CPanel.

Also I'm using Drupal 7.13 rather than 7.14. This may be why I can restore from the .gz file using the 2.3 module when others cannot. The .gz file still cannot be imported using PHPMyAdmin.

I can import an uncompressed backup made with the 2.3 module using PHPMyAdmin but it puts the site in maintenance mode and logs me out as an administrator. Since we only allow CAS logins I can't get back into the site to take it out of maintenance mode. To recover, I have been able to restore a .gz backup made with 2.2 module using PHPMyAdmin and then restore to a current .gz backup using 2.3 module.

It might help to explain what happens when I try to open the downloaded gzip file on my mac (OSX 10.6.8).

Say the download is example.msql.gz. When I open this with the mac archive helper I get example.msql.cpgz. When I open the .cpgz file I get a copy of example.mysql.gz and so on ad infinitum.

There's a related thread on Macrumors that has been live since 2006 and still going ( http://forums.macrumors.com/showthread.php?t=178295 ). None of the suggestions there works for me.

jack

I am also experiencing this issue. gzip files are created, and appear to be valid. They can decompress fine, and the resulting SQL file, when re-converted to .zip imports fine. However, if I use my desktop OS to re-convert it to tar.gz, Backup and Migrate could not decompress it.

Ubuntu 10.04 Server
PHP 5.3
Drupal 7
Backup and Migrate 2.3 (just updated)

Apparently, the fix for the time being is to use Zip instead of GZip.

My server will allow B M make the backup and zip it, but it won't allow for B M to restore a zipped file. (I can unzip the file fine from the command line).

So for me, the upshot now is that there are extra steps involved in restoring OR I use no compression and then have bigger files longer backup and restore times.

But let me explain that I think there is some urgency to this issue, even if there are pretty simple or slightly inconvenient workarounds: many, many people are backing their sites up to files that are either very difficult to decompress or impossible to decompress. They don't know they have a problem. Pushing out a version 2.4 with a fix would likely get the attention of a good proportion of those folks and take them out of the situation they have now, which is they don't even know they have a problem.

Shai

Gzip compression is apparently working for some users. Can somebody confirm that that is the case and on what environment it is working.

I wonder if it has something to do with differing PHP extensions. I found I could do it on my Ubuntu 10.04 32bit Desktop, but not Ubuntu 10.04 64bit server. It's likely that I have different PHP extensions installed on each. Do you know exactly which PHP and/or PEAR packages (and which versions) are necessary for the decompression of the gzip files?

Or maybe it's the difference between the 32bit and 64bit versions of those packages.

same problem here. i cant unzip or restore the file

Ubuntu 11.10 64 bit
PHP Version 5.3.6-13ubuntu3.6
php fpm
nginx
drupal 7.14
bm 7.23

Same as emosbaugh above.
Using Ubuntu 11.10 (64-bit).
Drupal 7.14

Problem started after I upgraded to latest version of Backup & Migrate module few days ago.
Backup Gzip files wont open.

Same problem here on Windows 7 64bit.

PHP 5.3.8
Nginx 1.2.0
zlib 1.2.5
Drupal 7.14
MariaDB 5.3.6
Backup and Migrate 7.x-2.3

Switched to bzip and it works.

Same for me.

CentOS 6
PHP 5.3
Drupal 7
Backup and Migrate 2.3

@Shawn DeArmond. I was also thinking that this may be a 32/64bit issue and tested to install Acquia Dev Desktop with Backup and Migrate on a Win 7 (64 bit) machine and on a Win XP (32 bit). But no luck. I am not able to open the Gzip file on any of the machines.

It happens indepent of underlying CPU architecture (x86/64).
I think it may have something to do with how the data streams from the mysql database are GUNZIPPED.
Definetely it has to have something to do with the gunzip zipping protocol, because bzip2 and zip all work - given the same database.

What would be nice is to take an actual .mysql dump file, and zip it manually (from command prompt) to .mysql.gzip. It would then be clear where the issue is...

@ilechcod: I did that. It didn't import.

Details:
--Used Backup and Migrate to generate a .gz file.
--Tried to Restore, but didn't work.
--Decompressed .gz file to .mysql
--Recompressed .mysql file (ubuntu, right-click, compress) to .gz
--Tried to restore, but didn't work.
--Recompressed .mysql file to .zip
--Tried to restore, WORKED!

Clearly there's something wrong with restoring a .gz file on some configs.

The error is not just with restoring, when I download a .gz backup and try to unpack it I get a "Error in packed file" message. It looks like the packed archive is not formatted correctly.

@ArjanLikesDrupal
Yes. Definitely, the problem is with the created gzip file - archive format is doubtful.
Arjan, please could you help try out the following (if you are on a Linux machine)

• Using Backup & Migrate module - do a normal DB backup (using no zip) - to produce a *.mysql file.
• Then right click on the file and manually do a gzip operation.

(1)Does the produced gzip archive open up (try transferring to another system) correctly?
(2) Can it be restored using Backup & Migrate module

I'm trying to figure out where in the module file the gzip operation is called - I have no doubt the problem could be from there.
Thanks

I also have this problem.
I cant extract the file, and when I try to restore I get error message/
I changed the default backup to zip?
whats wrong?
Thanks

I have updated to 7.14 and have issues with 7x-2.2. No tables are selectable within profile.

When I update to 7x-2.3 I could backup but restoring didn't work. For some reason a .info file is created along side the backup gzip file.

I was in the middle of creating a new site on D7.14 and tried using backupMigrate 7x-2.3 and the same issue is happening - .info file is created and when trying to restore, there is a MySQL error in phpmyadmin.

I have another site that I was updating all contribs first while backing up before every update. Everything was fine until I updated the D7.14 core. BTW, I was using b_m7x-2.3 during the entire update of contribs. Somehow D7.14 messes things up for b_m.

PS. I'm using the latest 7x-2.3 update from 4 May 2012.
E.

PS. See my post at http://drupal.org/node/1558680#comment-6026458 for my workaround.

On same exact server
I have one installation with 2.3 working
and a fresh install drupal7.4 with BM 2.3 not working with gzip

.zip seems to work .gz is corrupt reports my window 7

I run a daily backup and notice that it has been corrupt each evening since 5th May if thats any help. I have a feeling I installed an update of the module that day...

i recently had a crash of D7.14. gz'd B&M backups failed to gunzip. not a happy moment.

reading the thread here, and based on a 'lecture' or two I've gotten from my db-admin friends, the dependency OF backups ON the working state of Drupal & a module seems a really bad idea to me.

since I have root & mysql-admin access to my server -- not a situation for everyone I know -- I rm'd B&M.

i replaced it with AutoMySqlBackup (http://sourceforge.net/projects/automysqlbackup/). NO more problems with compression or extraction using either gz of bz2 formats. everything just works.

hth someone else.

EDIT:

> Can people having this problem weigh in on what server/os this is happening on?
> There's some indication it might be a XAMPP/WinXP problem.

drupal 7.14 + Apache/2.4.3 + PHP/5.4.5 on linux/64 + gzip 1.4

> Also, was gzip working before the upgrade?

gzip itself always works. gz'd B&M backup/restore *used* to work -- can't tell you when it stopped. only that it DID stop.

EDIT2:

for a 'failed' B&M backup.gz,

	file DEV-2012-05-24T20-50-46.mysql.gz
		DEV-2012-05-24T20-50-46.mysql.gz: data
	gunzip DEV-2012-05-24T20-50-46.mysql.gz
		gzip: DEV-2012-05-24T20-50-46.mysql.gz: not in gzip format

for an automysqlbackup backup.gz,

	file daily_DEVdb_2012-05-27_10h31m_Sunday.sql.gz
		daily_DEVdb_2012-05-27_10h31m_Sunday.sql.gz: gzip compressed data, from Unix, last modified: Sun May 27 10:31:59 2012
	gunzip daily_DEVdb_2012-05-27_10h31m_Sunday.sql.gz
	ls -al daily_DEVdb_2012-05-27_10h31m_Sunday.sql
		-r-------- 1 root root 3.9M May 27 10:32 daily_DEVdb_2012-05-27_10h31m_Sunday.sql

Hi;
I think that because that is a real issue, until release new version the must to do is:

1. change the gzip to zip format
2. create a new default backup with the change
3. save it

this bachup can be able to restore,

warning!
today I had the same problem on zip compress!!!!
watch out!

I have rolled back the code in the 7.x-2.x branch. There will be a new dev release today some time. If you're having this problem, please download the new dev and let me know if this it is fixed. I will then push out a new stable realease.

Thanks
Ronan

Hi Ronan,
Did you take into account the issue with D7.14 brought up at http://drupal.org/node/1558680. I found that everything works okay with my work around at http://drupal.org/node/1558680#comment-6026458. Can you please confirm that this is the right way to go?
E.

Grabbed the fresh dev release and did a manual backup in GZip format. Downloaded it and imported it into a test database. All seemed to go just fine. A quick check of tables and data and it all seems to be there. Drupal 7.14.

Forgot to add that I used phpMyadmin to import to the test db.

@erok415, Yep, I just rolled back the gzip patch. All other fixes and changes should still be there.

@ronan, thanks and I'll make sure to test it today and provide feedback.

I'm reporting back as per @ronan's request in #48.
1. I'm having the reported problems of backup_migrate 2.3 gzipped files not being able to unarchive and those gzips also not being able to be used by backup_migrate to restore site: as per my report in #21 above.
2. I downloaded and installed the May 30, 2012 dev version of backup_migrate.
3. Using the default profile which gzips, I backed up D 7.14 install to my desktop.
4. Using the backup_migrate UI, I restored the site using the gzipped file I backed up to my desktop.
5. The site was restored without incident.

In short, the latest dev solves the problem I faced with 2.3.

Looking forward to the 2.4 release.

Thanks much,

Shai

re: #54 report of drush bam-backup not working after installing latest dev version of backup_migrate.

On an installation that has the latest backup_migrate dev installed (May 30, '12), I ran drush bam-backup and it created a backup without error.

I'm running Drush 4.5.

Shai

Is there any way to salvage the corrupted gz files? I'm in a worst case scenario as far as this issue goes. I've been using this to back up the site that I'm currently in the process of developing. Unfortunately (apparently) all of my backups, besides one or two from before this version of the module, are all corrupted. I just did a database update for a couple updated modules and, you guessed it, website is dead. As anybody can imagine, that's a lot of work down the drain since I can't backup from the "backups" that I have saved.

If nothing else gets accomplished from this post, I'd suggest pushing out an emergency version of the module where the default backup mode is .zip instead of .gz or something like that. I would imagine that there's got to be more people than just myself in the same situation. Had I found this before, I would have obviously changed the default mode of compression to something else to avoid this headache.

But anyway, my initial question, is there any possible way to salvage these corrupted backups, or are they completely lost?

EDIT: Also, any idea what exactly is wrong with the backup file? I managed to restore the database, but for some reason the site is still broken (only displays blank pages). All tables appear to be intact from what I can tell. So, would it just be the relationships between them? Can that be fixed?

EDIT 2: I was somehow able to get the site up and running again by changing its php memory allocation. For some reason it was a version of the site from a couple days ago, but that's fine with me at this point. The fact stands that, in my opinion, a module that people base their site's well-being on should not be left in a state that creates corrupted database backups by default.

@mrweiner,
Please check comment #53 above. I think there's a development version of this module sitting out there - without the problems of corrupted gzips. You might want to run an update on your Drupal site to upgrade.

FOR ANYONE ELSE STILL HAVING ISSUES WITH THIS MODULE, PLEASE READ COMMENT #53 ABOVE. YOU NEED TO UPGRADE TO DEV VERSION,

Regards All,

@ilechcod
I must have skipped over that comment somehow, so thanks for directing me to it. However, I think that the problem lies with the fact that the fix is only within a dev version of the module. Dev versions don't show up in the module updates unless you have a dev version already installed (unless I'm mistaken). As such, people who aren't actively checking issue logs will never know there is a dev version to fix this problem, or even that there is a problem to begin with.

They then end up in a situation similar to mine, where the backups that they assumed were fine are in actuality corrupted. They then cannot restore to a recent version of their database. This is why I suggested a standard release of the module with some sort of temporary fix. If a standard release is in the works with the new patch, and will be released very soon, then I suppose this can be disregarded. However, with over 150,000 reported sites relying on this module as their (more than likely only) form of backup, a version with compromised default functionality should not be allowed to sit as a release for as long as this one has.

@mrweiner,

I believe that @ronan (the module maintainer) is fully aware of the gravity of the situation. In #48 he describes how he plans to release a new version as soon as possible.

But he has also asked for people to test the dev and report back. Few people have. He needs testing feedback in order to prevent pushing out something that may have another problem. This problem seems pretty bad, I agree, but you don't want to make things worse by putting something out that isn't tested. And this problem wasn't affecting all people, it seems to have depended on server configuration. I'm not minimizing it, I'm just calling for more people to test the dev version and report back. Even if you didn't have a problem with 2.3 it's important for people to test it and report back.

I'm sure that after a few more people have reported back, assuming no problems, that @ronan will push out 2.4.

Shai

@Shai

Fair enough. I guess I should have assumed that there was a reason for everything, but I just felt the need to voice my concerns about the project. I'm glad that there are likely plans to push this out a quickly as possible.

For what it's worth, I downloaded the latest dev, backed up as gzip, and was able to restore from the backup without any issues.

Same problem experienced.

Drupal 7.14
Backup and Migrate 7.x-2.3+1-dev (2012-May-30)

CloudLinux (prod)
WinXP/XAMPP (dev)

Database saved with Backup and Migrate using GZip compression
All tables dropped via phpMyAdmin
Database restored with phpMyAdmin

Production and development systems both functioned perfectly after performing the above operations.

In respect of the title of this issue, as of 30 May, GZip no longer creates corrupt backups.

This now has a stable release:
https://drupal.org/node/1617702

Thanks everyone for testing.

Hi to all,
ronan, after update the new Version 7.x-2.4 I have the same problem!!!
Maybe the reason is that because upgrading CentOS from 5.5 to 5.6 last week?!?
when I download the gz file anf extract it the size is the same.... and not working...
There is more that still have this issue?
Thanks

ntigh52:

Can you downgrade to version 2.2 and see if you're still having the problem?

Hi @ronan,
I found that the backup file works but it is creating a .info file along with the sql.gzip file. Do you know why that's happening?
E

There is a duplicate issue for the .info file problem.

http://drupal.org/node/1555742

Eric

I've downloaded, installed, and tested (including restore) the 2.4 version and it is working great on my server.

Shai

Yes, ronan, I will check it on my test server.

ronan,
I created a backup to gz file on my test localhost ( win7 + wamp - apache 2.2.21 - php 5.3.10 - mysql 5.5.20 ) and I could not extract the file ( the same error ).
I install the 2.2 release but I discovered that the default backup is sql file.
I cliched on Advanced Backup page and I got a errors.

Notice: Undefined index: name ב-backup_migrate_destination_db_mysql->_get_tables() (שורה 186 מ-C:\wamp\www\mysite\sites\all\modules\backup_migrate\includes\destinations.db.mysql.inc).

a lot of time.
I think that in the 2.2 release we didnt discovered the bug because the default was sql not Compressed file.
If I try to Backup the db I get the same error....
I wil check it again and comment it here.

After installing the version 7.x-2.4 erverything worked fine for me. All my backups went well, the gz-files saved to my ftp-server where not corrupted and the restore worked like a charm.

But today I had to migrate a database from the dev-server to online. And for being fast I did a "Quick Backup" of the database with the destination "Download" using gz-compression.

Well I could open this archiv. But the backup-file was corrupted. Instead of an ascii-file with sql-statements the mysql-file in the archiv contained strange characters. And of course the restore was not possible:

PDOException: in backup_migrate_destination_db_mysql->_restore_db_from_file() (line 198 von www.myserver.com\sites\all\modules\backup_migrate\includes\destinations.db.mysql.inc).

The characters in the mysql-file looked like binary-code to me and so I tried to open this file in 7zip again. And actually it contained an ascii-file without any extension with the correct backup.

So it seems, that the backup is double gzipped if the destination is "Download".

I just stumbled upon this issue after trying to open a corrupt .gz backup file. Is there any way to manually get the actual sql data out? I've tried 7zip, rezipping, file editors, but all I can get is a .mysql file with scrambled looking characters.

Any help would be appreciated.

I think the best sollution for now is to change the default backup format to none = not compress = sql.

I just Found that if you using explorer to download the zip or gz file then it creates corrupt backup.
if you using chrome or firefox, everything is ok/
dont ask me how?!

Note that this issue (corrupt gzips) is also occurring with the Drupal 6x-2.6 version (latest out).

And now I can confirm no restores work at all - not zipped nor uncompressed ones. The "restore" button simply causes the browser to spin, waiting for a response...

Posted by ntigh52 on June 14, 2012 at 2:00pm

I think the best sollution for now is to change the default backup format to none = not compress = sql.

I completely agree...which is why I suggested that nearly a month ago when I had a very bad problem with this whole corrupted backup issue. Would it really be that difficult to push out a temporary fix which just changes the default backup method to uncompressed? That way, people can still report on whether or not there are gzip issues, while at the same time non-knowing users can enjoy working backup files.

Like I said before, people don't realize that there is a problem with their backups until they need to restore them. Why would they? At that point, it's too late, because the last working, non-corrupted backup that they have will be from a while back. An old backup is better than no backup, but a working uncompressed backup is also better than a corrupted compressed backup.

For what it is worth:

I have been using 7.x-2.4 and Drupal core 7.14 for months now on a dev site, making gzip backups and restoring them -- all through the UI, seemingly without problem. I have done so a couple/few times. Also, during this dev period (since ~ Feb 2012), I believe that I have updated to Drupal 7.14 (from 7.12) and possible updated B&M to 7.x-2.4. I have attached my B&M settings for reference (note, I am actually using the default profile), as well as my server settings.

Could someone outline the symptoms of corruption for others benefit:
1. Error messages: ___
2. Site display failures: _____
3. File irregularity

So others can possibly avoid the failure (if using a release version), please state how you are making the "corrupted" backups -- few of you have stated.
Are you using B&M through Drush - both backup & restore?
Drupal UI - both backup & restore?
PHPMyAdmin for restore?
What backup settings are you using?

oops. attached server settings

I finally could get my mysql backup by using this commands after B&M gzip download :

$ gunzip mybackup-2012-08-22T11-59-12.mysql.gz 
$ mv mybackup-2012-08-22T11-59-12.mysql mybackup-2012-08-22T11-59-12.mysql.gz
$ gunzip mybackup-2012-08-22T11-59-12.mysql.gz 

When I try to unzip the same file with File Roller 3.4.1 in ubuntu 12.04 it show me that there's mybackup-2012-08-22T11-59-12.mysql in the gzip, I double click on it and it try to unzip it anew then it show me a file mybackup-2012-08-22T11-59-12 and if I double click on it, it says that an error occured during extraction !

I'm using B&M 2.4, here's my server status.

PS. I have the same problem / same solution with B&M 6.x-2.6

Looks like the follow-up information @ronan requested was provided in June, but the issue never got set back to Active. It would be good to get this resolved since it seems like there are still restoration problems under some circumstances.

Today I had to restore a backup again, which had been saved via the scheduled backup to my ftp-server with gzip-compession. And this failed: I got an error:

PDOException: in backup_migrate_destination_db_mysql->_restore_db_from_file() (line 198 of www.myserver.com\sites\all\modules\backup_migrate\includes\destinations.db.mysql.inc).

To make a log story short: Like mentioned in #73 and #81 this is because the database is double gzipped. I had to unzip it twice and then could restore it as a uncompressed mysql-file.

Seems that the problem I reported in #73 occurs not only when downloading the backup directly but also when using the scheduled backup.

Can people still having this problem check or confirm the following:

1) Have the backups files you've been having with downloaded through using a web browser at any point prior to restore?

2) If you unzip the file locally and then add .gz to the filename and unzip it again (as nikosnikos described in comment #81) do they work?

There is a problem with certain setups of mod_deflate/mod_gzip and certain browsers where an already compressed file can get compressed again upon download. I'm wondering if this bug is the source of the lingering issues.

Also: People having the gzip corruption issue:

Can you check if the file is still corrupted if you save the file to the manual backup directory and then download it via FTP or similar?

double-posted

#84

1. They were downloaded with both firefox and chrome.
2. If I unzip the file, re-add the .gz extension and re-gunzip the file it works correctly.

BTW, I'm using nginx. If I fire up apache server for the same website, it doesn't give me these issues at all.

nginx.conf:

    gzip on;
    gzip_buffers 16 8k;
    gzip_comp_level 1;
    gzip_http_version 1.1;
    gzip_min_length 10;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-font-ttf;
    gzip_vary on;
    gzip_proxied any; # Compression for all requests.
    ## No need for regexps. See
    ## http://wiki.nginx.org/NginxHttpGzipModule#gzip_disable
    gzip_disable "msie6";

@heddn

Thanks for checking into this. That is great info. B&M SHOULD be sending a MIME type of application/x-gzip and nginx/mod_deflate SHOULD be skipping the compression on these but somehow this is not happening.

Do you have the means to check the headers of the downloads to see if the mime type is correct? This can be a bit of a pain because you have to be logged in (valid session cookie), and your browser will trigger a download so the web inspector doesn't work. What I did was installed this chrome extension (https://chrome.google.com/webstore/detail/http-headers/hplfkkmefamockhli...), right click the download link, copy the url, paste it in that extension window and it shows all the headers.

I'd love to be able to get to the bottom of why this is happening, but I have not been able to reproduce it consistently.

I think what I might have to do is be tolerant of double-compressed files (ie: decompress twice if necessary). This isn't ideal because those files won't necessarily decompress via other means, but at least the backups will work.

I am still interested in other people's findings so weigh in on this thread if you have more info for me.

It's a lot easier than the chrome extension. The built in chrome Developer Tools let you do it too.

Request URL:http://example.com/admin/config/system/backup_migrate
Request Method:POST
Status Code:200 OK
Request Headersview source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:en-US,en;q=0.8
Cache-Control:no-cache
Connection:keep-alive
Content-Length:238
Content-Type:application/x-www-form-urlencoded
Cookie:SESS5619dc50d0a69c3afbfe1080fa4a3582=vZoqSW0hRObySvmM_-LBUFSF5MkNTyA4cQcExoRGNVM; __unam=40f2873-13e4792a270-23619d58-1; Drupal.tableDrag.showWeight=1; ctools-collapsible-state=views-ui-advanced-column-site_search%3A1; SESSbdd9229e7cb8a5fec3f2cab38f2a7954=3ecF6XBpAQO5EslEx_rSmebJBWDFk3ArQxQuPEwXBys; has_js=1
Host:example.com
Origin:http://example.com
Pragma:no-cache
Referer:http://example.com/admin/config/system/backup_migrate
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22
Form Dataview sourceview URL encoded
source_id:db
destination_id:download
profile_id:default
op:Backup now
form_build_id:form-4kqtS6JQ5qPsl-jrKMp2LW-DfJpNZu2Ezxh5AZj6mEY
form_token:2FxICtTkuuhdSTGNYeiU7WFRx4mSxd69l2voK6sLXxQ
form_id:backup_migrate_ui_manual_quick_backup_form
Response Headersview source
Cache-Control:no-cache
Connection:keep-alive
Content-Disposition:attachment; filename="localhost.dom-2013-05-08T12-14-14.mysql.gz"
Content-Encoding:gzip
Content-Type:application/x-gzip
Date:Wed, 08 May 2013 16:14:15 GMT
ETag:"1368029654"
Expires:Thu, 01 Jan 1970 00:00:01 GMT
Keep-Alive:timeout=10
Last-Modified:Wed, 08 May 2013 16:14:14 +0000
Server:nginx
Transfer-Encoding:chunked
Vary:Accept-Encoding

I did some tests today an found out, that the cause of the double gzipping of the backups must be the download with a browser.

If I did a "quick backup" with the destination "download" the backup-file was double gzipped.

If I did this backup with the destination "FTP Server" or "Manual Backup Directory" the backups were correct after download with a FTP client.

But if I do the download through the Web-Browser (Backup and Migrate > Destinations > MyFTP-Server > List Files > Download) the downloaded file is double gzipped.

@smitty, @heddn: Thanks for checking this out. The only way I've been able to reproduce this locally is to specifically include gzip files in the list of files that mod_deflate attempts to compress. I've made a change in dev that sends a 'Content-Encoding' header for gzipped files which (on my system) tells mod_deflate to leave the file alone.

Can you guys test the latest dev when it rolls sometime today and let me know if the problem still exists?

Thanks

I couldn't test sucessfully. Even when I removed the recent change for 'Content-Encoding', 'value' => 'gzip', I am getting a malformed zip file. It would appear that the dev version is non-functional for me. Maybe someone else will have different luck. The error I get is unexpected end of file. I get the error even if I select zip rather than gzip.

@heddn:

Can you test with the 'Content-Encoding' header in place but the 'Conent-Length' header removed?

Thanks

Tried it with 'Conent-Length' header removed. I was able to create a working gzip file, but the browser still double compresses it.

Ok, looks like this change is a bust on nginx at least. I've rolled it back and attached it as a patch here. If any apache users out there are having double compression issues please try one of these patches (for d6 or d7) and let me know how it works.

using 7.x-2.7

zip works, gzip no

When restoring to local dev from live site using gzip get error below. The other direction works fine with gzip.

local dev = XAMPP v3.1.0
live = cpanel v11.38.2 build 6 (apache 2.2.22, php 5.3.16, mysql 5.1.68-cll, linux

---------------------------

Type php

Date Monday, September 23, 2013 - 8:47am

User admin

Location http://localhost/*/admin/config/system/backup_migrate/restore?render=ove...

Referrer http://localhost/*/admin/config/system/backup_migrate/restore?render=ove...

Message PDOException: SQLSTATE[HY093]: Invalid parameter number: mixed named and positional parameters in backup_migrate_destination_db_mysql->_restore_db_from_file() (line 201 of C:\xampp\htdocs\*\sites\all\modules\backup_migrate\includes\destinations.db.mysql.inc).

Severity error

Hostname ::1

Hey, we started experiencing this issue on 7.x-2.x-dev since we migrated to another server.
So this might be related to some server specific configuration.

I'm fairly certain it is server specific. I just haven't figured it out yet. Perusio's nginx config seems effected but omega8's doesn't. That's as far as I've gotten with with the limited time to troubleshoot.

@netol: Are you using apache or nginx on the new server? Is mod_gzip or mod_deflate on? Can you try the patch in #96 and see how it works.

The D7 patch above does not fix the issue.

Our Apache server is running mod_deflate.

@netol: Are there any obvious differences in how your two servers are configured? If I could reproduce this locally I'd have a much better chance at solving it.

R

I'm not seeing any obvious differences so far, I will have another look.

Is anybody who is having this problem using an inexpensive commodity hosting company? Something I could set up quickly to try and reproduce this without spending a bunch of time and money?

Would anybody be willing to let me have access to a web root on a server they know to have this problem?

I am at a loss as to how to reproduce and fix this.

Also: Can apache people please try the patch in #96 and weigh in? That patch works in the only test I've been able to reproduce. If that's a fix for apache I might move forward with even if it doesn't fix the issue in nginx.

R

Just to remind that in my site (which runs apache) patch #96 did not solve the problem.

I tried setting up for you ronan but...

When testing to see if I am able to replicate the issue on the Inexpensive Shared Hosting server (our live-test environment), I cannot replicate the issue, everything works great.

cPanel Version 11.38.1 (build 15)
Operating system linux
Kernel version 2.6.32.59-sg3

GZIP transfers from Live-Test to Local Dev with no issues.

-----------

When testing to see if the problem still exists from Live Site to Local Dev, the issue still exists. All have the same code base and database. We are using a common code base for all sites here https://github.com/lsteam/unicron

I'm unable to give access to this server where the issue can be replicated. However, this server has different and more specs displayed compared to inexpensive shared server.

cPanel Version 11.38.2 (build 6)
cPanel Pro 1.0 (RC1)
Apache version 2.2.22
PHP version 5.3.16
MySQL version 5.1.68-cll
Architecture i686
Operating system linux
Perl version 5.8.8
Kernel version 2.6.32-358.2.1.el6.i686

---------------

Everything works fine from local dev going up to both places. The only issue is the semi-dedicated server coming down to local dev (XAMPP v3.1.0)

I just use ZIP in this case and everything is good.

attached is visual of error

below is the only log response

Type php
Date Wednesday, October 2, 2013 - 9:34am
User admin
Location http://localhost/*/admin/config/system/backup_migrate/restore?render=ove...
Referrer http://localhost/*/admin/config/system/backup_migrate/restore?render=ove...
Message PDOException: SQLSTATE[HY093]: Invalid parameter number: mixed named and positional parameters in backup_migrate_destination_db_mysql->_restore_db_from_file() (line 201 of C:\xampp\htdocs\*\sites\all\modules\backup_migrate\includes\destinations.db.mysql.inc).

I have an affected server (running Apache) that was exhibiting this behavior. Applying the patch from #96 has resolved the issue. So here's one RTBC.

This solved the issue on both my local and staging servers at work. My local runs Ubuntu 13 with Apache, and staging CentOS 5.4 with apache. The issue only surfaced a few weeks ago, so not sure what caused it, perhaps when I updated to latest version.

Ok, I've reapplied #96. This should hopefully fix the issue for Apache users.

nginx users: if you're still having issues, let's open a new issue so we can see if we can figure it out.

I just updated to 7.x-2.8 which seems to have this patch in it and I'm sorry to say this still hasn't fixed it for me with Apache 2.2.22 + Ubuntu 12.04

Anyone else that can confirm if this is (not) working in the latest stable release?

I'm not using the last stable release, but as written in #106, the proposed patch/solution did not work for me

Folks,

@ronan, thanks for the latest stable release.

@ronan and @Leetaq,

I would be happy to test. Can you please summarize here:
1. What exactly you want me to test... this issue is so long I'd rather just hear from one of you what precise steps you want us to do in the test(s).
2. What exactly do you want to know about my environment (e.g. http server and version, etc)

Thanks to all who have worked on this tough issue.

Shai Gluskin

Hi Shai,

Thanks for helping out. Frankly I don't know what info will help me get to the bottom of this. The only way I have been able to reproduce this issue is to misconfigure mod_gzip so that it attempts to re-compress .gz files. This is not standard configuration and none of your servers should be set up like this. Nonetheless, when I did this on my local, it exhibited the same behavior as you guys are describing. The fix I added to the latest version fixed it for me. It also fixed it on one of our production servers that was exhibiting the issue.

I guess at a minimum I need:

Server Software (Apache, Nginx, etc.)
Version
If mod_deflate, mod_gzip is installed.
What mod_deflate/gzip settings are in php.ini and .htaccess etc.
What browser you're downloading with.

Maybe we can find a pattern there.

Thanks all,
Ronan

Also:

If anybody would be willing to give me ssh/ftp access to a test account on a server which is exhibiting the issue, that would really help out. Use my contact form to get in touch with me if this is doable.

In reference to #107&108, when restoring from live to local using ZIP, there is a consistent visual issue of two buttons not showing. See image, the two arrows point to the missing buttons. The data restores fine but the page doesn't seem to want to display those buttons. This may or may not be related to the gzip issue.

https://drupal.org/files/issues/restore-zip.gif

Wish could give access to live site but can't. It is the only place for which the issue originates.

I just found it on one of my sites, and it's consistent with the description: the backup file is getting gzipped twice.

One confusing element is that if you expand sitename.mysql.gz using Mac OSX's GUI tool, it will expand both layers of gzip silently. That makes it LOOK to the user as if everything is OK.

On my local ubuntu though, I noticed that even after expanding the file and losing the .gz extension, Nautilus still detected it as a compressed file. gunzipping it a second time made it work with backup_migrate.

On the server which generated the file:
* apache2, up to date installation on centos 6.5 (blackmesh standard config, nothing funny)
* mod_deflate is installed
* mod_gzip is not installed

I'm downloading with Firefox latest.

Contents of /etc/httpd/conf.d/mod_deflate.conf:

<IfModule deflate_module>
  AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css application/x-javascript text/javascript
</IfModule>

I hope this is helpful... I can't give you direct access to the client server, but I'm happy to do any testing you like.

@ohthehugemanatee Thanks this is helpful info. Sounds like you're on Blackmesh so I may be able to do some debugging on a Blackmesh environment I have access to.

Can you confirm you are using the latest 2.x dev and still having the problem? Specifically I want to make sure that this commit:

http://drupalcode.org/project/backup_migrate.git/commit/2a730f36c02ea056...

Didn't fix the issue for you.

Thanks
Ronan

I have a similar problem. My backups restore properly on my local machine and Bluehost (my provider) but the same backup does not work on my client's server who is Westhost. Here are the specs on the servers:
Westhost (Restore doesn't work):
Apache version 2.0.52
MySQL version 5.0.27
PHP version 5.2.14

Bluehost (Restore works):
Apache version 2.2.29
MySQL version 5.5.38
PHP version 5.4.32

My local machine (Restore works):
Apache version 2.2.26
MySQL version 5.5.9
PHP version 5.2.17

I was able to work around this problem by using phpMyAdmin and importing my backup directly into the database.

Several people have mentioned that the backup file is getting gzipped twice. At least for the "download" link in the operation section on the "Saved Backups" tab, I'm seeing quite different behavior. As expected, the backup file on the remote server is gzipped once -- i.e., if I download it directly using FTP, it is a valid archive file, and when unzipped, is a valid MySQL file. But if I download it using the "download" link on the "Saved Backups" tab, then the file downloaded has the extension ".mysql.gz" but it is not a gzip file; rather, it is the non-zipped MySQL file.

If the actual location of the download file is sites/default/files/private/backup_migrate/manual/SiteName-2014-11-11T10-55-45.mysql.gz (for instance), then the URL of the download link is admin/config/system/backup_migrate/settings/destination/downloadfile/manual/SiteName-2014-11-11T10-55-45.mysql.gz. I'm not sure what the module is doing at settings/destination/downloadfile/manual in preparing the file for download.

@ronan: Earlier you asked for this information:

• Server Software (Apache, Nginx, etc.) = Linux 2.6.32.59, Apache 2.2.23, PHP 5.3.29, MySQL 5.0.91, Zip 1.11.0, Libzip 0.10.1, BZip2 1.0.3
• If mod_deflate, mod_gzip is installed = not installed
• What mod_deflate/gzip settings are in php.ini and .htaccess etc = N/A
• What browser you're downloading with = Firefox 33.1 (the latest as of this writing)

I also found some strange behaviour in the "download"ed files. .gz files donwloaded via FTP are OK, but when using the "download" link, seems to be a double-compressed file.

I also found this patch:

https://www.drupal.org/node/2129019

and it also solved this problem - and another, with an "illegal string offset" warning.

Should the two patches be merged in one?

@ronan I just ran into this problem again on another site I maintain, so I got to test your commit. It fixed the problem for me! The downloaded file is only Gzipped once. THANK YOU for your hard work and persistence on this long-standing bug!

I have the same problem with version 7.x-3.1, but only for offsite backups (hosted by NodeSquirrel). The local copy is just fine.

I also have the same problem with 7.x-3.1 when backing up to download.

Like @mjross in #125, we're getting the downloaded file labeled with .gz but actually uncompressed in 7.x-3.1. If I comment out

    if ($this->mimetype() == 'application/x-gzip') {
      $headers[] = array('key' => 'Content-Encoding', 'value' => 'gzip');
    }

it works fine. But here's a clue: on server #1, Chrome causes the problem, Firefox does not. On server #2, both Chrome and Firefox have the issue.

Info for @ronan from two different server setups:

1. Server Software (Apache, Nginx, etc.)
   #1 CentOS 5.11, Apache 2.2.16, PHP 5.3.3, MySQL 5.1.58, Zip 2.31, BZip2 1.0.3
   #2 CentOS 7.2.1511, Apache 2.4.6, PHP 5.6.17, MariaDB 5.5.44, Zip 3.0, BZip2 1.0.6
2. If mod_deflate, mod_gzip is installed = Deflate yes on both, gzip no. They shouldn't be run together, right?
3. What mod_deflate/gzip settings are in php.ini and .htaccess etc = Both have
   

       # Serve correct content types, and prevent mod_deflate double gzip. 
   RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1] 
   RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

4. What browser you're downloading with = Chrome, Firefox

Let me know if there's anything else I can test, since I have this weird variation between servers...

Patch in #96 seems to be the cause of files being decompressed when downloaded through certain browsers. At least that is the direction the discussion is going over in #2494009: compression not happening on database backup. I don't have time for a fix right now but hopefully it will be helpful to connect the dots...

I can confirm downloading .gz from current Chrome breaks the .gz. In Firefox the .gz works as expected

B&M.module 7.x-3.0

Chrome 56.0.2924.87 (doesn't work)
Firefox Developer 53.0a2 (2017-02-27) (32-bits) (works)
Firefox 43.0.1 (works)

I'm having the same problem unzipping the .gz files directly on the server...is this the same issue? Please tell me my backups aren't corrupted and useless?!?

Confirmed Gzip compression breaks in Chrome. Firefox is working fine.

Backup and Migrate 7.x-3.1
Chrome 56.0.2924.87
Firefox 46.0.1

As far as I know, the problem is that Chrome simply UNCOMPRESS those .gz files while downloading. So: on my server just they stay compressed. If I download them via binary FTP, I can uncompress them on my pc (winzip, 7-zip or similar). If I download them with Firefox, it's the same. If I download them with Chrome, the resulting file is already uncompressed, but with still the original name. e.g.
mydatabase.mysql.gz. If i open it with a text editor, I can see the Mysql content :-) Anybody could confirm?

Is it possible that the problem is related to the "Accept-encoding" directive in the standard Drupal .htaccess?

Would it be possible for developers with this problem to transition to the 3.x branch and see if that helps? The 2.x branch has not received a code update in quite some time, but a new 7.x-3.2 version was released September 27, 2017. Also be looking for a new stable release of D8 Backup and Migrate coming soon.

@DamienMcKenna has proposed that the 7.x-2.x branch be marked as unsupported once the upgrade path to 7.x-3.x is verified. See this issue: Verify 7.x-2.x -to- 7.x-3.x upgrade path, mark 7.x-2.x as unsupported

In order to reduce the maintenance burden the 7.x-2.x branch should be deprecated. In order to do this the upgrade path to 7.x-3.x has to be verified and any issues fixed.

Reverting back to postponed because the bug may or may not still exist with the new 7.x-3.2 release.

Updating version.

I can confirm the bug still exists in 7.x-3.2.

Thank you @hcethatsme.

I also can't unzip this file using this command on terminal on my macOS High Sierra 10.13.3

gunzip site.com-2018-03-01T14-26-27.mysql.gz
gunzip: site.com-2018-03-01T14-26-27.mysql.gz: not in gzip format

Drupal core 7.58
backup_migrate 7.x-3.5 (version used to create the gzip backup)

Bug still exists in version: 7.x-3.5

Drupal 7 is EOL. This issue will be closed.