generic import mechanism for external database dumps and site backups [#322788]

We are working on a system to allow users to upload database dumps and site backups manually through a site, to ease import of external drupals or just for testing.

The requirements are as follow:

* allow arbitrary sized files (through an FTP incoming directory for example) and uploads through the file browser
* allow simple SQL dumps or complete .tar.gz, specially-formatted backups
* restrict restore to files/ or modules/ or...
* security: check the archive for files outside the drupal sites/ root, run the SQL queries with the right user...

The latter will require some changes to the restore system, i think.

Comment	File	Size	Author
#7	provision_backup_import3.patch	2.9 KB	avoine
#7	hosting_backup_import3.patch	4.45 KB	avoine
#4	hosting_backup_import2.patch	4.45 KB	avoine
#4	provision_backup_import2.patch	3.03 KB	avoine
#1	provision_backup_import.patch	3.46 KB	avoine
#1	hosting_backup_import.patch	3.41 KB	avoine

Comments

Comment #1

avoine commented 17 October 2008 at 21:32

Status:

Needs work

» Needs review

Status	File	Size
new	hosting_backup_import.patch	3.41 KB
new	provision_backup_import.patch	3.46 KB

I attach the two patches, one for the hosting module and the other one for the provision module.

Comment #2

adrian commented 21 October 2008 at 14:55

i thought i had updated this the other day.

I like the feature, but i think that we should add the upload field to the restore page.

ie: choose this version, or upload your own.

I have concerns re: the incoming thing, especially when it comes to multiple servers / platforms. would like to discuss it more.

Comment #3

anarcat commented 28 October 2008 at 16:24

Status:

Needs review

» Needs work

This breaks platform verification:

Manual import configuration path exists.
Manual import configuration ownership of path could not be changed to hostmaster. (chmod failed on /var/hostmaster/incoming)
Manual import configuration permissions of path could not be changed to 511. (chmod failed on /var/hostmaster/incoming)
Manual import configuration path is writable.

That mode should be 1777 or the 1770 with group www-data.

Comment #4

avoine commented 27 November 2008 at 21:13

Status:

Needs work

» Needs review

Status	File	Size
new	provision_backup_import2.patch	3.03 KB
new	hosting_backup_import2.patch	4.45 KB

I worked more on the two patches.

Now, the upload form for a new backup is on top of the restore page. I also changed the mode of the incoming directory to 1777.

It's still primitive but it's working.

Comment #5

anarcat commented 28 November 2008 at 00:58

Status:

Needs review

» Needs work

Recent changes to the way the platform is configured makes this patch not apply cleanly anymore. Porting should be fairly easy however.

Comment #6

anarcat commented 28 November 2008 at 00:58

Status:

Needs work

» Patch (to be ported)

Comment #7

avoine commented 28 November 2008 at 20:02

Status	File	Size
new	hosting_backup_import3.patch	4.45 KB
new	provision_backup_import3.patch	2.9 KB

Those patches are suppose to be apply without problem.

Comment #8

anarcat commented 2 December 2008 at 02:28

Status:

Patch (to be ported)

» Needs work

So I can't get this to work.

I was able to upload a backup from my blog into a new site (new.test.hostmaster.koumbit.net). However, that backup doesn't show up in the list of backups to restore from. As it is now, that backup is sitting in the incoming directory, waiting...

So the problem seems to be that the "Incoming" tasks never gets created.

Comment #9

memtkmcc commented 14 June 2011 at 12:46

Version:	5.x-0.1-alpha3	» 6.x-0.4-alpha3
Priority:	Minor	» Normal

Note: it is already somehow listed on the 2.0 roadmap: http://community.aegirproject.org/roadmap/2.0#Standard_archive_support

There are already two working attempts to make this happen, but both are not enough generic to include them in the core:

Hosting Drupal Gardens import
http://drupal.org/sandbox/darthsteven/1178192

Hosting upload
http://drupal.org/project/hosting_upload

The Gardens importer is promising, but the UX needs to be improved (to remove the step with creating failed add platform task) and it could support also other known archive formats, like exports from WebEnabled, plus allow archives to be in a .zip or any other format other than hardcoded .tar.gz. Anyway, available already ability to create both platform and site is a huge step forward.

The Hosting upload module allows to create new platform from uploaded or remotely fetched archive, and has also plans to extend this to support managing modules and themes (which is beyond the scope of this feature request), but offers rather good UX, and doesn't require provision part, like the Gardens importer.

I imagine that both modules could be merged to offer all-in-one new feature - ability to import a platform and existing there site (or sites) in an elegant way, with option to either define the path, URL or accept browser upload.

Of course this would require some improvements in the Create Platform form, as I wrote already here: http://drupal.org/node/1144968#comment-4474304

Comment #10

omega8cc commented 14 June 2011 at 12:55

It was me - Grace, above :)

Comment #11

steven jones commented 14 June 2011 at 13:14

The Gardens importer is promising, but the UX needs to be improved (to remove the step with creating failed add platform task) and it could support also other known archive formats, like exports from WebEnabled, plus allow archives to be in a .zip or any other format other than hardcoded .tar.gz. Anyway, available already ability to create both platform and site is a huge step forward.

Ah I actually quite liked the UI, minimal code to maintain for me, and minimal UI changes for the user.
As for supporting other export formats, Drupal Gardens only does tar.gz so there was absolutely no point in complicating it further by supporting others.

Is the archive format in drush now then? If it is we should totally make this happen!