When publishing a platform/site on a remote web server, any uploads that happen are stored on that server.
The backup tarball used to migrate however, is made on the hostmaster server. This means those file changes
aren't contained in the backup.

When you then migrate, the new sites directory will be restored from the incomplete backup, and the original site directory
on the remote server will be removed. Meaning those files will go missing.

Possible solutions :
1. Store files directory out of the sites directory, using either a symlink or private downloads.
2. All servers do the same, including hostmaster. This will mean that the files are no longer in the backup tarballs, which will improve speed.
3. Synch changes (to file dir only) down from the front end servers whenever we make a backup. (might be slow)
4. Use some combination of the above with a shared filesystem mount. 3 would conflict with this.

Comments

omega8cc’s picture

We have many users with files directory over 1 GB and mirroring it between remote servers and hostmaster is probably not a good idea. It also makes even simple site migration between platforms on the same server very resource intensive and ends up with duplicated GB of data in backups created by Aegir.

I strongly support the idea to exclude files directory from backups and to keep (and move - yeah, that can be slow) files with site when it is migrated between servers (which is rather obvious requirement).

Excluding files from backups is not a problem (and anyway, it is rather sysadmin level job to maintain filesystem backups), but how to keep it under hostmaster control so migration between remote web heads will still work?

Maybe hostmaster should maintain backups remotely? [EDIT: what a stupid suggestion, sorry, just forgot we don't have provision on the remote servers] Also, how it is supposed to work - how hostmaster will keep track of modules and themes uploaded on the web head to compare versions of modules (re-verify) to prepare migration task, even between platforms on the same remote web head? I must admit I didn't try it yet, so I'm not sure how is it done now - are all remotely done uploads synced back (mirrored) to the platform and sites code (core + contrib) maintained on the hostmaster server?

adrian’s picture

it doesn't pull code changes back. It's also not possible to do that reliably, especially when web clusters get involved, because changes can clobber each other.

All changes to the code need to be made on the hostmaster server and then pushed out to the remote server. So the process is 'update the platform on hostmaster, sync changes to remote'

Back on the issue :
For now i'm just going to add the rsync fetch during the backup routine, to make sure all the file dir changes on the remote side are present locally.

I do however think we need to move file storage out of the backups eventually.

omega8cc’s picture

So it is not only about files directory, but also about any modules/themes uploaded on the remote server to sites/domain.com. Migration based on backups created on hostmaster will just break remotely hosted sites completely (not just files will be lost) if there are modules/themes stored/uploaded there, and the problem affects both sites/domain and sites/all - just anything uploaded and used on the remote web head.

omega8cc’s picture

If the code has to be maintained only on the hostmaster (there is no remote control, just push the data and config), so how the user can maintain anything on the sites custom level (sites/domain.com) when FTP access will make sense only to the hostmaster server (and every change in modules/themes would require site re-verify to rsync the code?), but the IPs will be different, for obvious reasons? It looks like using remote server breaks possibility to maintain and track changes (on site verify task). This reduces the entire idea of remote servers to just "closed" web/db heads, and doesn't allow to customize the sites the way it is expected by users - they no longer can modify the code (in sites/all or sites/domain.com) after the site was deployed? Is it expected result?

adrian’s picture

The 'master copy' of all the files is on hostmaster.
They can modify the code on hostmaster and push the changes out.

people uploading the files to the web head via drupal dont have the same option.

omega8cc’s picture

Ah, so I probably over interpreted the "remote server" idea. And yes, it is expected the remote server will be just a "static" web/db head since there is no provision there. I'm still thinking about it the old way, probably, when provision was expected/required on all servers.

adrian’s picture

yeah. there's no remote 'masters'.

doing it any other way requires a lot more setup, such as shared mounted filesystems and so forth.

jerodfritz’s picture

Could an rdiff be performed prior to each push that prompts further action from the user to make the decision to sync back those changes from the remote server(s). Possibly a choice of which remote server in the cluster to sync back to hostmaster or to just blow them away and go with what is on hostmaster?

adrian’s picture

so the first part is fixed in head.
it pulls the files down and tars them up.

the new problem is that we run into a permission problem when attempting to push them back out again (during verify etc).

The files are uploaded belonging to www-data, when they get pulled down they become owned by the aegir user.
During verify we set the optimal permissions for the files, and when we try to synch them up again, it gives us a permission error,
because it can't change the user permissions for the file on the remote side.

adrian’s picture

a few possible solutions to the perm problem.

1) ensure that the files get created with the permissions in the first place
2) figure out how to not let rsync change the user permissions.
3) don't change perms on the files (but we still need to on the dirs, to ensure they are good)

adrian’s picture

Status: Active » Fixed

this is fixed in head

i solved the last part of the problem by no longer recursively changing permissions in the files directory.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.