The is a difference between
a) your backend script service (ie : what drush is being run on)
b) your file service (ie: where your files are stored)
c) your web service (ie: where your files are being served to the public)

At the moment, the assumption is that all of these things are provided on the same server.
Hence when you set up a remote web server, you also have to enable the (currently misnamed) 'server: basic configuration', to be able to make the drush call and modify the files on that server.

The real interaction of all this is not so clear however. To successfully host a site you need to have each of these three, but it's not necessary for all of them to be the same server.

You could have multiple servers providing dumb http web heads, that are using a shared file system (ie: file service) to share the files, and are having the backend script run on a server that isn't even publicly accessible.

The nature of the file service, and how it differs from the backend/drush service is the crux of this problem. For Drush to work, you need to point it at a local (to it) copy of the drupal platform / sites directory. Which means that it will need to have full (local) access to what the file service should provide.

I suspect that the current provision.path.inc will end up forming the basis of the file service class, and that we will need to restructure our codebase so that all calls that modify the file system will need to go through this class (which means abstracting the config file generation into it too).

Because of how drush operates though, i suspect we will _always_ need to have the 'local' implementation of the file service available, so it will maintain the files in their pristine condition on the relevant script server. It will need to call an additional implementation of whichever required service it needs to trigger those hooks too (think rsync to all the right servers).

Comments

adrian’s picture

one thing that occurred to me is that we should consider dropping the ability to have multiple backends (for now).

The entire problem space becomes _drastically_ simplified when we make that assumption. The canonical platforms are thus stored on a single server , and each of the possible web heads gain access to them through one of the possible file service implementations (ie: through a mount of through rsync).

The requirement to be able to distribute and gain access to the backup files across various servers becomes a non-issue, because it's the backend service that needs access to the backup tarball, not the web heads. It turns the migrate between server problem on it's head, in that all that's changing is which web server is serving the site to the world outside, not where the drush commands are being run.

I'm still trying to figure it out, but i suspect that by focussing on the 'spoke model' described above and getting it properly implemented first, we will actually hit more than 90% of our actual use case requirements.

When we get to the point where we need to have multiple backends (mesh model), we will have reached the limits of the current "push" model of dispatching tasks from the queue, and we reach a situation where we _SERIOUSLY_ need to consider solutions such as REDIS/RESQUE to actually take advantage of the additional processing power.

drumm’s picture

Status: Active » Needs review
StatusFileSize
new55.49 KB

Here is a hopefully-straightforward patch for the filesystem interface. The whole ->path() construct is an $op anti-pattern, but that can be changed later.

drumm’s picture

StatusFileSize
new57.03 KB

New revision

drumm’s picture

StatusFileSize
new56.72 KB

A bit more work on this.

adrian’s picture

I think we can make this better, but i like the approach here ...

Specifically instead of the path(), we need to make it the actual method (for now).

so :

provision_service('file')->unlink($filename);

i wonder, if _later_ , we can have the methods return objects too. so you can do

provision_service('file')->chmod('filename', 0664)->success("message")->fail("message");
adrian’s picture

I think we can make this better, but i like the approach here ...

Specifically instead of the path(), we need to make it the actual method (for now).

so :

provision_service('file')->unlink($filename);

i wonder, if _later_ , we can have the methods return objects too. so you can do

provision_service('file')->chmod('filename', 0664)->success("message")->fail("message");
adrian’s picture

Status: Needs review » Needs work

I committed the patch here, so we're on the same page again for the following bit of work.

drumm’s picture

Thanks for committing, I started removing ->path() in the first patch, but didn't see any PHP tricks to make it really clean. I decided against changing to much for less debugging.

adrian’s picture

this is basically what the apache config stuff starts looking like :

http://gist.github.com/346501

it can still be made a lot cleaner, and doesn;t really work yet.

the idea is that each service implementation, has a class for each config file it implements, and this config file object knows it's template file, and has a method to determine it's final resting place.

what the delete_site_config etc methods will eventually do is make sure we pass the right object / data set into the object constructor (which we will get from the aliases).

I still need to figure out what to do with the variables associated with the config files..

for instance.. we need a backend service that initializes and verifies the global backup and config file directories.
Some of the stuff (ie: drupal_root and the drupal sites paths, also need to be moved out of the apache / http context, i *think*).

this stuff is evolving fast, and i hope to commit some wokring code int he next few days.

drumm’s picture

StatusFileSize
new4.31 KB

To remove the ->path() anti-pattern: the attached path adds ->succeed() and ->fail() constructs for all services and uses file's ->writable() as an example.

drumm’s picture

StatusFileSize
new44.47 KB

Attached is the patch to convert everything except fopen. That one needs a larger change to 'write this string to this path'.

adrian’s picture

i committed this because it's exactly what's needed.

onwards =)

drumm’s picture

StatusFileSize
new3.32 KB

Moving the settings.php writing to the new config classes, one less fopen.

drumm’s picture

Moving another configuration file. This one isn't tested, and not quite as clean, but I think the SSL branch will be a moving target anyway. Removes a bunch of provision.inc cruft.

drumm’s picture

StatusFileSize
new6.46 KB
drumm’s picture

StatusFileSize
new8.6 KB

3/4 done. global.inc moved over, which was straightforward.

drumm’s picture

StatusFileSize
new14.62 KB

Final patch for now.. moves all use of fopen to config classes and removes ->path().

adrian’s picture

committed the last patch.

need to merge into envobject next.

drumm’s picture

StatusFileSize
new14.13 KB

This one is general cleanup, rolling some permissions management into provisionConfig, and using provisionService_file to write out files.

drumm’s picture

Status: Needs work » Needs review
StatusFileSize
new15.52 KB

More cleanup to make site install, disable, and delete work cleanly.

drumm’s picture

Status: Needs review » Fixed

I think this can be called done in the dev-services branch.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 5b3da29 on debian, dev-dns, dev-envobject, dev-koumbit, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newhooks, dev-nginx, dev-ports, dev-purgebackup, dev-restore, dev-services, dev-simplerinstaller, dev-ssl, prod-koumbit, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-subdir-multiserver, 6.x-2.x-backports, dev-helmo-3.x by adrian:
    Committed path 11 from #736706 - thanks drumm
    
    This gets rid of the path...
  • Commit 36eb841 on debian, dev-dns, dev-envobject, dev-koumbit, dev-log_directory, dev-migrate_aliases, dev-multiserver-install, dev-newhooks, dev-nginx, dev-ports, dev-purgebackup, dev-restore, dev-services, dev-simplerinstaller, dev-ssl, prod-koumbit, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-subdir-multiserver, 6.x-2.x-backports, dev-helmo-3.x by adrian:
    Committing the config file patch from #736706 , thanks drumm.