Closed (fixed)
Project:
Provision
Version:
6.x-1.1
Component:
Code
Priority:
Normal
Category:
Feature request
Assigned:
Reporter:
Created:
10 Dec 2010 at 02:11 UTC
Updated:
1 Jul 2011 at 04:11 UTC
Jump to comment: Most recent file
The commit: http://gitorious.org/aegir/provision/commit/3fc60fdd553e1e0358c74ea414b4...
Creating those subdirectories by default with correct and safe permissions 02770 resolves many issues with sites imported to Aegir.
| Comment | File | Size | Author |
|---|---|---|---|
| #8 | hosting_folders.patch | 666 bytes | EugenMayer |
| #9 | 587044a8237afab4653998cebce0e5e803a9987f.patch.txt | 4.06 KB | omega8cc |
| provision_drupal.drush_.inc_.patch | 2.32 KB | omega8cc |
Comments
Comment #1
anarcat commentedThis sounds like contrib stuff, and anyways: don't those modules create those directories by themselves? Why would we need to do this?
Comment #2
omega8cc commentedAs mentioned above, creating those commonly used subdirectories helps with imported sites. This is not an issue for new sites, but when the site is imported and user is uploading some existing files there (with some weird ownership/permissions), he will break the site and Aegir clone/migrate will fail due to wrong permissions set there. This is just a way to improve user experience and avoiding problems. The patch is a results of 50+ support requests so it is based on a real experience.
Comment #3
omega8cc commentedBTW: the patch is now outdated, since we no longer use
chownthere.Comment #4
steven jones commentedNeeds work then?
Comment #5
omega8cc commented@Steven
I'm using updated patch, but I assume it will be not included anyway, so I added #3 just FYI. Not sure if I should re-roll it here.
Comment #6
anarcat commentedPlease reroll.
Comment #7
EugenMayer commentedBefore you reroll omega8cc, wouldnt it make a lot of sense having a configuration section ( UI )? I mean this would be a textarea, one folder on each line. Those folders will be created on every platform this aegir rolls out.
I think it will not make this patch more complicated but useable much more widely. Any ideas?
Comment #8
EugenMayer commenteddeleted
Comment #9
omega8cc commentedThe contrib module idea looks interesting, but I'm not sure how it will work with remote sync when using overrides also for sites etc. Just needs testing probably.
In the meantime my patch rerolled: http://gitorious.org/aegir/provision/commit/587044a8237afab4653998cebce0...
Comment #10
EugenMayer commentedwell my patch does not work right now, because i actually got the context wrong. neither can call module_exists nor any module methods... its not a drupal bootstrap i guess.
Comment #11
EugenMayer commentedremove #8, moving delelopment into private repos
Comment #12
anarcat commentedomega8cc - could you rebase your patch from HEAD? Right now, that commit is related to the previous one... I don't think it's a big issue for me to review, but in the future it could make things harder for other reviewers.. (unless that's the way git review work now on drupal.org? :)
Comment #13
omega8cc commentedAh, sorry, I did it the wrong way probably, so it could work as a patch only, will try again.
Comment #14
anarcat commentedI am not confortable committing this as a complete fix - we need to figure out why importing (deploying?) sites will not respect g+s or set proper permissions. If it's only on import, maybe that's just a documentation issue. However, if this is reproducable on migrate, then we have a problem.
Note that this is also an issue for profiles that create files in the files dir, as we're running as the aegir user, see #1059006: /files/ctools/css permissions broken for an example of that.
I'll commit the workaround for now and will open a new issue for the deeper issue, if I can figure it out.
Comment #15
omega8cc commentedWell, it seems it is now a half-baked patch, because I failed (sorry!) to rebase to head before rerolling it after we removed the chown there, so now it requires another patch:
http://gitorious.org/aegir/provision/commit/88b2ca077436eaf5945d9ce3f528...
But it is nothing critical, and still it is just a dirty, but working workaround to all permissions related issues in the files directory. I'm not even sure if we can do anything more without requiring system level umask to be set to 002 (and even this will not help for files extracted from tar.gz archive, so there is still something to be explained in docs for imports.
The system level fix I'm using is mentioned here: http://drupal.org/node/908524#comment-4086544, yet, our current logic for setting permissions on files subdirectories (with predefined array of subdirs excluded from recursive actions) is probably wrong, as permissions for the files and directories there after some clone/migrate actions are a total mess, causing further issues and it is something we need to re-think probably.
Opening new issue is a good idea, I set this one to needs review to keep this patch waiting for rc2 maybe.
Comment #16
EugenMayer commentedMigrate:
- Backup Site + Files to the master
- Extract on the master into platform
- rsync to the remote
Should be too easy to undestand, where issues can come from. And i fully cover omega8cc`s opinion that the current approach does not work out for more rich features sites. Beside the permissions issue there is other one.( copying ALL files back to the master, backuping them then and then rsync them back) ... well works while you have ~100MB of files, or you migrate every half an year or you simply dont care. But if you deploy into customer-networks or even hit the range of 5GB of files, you will have to rethink this. Esp. if you host more then 10 sites.
- the backup size will be, as most files are already kind of compressend ( png, jpeg, odt... )~70% of the original space
- Extracting the files, 100% of the space
That means you nearly need
- (0,7 * [times migrated] + 1 ) * numbers of sites
Of space on the aegir master _all the time_. Not talking about all the CPU cycles for all the tgz`ting of all those files arround. Not to mention all the permission issues due the umasks on the master or the simple traffic and bandwith use.
--
That all works out with smaller numbers of customers or smaller sizes of files on simple blogs / websites and stuff. But AFAIK you guys do neither plan to have smaller amounts of customers nor host websites. I have seen all of you offer atrium e.g., which will grow the sizes above pretty fast, as people will start using it as a document reporsitory pretty early.
The current handling of file is a deal-braker in my eyes. The kind file-permissions are handling is fine for most websites, but i dont think that is the primary group of audience you want to deal with when thinking about aegir.
The way aegir is extendable is wonderful. The way you can create new task, base them on others or use the pre / post hooks for chaning + the way you can add those thing to the drush context is very powerful, so it is not hard kind of rework those things without creating too much work. Iam actually nearly done having a new migration task, based on the old one (which is stil there), dealing with those things above.
I also think it is worth it, because most of the other things in Aegir scale up very well.
Comment #17
anarcat commentedpatch from omega8cc cherry-picked to head and provision-0.4.
Comment #19
omega8cc commentedAdd support for D7 specific files/styles (imagecache in D6) and files/xmlsitemap directories
http://drupalcode.org/sandbox/omega8cc/1074910.git/commit/8493efb
Comment #20
cashwilliams commentedI can't make up my mind it this is a good idea or not. I can see how this would be nice, but it seems like it will complicate things, not to mention the fact that an exception would have to be made for every contrib module that creates a folder. This exception list would have to be maintained.
Seems like this is more of a documentation issue, and there need to be a few steps outlined to 'prep' a site to be imported. Then once its is fully imported properly, everything should run smoothly from there.
I just don't like the idea of building special cases into Aegir itself. Am I misunderstanding the issue?
Comment #21
omega8cc commentedIt is not only to better support imports. It is also to create workarounds for some known issues also for sites install - example: try to install Open Public without the patch from #19 to see what happens.
Of course we could just send patches upstream, in this case to xmlsitemap module developer, but then: 1. It will take time (weeks or months or never) to get it accepted, 2. It will help only for newer installs.
So we are using workarounds here to help with some popular modules support in Aegir. I don't think this list will need to grow (maybe a bit only), as it already includes the most popular modules.
Changing the status back to needs review.
Comment #22
cashwilliams commentedI've just installed OpenPublic with no problem (aside from password issues #1092220: Allow Open Public to work as an Aegir platform), are you using the make file to generate it, or downloading the tar.gz?
Comment #23
omega8cc commentedI'm using tarball. Check the site status page after the install. It complains about missing files/xmlsitemap directory.
Also, I never had any issue with provisioning Open Public sites, so not sure about the other issue.
Comment #24
anarcat commentedI would rather see this fixed upstream. Maybe we can add this as a workaround, but simply refusing to lobby upstream seems wrong to me. The original issue was fixed here. If there is more contrib to support, open another issue or do it... in contrib. :)