Problem/Motivation

Aegir doesn't support having multiple sites in "subdirectories" (like example.com/foo and example.com/bar) yet that is a multisite feature that is supported in core and widely used in higher-ed institutions or organisations that do not have good control over DNS.

Having this feature would also make it easier to have multiple sites in a local dev environment as only one DNS record would be necessary.

Proposed resolution

There is a working solution in contrib: Mig5's Aegir Subfolders. The current plan is to incorporate this extension into core, as an 'experimental' feature.

From a front-end perspective, it simply adds a 'Subfolder path' field to the site form, which is both saved to the database, and passed to the backend to be saved in the site context (alias/entity). It also requires that we set an alias in the format "yourdomain.com.<subfolder>".

On the back-end, we create a stub.d directory in the /var/aegir/config area that is *not* pre-loaded/included in the server-wide context, but rather is included by our platform's Apache config at the right time. On platform verify, we inject a snippet that instructs the platform-level Apache config to include our special RewriteRules per server across this platform, *before* the .htaccess is included.

On site install (and verify and deploy), we stick an Alias one-liner for this site in a server-wide config. This can't go in the site's .stub file along with its Rewrite rules, because that .stub file gets Included within a directive, which can't contain Alias directives. We then symlink the subfolder name into the top-level of the Drupal root itself. We also insert our RewriteRule overrides per site into a .stub file which the platform-level Apache config includes.

When we disable a site, we inject a 'disabled site' rewrite rule into our stub so the site is disabled under its 'subfolder' URL. On site deletion, we delete the RewriteRule stub file along with the subfolder symlink in the platform root and the Apache mod_alias stub file.

This module has a number of documented issues we'll need to resolve along the way. These, along with the tasks required to merge it into core follow.

Remaining tasks

The tasks required for merging in 2.x would be:

1. Move subfolder.drush.inc into provision/subfolder
2. Move hosting_subfolders into hosting
3. Add a Hosting Feature for subfolders
4. Apply the included patch to clone.provision.inc.patch
5. Refactor provisionService_subfolder as an Aegir 2-style service.

Those tasks would be optional, in priority:

1. Perhaps add/improve validation of the 'subfolders' field (re. "2. Specify a 'subfolder'. No opening or trailing slashes")
2. Perhaps implement a full service that attaches to servers, which would get us automatic 'verify' tasks on all platforms.
3. Add support for renaming the subfolder when a site is renamed, or get to set a new subfolder for a site that is going to be cloned
4. Delete the old subfolder stub files when it's renamed, so have a concept of 'old' subfolder name vs 'new'.
5. Autocomplete the subfolder name based on the first part of the subdomain name
6. Autocomplete the 'example.com.prefix' Site Alias too.
7. Add Nginx support

User interface changes

(New or changed features/functionality in the user interface, modules added or removed, changes to URL paths, changes to user interface text.)

API changes

TBD.

None so far.

Original report by redndahead

Talking with anarcat on irc it seems aegir doesn't support creation of mysite.com/site1 and mysite.com/site2 type of sites.

CommentFileSizeAuthor
#6 subdir.tar_.gz3.49 KBjasonlttl
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

adrian’s picture

Priority: Normal » Minor

unlikely to be supported, because multiple sites end up sharing the same virtual host entry in apache (or similar), and you will be required to upgrade/migrate all of the sites simultaneously because /site1 can't be on a different drupal root than /site2, they can't be on separate servers ever. and so forth.

it's probably not even possible to do cleanly in aegir, but i'm leaving this open so others can come across this.

anarcat’s picture

The problem we have with this is twofold:

1. right now sites are mapped to vhosts, and the coupling is pretty tight.
2. you can't have sites http://example.com/ and http://example.com/site1 otherwise example.com can eat example.com/site1

The first problem is a technical limitation of Aegir. I believe it is possible to host multiple sites in subdirectories in the same vhost, and that they point to different docroots. The trick is to use the Alias directive:

<VirtualHost *:80>
  ServerName mysite.com
  DocumentRoot /var/aegir/drupal-6.15-1.0-prod/

  Alias /site1 /var/aegir/drupal-5.20-1.0-prod/
  Alias /site2 /var/aegir/drupal-6.87-1.0-prod/
</VirtualHost>

The trick here would be to *add* stuff to the vhost, which is a fairly annoying thing to do, but that can be done. We need locking and a good parser (Augeas lenses anyone? in php?). We also need to abstract the "vhost" in the frontend (so it's decoupled from sites) and probably in the backend too.

The second problem is more a matter of policy: do you want to allow the site on example.com to completely break sites example.com/foo or example.com/bar? If so, who can? The same client? The same user? If not, you need to implement checks so there's no overlap, and maybe a setting there.

I believe tools shouldn't dictate policy, people should. Therefore, I believe this should be allowed, from a strictly technical perspective.

The thing is: domain names are cheap, contrarily to common opinion. You only have to get *one* domain and you can have as many sub-domains as you want. Hell, you can even have sub sub domains and have your whole domain tree down there, you're free to go. Providers often give out a free DNS namespace to their customers based on their username (full disclosure: we do :P).

The other thing is: /whatever is exactly what drupal does. It dispatches menus callbacks based those GET URLs. So trying to make aegir (or Apache + Drupal, rather) do that is rather backward. On the other hand, Drupal is notoriously bad (if you make exception of the great Domain contrib module) at dispatching requests based on domains, and that's where Apache, DNS, Aegir and all their happy friends gets in. :)

In short, it's not worth it. Or at least, it hasn't shown its usefulness to anyone in the 2+ years of this (new phase of this) project. :)

If you want to do it, I think you have a good recipe here (hopefully), and it will actually improve our APIs (hopefully).

redndahead’s picture

Let me give you my use case. I work for a university and we would like to give out sites to our graduate students. So we would have a domain gradstudents.mysite.com so we would like gradstudents.mysite.com/student1
gradstudents.mysite.com/student2
etc.

There are other options, some sort of organic groups or domain access, but if I can manage with aegir that would be best.

alozie’s picture

Hi,
Just wanted to second rednahead's sentiments.

I also work for a university, where we already have several sites managed under a multisite core (using the aliasing as described above) set up as follows:

In the next year there we'll have many more such sites as we migrate existing department "sites" (subsections of our main site) out of our old CMS and into Drupal.

For a number of reasons over the past few years we've been moving away from the subdomains except where organizational (or technical) distinctions deem it absolutely necessary. As such I too would like to be able to manage the instances described above via Aegir.

I've setup Aegir on a local test instance and think it can manage the sites as long as I manually enter the aliases in the httpd.conf and create the rewrite rules in the Drupal core's .htaccess. The only things that Aegir would seem to get stuck on are the links to the site itself (see: view site and renew admin password links). And other than in those places it would seem to work okay. If I get the alias and rewrites working on my local instance, I'll update this ticket with the confirmation.

In the meantime, I'd still like to echo the sentiments above that the hosting scenario described is important enough to factor in for future revisions of this great tool.

jasonlttl’s picture

Mostly a me too post...

I also work at a university and my college is looking to move towards aegir (which I think is awesome). We have a lot of departments with research labs that fit cleanly inside their parent department.

For example:
http://deptx.myu.edu
http://deptx.myu.edu/labs/somelab

Why use subdirectory sites? I opted to make urls a communications decision instead of a technical one. Subdirectory sites were deemed preferable. Personally, I agree deptx.myu.edu/blah tends to look better than blah.deptx.myu.edu. Four level domains can be a little tacky. We also have some organizational issues which would make dns for blah.deptx.myu.edu difficult to manage.

Anyways, we're about to redo our environment and Aegir will be part of that. We might end up trying the recipe above, however as we're new to Aegir I suspect whatever we do will be moderately kludgy. We're also considering a reverse proxying approach, mapping directory subsites on the front end to aegir managed subdomain sites on the backend. I expect reverse proxying will be difficult (particularly with shibboleth which we use for auth), but at least then all the subdirectory sites could be managed normally in aegir.

If anyone gets either mode working, I'd be interested in hearing about the solution.

jasonlttl’s picture

FileSize
3.49 KB

Attached is a primitive first stab at adding same server subdirectory site support in contrib without any hacks to aegir. Since I'm new to aegir, and since this is kinda tricky, I'm sure this is not well written so I'm not suggesting this be committed anywhere. I've also just gotten this working in a development area back home (against head), so it's relatively untested. It worked with a coupe of simple test sites, but has not seen much testing yet. This post is mostly to get feedback.

In making this, I followed the cheese extension example. Here's how it works.

1) Every site must have a valid primary domain subdir.company.com.

2) In this format, add a secondary alias company.com.subdir (this will create an appropriate symlink in sites folder)

3) Go to the site node for company.com. You will see a subdirectory sites field group with a checkbox for subdir.company.com and any other subdirectory sites for company.com. Basically, it's scanning the alias tables for things that might belong to it. Check a site to include the subdirectory site.

4) When you verify or install company.com, it should automatically include apache alias and rewrite rules in the vhost for company.com. You probably need to make sure your platform does not have additional rewrite rules in the .htaccess (as it normally would). In addition, because Aegir writes the db credentials to the vhost, and we only have one vhost and multiple sites, I'm using the settings hook to add the db credentials back into the settings.php files.

Other notes:

The aegir team considers the db credentials in settings.php a security risk because all settings.php files are readable by all sites. This may not be a problem for you if you are really all just one org. And if it is, IMO, a larger problem is that everything is executing in the same security context with drush and mod_php.

My drush rc is undoubtedly convoluted. I wasn't sure what the right way was to get some of the info. It will probably make real aegir developers cringe.

There is no validation (same server, site alias changes, etc). There is no real security beyond the parent needing to approve the child site. Some changes will require verifying both child and parent sites.

--

Hope this helps. I'll likely keep working on this and post a final version at some point. Feedback is welcome.

anarcat’s picture

Status: Active » Needs work

Can you please provide a diff instead of a tarball, it makes it much easier to review for us.... :)

Also mark issues with "needs review" when you want such a review.

anarcat’s picture

Issue tags: +FAQ
bowersox’s picture

sub

ember_’s picture

Maybe this is just a university issue. I'll enter another me too! request for this feature from another University. Thanks for adding it to the FAQ, wish I had found it a couple of weeks ago.

skwashd’s picture

subscribing

izmeez’s picture

subscribing

mpearrow’s picture

Just adding a "me too"; we were evaluating OpenScholar, which allows for the creation of sites under one domain, e.g. blah.univ.edu/{site1...siteN}, which is useful for getting someone up and running quickly.

We are not able to modify our school's DNS programatically to create new hostnames on the fly, so having a sort of communal hostname with subdirectory site is a major use case for us. We will probably have a lot of users who will eventually want to upgrade their site's URL to a dedicated hostname (e.g. http://drupalstuff.mit.edu/lab1 would become http://lab1.mit.edu) but right now that business process is all manual.

kolafson’s picture

Adding another "me too".. our team is also building sites for a University where this would be very helpful. Currently only able to use Aegir for sandbox sites w/different subdomains..

kmasood’s picture

Another university vote for this feature. Looks like the need for this feature is gaining momentum ... hopefully it amounts to something (-:

lkjairath’s picture

Hello Anarcat and all Aegis Maintainers,

This is a must feature in such a versatile tool as Aegir. Without this feature it's just short of what its meaning signifies ...no parties for the god(desse)s :]

Many thanks in advance.

Lalit

ergonlogic’s picture

Version: 6.x-0.4-alpha3 » 7.x-2.x-dev
Priority: Minor » Major
Status: Needs work » Needs review

I've personally heard many more requests for this functionality in Aegir, and it appears to be blocking a significant segment of potential early-adopters (educational institutions). Also, since Drupal itself supports this functionality, this limitation effectively imposes policy, which goes againt our "tools, not policy" credo. So, I'm bumping this up to "major."

Anarcat lays out several issues that would have to be resolved in comment 2 (above). My, admittedly fuzzy, understanding of them basically break down to the following:
* we need to abstract the "vhost" in the frontend (so it's decoupled from sites) and probably in the backend too.
* we need vhost locking
* we need a good configuration parser (there's an Augeas PECL package, if that helps any)
* we need to determine a policy around site naming to determine whether these are controlled at the user or client levels
* we need to implement checks so there's no overlap (e.g., between example.com and example.com/site1), and maybe a setting there.

I'm not too clear on the description in comment 2 of those last couple. It seems to me that example.com/site1 would potentially clobber a section of example.com, and not the other way around (e.g., a site at example.com/admin would be disastrous for example.com). I think implementing the Path Blacklist module on client sites making use of this kind of naming might alleviate some of the potential problems.

I believe there's also a potential issue of (sub)domain squatting, if wildcard records are used (i.e., nothing stops another client from creating a site at example.com/spam or spam.example.com).

If that's a fair summary of the situation as it exists, I'll post individual issues for each, and turn this one into a meta issue to track progress across all of them, even if they end up at the bottom of the backlog for awhile. So I'm also setting this to "needs review" to get feedback.

ashooner’s picture

Yet another university +1 for this. While Aegir appears otherwise perfectly suited for our needs, it isn't likely that I'll be able to push through the massive amount of redirecting and general site organization policy changes that would be needed to work around this limitation.

anarcat’s picture

Status: Needs review » Needs work

Good analysis ergonlogic, and you are correct about example.com not being able to pollute example.com/admin, i may have gotten this backward in my original description.

It's funny you mention Augeas, because it's the engine now used in Puppet to parse/edit config files... I am not sure, however, we need to *parse* config files, although that would probably much better than the current approach. I am hesitant in adding such a dependency however, as it would make porting to other operating systems much harder, ie. i could make a package of php-augeas for debian, but not for other platforms - and it would therefore complicate the install.

Something to be thought of. Anyways, the bigger problem is not parsing config files but decoupling vhosts from sites, so that there's a 1-to-n relationship there...

millaraj’s picture

Another vote from another university. We currently have Aegir deployed and map all sites to a wildcard subdomain. We have a main webserver that proxies requests through to the correct domain for the request. We are coming across problems whereby URLs are not redirecting properly and forms seem to be broken as well. This could potentially be down to flawed implementation on our part, but with the addition of a custom module that attempts to fix the forms we seem to get it working for the most part (bit of a nasty hack though).

The other issue that I haven't seen flagged up here is how users log in (and are kept logged in) to each site. It follows that users would expect to be kept logged across a domain, but not neccesarily across multiple sub domains. So for example if our primary domain is example.com and we have a seperate drupal install at example.com/site1, a user would expect to be logged into site1 if they had already logged into example.com and vice versa.

From what I can tell we would have to do some kind of table sharing across databases?

We're finding that Aegir is a fantastic product that is helping us roll out many more sites than would otherwise have been possible and the addition of this functionality would almost feature complete it for us.

ergonlogic’s picture

If any individual, organization or group was willing and able to sponsor this development, it would help us (Koumbit, anyway) free up the resources to work on it. Please contact me directly if anyone wants to pursue this avenue.

Steven Jones’s picture

Version: 7.x-2.x-dev » 6.x-2.x-dev
webmaster-eddie’s picture

Hello,

I have a slightly different need, that might already be taken care of in Aegir, but I can't figure it out:

I just want to be able to build a new site off of a valid aegir platform - but I want the site to be built NOT on the root of the new domain name, but on a directory, so that I can do something else - something outside of drupal on the domain root.

For example, I want to be able to build a new site off of a valid aegir platform on domain.com/network/ - and I can't figure out how to do this. I don't need to build any other drupal site on that domain.

Is it possible?

Can anyone help me do this?

Thanks,

Eddie

kim-day’s picture

It's been a few months since the last post re this issue ... is anything happening around it? I work at a university too, we currently use drupal in a multisite config with subdirectories and I'm reaching the point where I need a better way to manage site creation, etc.

I just wanted to post letting others know I need this and am willing to help. Is anyone else working on a solution?

anarcat’s picture

No, nobody is working on this as far as I know. If you do, please review thoroughly the ideas i proposed in #2. There is also a tarball of code that has been uploaded in #6, which I haven't reviewed yet.

If you could look into this, it would be great as a lot of other people are in the same place as you right now...

Dustin@PI’s picture

We're another higher ed. group with a similar use case/need. And I know of two other Universities in our area who have looked at Aegir, but ended up coming up with their own drush automation instead due to the lack of support for sharing DNS/managing Drupal in a subdirectory.

We've been working on expanding the solution here: http://www.computerminds.co.uk/articles/storing-data-aegir. We're setting up the sites with the address on 127.0.0.1 via the hosts file and using the provision hooks to setup the necessary reverse proxy rules so that appears runs in a sub directory. It's kind of ugly but meets our needs until Aegir supports sites in directories. I'll share the project once we've tested it a bit more.

- I would be able to put a couple thousand from my budget towards adding this feature to Aegir 7.x (if the work was completed by September and billed by the end of the July)

- There are number of legitimate reasons for needing this feature, so I would ask the Aegir team to stop saying things like "it's not worth it" [from FAQ] and start saying: "it's a big change, so we'll need developers to step up or sponsors to complete the work"

bowersox’s picture

+1 for the suggestions from @Dustin@PI.

Here at Pixo we are helping the American Library Association move about 70 websites into Drupal 7 with a multi-site setup of this style: www.ala.org/alsc/ , www.ala.org/pla/ , www.ala.org/advocacy/ , etc.

We used a similar approach: aegir and drush, plus our own custom Apache configuration lines with a custom drush module using provision hooks. We add Apache configuration rules to route requests to the right Drupal site.

It sounds like everyone on this thread is reinventing the same wheel. We would be better pooling our resources. I love the suggested approach: "this is a big change, so we need developers to step up and/or sponsors to fund this work".

anarcat’s picture

I have changed the FAQ. We (the Aegir project) are not opposed to providing this feature, it's just not part of our use cases so far, so it was assumed that one site = one domain, which makes changing this pretty difficult.

Koumbit may be able to provide the workforce to factor this in, although I cannot commit on timelines or estimates right now. The basic thing here is that we have no need for this in our current use case, as we control our DNS and use wildcards. However, if funding is provided, we could divert resources towards that goal and would be happy to push this feature in 2.x.

I absolutely like the idea of pooling funding. I feel that a lot of universities and organisations need this, but we lack a common way of pooling those resources. Any ideas on how to do this? Kickstarter? The paypal button? :) Otherwise, Koumbit could be contracted by one organisation to just push this as far as we can with the provided funding...

Let me know, we (Koumbit) are interested.

kmasood’s picture

If the higher-ed folks on this thread (or reading this thread) are interested in connecting at DrupalCon 2012 in Denver, there is a higher-ed unconf on Monday March 19th ... http://groups.drupal.org/node/211118 ... perhaps we can pool our resources and get this project funded? What say you?

Vidus’s picture

Another university user here. I was extremely excited for Aegir and it making my life easier, but we'll have to totally rethink our way of doing things with our sites if we stick with it. This would make our lives 1000x easier if we could just get sub-directory sites.

Dustin@PI’s picture

@bowersox would you be willing to share the solution you've come up with? We aren't very happy with the hacks that we've had to do to deal with the reverse proxying ... (setting baseurl and applying patches for bugs around the user path)

We would be willing to test your solution and provide bug reports with patches. This could be a good contrib project and I would definitly be willing to provide patches for issues and new features.

Dustin@PI’s picture

@anarcat

Regarding a way to get this funded ... higher-ed institutions/universities (at least here in Canada) can't spend money speculatively so funding in a kickstarter type of way would be tough to get through our various financial governance structures.

... but spending money as part of partnerships/collaborations is very much encouraged. If we had a contract where 4 or 5 institutions got together to spend x dollars and we're responsible to pay 20% of the total, not only would that be good it would be looked on favorably (while circumventing some very important, but impractical in this scenario, bidding processes).

I would definitely be willing to help with such a partnership or any other funding model.

bowersox’s picture

@Dustin@PI, I'll try to do a blog post some time in the next month and share an example of some of the configuration files and tweaks we used to achieve this. Right now I'm getting ready for a Drupal Camp event and then traveling for an out-of-town conference, so I'm sorry that it may take a while.

If a group of universities are interested in funding this, we would be glad to explore the possibility of helping to build this enhancement for hire and contribute it to the whole community. We've done that with LDAP, Solr, Accessibility and other Drupal contributions.

-Brandon Bowersox-Johnson

anarcat’s picture

@Dustin@PI - if you have cycles to attack this problem, please feel free to organise towards it as you see fit. Our priorities at Koumbit do not include this project at this stage, but again if there is funding we can reconsider.

Anonymous’s picture

I've written a (crude) extension for Aegir called aegir_subfolders, which can be found at https://github.com/mig5/aegir_subfolders

It's somewhat ugly in terms of how it injects the relevant Apache bits (the 'Alias' attributes can't exist inside vhosts, and the rewrite rules have to be read in *before* the rest of the platform's .htaccess), but:

a) It works (!), including across migrations

b) It doesn't use any mod_proxy hacks, so no duplication of requests through the Apache server, yet still avoids any external redirects

c) Works for both Drupal 6 and Drupal 7

d) Doesn't require any modifications to the platform's .htaccess

There are some caveats (see the README.txt), such as no Nginx, and it's still a work in progress / no doubt buggy, but I welcome anyone who wants to start using it / fixing it.

Thanks to darthsteven too, since I based this off his cm_subdirectories extension, with a lot of modifications.

Dustin@PI’s picture

@ anarcat ... my apologies for not replying I have tried to pull together interest for funding this work with a couple of other institutions, but have been stymied by a lack of an estimate for the work. I have interest but not commitment with out realistic budget amounts.

Dustin@PI’s picture

@mig5 thanks for sharing this work. My team is booked up for the next two weeks, but after that I can spend some time testing. How do you want issues and patches reported?

anarcat’s picture

@Dustin@Pi - sorry for not providing estimates! ;)

It's up to mig5 to decide how he wants those reports, but feel free to use the issues queues here for communication. This should probably be a branch from 2.x.

Anonymous’s picture

Hey Dustin@PI: feel free to report issues here or in this specific ticket (or perhaps this can become a meta-ticket wrapper for the other ones if it turns out there are many bugs :)

Thanks!

andrew-r’s picture

Hi All,

I'm really excited by this feature, I've been playing with mig5's solution (followed the install instructions with aegir installed from debian packages on ubuntu 12.04) and I've found an issue (I'm new to aegir so it could just be my setup). The subfolder works well until I try to create an aegir managed site at the main domain. I think it might be that my subfolder site is getting it's db login details from the main site's vhost or just can't find it's own (is this how it works?).

I couldn't figure it out at first because both sites were on the same platform so i suspect I was seeing 2 multisites with the same db settings, but when I deleted and recreated the main site using a different drupal 6 platform, I noticed the subsite (drupal 7) was throwing a fatal php error because it couldn't find the mysqli database includes. The main site has db_type set to mysqli.

I've set up my sub folder site at domain: "subsite.test" path: "test" and alias: "andrew.local.forge.test"
while the main site is at domain: "main" and alias: "andrew.local.forge"
I've tried a few combinations of the above but it only works when the main site is gone and it's running off a bare vhosts file set up outside of aegir (like in mig5's instructions).

I've got this set up on a virtualbox test server and if there are any combination you want me to try I'm happy to do so, also if you want any config files.

Cheers

chrsnlsn’s picture

I'm having a little trouble getting this going as well, a little confused by step 3

3. You *must* give your site a site URL Alias via the 'Site aliases' feature, that is consistent with the multisite/subfolder instructions for Drupal.
(e.g for example.com/foobar, add an alias of example.com.foobar)

[edit] is step 3 done after you create the site in hostmaster?

Is this asking me to put something in the 'Domain Aliases' field? I need a little more explanation here I looked through the multisite Drupal instructions and couldn't follow what steps from that I needed to include in this process.

any luck andrew-r?

[edit2] after further investigating it looks like I do have the two separate installs but like andrew-r they're both pointed at the same db

[edit3] so if I comment out the unset of the database variables in the settings.php for my sub site and dsm($_SERVER); I can see that it's getting the parent sites db settings. But if I look in the drushrc.php file for the site they are set correctly. So how is my sub site getting the wrong db credentials, but getting the correct modules and themes that are different from the parent site?

[edit4] Resolved, had to do this
Set $options['provision_db_cloaking'] = FALSE; in the site's drushrc.php and then re-Verify the site
Probably has to do with not actually having a subdomain or anything established for my sub site while testing so the apache vhost couldn't work it's magic to set the server vars correctly. Now it's picking the correct ones up.

Dustin@PI’s picture

From mig5's blog post:

The main domain should not be that of a Drupal site already managed by Aegir. I will change this constraint if I can get it to work..

I hacked around in the Apache settings and manged to get the "root" site to point to another site that was managed by Aegir, but installed on another domain, by setting the server name and copying a bunch of settings from the host file that Aegir created for the other domain.

Note: I haven't installed this in production yet and have only done limited testing.

andrew-r’s picture

Hi ChrisNelson,

Thanks for your update, I got it to work using your suggestion!

If it helps, here are the details of my setup now...

I created both sites (the subfolder one first) with their alias setting filled in. So ChrisNelson, my sites had them from the getgo. I used a Domain Alias initially for the main root site and a nonsense domain. I now have the main site migrated to the domain "andrew.local.forge" after deleting it's domain alias. The sub folder site still has the settings quoted in #42.

I updated my sub folder site's drushrc.php file with $options['provision_db_cloaking'] = FALSE; and it appears to be working now. I haven't applied mig5's clone.provision.inc.patch yet, but I will when I get a chance and start moving and cloning these sites around.

Hi Dustin@PI, you're right, I missed that in mig5's instructions.

joshrabinowitz’s picture

Hi all, new to aegir, running on top of sl5.

Running into a initial setup issue with mig5's [super awesome] extension.

aegir installed, successfully added a remote [web and db] server, but when adding (and initially verifying) a 7.18 platform, I get this error:

[root@drupaldev ~]# /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: httpd: Syntax error on line 210 of /etc/httpd/conf/httpd.conf: Syntax error on line 25 of /etc/httpd/conf.d/aegir.conf: Syntax error on line 12 of /var/aegir/config/server_drupaldev/apache/platform.d/platform_Drupal718_dev.conf: Include directory '/var/aegir/config/server_master/apache/stub.d' not found
[FAILED]

Not sure why the platform conf on remote server is getting populated with an Include for server_master, combing through /var/aegir/.drush/provision_subfolders/subfolder.drush.inc now though.

Thanks for any guidance!

[edit1]
I guess I could
aegir@drupaldev:~>mkdir -p /var/aegir/config/server_master/apache/stub.d/

But that's sorta janky

joshrabinowitz’s picture

Using mig5's subfolder extension, I can see all the filesystem magic happening, but apache keeps dumping me when i try to hit http://drupaldev:5080/sub1 (http://drupaldev:5080 loads the apache test page).

I am guessing this is due to the alternate port, maybe the alias needs to be different? I've tried changing the alias to 5080.drupaldev.sub1 per Drupal's main multisite documentation, but I still get 404s. One interesting find is

09:47:10 root@drupaldev:/var/aegir>tail -n 1 /var/log/httpd/error_log
[Tue May 07 09:32:21 2013] [error] [client 192.68.194.81] File does not exist: /var/www/html/sub1

Is this config not being respected? Why is apache looking in the default docroot, when another is specified for this subdir?

09:47:12 root@drupaldev:/var/aegir>cat config/server_drupaldev/apache/vhost.d/sub1
<VirtualHost *:5080>

  DocumentRoot /var/aegir/platforms/drupal-7.18_dev

  ServerName sub1
  SetEnv db_type  mysql
  SetEnv db_name  sub1
  SetEnv db_user  sub1
  SetEnv db_passwd  ********
  SetEnv db_host  drupaldev
  SetEnv db_port  3306

 ServerAlias drupaldev.sub1

Seems to be loaded correctly:

09:51:25 root@drupaldev:/var/aegir>/usr/sbin/apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
wildcard NameVirtualHosts and _default_ servers:
_default_:5443         drupaldev (/etc/httpd/conf.d/ssl.conf:81)
*:5080                 is a NameVirtualHost
         default server drupaldev(/etc/httpd/conf.d/aaegir_subfolders.conf:6)
         port 5080 namevhost drupaldev(/etc/httpd/conf.d/aaegir_subfolders.conf:6)
         port 5080 namevhost default (/etc/httpd/conf.d/aegir.conf:5)
         port 5080 namevhost sub1 (/var/aegir/config/server_drupaldev/apache/vhost.d/sub1:1)
                 alias 5080.drupaldev.sub1
Syntax OK

Made a main vhost here:

10:04:09 root@drupaldev:/var/aegir>cat /etc/httpd/conf.d/aaegir_subfolders.conf 
<VirtualHost *:5080>
  ServerName drupaldev
</VirtualHost>

Am I doing something wrong?

ergonlogic’s picture

As I understand it, the plan is to incorporate Mig5's Aegir Subfolders extensions into core, presumably as an 'experimental' feature.

Having quickly reviewed the state of that project, here's what I think needs to be done:

  1. Move subfolder.drush.inc into provision/subfolder
  2. Move hosting_subfolders into hosting
  3. Add a Hosting Feature for subfolders
  4. Apply the included patch to clone.provision.inc.patch
  5. Refactor provisionService_subfolder as an Aegir 2-style service.
  6. Perhaps implement a full service that attaches to servers, which would get us automatic 'verify' tasks on all platforms.
  7. Perhaps add/improve validation of the 'subfolders' field (re. "2. Specify a 'subfolder'. No opening or trailing slashes")
  8. Autocomplete the subfolder name based on the first part of the subdomain name
  9. Autocomplete the 'example.com.prefix' Site Alias too.
  10. Add support for renaming the subfolder when a site is renamed, or get to set a new subfolder for a site that is going to be cloned
  11. Delete the old subfolder stub files when it's renamed, so have a concept of 'old' subfolder name vs 'new'.
  12. Add Nginx support

We should perhaps look at prioritizing this list, and determine if perhaps a subset would be sufficient for inclusion as an 'experimental' feature.

anarcat’s picture

i prioritized that list in the issue summary, please do expand and repriotize as needed.

anarcat’s picture

Issue summary: View changes

add issue summary with remaining tasks

ergonlogic’s picture

Issue summary: View changes

update with proposed solution

anarcat’s picture

i'm starting to merge this stuff in the dev/subfolders branch on both hosting and provision.

anarcat’s picture

Issue summary: View changes

update proposed resolution after reading through the code.

ergonlogic’s picture

For testing purposes, I'm going with a very minimal "Refactor provisionService_subfolder as an Aegir 2-style service.", hardcoding the includes, etc.

It appears that we need to verify the server before platforms will verify, once the feature is enabled. But with that, it works! I'm committing as is to the dev branch, so I can start the proper re-factoring.

ergonlogic’s picture

We also need something like /etc/apache2/conf.d/example.com.conf which currently needs to be created manually:

<VirtualHost *:80>
  ServerName example.com
</VirtualHost>

We're discussing options for this in IRC.

ergonlogic’s picture

I've shuffled the subfolder code around, so it's now structured like a proper Aegir 2 service, with autoloading and such.

anarcat’s picture

Title: Allow creation of mysite.com/site1 and mysite.com/site2 type of sites » Allow creation of example.com/foo and example.com/bar type of sites

awesome, thanks.

so one thing i have just thought about, since #2000038: cannot generate multiple configuration files in nginx kicked in, is that we can now generate multiple configuration files per site. so for example.com/foo, a regular vhost would be generate for example.com and another special set of rules would be generated for foo in (say) ~/config/server*/apache/subfolders.d/exmaple.com/foo.conf. Then we use the extra_config hook to add include .../folders.d/example.com/*.conf into the example.com vhost!!!

the example.com vhost would have:

<VirtualHost *:80>
ServerName example.com
# [...]
Include ~aegir/config/server_master/apache/subfolders.d/exmaple.com/*.conf
# [...]

the foo.conf file would be a simple Alias with maybe a Location block for the site-specific stuff?

  Alias /site1 /var/aegir/drupal-5.20-1.0-prod/
<Location /site1>
  SetEnv db_type  <?php print urlencode($db_type); ?>
# [...] all the regular stuff that goes in the vhost normally goes here
</Location>

seems simple clean and logical. The only problem is if someone creates the site example.com, then that vhost will be destroyed with example.com/foo will be created, and will lack the proper data. but i am sure i can think of a way to work around that, either by requiring that site to be created first (bleh) or simply by not overwriting stuff.

next step if to test if such an apache (or nginx!) configuration works.

ergonlogic’s picture

<Location> blocks appear to be very limited in what they can contain. For example, they cannot contain:

  • DocumentRoot
  • ServerName
  • ServerAlias
  • <Directory> blocks

This appears to eliminate pretty much everything but the SetEnv's from what normally goes in a vhost.

ergonlogic’s picture

I figure that I'll take a few minutes to list the steps to create an 'example.com/foo' site, and catalogue all the Apache config files and such we currently generate.

First, we need to create an Apache vhost by hand for the domain (example.com). To create the site, we give it an arbitrary Domain name, since this isn't used as the URL: 'foo-site'. Then we add an alias to 'example.com.foo', and redirect all domain aliases to this new alias. Finally, we add a subfolder path: 'foo'.

Here's are the resulting filesystem artifacts:

/etc/apache2/conf.d/example.com.conf (manually created):

<VirtualHost *:80>
  ServerName example.com
</VirtualHost>

A symlink from a 'foo/' directory in the platform root, back towards the platform root:

/var/aegir/platforms/drupal-7.22/foo -> /var/aegir/platforms/drupal-7.22/

/var/aegir/config/server_master/apache/stub.d/foo.stub:

SetEnvIf Request_Uri ^/foo/ db_type=mysql
SetEnvIf Request_Uri ^/foo/ db_name=foosite
SetEnvIf Request_Uri ^/foo/ db_user=foosite
SetEnvIf Request_Uri ^/foo/ db_passwd=7ERnyCQNnz
SetEnvIf Request_Uri ^/foo/ db_host=localhost
SetEnvIf Request_Uri ^/foo/ db_port=3306
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteCond %{REQUEST_URI} ^/foo
RewriteRule ^ /foo/index.php [L]

In /var/aegir/config/server_master/apache/platform.d/platform_Drupal722.conf, we've added:

[...]
<IfModule mod_rewrite.c>
  RewriteEngine On
  Include /var/aegir/config/server_master/apache/stub.d/*.stub
</IfModule>
[...]

/var/aegir/config/server_master/apache/pre.d/000_foo_alias.conf:

Alias /foo /var/aegir/platforms/drupal-7.22

/var/aegir/config/server_master/apache/vhost.d/foo-site:

<VirtualHost *:80>

  DocumentRoot /var/aegir/platforms/drupal-7.22

  ServerName foo-site
  SetEnv db_type  mysql
  SetEnv db_name  foosite
  SetEnv db_user  foosite
  SetEnv db_passwd  7ERnyCQNnz
  SetEnv db_host  localhost
  SetEnv db_port  3306


 ServerAlias example.com.foo

<IfModule mod_rewrite.c>
  RewriteEngine on
 RewriteCond %{HTTP_HOST} !^example.com.foo$ [NC]
 RewriteRule ^/*(.*)$ http://example.com.foo/$1 [NE,L,R=301]
  RewriteRule ^/files/(.*)$ /sites/foo-site/files/$1 [L]
  RewriteCond /var/aegir/platforms/drupal-7.22/sites/foo-site/files/robots.txt -f
  RewriteRule ^/robots.txt /sites/foo-site/files/robots.txt [L]
</IfModule>

# Extra configuration from modules:

    # Error handler for Drupal > 4.6.7
    <Directory "/var/aegir/platforms/drupal-7.22/sites/foo-site/files">
      SetHandler This_is_a_Drupal_security_line_do_not_remove
    </Directory>

    # Prevent direct reading of files in the private dir.
    # This is for Drupal7 compatibility, which would normally drop
    # a .htaccess in those directories, but we explicitly ignore those
    <Directory "/var/aegir/platforms/drupal-7.22/sites/foo-site/private/" >
       SetHandler This_is_a_Drupal_security_line_do_not_remove
       Deny from all
       Options None
       Options +FollowSymLinks
    </Directory>
    
</VirtualHost>
anarcat’s picture

Let me try this in a better way. I think all we need is, for URL example.com/foo...

...in vhost.d/example.com:

<VirtualHost *:80>
  ServerName example.com

  Include /var/aegir/config/server_master/apache/subdir.d/example.com/*.conf
</VirtualHost>

This file (re)generated on verify/install by any hosts using example.com as a vhost.

... in subdir.d/example.com/foo.conf:

Alias /foo /var/aegir/platforms/drupal-7.22

<Location /foo>

  SetEnv db_type  mysql
  SetEnv db_name  foosite
  SetEnv db_user  foosite
  SetEnv db_passwd  7ERnyCQNnz
  SetEnv db_host  localhost
  SetEnv db_port  3306

</Location>

    <Directory "/var/aegir/platforms/drupal-7.22/sites/example.com.foo/files">
      SetHandler This_is_a_Drupal_security_line_do_not_remove
    </Directory>

    # Prevent direct reading of files in the private dir.
    # This is for Drupal7 compatibility, which would normally drop
    # a .htaccess in those directories, but we explicitly ignore those
    <Directory "/var/aegir/platforms/drupal-7.22/sites/example.com.foo/private/" >
       SetHandler This_is_a_Drupal_security_line_do_not_remove
       Deny from all
       Options None
       Options +FollowSymLinks
    </Directory>

Also, we need to create the site in ~aegir/platforms/$PLATFORM/sites/example.com.foo instead of sites/foo-site (or make sure the sites.php file is up to date).

There's the question of whether we want to allow subdirs in the main site URL: this would mean having slashes in context names, which is impossible, so instead we would need to pass (say) example.com.foo as a context name, which could be confusing.

Instead, we should use the alias fields for now, which we would validate differently from the main site URL (in the frontend) but pass as-is to the backend (ie. the backend receives example.com/foo as an alias). In this case, the main URL of the site could be foo.example.com (or whatever the user chooses, but *not* with a subdir) and example.com/foo would be an alias, in which case the site would be created in sites/foo.example.com and example.com.foo would be added as an alias in sites.php (or a symlink in D6).

The most complicated case is if you have 3 (or more) aliases (example.com/foo, example.com/bar and example.org/baz), you end up with *6* configuration files:

vhost.d/foo.example.com
vhost.d/example.com
subdir.d/example.com/foo.conf
subdir.d/example.com/bar.conf
vhost.d/example.org
subdir.d/example.org/baz.conf

Maybe we should limit ourselves with only one level of subdirectories (ie. not allow /foo/bar as a subdir) to keep things a little simpler for now. :) So this should also be validated in the frontend.

One limitation of that system is that you can't have an "example.com" site. One way to work around that would be to generate normal sites the same way as subdir sites. Basically, the current vhost config would end up in (say) subdir.d/_.conf (or whatever) and that way the example.com file wouldn't be overwritten. This would be significant parsing overhead for Apache, but it would work and allow us to have a regular site on example.com, which is not possible right now.

Of course the best approach is to eventually be able to respect the example.com context and just inject that Include in there in an existing site... But that may be tricky in the backend - maybe by loading the context and verifying it?

So, in summary, this is what we need to do:

1. ditch the "subfolder" field in the frontend and backend (maybe that means just starting a new branch)
2. parse the alias fields for subdirectories, validating separately the hostname and the subdir parts (not sure what should be legal as a subdir, at least don't allow multi-level subdirs)
3. send the alias field as-is to the backend (with the slash)
4. in the backend, make sure we provision-save example.com.foo as an alias instead of example.com/foo
5. generate subdir.d/example.com/foo.conf (instead of stub.d) in the site context
6. generate vhost.d/example.com in the site context
7. repeat 5 and 6 for every alias
8. make sure the main URI and aliases are validated against the base URI of subdir sites (so that users are forbidden to create example.com when example.com/foo exists)
9. *eventually*: allow the main site URI to have subdirs - we would then need either need to:
a. have a subdir field the clarify to the backend what part of the URI is a subdir
b. keep only the full URL (with slash) in the frontend and let the backend figure it out
c. probably more shit

My head hurts now, but I think that's the way to do it, and it should actually be possible, especially now that we can generate multiple files per context. :)

ergonlogic’s picture

I'm starting this from scratch in dev/subdirs. I'll leave dev/subfolders around for awhile though.

For (2) in #56, we can't easily override the alias field validation, since it happens in hook_nodeapi(), rather than as a form validation function. I'm not certain whether there would be side-effects to moving this to a form validation. So, I just wrapped the relevant checks in 'if (!module_exists('hosting_subdirs'))', which is ugly, but functional. However, keeping it this way continues to make overriding this (for example in contrib) difficult.

ergonlogic’s picture

I've got the front-end working properly, so (1) through (3) of #57 are done. (4) appears to only apply to Drupal 6 sites (so I'm ignoring it for now).

I've been banging my head on (5) and (6). In particular the "in the site context" part. I had thought we'd just need new Config classes, and possibly a Service... but apparently we'll need a new Context too. My understanding of Contexts, is that they are Drush aliases used to store relevant data from the front-end, so that Provision can operate independently of the front-end's database. Since we're storing the alias with the subdir directly in the site's alias, I'm not sure what role a new Context sub-class serves here...

ergonlogic’s picture

I've taken a shortcut approach, for the time-being. To be able to advance on generating the proper configuration files, I'm currently just rendering the templates directly, in a drush_hook_pre_provision_install(). This is worked nicely to generate the files, but resulted in being forwarded to install.php.

It turns our that sites.php was getting the alias with the '/' in it. I fixed this by overriding the template used to generate sites.php with hook_provision_config_load_templates(). The template just replaces any '/' with a '.' in the alias records.

So subdirectories now work when installing a site simply by adding an alias with a subdirectory. We still need to:

  • create config files during verify, migrate and deploy tasks;
  • delete config files after migrate and delete tasks;
  • support disable and enable tasks.
ergonlogic’s picture

Status: Needs work » Needs review

Subdirectory aliases now persist across migrate and verify tasks, get cleaned up during delete tasks, and can be disabled/enabled. I believe this feature is fairly complete, so I'm marking as 'needs review'. The code is in the 'dev/subdirs' branches of hosting and provision.

Note that I still believe it should be re-factored to use proper Config, Context and/or Service classes. The class autoloading code and directory structure already exist to support doing so. However, we now have a working feature in the mean time.

ergonlogic’s picture

Status: Needs review » Fixed

Merged in b3bce19ec3e222

anarcat’s picture

can you open issues about the remaining problems here? we talked about:

* multiserver support
* refactor
* allow example.com and example.com/foo domains (or at least forbid them...)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

anarcat’s picture

i created a meta-issue to followup on those instead of having to copy-paste the same blurb in all relnotes.

#2046167: [META] Remaining issues with subsite support (Part II)

anarcat’s picture

Issue summary: View changes

progress

colan’s picture

Title: Allow creation of example.com/foo and example.com/bar type of sites » Allow creation of example.com/foo and example.com/bar type of sites (Part I)

Added meta issue part number.