Move all core Drupal files under a /core folder to improve usability and upgrades

I would prefer:

public_html (root directory)
public_html/index.php
public_html/.htaccess

public_html/drupal
public_html/drupal/core

public_html/drupal/config
public_html/drupal/user

config containing sites and any other config files
user containing all user additions including modules, themes etc.

using 'local' is not very intuitive IMO, and I think config should be separate.

I am however unsure how this would pan out for multi-site configurations, as a big plus point is having modules unique to certain "sub-sites" etc.

Regards,
-K

Some of Drupal's top level directories, like "scripts", are clashing with directories already on my site. Hope this will be implemented soon!!!

The solution for me to install everything of drupal in another directory and make it 'feels' like it is in the main page is this:

  chdir("drupal-4.6.3/");
  include("index.php");

Works like a charm... And you don't have to worry about your other directories...

  chdir("drupal-4.6.3/");
  include("index.php");

I tried this method, but when the index appear, css didn't load and the node links don't work.

Update, the above method work. But you need to enter your base url in your settings.php

But in our nodes url, the /drupal/ directory still appears.

Any way to remove that without causing any problem?

I've been looking everywhere for this functionality. The above code works very nicely for redirecting from the root directory to the frontpage under the /drupal directory, but it still leaves the URLs looking like and accessible only at http://www.example.com/drupal/links, for example. I would really prefer if it could be accessed directly from the root directory (I.E., http://www.example.com/links).

Oops, fixed version field back to cvs.

Another method that work via htaccess.

http://drupal.org/node/135206

But we still need to know how to remove the /drupal/ portion from the urls.

I solved this problem by using a RewriteRule in the .htaccess file.

I also found that the key to removing "/drupal/" from the URL is in setting $base_url appropriately.

See my comments here and here for how I did it.

Marco

I have a somewhat unique situation that seems to be a bit direct but, I would like someones input on this. I currently have Drupal 6.1 (w/no additionally installed modules or themes - only written nodes and some edited CSS files for themes) installed and running in a sub-domain, ie: http://test.mydomian.com, whereas test, being the sub-domain. I am looking to install/upgrade Drupal 6.2, but eventually I will have it located in my main domain, ie: http://www.mydomain.com, whereas mydomain, being the main domain.

I have an option, I believe, that would allow me to just "install" a fresh copy of 6.2 in my main domain and move effected files and such to the new installation from my sub-domain version. I thought this would be less extensive than updating 6.1 first, then moving it into my main domain later. Would this be the most appropriate choice? (I am ready to have it go public in my main domain at this point!)

Also, I have a concern with the existing database information. Will the new installation over-right any db entries? Will the current users still be there? etc. (I have backed up my db and such already) I hope this is not off topic but, the thread that contained all these concerns directed me to this node for any new posts.

Thanks folks....I'm really impressed with this Drupal website maintenance environment!

Roger

Please don't hijack issues.

Never mind....again!

I guess this issue is outdated anyway. Drupal cleanly separate core files from custom ones (that lie in /sites). So marking that as won't fix.

This is not an outdated issue. Drupal 6 still expects everything to be installed in the root. This request is to allow everything to be installed in a /drupal subdirectory instead and potentially move /drupal/sites outside the core installation (e.g. /drupal_sites) -- all WITHOUT including "drupal" in the URL.

While this can be done currently, it is less than straight-forward and requires hacking the .htaccess rewrite rules and settings.php. It would be ideal if Drupal would support this configuration natively instead of hacks that work to obscure its relocation.

No, Drupal does not requires you to have a "drupal" subdirectory or to include "drupal" in the URL. You can put everything in the root directory of your website (do you see any "/drupal", here on drupal.org?). The separation between Drupal Core and custom code is *very* clean (every thing custom is put under /sites), so there is no point in that issue anymore. If you really want to physically separate the two, you could use a link at the operating system level (symbolic link or anything else).

I'm closing that issue again (won't fix, because you can't fix what ain't broken). I see no point in hacking rewrite rules to do something that is done by default.

Please explain your needs more clearly if you want to reopen that issue.

The original post explains the issue very well.

I believe you are misunderstanding that the primary issue is to ALLOW Drupal to be installed transparently in a subdirectory. Currently, Drupal expects everything to be installed in the document root, causing problems described by the original poster.

This issue was discussed at length in the developers forum (link in the original post), with a general consensus that this is a good idea that should be implemented.

Please don't disregard this issue as being pointless without reading that thread.

then we'll mark it by design.

Does it make sense to mark a feature request as "by design"?

About the original feature request, just linking the /sites/[your site] to another directory would allow complete separation of Drupal Core and custom files.

First off, new features go in 7.x.

I'm against putting parts of drupal outside of the document root, mainly because in a lot of situations where access to the directory below public_html is not straightforward. My reasons are as follows:

1) It adds clutter: Having 2 locations depending on each other, but keeping them at separate locations is counter intuitive and is prone to errors when moving installations around.
2) There are cases such behavior will cause annoyance and perhaps even breaking. All of my websites are on servers which use virtualdocumentroot where folders in /home/username are sub domains of the main domain, and don't have access below that directory.
e.g.
/home/domain.com/www = www.domain.com
/home/domain.com/blog = blog.domain.com

If drupal would need /home/domain.com/drupal/ for includes and stuff and /home/domain.com/www/drupal/ for the rest of the files, that would prohibit me from using drupal.domain.com as a functional subdomain.
3) There are a lot of situations (including the above) where you don't have a directory directly below the document root that is accessible to the users. For example if you have a server on which you have your document root (user-writable) in /var/www, you would need to give users write-access to /var to make up for the missing public_html structure.
4) On some servers Apache (and PHP) is prohibited to look outside their document root for files to run.

Thus, if this feature gets into core it will break a lot of websites on shared and access-limited servers, and will seriously hinder platform independence of Drupal.

I'm urging to mark this won't fix.

maartenvg, I don't think we'd ever require that the Drupal core files live outside of the webroot, just allow it. Ideally, part of this re-org would be adding a variable with the path to the drupal directory, so that it could be moved out of the webroot.

My take on this thread is that it would be great to consolidate all Drupal core files in a "drupal" subdirectory, which I whole-heartedly support. My ideal layout would be something like this:

.htaccess
index.php
robots.txt
drupal
sites

This would enforce a clear separation between core/system files (drupal) and user files (sites). Everything that currently lives in the webroot would be moved into the "drupal" directory (except those listed above). This would also have the side effect of discouraging users from editing/hacking files in core (or dropping contrib modules into the top-level "modules" dir, etc). And it would make upgrading (conceptually) easier, just replace the "drupal" directory.

Finally, Drupal is currently quite rude in that it assumes it's the only script living in the webroot -- making it difficult to integrate with other systems that might be in use. This re-org would make this task a little bit easier (in that there would be less potential conflicts).

discouraging users from editing/hacking files in core (or dropping contrib modules into the top-level "modules" dir, etc).
Yup, putting new modules into the top-level "modules" dir was the first thing I did as a newbie.

So a big +1 to this because of this and of all the reasons stated in this issue.

I like your (and the original) take on it; it seems the next step in separating core from contribution. I will review & support any nice patch that floats by.

On a side note: Making it possible for D to live outside the document root is okay, but definitely shouldn't be the default behavior. Perhaps leave it almost undocumented because otherwise it will introduce a wagon load of cleanurl- and multisite-like questions which limit the user experience and increase the support load.

I've marked #311783: Directory called "modules" is not for modules as a duplicate. The focus there was just on the modules directory, but I think this would be the better option.

Agree with #25, and also keep install.php, update.php, cron.php and xmlrpc.php in the root directory for backward compatibility. My initial suggestion would have been to move /modules and /themes into /core/modules and /core/themes, but this one is more comprehensive and makes more sense.

Renaming files and folders is difficult in CVS, how do you propose we roll a patch for this issue?

See also : #22336: Move all core Drupal files under a /core folder to improve usability and upgrades

Hi,

I am really happy to find this thread for D7.

It seems not needed to have drupal outside the webroot. But i think it can be very handy.
Then a SERVER-side install of Drupal will become possible!! This would be a major feature, and is not so difficult to make.

To get this done for our install I had to hack Drupal 6 very much. Sorry :(
I wrote an article to explain what we did. I will be happy to write more info, if that is needed to get D7 to adopt this . ;-)

If Drupal 7 would support variables to all the important paths anyone can decide for himself how he wants to handle its installation. And Drupal 7 can boost that it can be installed SERVER-SIDE.

What is needed:

• configuration to set the paths (also to /modules, /themes, /includes etc.)
• configuration to set symlink-paths
• a class/script/functions to
  • search the global and client paths and decide what to send back
  • it should also test if the path exists, handling files and dirs
  • give back root-paths and url-paths
  • give back a relative path to store in the database
  • keep a list of paths, so each path is searched for only once
• change drupal scripts to request paths by this class (ohoh.....)
• no hardcoded paths ever; even the class gets its info only from the init-settings

Greetz, Joly

Ohoh, I cant get used to this kind of forum ;)
I keep editing settings and tags, thinking it is only for my comment.
I do hope I learned it from now on.

By just symlinking ../sites/[client site name] to another location on the server, you get separation of server/client files. Then provide the client with access to that area only.
e.g.
$ cp -R /var/www/drupal/sites/* /var/www/public_html && rm -rf /var/www/drupal/sites
$ ln -s /var/www/public_html/ /var/www/drupal/sites
$ chown -R client1:client1 /var/www/public_html/client1-site.com/
or something like that. This could be done across NFS mounts, if you have multiple servers.

It's what I do, though I am a small shop with under a dozen clients.

The individual site directories can contain themes, modules, and files - pretty much the only things that you cannot separate out are theme engines and core modules; if you need to separate these out for some reason, then you'll just have to give each client their own copy of core. Which is really not a big problem if you are using revision control (svn/cvs/bzr etc.)to keep every install updated.
For site-specific apache configuration, use per-site directives in htaccess or apache config file. Sure, you'll actually have to do the work of changing the config files, but if you are a service provider your worth is determined by your toolchain, and so you probably would write a script to change the apache config (as well as create and populate the new site directory) for each new site.

I just find it very curious that you hacked core so much (Ive read your article), and wonder exactly what circumstance would require you to do so given that drupal + linux provides much of this functionality by default. Call me curious ;)

EDIT: also be sure to check out Aegir, a hosting platform for Drupal. http://groups.drupal.org/aegir-hosting-system

Note: this would make life easier for #606592: Allow updating core with the update manager

This is a long standing issue I've had with Drupal since I started back at D6.1 -- and continue to wrestle with on a daily basis. All of the core files are "up front" and all of the "you should edit these files" are hidden under that strange name "sites."

For the usability experts, tell me which folder you are likely to drop your very first module into...

• modules
• sites/all/ —> mkdir modules —> sites/all/modules/
• scripts

Exactly. This is a huge Drupal-WTF. The best proposal so far from the comments above is this:

.htaccess
index.php
robots.txt

core/
sites/

What if we did something more along the "Drupal as a framework" lines...

.htaccess
index.php
robots.txt

drupal_framework/
drupal_config/

At least then we are not asking a new user to understand the difference between 'modules' and 'sites.' In fact, "Config" is 1000% more understandable without documentation than "sites." I also recommend that we move all non-essential *.txt files to drupal_framework/ as well.

Ideally, this folder structure should be very lean and very controlled. Sadly, I don't think drieschick would consider this kind of change so late in D7. But that's ok because I wouldn't even know how to roll a patch with this ;P

Please, can we have others weigh in on this issue? Tagging for exposure.

drieschick == dries + webchick ?

I'm not sure I like the idea of hardcoding the name "drupal" into the folder structure. It makes things more confusing for those repacking Drupal.

/core and /sites seem really good to me. However, I think all of the *txt files, and special files like authorize and update.php would need to remain in the root, unless we require mod_rewrite?

Current D7 List:

includes/
misc/
modules/
profiles/
scripts/
sites/
themes/

.htaccess
CHANGELOG.txt
COPYRIGHT.txt
INSTALL.mysql.txt
INSTALL.pgsql.txt
INSTALL.sqllite.txt
INSTALL.txt
LICENSE.txt
MAINTAINERS.txt
UPGRADE.txt
authorize.php
cron.php
index.php
install.php
robots.txt
update.php
xmlrpc.php

New proposed structure based on #37 and my own thoughts:

core/
    includes/
    misc/
    modules/
    profiles/
    scripts/
    themes/

    cron.php          <-- move files that admins don't have to type up 
    authorize.php         into the core/ folder

readme/ <-- folder that contains txt files like changelog.txt, etc.
sites/

index.php
install.php
update.php
robots.txt
xmlrpc.php <-- I don't think we could move this, not sure though

Edit: Sorry, I meant to click preview originally, I just changed some formatting and moved a few files.

WHy not also move install.php and update.php into core? If we can move cron, why not those?

Like this structure, but also like config better than sites, but I'm a bit on the fence on that one.

I'd recommend naming "readme" "docs" instead. And don't forgot the .htaccess!

Building on #38, #39 & #40 I would propose the following

core/
    docs/   <-- Everything proposed for readme/ seems to be core related.
    includes/
    misc/
    modules/
    profiles/
    scripts/
    themes/

    cron.php
    authorize.php
    install.php
    update.php

custom/  <-- It should be clear that this folder is for more than just config files.
    modules/  <-- What is now contained in sites/all/modules
    themes/    <-- What is now contained in sites/all/themes
    sites/        <-- For multi-site specific files

    settings.php  <-- What is now in sites/default/settings.php
    readme.txt    <-- A file explaining not to hack core and to put your files here.

.htaccess
index.php
robots.txt
xmlrpc.php <-- If possible I'd prefer in core/

My thoughts are, while config is clearer than sites, it also encourages new users to think only configuration files go there not modules, themes, etc. I think that custom is clearer. I also believe that a single instance install is much more common than a multi-site install. If that is true I think Drupal should be optimized for that. Merging the sites/default and the sites/all folders to the root of custom I think accomplishes that, and leaving the sites folder there notifies people of that capability but doesn't require that they understand it just to get a single site install running.

That is my two cents worth.

.htaccess
CHANGELOG.txt
core/
  css/
  images/
  includes/
  js/
  modules/
  scripts/
  themes/
index.php
install.php
INSTALL.txt
README.txt
sites/
  all/
  default/
update.php

I am in favor of leaving the *.txt files at the top level where they will be easy to find.

IMO we should consider this strongly as an early entry into D8 so that we can get it done and out of the way while there are still few patches against D8 (since it will invalidate, oh, every patch. And then some).

I don't believe the .php files can be safely moved out of the top level directory, because technically we do not rely on .htaccess. We only rely on that for clean urls.

I'm not sure I like 'custom' because it suggests that it's custom to a site. I don't know that it's worth changing it from sites/all, especially since we now actually include sites/all/modules

I was envisioning custom as in my custom files, modules, themes, etc.. However, I'm not set on that particular name.

I'm really opposed to sites/all and sites/default. I've used Drupal for quite a while now and the distinction between them only matters for multi-site. Thus the current stucture requires understanding the multi-site capabilities just to "properly" get a single instance site up. I feel that is a major usability problem. I have also witnessed firsthand the problems it creates for new users.

Actually it's documented on drupal.org somewhere (I'm sure it's moved since the page was written) to use sites/all for contrib modules and sites/default for custom-written modules. That makes a perfectly good distinction even for non-multi-site sites.

Of course, the D7 update system breaks that horribly by putting all modules in sites/default, even though drush puts its downloaded modules in sites/all. I think that's a critical core bug, but I was overruled. :-(

That doesn't even work for multi-site, since sites/default is for just one site. =(

#45: Yes, but would this usability problem exist if 1) 'modules' wasn't at the top level, and 2) 'sites/all/modules' already exist? Particularly if the INSTALL.txt instructions told you precisely where to put modules you downloaded? With a separated 'core' directory, it's easier to say "DO NOT MODIFY THIS STUFF" AKA "DON'T HACK CORE".

to throw more bikeshedding into the custom folder name debate, when we build sites, we use:

sites/all/modules/contrib
sites/all/modules/custom

'contrib' for contributed modules and 'custom' for modules written specifically for a site.

and subscribe, and +1 to this great idea.

#48

1) Helps a lot.
2) Helps some.

However I really like the idea of the sites/all/ content being at the root of some custom/ (or your favorite name) folder. It seems to me that logicly the contents of those folders would be avaliable to all the deeper folders. It is using the folder structure to reinforce the actuallity of what is happening. For instance.

Currently.

sites/
    all/          <-- Avaliable to all domains.
    default/      <-- Avaliable only to all "Default" domains.
    example.com/  <-- Avaliable only to the example.com domain.

My Proposal

custom/         <-- Insert favorite name here
    modules/          <-- Avaliable to all domains including 
    themes/                    multi-sites living in the sites folder
    ...
    sites/
        default/      <-- Avaliable only to all "Default" domains.
        example.com/  <-- Avaliable only to the example.com domain.
        ...

This also makes getting a single-site up clearer. However, I can see your point that it will also discourage proper multi-site setup over time. (My use case of Drupal has multi-site installs causing far more problems than they solve, so I am a bit biased I suppose.)

Having said that any of these proposals are significantly better than what there is currently.

Let us not forget settings.php. That has to be per-site by nature. And there's also now sites.php. Multi-site is rather intrinsic to Drupal's architecture at this point, like it or not.

Throwing my two bits in, having default / multi-site-wide modules and themes under sites/all is a bit of a pain from a backup standpoint. It would be better if 'sites' was just that... sites... and modules and themes (the contrib and custom kind) were held elsewhere. I'd frankly prefer something like

core/
contrib/ (or custom, although i think i like contrib better, as in my mind it covers custom too)
sites/
   default/
   sitea.com/
   siteb.com/
   ...

I'm working with multisite setups, and this seems much cleaner to me.

What about a (slightly) different approach. Are we completely wedded to the sites folder, or could we have something like:

core/         <- everything Drupal that shouldn't be modified ever, and can be completely replaced when upgrading
.htaccess
index.php   
README.txt
robots.txt

files/
    all/
    example.com/
    example.net/
modules/
    all/
    default/  <- do we really need this? Isn't it too confusing for most users?
    example.com/
    example.net/
settings/
    default-settings.php
    default-sites.php
    example.com/
        settings.php
        sites.php
    example.net/
        settings.php
        sites.php
themes/
    all/
    default/   <- do we really need this? Isn't it too confusing for most users?
    example.com/
    example.net/

My thoughts behind this layout is that it becomes instantly obvious to a new user where to put their modules and themes. A slightly simpler option for single-site installs would be to drop the subfolders of modules and themes, and allow multi-site to add example.com/{modules|themes|files} folders for site-specific files.

Do we really need a default folder for modules and themes? Can someone let me know where you would use default in place of example.com, especially now with the ability of sites.php to provide a different domain and/or path for development.

As an aside, maybe the index.php should as minimal and free from change as possible. Maybe even to the point of simply being a list of include statements, so that when upgrading users do not have to think about copying the index.php file into the root. That way, the Drupal tarball could be extracted into the folder and renamed core and the upgrade performed.

For the normal case of single-site, having a consistently named directory is very helpful. Especially when you're spinning off a lot of non-multisite sites on a regular basis (like many dev shops do).

I'm really not wild about splitting settings files from site-specific modules. That only makes backup more of a challenge since your not-from-d.o files are now spread out to 2 or more locations. (Remember the files directory.)

I agree with #55 in general, though it's missing robots.txt, UPGRADE.txt, INSTALL.txt, CHANGELOG.txt, and hopefully #607028: Add README.txt to core.

What's sites/all/addons? The same as libraries?

Side note, as previously mentioned on the dev list I am now working with the PHP Standards Working Group (such as it is) on naming conventions as they pertain to classes and namespaces. It's almost certain that such standards will influence file paths. I am trying to keep this thread in mind as I discuss the point. We'll see what happens. :-)

Just trying to keep folks in the loop.

I agree that the Drupal root folder needs cleaning, but going from bloat to the minimalist 2-folders 3-files (a la #55 + #57) seems also overly archaic for noobs as well as seasoned developers. Here's a disclaimer: I guess I'm kinda new-ish to Drupal, but ... anyone can dream, and think outside the box. I hope not to offend anyone with my galactic translucent bikeshed paint :) I'm coming at this from an end-user friendly p.o.v.

As developers and web architects (noob or veteran) using, testing, and building with Drupal, we all live with three components not simply two: core AND contributions AND our site(s). After installing a fresh Drupal instance, the first thing you have to do is add your favorite modules... so burying module code four folders deep inside the sites/all/modules directory (#55 suggests five folders deep!) just doesn't add up to a "sensible default". Neither does five folders deep (!?!) for my own cherished custom code (sites/all/modules/custom/my_module) I want to SEE my code, and know its there, and allow others on the team to readily see and know that i have custom plugins in any given Drupal install. While we are dreaming up ways to improve Drupal folder structures lets go ahead and improve sites/ as well!

In addition, as people move towards automating their update mechanisms (be it through RCS update commands, scripts like drush, or web-based Update modules) EVERYONE needs to deal with and distinguish between "core" updates and "contrib" updates. Lets make a local folder structure agree with the server repository logical division in order to foster usability and instantaneous grokability with what people will be checking out and keeping up to date. "As above, so below."

I hope my structure makes sense, its heavily doted with footnotes.

core/          <--- contents already hashed out in prior comments. 
  .../              (certainly core modules, themes, profiles, and php and txt files
  .../               generally EVERYTHING that could potentially be touched by a drupal minor version update.
  -misc/       <--- misc folder needs serious cleanup, move visual elements to theme. 
                    move base jquery to a module handler which can be overridden by contrib
plugins/       <--- replaces sites/all/ to embrace the idea of "core"  and "contrib"  (*) (***)
  3rdparty/    <--- makes official and replaces the unofficial convention of 'sites/all/libraries', aka 'sites/all/lib'  (**)  
  custom/      <--- a second-level folder that users can create to single out their own code (**) (***)
  modules/     <--- the sensible default place for all your favorite "contrib" modules; replaces sites/all/modules.  (***)
  themes/      <--- the sensible default place for base themes from "contrib"; replaces sites/all/themes
    engines/   <--- the sensible default place for theme engines from "contrib"
    zen/       <--- an example base theme
  profiles/    <--- the sensible default place for installation profiles from contrib
  translations/<--- the sensible default location for translation packs
  .../         <--- leave room for other futuristic contributions here  (widgets, or frameworks, or feature-sets)
  Readme.txt
sites/        <---  bundled with packaging script
  default/     <--- yuck, but oh well, i don't have a better solution for this :/ (****)
  example.com/
    files/
    modules/
    themes/
      my_zen_subtheme/
  Readme.txt
index.php      <---  Simple one-liner that points to core. (*****)
.htaccess
robots.txt

(*) Despite the idea of introducing an organization that mimics the 'core' and 'contrib' structure, I propose 'plugins' to reenforce the idea that contributions not necessarily limited to the official Drupal "contrib" repository; Take the 3rdparty folder special case as an example (details below). The advantage of having a plugins folder is that it leaves room for future growth, and welcomes healthy additions on down the road, for things that dont classify as modules or themes or profiles or translations. Introducing and really embracing the term "plugins", and noting that modules are really a subset of the plugin cosmos is something Drupal could use to reduce the initial 'nomeclature' barrier.

(**) The 3rdparty folder is THE special case inside the plugins folder that wouldn't necessarily contain elements that sync with drupal contrib repository. Also, the name '3rdparty' makes more sense than 'libraries' to non-programers, because it truly distinguishes this section for what it is: ANYTHING non-drupal that various modules might put to use, that may or may not be added to the Drupal repository (for Licensing reasons or other). Also, very often, the external code packages that go in there aren't technically "code libraries" in the strictest sense, as they are often full-blown applications and may be portrayed online as 'packages', 'components', 'libraries', frameworks, etc. If we stick with the generic term '3rdparty plugin' we can use it everywhere.

(***) Since this folder will potentially house drupal modules and sub-modules, leave it up to the user to determine folder structure inside the directory, and recurse the folder (if it exists). Update: after meditating on the structures here. This point to me is the one that sells the idea of a plugins directory. Essentially, plugins will be a better modules/ directory. One that has sensible defaults, but since its recursed, its internal structure could be customized to anyone's desired structure. For example, if you're going to distribute a drupal+specialsauce solution (eg OpenAtrium), you could isolate your special sauce into plugins/specialsauce/modules plugins/specialsauce/theme.

(****) The sites/default folder should go away if at all possible and be replaced with sites.php or other type of general config file. Disclaimer: I'm not up to speed on D7 changes in the works here.

(*****) A user may choose to customized 'index.php' to point to core outside the webroot. So as not to "overwrite" these changes upon a core update, this file 1) should not be in the repository 2) should never be included in any update scripts and 3) should be added by the packaging script. It's contents should contain a link to more information on how to move "drupal core" out of the webroot.

My take on the setup:

core/          <--- I like the idea of everything core related going into one directory. This also helps with the "don't hack core" ideal.
  .../           
  .../               
  -misc/       
------------------           
plugins/       <--- 'plugins' is a bad naming scheme since it is used nowhere in the drupal community. "contrib" would be better than "plugins" 
  3rdparty/    
  custom/     
  modules/   <-- If we are going to continue to use the /sites directory, I think we should use sites/all instead of this.
  themes/           //
    engines/        //
    zen/             //   All same as above. Let's either break /sites up or consolidate all site specific stuff with it.
  profiles/          //   We don't need /core/modules, /plugins/modules, /sites/all/modules. Same for themes, etc..
  translations/   //
  Readme.txt
-----------------
sites/        <---  Easy to understand if using Drupal in a multi-site setup, not as user friendly in a one site install
  default/     <--- "default" is an awful name. One way to make it more intuitive would be to change the directory -> 
  example.com/     name when install.php runs (i.e force $base_url or $_SERVER['PHP_SELF', DOC_ROOT, etc...]) 'default' => 'mynewsite.com'
    settings.php     
    files/
    modules/
    themes/
      my_zen_subtheme/
  Readme.txt
-------------------
index.php      // Nice and clean root directory. I don't really have an opinion on moving the README, CHANGELOG, etc.. into a 'docs' directory. Both 'root' or 'core/docs' seem like logical places.
.htaccess
robots.txt

I also looked at Wordpress and Joomla to see how they handled their 'core/contrib' separation (from a file system perspective only). Wordpress keeps all plugins and themes in 'wp-content', which doesn't seem all that intuitive. Joomla pretty much throws everything into the root directory (i.e administrator, components, plugins, modules, logs, etc...). Joomla seems the most intuitive yet it fails to distinguish between multi-site setups (if Joomla can even do multi-site setups?).

Doesn't this raise the question that contributed modules/themes should have a different filesystem?
Something like..

mymodule/
mymodule/mymodule.info
mymodule/mymodule.module
mymodule/mymodule.install
mymodule/inc/
mymodule/inc/include1.inc.php
mymodule/css/
mymodule/css/style.css
mymodule/lib/
mymodule/lib/jquery.js

..and so on. I understand that this is partially up to the developer, but a little encouragement never did anyone harm.

For the D7 part; I'm still with #43. Why do we think this is a usability issue? Why don't we document it better? Show explanations of the directories in the installer or admin/help, include a readme, refer to the readme, etc. Someone who uploads his D7 folder using ftp isn't looking for .txt files in the directory, he just uploads everything and clicks 'Install'. .txt files are not (new-)userfriendly.
Look at http://drupal.org/node/43816 for example. It's the first google hit for "Multisite drupal", but as soon as I load that page, I get blasted with irrelevant information. I only see "4.x" and "5.7" in the first comment. Where can I find 6.x? 7.x? Why isn't the first link on the page working?
D.O needs a cleanup. Bigtime. Don't get me wrong, there are tons of useful information available here, but there is even more irrelavant data around it, which makes it hard to find.

The problem isn't in the filesystem, it's in the documentation (of the filesystem, in this matter).

http://buytaert.net/8-steps-for-drupal-8 let's stop saying smallcore.

@chx - does that mean you're opposed to creating a folder called "core" ? Because, no one has mentioned "smallcore" until now.

Ok, folks, I just read through this whole issue for first time in a while -- we have some well meaning posts, but everyone's ideas are getting mangled and hard to follow. Let's create a wishlist of changes so Dries can weigh in on each idea.

@Crell or @merlinofchaos or @chx -- you guys have big names in the community, please weigh in and try to personally ask Dries to buy in to Earl's idea that this should be one of the first D8 commits.

Here's my first crack at our wishlist (we'll start with letters, please identify your remarks using letters or assigning your thoughts to new letters)...

A) Move all folders except sites (misc, modules, themes) to a single parent folder called "core" or "admin" or "something".

Pro: We can now let core upgrades work: #606592: Allow updating core with the update manager, our documentation that says "don't hack core" now has a real folder correlation.

Con: The only downside is the perceived amount of work involved. No one has yet argued against it.

Impact: Less "first time user" accidental core hacks, makes "sites" folder instantly recognizable as the only "non core" folder, may solve "D" issue right away

B) Move as many core PHP files (update.php, install.php, xmlrpc.php, etc.) as possible from root to parent "core" folder as possible. Keeping critical files (like robots.txt, index.php) in the root directory is implied.

Pro: Moving non-critical php files outside of the root directory "hides" the mess and makes the "core" folder concept much more powerful.

Con: People used to typing in "/install.php" or "/update.php" will get 404 or WSOD. Adds cruft to common administratable urls in the name of "clean up."

Impact: At best, we will have improved the concept of the "core" folder and made it that much harder for a first time user to tweak core php files. At worst, long time developers will feel alienated and frustrated. Middle ground might be to .htaccess some of those common admin files to their new locations.

C) Move documentation files (README.txt, INSTALL.txt) out of root

Pro: Having sooo many *.txt documentation files alongside critical core files is confusing and amateur

Con: The argument for keeping all doc files in the root is that they are right there when you install, obvious documentation, and already in the standard place for most software

Impact: At best, this change would not confuse first time users. At worst, first time users would be very confused. Middle ground might be to have one documentation file (readme.txt?) that points out a folder somewhere in the installation to check out further tips, like installation instructions, etc.

D) Rework "sites" to mean something to people who need to know where to drop custom files. (See comment #53)

Pro: A new user should be able to instantly know where to put their modules and themes

Con: The "sites" convention is well understood for all serious Drupal developers and this would pose a major WTF for people familiar with NOT adding files to the folder called "modules," etc.

Impact: At best, new users would have instant understanding. At worst, 99% of Drupal developers are facing major re-learning.

E) Keep "sites" but create a third "non-core" folder called "contrib" to replace "sites/all" (see #59)

Pro: Helps new users get started without the issues of D, Provides a place for installation profiles to hide their "special sauce"

Con: This creates a strange special case that may be so different that most people won't "get it." Adrian will probably hate the multi-site implications

Impact: At best, this helps new users and installation profiles. At worst, this confuses everyone.

F) Move current "index.php" to "core" folder and make the new "index.php" be a one-line include.

Pro: It's likely includes won't need upgrading. It's also likely this *might* be the first step in moving core off root.

Con: We're promoting indirection for the sake of clean-up and upgradability

Impact: At best, this makes upgrading that much easier. At worst, we've added a layer of indirection that causes many issues.

G) Keep "profiles" as "non-core" folder

Pro: It should be possible to upgrade an installation profile without hacking core.

Con: Creates confusion as to the difference between "sites" and "profiles"

Impact: At best, all major installation profiles gain some upgradability outside of core updates. At worst, we've added additional confusion to the whole "less is more" argument

Simplest set of folders I think is possible:

core/
sites/
profiles/
index.php
robots.txt
.htacess

H) Create modules, themes and files directories in root and leave sites for advanced users.

These 3 directories could be direct replacements for the existing sites/all/* directories, and operate as such. For users who want a multi-site setup, they can create a sites hierarchy in the same for as we currently use, with the same rules. settings.php would then be placed in the webroot.

Pro: It is very obvious for users who run a single website from a codebase (the majority case) as to where to put their downloaded files. It would also allow G to make sense.

Con: Create confusion for experienced users as to why the sites folder is missing.

Impact: At best, new users know exactly where to put their modules and themes. At worst, experienced multi-site users are confused as to the lack of a sites folder.

Problem with H: Where do we then put settings.php et al?

Honestly I'm -1 on H. Even when I started with Drupal /sites/ was never confusing to me. Really, the only confusion I had about file placement was fixed by adding sites/all.

There are two advantages to H:

1. less confusing for newbies, immediately visible where themes and modules should go,

but also, and while it sounds silly I think it's a major point, it

2. SAVES TWO CLICKS in your editor/IDE/FTP client every time you edit or upload a module or theme.

most custom work is under sites/[all|default]/[themes|modules]. If it was just /themes and /modules, it would probably save me 20 clicks or 10 seconds per day. But estimating 10.000 Drupal developers, it could save us over 5000 man-hours/year, better spent on hacking Drupal :)

As for settings.php, it can still live under sites/default, nothing wrong with that, rarely needs to be touched.

Fixing typo.

Since in actuality the module and themes folders already exist in the webroot folder, the net effect of H is not to "create" those directories, but to repurpose them for simple site installs and have settings.php default to webroot. I assume then that H would have to imply A (moving the contents of the modules and themes folders in the webroot to a 'core' folder).

Problem with H: Where do we then put settings.php et al?

it can still live under sites/default, nothing wrong with that, rarely needs to be touched.

Note that in #66, H's OP says: settings.php would then be placed in the webroot. Its a plus that it could exist under sites/default, for say, upgradability from the existing sites folder system, but I understand that the point of H is to remove the need for sites unless you intend to develop in a multi-site environment. If thats true, then everything in sites/default would (for lack of a better term) default to the public webroot folder for regular default site installs.

I can certainly identify with #68.2; Anecdotally, how many of you added contrib modules to the /modules folder the first couple times you played with Drupal? Repurposing the modules, themes and files folder is a lofty idea, but if I had to choose, I'd say retaining and imposing the sites structure is easy to learn with good documentation, and not a problem if the aforementioned folders aren't there in the webroot to confuse new users in the first place (in other words, if A gets approved for consideration).

Perhaps the best ideas to keep in mind in the thread are to aim for a webroot folder cleanup that permits core "updatablity" while maintaining site/other/custom/stuff "portability" independent of core; and to not confuse the heck out of newcomers, existing users, and important contrib maintainers.

This is the best comment to date and I'm so ready to rally around it:

Perhaps the best ideas to keep in mind in the thread are to aim for a webroot folder cleanup that permits core "updatablity" while maintaining site/other/custom/stuff "portability" independent of core; and to not confuse the heck out of newcomers, existing users, and important contrib maintainers."

We have to remember that if we change folder structure it will totally create more confusion than necessary. But a little pain in the transition (if done in the name of "clean-up" and "making it simpler") could be just what the doctor ordered.

To that end, and for general maintainability and performance, we want to have a clear, small number of places that "stuff" can live. That's one reason I don't think settings.php in / is a good idea.

Currently, Drupal safely assumes that everything it worries about in the installer is in /sites, usually /sites/default. A lot of Drupal management software (Drush, Aegir, and custom stuff that I and others have written) assumes the same pattern. If that pattern becomes more complicated, that makes Drupal and all of that auxiliary code more complicated as well.

As one of my professors in school once said, "overly complex code leads to overly complex bugs". :-)

That is, KISS, both from a documentation perspective, a performance perspective, and a bug-avoiding perspective.

KISS must be most definitely THE guiding principle for this thread having any chance of getting accepted. I agree with Crell that technically H is introducing more entropy than order and simplicity.

We have a good number of ideas all summarized now, is it time to start analyzing the deeper questions? It sounds like options A+B+C are generally accepted by everyone participating as a really logical thing to do in the name of cleanup, usability, upgradability.

But Option D, Option H and various previous posts (eg #53) are all attempts at addressing specifically one main issue: question #73.1: Is sites/ structure really that confusing or overly complex?

Purely speaking in terms of a perceived complexity, for which this thread exists to confront, and rash disregard for the necessary code changes that may be implied by some of the proposals, it seems like everything is boiling down to two (well, three) camps. Those who think sites/ structure as it exists today is the golden ticket, those that don't, and those that are still riding the fence.

Newcomers, non-developers and perhaps even some developers in the community probably think its not an optimal (not intuitive?) structure.

Here's a quick run-down of some excellent and important features of sites/ as it works today.
(Note that I'm using the word "project" to generalize themes/modules/profiles/etc, afterall, its not drupal.org/module/devel nor is it drupal.org/theme/zen)

* permit multiple site configurations: example1.com-settings.php|example2.com-settings.php,
* inherently distinguish site-specific projects: all|default|example.com,
* inherently distinguish project types: themes|modules|libraries, (what about profiles? are they the black-sheep? technically they are "projects" in contrib repository too)
* inherently distinguish project sources: contrib|custom|fservers,
* inherently distinguish project groupings: features|kits|distros|modules-within-modules,
* permit nearly any structure you can come up with, within the "modules" folder (ie, recursive folder exploration for *.info;)

Important maintainers may argue that sites/ works really well and that any issue regarding complexity is outweighed by the perceived effort and risk in changing existing code, which is to be sure, a natural fear of a paradigm shift that could potentially affect code integrity, as does the whole premise of this thread. (woah, did I really say paradigm shift?) But I remind we of the goal: usability; and the guiding principle: simplicity.

A goal of the Drupal project is to systematically embrace the concept of permitting people to easily override someone else's groundwork of sensible defaults, and BUILD on top of a great foundation. As a Builder, its generally accepted now as reeeely baaaad to have to come in contact with the man behind the curtain (ie hack core), thus, the guiding principle is that Drupal defaults shouldn't get in the way of site customizations. This principle should apply both in code as well as in the filesystem. This is why most people are for moving 'core' out of webroot.

The next logical step, then, is to draw a line between the needed drupal contrib projects, 3rd party contrib projects/profiles/features, and custom or glue code that all work together to hold a particular website together. But the way things seem to be heading with "distros", feature servers, kits, base-themes, etc. I'm seeing things standardizing on a system of yet more and more subfolders to handle these latest inventions of the community. Its almost like, the goal is to see how deep of a folder structure we can create to handle all these inventions. The recursive exploratory nature of the modules folder structure is pushing along nicely the adoption of this depth-first strategy. Following emerging trends, its not far-fetched to envision a larger single-site project with a structure like this:

core/...omg_easy_peasy_upgradable_core_in_a_single_folder_yay

sites/all/modules/contrib/...a_standard_array_of_contrib_modules
sites/all/modules/custom/...a_dab_of_glue
sites/all/modules/development/...the_tools_for_the_job
sites/all/modules/features/...exported_gui_settings
sites/all/modules/migration/...custom_drush_build_and_upgrade_scripts

sites/all/themes/base_admin_theme_from_contrib
sites/all/themes/base_site_theme_from_contrib
sites/all/themes/custom_skin_theme_based_on_base_site_theme

sites/all/libraries/...third_party_goodies

profiles/distro/modules/contrib/...
profiles/distro/modules/features/...
profiles/distro/themes/base_theme
profiles/distro/themes/skin

Do I understand this structure completely? Sure. But, I'm still scratching my head a bit... Is this an elegant solution? While it may be great for (paraphrasing Crell) "bug-avoiding, documentation, and performance", architecturally (perhaps visually), it appears to be about the farthest thing I can think of from the KISS methodology.

One perhaps philosophical concern I'm seeing being brought up by a few people now is 'Why does this important stuff need to be buried so deeply?'

What if we step outside the box and think a second, be a kid again, ahem, i mean.... a software architect. Temporarily forgetting the lines of code written on existing assumptions, could there be something better?

Thus question #73.2: is there a more optimal folder structure that can accommodate for and distinguish between the existing features and attributes of sites/ (some mentioned above) but has some sort of inherent simplicity or other added value that prepares Drupal for the future/long haul?

Its cool if there there isn't a better way, I'm just thinking out loud. anyway, fwiw, I apologize that I don't have any good answers... just food for thought, and, sorry for being so damn longwinded. This will probably be my last post in this thread, meaning ya'll can either ignore all this blahblahblah, or hopefully, if i said something worth while, someone will run with it.... so I'll leave with just a few more thoughts that crossed my mind while writing this up.

Lots of web projects have an intriguing security feature that amounts to module code residing outside of webroot. The whole 'core in its own folder' is a move towards that, but then what about contrib modules, which arguably are more susceptible to attacks than core? Leaving sites/ 'as is' certainly wouldn't permit that. Maybe this calls for introduction of a distinguishing between public web files (css/js/files/theme) and stuff that could potentially exist outside webroot.

Also, the special case that exists today of "profile" type projects being okay to exist in root but modules and themes not seems like a strange aberration in the of order and logic argument for the sites/ structure. If Option A were to be adopted, then what? 'profiles' will be left behind (outside of core/ and outside of sites/). A move towards order or special cases? Or are 'profiles' THE special case (versus features/modules/themes/X-new-invention)?

Is it not at least partially true that the momentous move towards adopting the sites/ structure was born at least partially out of 1) a need to have a place for people to put site-specific files, modules, and themes, 2) and so that folks don't 'dirty up' those inconvenient Drupal core folders in webroot, 3) in order to promote upgradability and code sanity among other things. Again, assuming we adopt option A, and get core off by itself, does that leave us with a really robust solution for a need that simply no longer exists? The only remaining case for a 'sites/' structure would then be exclusively for mult-site support (a la option H)... And around we go again :/

I am completely in agreement with kbahey.

IMO if you want to stick to the KISS principle then you have to stick to fixing the problems that are identified. The big problem that needs to be fixed is upgrading and preventing people from putting their files in the wrong place. Making upgrades as painless as possible is, IMO, the primary goal here.

As a proponent of discussing the value of the sites hierarchy, I vote to get the core move in, and discuss that later. To that end, I have opened up #760992: Where should site specific modules, themes and settings be kept? to discuss the sites issue, so we can focus here on moving core Drupal to a subfolder.

I also agree with kbahey. That sounds simple and doable for now.

Do we want to have DRUPAL_ROOT as '/' or '/core'? It seems to be a bit simpler to redefine DRUPAL_ROOT, but I'm wondering if we need WWW_ROOT, DRUPAL_CORE, and DRUPAL_SITES as global constants in index.php and so on.

I like the structure as proposed in #74, along with DRUPAL_ROOT, DRUPAL_CORE and DRUPAL_SITES.

+++++++++++1!!

I agree with #74
Marking as critical because this needs to be done as early as possible in development or it becomes impossible.
(sticks his neck out...)

This isn't critical. Definition of critical: "When a bug renders a module, a core system, or a popular function unusable." Source

I agree with kbahey in #74 (and not just because it's what I proposed in #25 :)

Anyway, can moving these files around be done in a patch, or is it something that can/should be done on the server with crazy archaic cvs commands to save histories (I seem to recall this being a possibility)? Obviously there will still need to be a patch to update the paths _within_ modules, etc.

I'm willing to tackle the patch part, just wondering if it should be a giant monolithic thing that does everything, and if it should actually move the files (remove/add)?

A CVS patch can't do this. There's some hacky back door that only dww knows how to use that involves mucking about in the CVS history directly that I try to not even think about.

A git commit object, on the other hand, can move files around like this without even blinking. :-)

oh, trust me, I know. I can't wait for the future.

so best way to proceed? how's this:

- map out dir structure
- create patch to update paths to new structure within drupal (inc, misc, etc)
- bribe dww to actually do the work to move the files in cvs

Alternatively:

- Map out dir structure.
- Wait for git.
- Bribe Dries to actually do the work of moving stuff around.

We still need to finish step 1, however.

a) At the presentation I gave about this at the Drupal Core Developer Summit, I warned that the single greatest threat to this issue is the high likelihood of a monster bikeshed, not any technical problems. Please, let's not "map out dir structure". It's already mapped. If we keep debating it, we'll never see any improvement at all.

b) http://drupal.org/node/369445 tells all about CVS renaming. Yes, doing this in Git would be better, but we don't have to block this on Git. If Git is ready when we create the D7 branch and open up HEAD (aka "trunk") for D8, lovely. If not, at the break-out session on this at the core dev summit, we all agreed it would be better to just make this change, regardless of CVS vs. Git. The history is preserved, and we even know that our CVS -> Git migration handles it properly (since we've moved around core files in CVS before).

So, please, no more FUD in here. ;)

Thanks,
-Derek

by "map out dir structure", I absolutely mean to use what's been mapped out in #74.

what I really meant was more "map out what changes need to be made in code to point to new directory structure". poor wording on my part. and maybe not even necessary, as those changes will need to be made by patch anyway. however I wrote "map out" because it will still be a massive, far-reaching commit, and keeping it up-to-date may be a pita (hence the "map out", i.e. keep track of what _types_ of changes we're making)...

and, yes I absolutely think git is not a blocker. it's also nice to know there's a script to do cvs renames--every other time I've seen it brought up, it was said you were the only person with those magical capabilities. :) hope I wasn't spreading FUD, because my intent was actually to get this thing moving...

Agreed with @kbahey, @dww, @merlinofchaos, @chx, et al.: /core & /sites sounds great. I think there was another issue along these lines which should be flagged duplicate of this one. (Or has it already been?)

subscribing, I had no idea this discussion has been going on since 2005 !

Like the original feature request I have wondered about the clutter.

When I first started using Drupal beginning with 6.1 the directories were confusing.

As a newbie, I was surprised that Drupal was not typically installed in its own sub-folder of the web root. Instead Drupal is typically installed in the web root, public_html folder where a number of files are placed and several folders with files, etc.

It would be great to have all the Drupal related files and folders in one directory.

From a usability perspective I like "core" and "sites" and would like the Drupal folder to have these two main sub-folders and all the other files currently in public_html folder.

From the various threads I have come across, including this one, I realize there are many considerations and issues including multisites, clean url, duplicate urls, seo etc.

And a variety of solutions including using symlinks, .htaccess, and possibly $base_url in settings.php although I have wondered if configuring $base_url in D6 causes filenames to be stored as a full filepath name rather than relative which can cause problems when migrating.

All of this is confusing to a newbie who is just trying to have a tidy server, but I have come to learn is part of the Drupal journey.

The features requested in the original post in 2005 resonate with my old newbie Drupal dreams,

1. Less clutter in the root directory.

2. Separation of core and sites files without intermixing.

3. This would simplify upgrades.

Subscribing. Also extremely surprised that this discussion has been going on for sooooo long.

I am planning on using a work around described at http://drupal.org/node/144643, to get around this short coming of Drupal. But like all work-arounds, I am sure it will bite.

Yes, this issue is active since 2005. The topic of this issue was my first thought when I downloaded Drupal 5 and Drupal 6 a couple of years ago. In Drupal 7.0-alpha7 we still have the following.

Folders in web root:

includes
misc
modules
profiles
scripts
sites
themes

Files in web root:

.htaccess
authorize.php
CHANGELOG.txt
COPYRIGHT.txt
cron.php
index.php
INSTALL.mysql.txt
INSTALL.pgsql.txt
install.php
INSTALL.sqlite.txt
INSTALL.txt
LICENSE.txt
MAINTAINERS.txt
robots.txt
update.php
UPGRADE.txt
web.config
xmlrpc.php

Having the Drupal admin folders and files together in one or two folders (excepting .htaccess, index.php and robots.txt) would make maintenance, permissions, updates, etc. easier. In our case, we have a lot of other folders and files in addition to the Drupal ones, and they are now mixed, which increases errors and security risks.

It would be indeed great to take this issue into account now that Drupal 8 is near to the beginning of its development cycle. Later it would be not so easy as now.

merlinofchaos wrote:

IMO we should consider this strongly as an early entry into D8 so that we can get it done and out of the way while there are still few patches against D8 (since it will invalidate, oh, every patch. And then some).

Yes, it looks like the best moment for this task is coming soon. D7 alpha7 has been released today, the first D7 beta will be probably out one of these days (there are just two remaining issues blocking D7 beta right now), and developers will soon be able to start concentrating more and more in D8.

Patches for this will affect multiple files. The following seems a good way:

kbahey wrote:

Perhaps we need DRUPAL_ROOT to stay the same, then DRUPAL_CORE is really DRUPAL_ROOT . '/core' and DRUPAL_SITES is DRUPAL_ROOT . '/sites'. Those can be the first 2 lines in bootstrap.inc (and friends, such as install.php ...)

Also, as mentioned in this issue, this kind of folder tree change seems much easier for Git than for CVS, but doable, especially if the change is made just at the beginning of the Drupal 8 development.

Seems that we have an agreement here.

Let's postpone on the Git migration.

I don't think this is an issue to be be postponed until a total CVS -> Git migration of all the historical code. If we postpone it until 2011, it will be probably postponed again until Drupal 9. I think all Drupal 8 development, with the new folder tree, should be done from the start with Git only, even during the migration of the old versions.

As said recently in Git Transitional Workflow Discussion (2010-09-06):

Even though the Drupal CVS to Git migration will take a few more months, it's possible to start using git now for nearly everything you used to do.

So, what is the consensus on this task: postponed until 2011 (CVS ending) or later (D9), or active for D8 and transitional Git (2010)?

No, it's not possible to start developping in Git and migrate old revisions later. The Drupal 8 tree is not open yet, and will not open before the 7.0 release. This can easily wait that we switch the Drupal Core to Git.

See comments #84 and #88. The CVS vs GIT issue has already been discussed and decided. It was already explictly decided that this issue should not be postponed due to GIT. Let's please keep it that way.

Damien Tournoud wrote:
> No, it's not possible to start developing in Git and migrate old revisions later.

Well, now I think probably that point of mine was a too unpractical idea, sorry. There is now a Git mirror where it seems everything works except committing. So it will be simpler to disable committing on CVS and enable it on Git for everything at the same time, rather than first for D8. Let's hope this will happen sooner than I expected, hopefully before Drupal 8 is open for development.

antgiant wrote:
> It was already explictly decided that this issue should not be postponed due to GIT.

You are right, dww -who for example designed the current Drupal release system- has said on this issue:

Yes, doing this in Git would be better, but we don't have to block this on Git. If Git is ready when we create the D7 branch and open up HEAD (aka "trunk") for D8, lovely. If not, at the break-out session on this at the core dev summit, we all agreed it would be better to just make this change, regardless of CVS vs. Git.

So it seems this is coming for D8.

Note: the fact the version says "8.x-dev" means this is effectively "postponed" anyway. Y'all can get in a pointless pushing match about it, but the facts remain that: a) we can't do anything about this right now, b) we know what we want to do (at least to start), c) we all want to see this happen in D8. Please let's not generate excess noise.

Thanks!
-Derek

now that we're officially in d8 development, and very close to TGGM, I thought I'd bump this topic so it's not forgotten about. Not sure if it still needs to happen asap after head is opened up (what with git's easy moving and all), however...

@joshuajabbour

The Drupal 8.x-dev page is still just a placeholder for D8 issues like this one. So, although I can be wrong, it looks like D8 is maybe waiting some days until the CVS -> Git migration on February 17.

Yes, there was an agreement about this issue being a task with significant repercussion -that is, major I think- needing to be done at the beginning of the D8 development. As dww said, it could be done on CVS -even if it's simpler on Git- but, since Git is coming so soon and people is focusing on the migration now...

However, I haven't seen yet an official word on a D8 opening date for actual commits, with dev releases, etc. Just the placeholder for this waiting issue queue.

Rather than, e.g.,

/core
/sites
index.php
.htaccess

surely it'd be a good idea to prefix the directory names so they don't interfere with paths that might be needed for clean urls, e.g. use

/d_core
/d_sites
index.php
.htaccess

If I'm right in thinking that we get 403 errors on any alias matching a folder in the drupal root, then even a reduced number of folders at that level would still prevent a webmaster using e.g. www.drupal.org/core or www.drupal.org/sites.

I found this issue due to getting a 403 Forbidden error from a page using the url alias "sites", which led me to http://drupal.org/node/121362, and then here.

If there's some way of viewing pages using aliases matching those drupal folders (sites, misc, themes, etc) that I've missed, I'd be very happy to hear about it...

EDIT:

1. I agree with @joshmiller - sorting the drupal directories by only using an underscore for the prefix would be good.
2. I've found that I can use aliases on existing directories (e.g. "sites") by amending the clean URLs rewritecond (http://drupal.org/node/246093).

Not to continue the bikeshed that we've been warned about, but @Dave_B makes a good point in the above comment.

What if we were to name the folders with one character instead of two characters. In most modern operating systems that would make these folders group together and be on top of a listing.

/_core
/_sites

Just a thought...

Some kind of prefix seems indeed a good practice. For example, I think another well-known open source CMS (WordPress) has three folders: wp-admin, wp-content, wp-includes. The suggestions on previous posts (d_..., _...), or something like "drupalcore" and "drupalsites", could be also good.

Subscribe.

And also +1 for just move things to /core first and deal with sites/custom folder after.

Good point made here, though I think d_core and _core are kinda ugly, and we don't do ugly round here ;)

'drupal_core' will do just as well, surely. Also, the standard form for menu paths is to use a hyphen to break words (eg admin/settings/date-time), so if someone does want a page to talk about drupal core, they'd probably alias it 'drupal-core' :)

I created a new sandbox trying to start #65 A.

There will be a lot of work, so my idea is to divide it into small moves for each directory (misc, includes, profiles and "modules and themes"), and branch it. Thanks to Git will be possible to merge all them easily later.

It's in http://drupal.org/sandbox/danillonunes/1098200. Help will be welcome.

I think this approach of multiple branches is cumbersome, you need to have all the changes made in the same branch to test whether it works or not.

Most importantly, instead of making manual changes into the core codebase, I think we should work on a script for automatizing this, like what kbahey started to work on in #65. I've imported his script from the legacy RO CVS into http://drupal.org/sandbox/scor/1102276 and started to make changes so that it works with git (move files around and do string replace across core). The script is about 120 loc so far, which will be much easier to review and agree on than an actual branch diff (which currently spans about 2000 lines).

Another thing to bear in mind is that in order to preserve file history, the commit that moves a file must not change it at all.

Hence we ultimately will need a commit on the 8.x branch that moves all the files and *breaks* Drupal, as no paths in code will be changed.

Another thing to bear in mind is that in order to preserve file history, the commit that moves a file must not change it at all.

Why? What if we took this gradual approach:

1. add a new folder to root: \drupal_core\
2. add relevant subfolders to the \drupal_core\ folder:
   \drupal_core\includes\
   \drupal_core\misc\
   \drupal_core\modules\
   \drupal_core\profiles\
   \drupal_core\scripts\
   \drupal_core\themes\
3. start, slowly & one-by-one, moving the files & folders to the corresponding \drupal_core\*\ folders, making making sure each 'push' is tested and functional.
4. lastly, rename the sites folder to drupal_sites

I don't think steps 1 & 2 would actually break anything. Those empty folders could simply be added without any ill effects (might need a dummy placeholder file).
Then, the real work could be split up and done in #3 in steps or stages. Thus files and folders could gradually migrate into the new folder system.
Lastly, step #4 should be fairly trivial (maybe optional, even).

Does that makes sense, or am I talking about things of which I know nothing about?

What I mean is that AFAIK if you rename or move a file in git (which is the same thing), the file must not change during that move if you want git history to be preserved. I seem to be repeating what I said in my earlier comment, I'm afraid, but I don't know how to phrase it differently.

If we want, say, the file system.module to preserve its history, then commit X must move it from modules/system/system.module to core/modules/system/system.module and NOT change its contents in that commit.

Hence after that commit, Drupal lies broken until the next (jolly quick!) commit fixes all the paths.

Another thing to bear in mind is that in order to preserve file history, the commit that moves a file must not change it at all.

--follow will give you all the history, including the changes the file had during the move. for a file moved and changed during the same commit, git log -p --follow core/authorize.php will say:

commit ddebf3a9f8ceadb703bddbcf4ed95e938ec4e80b
Author: scor <scorlosquet@gmail.com>
Date:   Wed Mar 23 11:17:47 2011 -0400

    move Drupal core files into /core folder

diff --git a/authorize.php b/core/authorize.php
similarity index 91%
rename from authorize.php
rename to core/authorize.php
index cd3df50..bc1ed26 100644
--- a/authorize.php
+++ b/core/authorize.php
@@ -23,7 +23,10 @@
 /**
  * Root directory of Drupal installation.
  */
+chdir('..');
 define('DRUPAL_ROOT', getcwd());
+define('DRUPAL_CORE', DRUPAL_ROOT . '/core');
+define('DRUPAL_SITES', DRUPAL_ROOT . '/sites');
 
 /**
  * Global flag to identify update.php and authorize.php runs, and so

About the points on folder naming in #107, if d_core and _core are kinda ugly, and hyphens aren't convenient, then other suggestions from above comments are:

1. core, sites
2. drupalcore, drupalsites
3. drupal_core, drupal_sites

I think probably any of these can be good, maybe better with the "drupal" prefix.

I was under the impression that the first step was moving to "core", the cleanest and simplest of the above options. Then, the second step would be to make the location of the core folder over-ridable (eg through a variable defined in index.php), so if someone needed the url "/core" or wanted to move core completely out of the webroot, they could achieve that.

I'd propose to leave "sites" alone for now, as 1) it seems out of scope for the issue as it stands now, and 2) it will add confusion and make upgrading a lot more challenging (see Earl #75).

Just joining in to say that putting everything in "core" that we can sounds great. Eventually I'd like to eliminate "sites/all" and put modules and themes directories right in the root (option H from the long discussion above), but I understand that we've had enough bike-shedding for now. Let's make the core directory and update the code. We'll save further restructuring for later.

To move forward we should outline the git commands needed to move the files from one location to another. That way Dries actually has some way of implementing the recommended changes. As scor said, we do NOT need to break core to have these changes made. We'll just use one series of move commands, do the git commit of the moves, apply a patch to the moved files, commit the patched files. Then push the series of commits back up to origin. No problem, no tests ever broken.

I also think this should be the first thing we handle in Drupal 8 (the nice thing being the "sites/all" directory has absolutely zero impact on further core development). After #1097100: Remove all 7xxx update functions and tests (D6 to D7 upgrade path) is completed (it's only a matter of hours away), Drupal 8 development will open and we'll need to move on this immediately.

+1 subscribe to the core, sites, profiles move for simplicity and beauty reasons (and i think this get us closer to my sysadmin's dream of having core installed in /usr/local/bin or somewhere i'd never find ;-) )

most of the work will be writing the script to replace all the paths... the latest git-aware version is at http://drupalcode.org/sandbox/scor/1102276.git/blob_plain/HEAD:/22336.php

I don't think a script will be able to handle all the use-cases we need (though it might help if this task takes a long time). I didn't use the script for this patch, I just did it "the old manual way" with a search and reading every place we do an include of Drupal core; call update.php, install.php, or cron.php; and all calls to DRUPAL_ROOT (plus some other searches).

RE: #78 and #79: I do not think we should make a new constant for DRUPAL_CORE, because there are many places where we need the string "core/" without DRUPAL_ROOT attached to the beginning of it. While we could make a constant for just the string "core/" and then another for DRUPAL_ROOT . DRUPAL_CORE, I find it would not actually save us any typing or be any more clear than just using the actual path string. Moreover we'd need to redefine that constant in update.php, install.php, authorize.php, index.php, and several tests. Generally just not worth it.

Another interesting hold-up is what we should do with the "profiles" directory. Some people have noted above that the "profiles" directory could be modified by distributions. Unfortunately profiles do not yet have the flexibility of modules and themes to be located in multiple places. Since we don't want distributions to "hack core" (which is what they'd be doing if they had to change /core/profiles), the profiles directory must stay in the Drupal root until we add more flexibility to install profile locations. This isn't too bad because you can still upgrade core just by replacing the "core" directory, since you'll never need to update the Drupal core install profiles after you've installed Drupal anyway. Unless there are large objections, let's save the "fixing" of profiles for a later issue.

This patch is already quite large, which goes to show just how big of a change this is even without further changes. This approach follows the recommendations of kbahey, merlinofchaos, chx, and others; so I'd like to see us start with this and fix all of the following later:

Things for later:
- Moving the "profiles" directory to "core/profiles".
- The "Option H" described in the comments above, moving "sites/all/*" to the root directory.
- Any other changes to the "sites" directory, such as removing "sites/default" or relocating settings.php
- The addition of a "docs" directory.

All of these things are great ideas and I'm personally interested in contributing (thoughts or code) to all of them, but this patch is big enough as long as we all agree on the direction being suitable. I've attached two files, one that provides a list of git commands to move the files to their new homes, and then a patch to be applied on top of the new structure that makes all necessary code changes. What say you guys?

Reroll of the patch against the proper Git branch (last one was against 7.0 apparently). Passes SimpleTests! Very cool. Regarding an upgrade path from Drupal 7, I think the change should be pretty simple. The only thing that looks like it's absolutely necessary is to rebuild the file/object registry immediately upon visiting update.php, since all the existing paths will be out of date. We'd probably need to do this eventually anyway if we ever added any new files to the database system.

@@ -1040,7 +1040,7 @@
   // Escape the periods in the extension.
   $regex = '/' . str_replace('.', '\.', $extension) . '$/';
   // Get a listing of all template files in the path to search.
-  $files = drupal_system_listing($regex, $path, 'name', 0);
+  $files = file_scan_directory($path, $regex, array('key' => 'name'));
 
   // Find templates that implement registered theme hooks and include that in
   // what is returned so that the registry knows that the theme has this
@@ -1200,7 +1200,7 @@

Is this necessary for this patch?

-  $files = drupal_system_listing($regex, $path, 'name', 0);
+  $files = file_scan_directory($path, $regex, array('key' => 'name'));

Is this necessary for this patch?

Yes this is an intentional change. When the theme system goes through and tries to find all the template files for a particular theme, it passes in the entire structure, such as "themes/garland" or "sites/all/theme/mytheme". It's mostly by accident that drupal_system_listing() matches files against "sites/all/themes/mytheme", since what it's actually looking for when it successfully matches is a theme in core. It also causes unnecessary checks at the location "sites/all/themes/sites/all/themes/mytheme" and in "profiles/sites/all/themes/mytheme". Clearly this isn't intentional behavior.

In any case, the reason it's needed for this patch is because trying to find the templates for a core theme needs to look in "core/theme/garland", not "core/core/theme/garland". I think the use of drupal_system_listing() entirely incorrect, since you really should only find template files in the same directory as the theme, not in one of several directories. Note that this only affects discovery of template files, not the discovery of themes themselves.

One more fix in system.module I found due to a SimpleTest failure of AJAX requests.

EDIT: Difference in patch size is due to rolling with different diff options (this time with -p to show functions).

quicksketch, on #121 and another constant -- it would worth it! Put the constants in constants.php and include it always that saves you the work of defining of it N times. And then aspiring people can put their Drupal where they please. Like, outside of the document root. Put the php files being used in docroot (just index.php in production, most of the time) edit constants.php and have fun. The point, of course, is somewhat heightened security.

quicksketch, on #121 and another constant -- it would worth it!

@chx: I really don't think new constants are worth it. There are literally hundreds of places in core where the path is already relative to the DRUPAL_ROOT variable, such as variables for cache.inc, password.inc, menu.inc are all defined as such: variable_get('cache_inc', 'includes/cache.inc'). Would changing this to variable_get('cache_inc', DRUPAL_CORE . '/includes/cache.inc really be an improvement? Like I said above we'd actually need not just one constant this way, we'd actually need two, (say DRUPAL_CORE and DRUPAL_CORE_ROOT), or we'd just end up using both constants all the time like this:

include_once DRUPAL_ROOT . '/' . DRUPAL_CORE . '/includes/database/prefetch.inc';

Or at best:

include_once DRUPAL_CORE_ROOT . '/includes/database/prefetch.inc';

I don't find this to be an improvement at all. The ideas of having a swappable "constant.inc" or moving the core directory outside of the Drupal root aren't necessarily bad, they're just not part of the issue that we're trying to address here, which is to improve the structure of new Drupal installs. The patch is already 160KB and it doesn't change anything beyond moving the files to their new locations. Let's keep it that way and keep re-architecting things in a different issue. At the very least, this patch makes future find/replace tasks much easier, because it adds the very-searchable string "core/" everywhere.

So, following quicksketch's (#128), chx's (#126), and others' suggestions, the current issue creates an unified core folder, and a follow-up issue could develop chx's really tempting proposal, for example #1116144: Customizable locations for Drupal core folders (constants DRUPAL_CORE, DRUPAL_SITES) (just opened for D8; there seemed to be no issue yet on that specific task).

Oh right... of course this patch isn't going to work if it's against the new file structure and not the old one. I'll need to roll a patch like scor demonstrated in #115 in order to get testbot to move the files to their new locations and then apply the changes.

Thanks juan_g for the new issue, I've posted some of my thoughts over there.

The patch including all the file move instructions is 19MB, which is over the 3MB limit allowed on d.o :(

And we really need to use an ordinary git patch that has moves. #1107552: Make git patch application more robust and use git for patch application

thanks DamZ

filed #1116914: failed to set database information error during installation, most likely due to the fact that the testbot excepts to find install.php next to index.php...

@scor the patch is big because it's a copy of all moved files. Also, if some file is changed, the patch will be broken. Instead of a patch, just send instructions to move directories like git mv modules core/modules

@danillonunes we're trying to get the testbot to test this. the testbot does not understand git mv instructions, and will likely never will. At best it will support git patches.

Uploading a git patch (to be applied with git apply or git am only) for when the testbot supports it. The file size goes down to 473KB. instructions on how to generate and keep this file size low are at #1116900: git format-patch displays file moves as whole verbose diffs.

After consulting with quicksketch it seems I already have what I want 'cos all PHP files are already DRUPAL_ROOT bound so please disregard #126.

@scor's #138 is definitely the way to go if we can get testbot to apply patches with git apply or git am, but this patch is just another huge patch using the old format to see if testbot will pass with a few redirects set up manually (install.php to core/install.php) in htaccess.

Confirmed bug in redirect if running in sub-directory which is preventing testbot form working since it does install.php?profile=standard&locale=en.

Okay boombatower figured out the issue being that my new redirects simply didn't work when Drupal was installed under a sub-directory of the web root. Since this is what pifr does, the redirect didn't work. Here's a new patch in the new format (two separate commits) instead of the giant format. We're going to see if pifr can handle it.

SimpleTest browser chokes on /core during install (pifr uses), so I tweaked pifr to force it to work. Another reason why we need the good old browser rewrite in core. Anyway here is the issue to track #1120030: Support moving of core files to core/ directory.

I fully support this initiative. Absolutely.

Would it be possible to postpone it until the huge backlog of D7 patches gets at least a bit cleared? This will break every patch, of course, and since we have about 200 patches that need to be cleared through D8 and then back to D7, it will make the process easier if we defer this one.

Marking postponed. Just move it back if you disagree.

BTW: Deploying PIFR code today that will allow git copies and moves to be allowed in patches (it now uses git to apply). #1107552: Make git patch application more robust and use git for patch application

#145: drupal_core_move.patch queued for re-testing.

@rfay: I would say we shouldn't postpone this issue since what we're attempting to do is get it to pass test bots at this time (which we can't do if it's marked postponed). I definitely agree that we should try to clean out the RTBC queue for D8/D7 first, but we likely still have a ways to go on this patch I'd like to continue pushing forward until it's RTBC, then we can give other RTBC issues priority above this one.

Thanks for the heads up on test-bot, I marked the last patch for a new review to try it out.

@quicksketch, right-ee-o. I'm all for getting this all ready. Jus twould hope it shouldn't get committed until the backlog is cleared.

Heh, apparently it honestly needed a reroll this time. New patch off of HEAD.

So glad we have git during this move :-) .

Would it be possible to postpone it until the huge backlog of D7 patches gets at least a bit cleared? This will break every patch, of course, and since we have about 200 patches that need to be cleared through D8 and then back to D7, it will make the process easier if we defer this one.

I worry that the backlog will not really go down that much: as some RTBC patches get committed, more NR patches will become RTBC. How many weeks would it take do you think? maybe we will be half way through the D8 release cycle by then... From what I can tell looking at the RTBC queue for D7 backports, we're only at 39 issues. Yes, this pacth would break many patches, but all it would take to fix these patches is:

s:--- a/:--- a/core/:
s:+++ b/:+++ b/core/:

I'd be happy to help rerolling these patches.

Another sad, destined-to-fail reroll. Just keeping it up-to-date while Pifr gets sorted out.

Oops. I confirm that #158 does not apply manually.

rfay@rfay-gazelle:/tmp/drupal$ git apply drupal_core_move_4.patch
error: patch failed: core/includes/theme.inc:1760
error: core/includes/theme.inc: patch does not apply
error: patch failed: core/modules/simpletest/simpletest.pages.inc:82
error: core/modules/simpletest/simpletest.pages.inc: patch does not apply
error: patch failed: core/modules/update/update.report.inc:57
error: core/modules/update/update.report.inc: patch does not apply

I know you'll apply the patch and run at least a couple of tests before the next one :-)

Would a git fork and pull be a better solution for this?

Weird, yeah right now I can git pull and it merges successfully. I then do a git format-patch per scor in #138, but then immediately it doesn't apply to a fresh install. I think I'm probably missing something on the git side of things. I'll investigate and see if I can get an adequate patch.

@Rob Loach: It probably would be a very good scenario for a pull request, though we'd need to get testbot testing that branch (though I suppose you can theoretically have testbot enabled on other projects, it might work). In any case we have a conflict (or something) right now that just requires manual work, I don't think it's a problem with our tools at the moment.

Yea, it would make it much more reliable to test that the testbot can work with /core...we can deal with patch later.

+1

New updated patch that I've tested against the 8.x branch and I'm sure it should apply. It seems that merging my conflicts increased the number of commits in this patch from 2 to 3, plus I forgot robots.txt in my commit so now there are 4 commits/patches total included in this file. If this happens every time we get conflicts this could make for quite a few merge commits being included in every reroll. I'll set up a sandbox we can enable testing on.

Well, dang. Testbot still can't install. No different from #134 and #140.

I've pushed my new sandbox up at http://drupal.org/sandbox/quicksketch/1147512. Perhaps we can get testing on it somehow?

It looks like drupal_core_directory_move/modules/dashboard/dashboard-rtl.css and drupal_core_directory_move/profiles are still there. Seems like these should be in drupal_core_directory_move/core too.

Is this all that's required to move the profiles?

git clone http://git.drupal.org/sandbox/quicksketch/1147512.git drupal_core_directory_move
cd drupal_core_directory_move
git checkout 8.x
git mv modules/dashboard/dashboard-rtl.css core/modules/dashboard/dashboard-rtl.css
git mv profiles core/profiles
wget http://drupal.org/files/issues/dashboardcssandprofiles.patch
git apply dashboardcssandprofiles.patch

Thanks Rob Loach for the heads up on the dashboard CSS file. Regarding profiles, see my comment in #121 about why they need to stay where they are (for now).

Yeah, sounds reasonable. After we do this move, we should consider moving profiles to sites/all/profiles and/or core/profiles, with the ability to enable/disable certain profiles in settings.php, or something. I've always wondered why we don't have a sites/all/profiles.

well by that I would assume you mean profiles directory in any sites directory which seems logical and is something I have actually looked for on a few projects.

Fixed up run-tests.sh.

git diff -C -M core/8.x origin/8.x

We had another issue about sharing a set of modules across multiple profiles, which is something core doesn't support right now. It would be good to consider that problem space as part of this re-organization.

@boombatower: So testbot should in theory be able to handle this now? Any idea why #172 is still failing to install Drupal during testing? Once testbot is able to install I'm all-over getting 100% of tests passing. Right now it still seems like we're somehow hung-up on our tools.

@pwolanin: This patch does absolutely nothing outside of moving files around and adjusting the code to support the new locations. There are plenty of things we can do to improve our file structure, but we're saving them for follow-ups. As a 300-400KB patch, this is already a huge issue. See #121 for other reorganization ideas that are also as follow-ups.

We just need to deploy it now. With the redirect it gets farther to the point where it got stuck locally. Rfay has some things he'd like to get in as well so hopefully we can push a deployment this week.

FYI, the code to support this in PIFR is committed and I'll try to test it in a real testbot environment probably next week.

So sad to say this... This patch is stale again and does not apply.

git apply ~/tmp/drupal_core_move_6.patch
error: patch failed: modules/node/node.module:1455
error: modules/node/node.module: patch does not apply
error: patch failed: modules/simpletest/tests/batch.test:366
error: modules/simpletest/tests/batch.test: patch does not apply

That means I can't test the testbot against it. I want to get that stuff committed to PIFR so will go forward without testing this part of the release unless a reroll shows up shortly. It's possible that with an updated patch here it will work fine.

@quicksketch: Just noticed your request to test your sandbox. We can't test sandboxes, but can test full projects. Just promote it and let boombatower or me know and we'll turn on testing. (We've never tested a drupal fork before, so that may cause its own problems... I'll bet it will...)

In sandbox:

git remote add drupal git://git.drupal.org/project/drupal.git
git fetch drupal
git rebase drupal/8.x
git diff -C -M drupal/8.x 8.x

It seems there was just one merge collision in common.test; I think I got it right.

This is committed to the sandbox (didn't know I had commit rights, but to keep things going I'll commit)

This is committed to the sandbox (didn't know I had commit rights, but to keep things going I'll commit)

Yep, I gave both you and boombatower maintainer access when I set it up. Please feel free to push code as necessary! Thanks for your help on this, I was really beginning to believe we'd just be stuck forever on testbot failing.

Actually, I did fail to commit, as you can't rebase a sandbox.... But I am in the process of deploying this today, so maybe we'll be done!

That or you could push to a new branch?

OK, changes to testbot deployed. What will happen? Fingers crossed.

#180: drupal.core_move_22336_180.patch queued for re-testing.

IT PASSED! YAY!

Note that I did *not* commit this patch. I think in the future the way to go forward will be to create new branches, since that's the drupalish way to rebase (since you can't push rebased stuff)

Ooops. That was a "0 passes". Yuck. We're going to have to work this one.

My current theory is that these are caused by the testbot timing out or something, although in this case it didn't have time to time out (90 minutes).

See #1156976: 0 tests passing is not green for that problem.

Followup in #1192680: Testbot fills up /tmpfs partition.

Sadly.

Why did the patch change from 345.86 KB to 7.02 MB? Yes it is creating the /tmpfs problem, but seems like patch just needs a reroll or something to get the size back down?

I was surprised at the size change as well. Maybe you can get it down.

However, I don't think the size of the patch has anything to do with the tmpfs fail.

This patch is still large, but it applies and I believe is correct. I don't know why it's so large.

In sandbox...

git checkout --track origin/8.x -b sandbox_8.x # New branch to rebase on
git fetch drupal # the actual commits from core
git rebase drupal/8.x
# Fix the js merge conflict in common.test again since I lost that commit
git rebase --continue
git push origin sandbox_8.x

We'll either need to delete the sandbox_8.x and recreate it each time we rebase, or create new branches for every rebase.

This will be pretty painful if it takes months. Since we touch so many files (all of them) merge conflicts happen awfully easily.

Yay, we got a test result through!

In addition to the test #fails, it seems that simpletest is not deleting its tables as it goes. We got up to 2500 tables in the drupaltestbotmysql database before this was over. I had increased the available memory for the mysql databases. It had in fact hit the limit in earlier tests.

BTW, Fabianx suggests that doing a git merge will be every bit as effective as the rebase, and still allow pushing the branch without renaming it. He's right.

Needs work, of course.

Several of the failures were due to the lack of core/ on the front of includes/language.inc.

Sorry if this is a stupid question, but did you forget the "-M" flag when rolling the patch? The earlier patches were actually renaming the files while the newer patches are deleting the old ones (thus, printing their entire content) and adding new ones (thus, printing their entire content again).

A couple of more low-hanging fruit. There are some items in here that are harder, but these are a couple of easy ones. Commit f1fd0963d844ed1e39d33ada8b440f54e748ed9a from the sandbox.

@tstoeckler, yes, git diff -C -M. If you look at the patch I think you'll see lots of renames... just not as many as we wished.

Everybody, of course, can check out the sandbox and see if they can wrangle the patch creation better.

git clone git://git.drupal.org/sandbox/quicksketch/1147512.git --branch sandbox_8.x
git remote add core git://git.drupal.org/project/drupal.git
git fetch core
git diff -C -M core/8.x

This version resolves the xmlrpc tests and also allows drupal_system_listing() to peer into the profiles directory, which inexplicably is not in core/profiles. Is that OK? It sure seems like there should be a core/profiles directory and also a sites/all/profiles.

Sandbox commit 5605366425cad75abfc2c49687c07aa77c8eb96c

I believe that only the https tests will still be broken after this one. Can't test those until I set up https on my (nginx) local :-)

This one fixes up the https url used in the session testing. It finds its way directly into the test directory, so needs core.

Commit 7aa7403d33275c253cfa83a27390417d7e27241f

Well YAY! Clean tests.

I encourage you all to move quickly on this, as it's quite difficult to maintain. Go lobby those maintainers quick!

One thing I noticed (since I run nginx on my local) is that the built-in assumption you can use /install.php and friends is invalid, since nginx doesn't respect .htaccess. Of course, we've always looked the other way with configurations that didn't use .htaccess, so perhaps this is nothing new.

And restating: Do we really want the core profiles directory in /profiles?

• Applied cleanly, except 2 warnings:

  warning: squelched 17 whitespace errors
  warning: 22 lines add whitespace errors.

• Fresh installation: Site is running. (Didn't check the update process though.)

Good work!

I tried to split the above patch in 2 commits. One purely for renaming and one for all other changes. This gets the patch down to about 300KB, and I'm crossing my fingers that it passes. If not disregard this patch in favour of #204.

#204: drupal.core_move_22336_204.patch queued for re-testing.

This strikes me as an issue that should not be done via patches at all, but via a git sandbox and just rebased/merged when we're ready. The patch workflow is just not suited to this sort of change.

@Crell... It's been in a sandbox the whole time. But we needed to get it tested.

Then I'd vote for getting the test bot working nicely on sandbox clones of core... Doesn't that already Just Work(tm)? If not, is/are there an(y) issue(s) about why not? We'll need that more and more, especially with D8 initiative sandboxes, so we might as well invest the effort in getting that part working, instead of rerolling this monster patch...

Furthermore, as much as I'm excited to see this move forward, this is exactly the kind of patch that's going to make life miserable for #1050616: Figure out backport workflow from Drupal 8 to Drupal 7, and there's still a *lot* of D7 fixing to do. I think I'd be in favor of postponing a merge from this sandbox into core until the D7 major + critical counts are a lot more under control.

Thanks,
-Derek

It should work fine with existing pifr and testbots, but you will need to trigger it manually since the d.o integration doesn't support a sandbox that replaces core (although qa.d.o and testbots do). d.o only sends them as contrib modules.

Take one setup (I can do it if we want) and then we simply press the requeue button on qa.d.o.

Nope, there are at least two major problems

1. All testing is based on *releases*.
2. All testing assumes that *drupal* is the basis.

So both of those will need to be solved. IMO testing should be based on tags or other committish, but it's currently based on releases.

Instead of bogging down this issue with the details, can the testbot team create the appropriate issues for these subtasks (if they don't already exist), perhaps standardize on an issue tag (if needed), link to those from here, and then we can safely leave this issue for discussion of the file restructuring itself?

Thanks!
-Derek

@rfay: pift makes those assumptions (as I noted), pifr does not. You can override the core repository as Examiner and a few other companies using it do. Although after thinking about it I realize that you would need to alter a variable on the workers.

To test a core fork using the current PIFR/PIFT requires among other things:
- alteration of a variable on the workers
- manual setup of the job via an XML-RPC call

Given that we only want to run this fork on a single issue, we would need to set up an alternate qa.d.o to run only this job to. Otherwise, if we changed the existing workers for qa.d.o all patches would use the forked repository.

This use case was incorporated in the design of ReviewDriven and is therefore supported out-of-the-box with ease. On behalf of ReviewDriven, we would like to offer use of the service for this issue. We will make available an account for two weeks to test this issue daily (or on demand). Those interested in this issue, please comment whether you would like to utilize this offer.

@boombatower: awesome, please do so. Thanks ReviewDriven!

Is this still as per #74 about making the top level dir structure like this:

index.php
.htaccess
robots.txt
core
sites

?

To the extent that this is a usability issue: please go ahead. It was pretty scary figuring out where to put what when I started using Drupal. Having only two folders 'core' (sounds like a don't touch, great) and 'sites' (yeah, I want to make sites, cool!) would certainly help.

Oh, but where will readme.txt live now? Without reading all 220 comments here I'd expect readme.txt to live in root.

+1 for including a README.txt file in the root directory, but only one such file. As I believe someone already suggested, it might be nice to move all the other text files currently in root to a new "docs" directory. The main README file could then refer users to specific docs as appropriate.

I'd also recommend adding a file at sites/README--about_sites.txt. This, more than anything else, would help newcomers to Drupal in getting their sites set up. This readme file could explain exactly what goes where in the sites directory, including for multi-site installations. It would duplicate much of the documentation in sites/default/default.settings.php, but that's a good thing. Newbies don't know to look in sites/default/default.settings.php for instructions. I sure didn't.

WOW, great work rfay! I didn't think I'd see this moving anywhere for quite some time. I'd really just about given up on it. I also would really love to see this committed sooner rather than later. The number of patches in the queue only goes up, not down. Though as I've heard, we've got a long ways to go towards maintaining the "no new features when there are more than 10 criticals or 100 majors". The number of blockers on this issue is ridiculous and the more compromises around testing people are willing to make the better. This has already been a herculean effort by the testing team to get this as far as it is currently (just passing the tests through patch form). As long as our patches are properly formatted with file moves instead of file delete/add, we're in good shape.

As for the README.txt file, please look at the actual sandbox: http://drupalcode.org/sandbox/quicksketch/1147512.git/tree/refs/heads/sa.... REAME.txt is still in the root, all other .txt files are in the "core" subdirectory.

Making the "docs" directory is a good idea but it's a followup. Same goes for moving the "profiles" directory. Please see #121 for a list of followup tasks.

I've merged all of @rfay's patches back into the 8.x branch in the sandbox (that's http://drupalcode.org/sandbox/quicksketch/1147512.git/tree/refs/heads/8.x). Let's continue using the 8.x branch for our work.

This patch is slowly getting bigger because every time we make compensating patches to fix conflicts it's another commit bundled into the patch. However with this reroll, all files are renamed properly and the patch size is a much more reasonable 900KB instead of 8MB.

For future reference, do NOT roll patches with git diff -C -M drupal/8.x 8.x. This does not handle file moves properly. Use this instead:

git format-patch -C -M origin/8.x --stdout > patch.patch

Okay, the problem with that approach is that it makes Git attempt to re-apply changes that have been fixed in later commits, thus causing conflicts when you try to apply them all at once. Here's the approach I used to fix this problem by making a new sandbox branch:

git checkout origin/8.x
git branch sandbox2_8.x
git checkout sandbox2_8.x
git am 1-core-move-files.patch # This is the first commit pulled manually from #224's patch.
git diff sandbox2_8.x sandbox/8.x > 2-core-update-files.patch
git apply 2-core-update-files.patch
git commit -a -m "Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory."
git push sandbox2_8.x
git format-patch -C -M origin/8.x --stdout > patch.patch

It's not real pretty but I think this will work. If this patch passes, I wouldn't mind deleting the existing sandbox 8.x branch and replacing it with this new cleaner branch for ease of maintenance.

Re #221, #222: As you can see in the patch README.txt is left in the root directory, all other text files move to core/. Moving those into a separate docs/ directory is for another issue, or a follow-up at least.

Yes, thank you. It was not obvious for me to know where to look for what is being changed here. Please do continue :)

With all tests passing, my manual testing not showing any problems, and another review of the patch looking good also, I'm going to mark this RTBC.

Note that I'm not necessarily advocating this patch be immediately committed (although I'd personally be happy with that). We're still working on getting the Drupal core major/critical queues down lower, but I'll defer to maintainers to weigh in on the situation. (Though I will say that right now is an ideal time, with 7.4 just released and hovering right around the new official cap).

This patch does nothing beyond A: Moving most of the root files into a "core" directory and B: Updating files so that the new location works. There is also a side-effect change to the way folders are scanned for new themes/modules (see #124 for the explanation).

Please see #121 for a summary of what this issue does not cover.

On commit timing, it would be nice to see this queue a bit shorter as well as any thresholds before we break every patch in it:

http://drupal.org/project/issues/search/drupal?status%5B%5D=14&version%5...

Hmmm......

/var/www/drupal/8 $ git apply drupal.core_move_22336_227.patch
error: patch failed: core/modules/simpletest/tests/file.test:427
error: core/modules/simpletest/tests/file.test: patch does not apply

#227: drupal.core_move_22336_227.patch queued for re-testing.

Yea, the endless cycle of rerolling. Need a firm data point when this can get in or just get on with it, but no sense continually rerolling this with the _hope_ that it gets committed.

Well, one thing I definitely do know is that this can't get in before our thresholds are met. So if you want this in faster, start fixing some major/critical bugs/tasks. :)

I was pointed to this, with big surprise about @webchick's comment. What you are basically saying, any major change cannot happen unless we meet our thresholds. Even criticals, its unlikely we meet the standard anytime soon.

Sorry but enforcing this standard to prevent improvements to core, will halt innovation and most certainly demotivate major contributors from working on new stuff. Please reconsider or at least discuss this standard, because it leaves me in big worry about the future of Drupal 8 (which frankly hasn't moved much since Drupal 7 release).

That's exactly what I'm saying, yes. See http://drupal.org/node/1050616 for the full story. The goal is so that we never again end up with a release we can't ship for 18 months due to clean-up requirements.

I'm not exactly sure what you're concerned about though. We're only 3 issues off "unblocking" Drupal 8 (major bugs / tasks are > 100), and several of them are marked RTBC. I anticipate this problem being solved by roughly Tuesday.

Yes, we are close - which is very nice. But the bad precedent is here that we should focus on the critical/major issues and not on new issues that innovate. By saying, guys stop even thinking about getting this in before our thresholds are met - you are also saying stop thinking about it. I know that's not your goal at all, but I have seen this "stall" a number of larger issues.

Only time will tell if we will stay below the threshold long enough, to burst activity on these kind of major tasks. But I will shut up till Thursday and hope, this period of us being above the threshold doesn't occur too often. Because when it does, its very effective in stalling progress on issues that are about innovating/changing something big.

I think its at least a worthwhile consideration, if it does occur more often to have committers feedback or some kind of "can be committed when our thresholds are met" or even just some feedback.

I know its a hard policy to enforce, so I understand that being strict is important.

@bojhan: Can we discuss the policy at #1050616: Figure out backport workflow from Drupal 8 to Drupal 7 and leave this issue for discussing the file reorganization? ;) Thanks.

Arg... well set out to reroll this today and after 30 minutes found it's not worth my time. Core has got not only a new conflict but a couple of new files. LMK when this is worth it to actually update. :P

Yeah, this issue is "special" because once it's committed, every. single. patch. thereafter gets immediately more difficult to backport. It also immediately breaks all existing patches in the queue, causing weeks or months of additional work doing patch re-rolls. So it's important to be strategic about the timing (assuming Dries is willing to commit it, which he hasn't said yet, afaik).

I'm going to try and bring this up with Dries this week.

every. single. patch. thereafter gets immediately more difficult to backport.

It's not going to be difficult to backport to D7, it's going to be ignoring 2 directories instead of 1. People applying patches regularly are using -p1 right now anyway to skip the a/b prefixes, just use -p2 and all D8 patches apply to D7 as well as they do today. Re-rolling existing patches means adding "core/" in front of existing patches.

I still maintain that the longer we wait, the MORE painful this becomes, not less.

YAY! Now that #1036132: Provide a mechanism for issue summaries is deployed, at 244 replies, this issue could really use one. :)

Could someone please edit the issue body above using the issue summary template? Dries said he can probably take a look at this issue tomorrow night, and this would save a bunch of time.

Possibly scor's #157 (on backports to D7) suggests a reasonable solution for this timeline dilemma. After all, it's a "core/" string replacement.