protect Drupal core .txt files

if we do that we need to take care we don't accidentally "protect" files in our /files directory too.

what about something like this in ROOT/.htaccess:

# prevent site fingerprinting via release related files
<FilesMatch "(^LICENSE|CHANGELOG|MAINTAINERS|INSTALL).*\.txt$">
  Order deny,allow
  Deny from all
</FilesMatch>

this *would* prevent files with above names from being downloaded from files/ directory. though that can be undone with an additional entry in files/.htaccess if it's a problem.

Or, move the .txt files all into something like docs, and add an .htaccess file there.

pretty conventional to have README, INSTALL, etc, at the root dir of a software distribution.

While it is common for a software distro to show up with such files in the root, that doesn't mean that it is is common for them to end up in the root of an install.

At the risk of delivering a boat load of red herrings, this, to me, points up a general irritation with drupal -- the assumption that you will simply unpack the tarball in the web server directory corresponding to the URL. This makes it hard to stage/upgrade new versions. I've set up a redirect structure so that my actual drupal file set is not in the web server HTML directory root directory. Thus, when a new drupal comes out, I can unpack it parallel to the old version, and switch the redirect when I am good and ready.

it really would be nicer if the only files that landed right in the web directory were index.php and some sort of setting file to go with it. The distro could be unpacked someplace out of the line of fire, and then come with a 'make install' sort of thing that push the necessary materials into the web directory.

I'm downgrading from critical as this doesn't impact the proper functioning of a drupal installation. It is still a matter to be considered. If you want to get nitty gritty thought I can probe the url callbacks for not founds and access denieds to fingerprint a site or look for some key lines of text in the rendered page... variations in css locations the contents of drupal.js, or many other locations. I think this is simply an moot issue considering the number of ways a site can be finger printed.

Please let's also encrypt the HTML that Drupal spits, because you can signature check on that as well.

(Psst. I agree with dopry. This is getting on the verge of ridiculous.)

I hadn't noticed this previous thread when posting http://drupal.org/node/105183, but as I noted there, the new install system allows us to have these files automatically removed after install.

That way, they are still there in the distribution, but don't give such gratuitous easy access to the site level information.

Also, since this affects the no-yet-released 5.0 as well, changing the version info.

I personally like Harry Slaughters suggestion.

I know there will be many ways to probe a site for what versions or what CMS its running - however (and this is my personal belief) if one believes "well there are other ways to find out so there is little point blocking the easy ones" then there isn't a lot of point bothering with security updates at all - they're gonna find out anyway, might as well not bother...

Although holes get patched up, if someone only needs to open CHANGELOG.txt to get an idea of which version you're running then they immediately have a list of all known security holes between the version in CHANGELOG and right now. Its then a simple matter of poking them and seeing which ones break it.

I agree that it probably wont stop a hacker with a 'good' brain and an attention span long enough to figure out a way to break your site - but I think it might go a long way to stopping stupid hackers and spammers who just find loop holes in things just to advertise Viagra on your site.

At the end of the day - adding Harry's solution simply blocks access to those files while leaving them there for an admin to read. And its only 4 lines - I cant see that hurting server performance can it?

Just my 2 cents..

@fgm, we can't remove the files on install unless the entire Drupal directory were writable by PHP, which is a much, much bigger security threat than a few simple text files.

From http://drupal.org/node/213921, which was marked as a duplicate of this issue:

We discussed in #drupal-dev for a /long/ time about this. While there are numerous ways to fingerprint a Drupal site, nothing is quite so straightforward as a text file that tells you exactly. ;)

Joomla's solution to this is 1) a hack and 2) do-able because they're using a revision control system that can track revisions through file name changes. CVS cannot, without admin intervention.

So if we employed the same fix, we'd of course want to do so across both core /and/ contrib, which means driving dww completely batty with requests to manually edit server-side CVS files. Blarg.

So we decided instead to fix this in .htaccess, and disallow direct viewing of specific file names.

This both:
a) fixes the problem for core and contrib both.
b) enforces a good naming convention in contrib for these types of files.

This is also the same fix that prevents direct viewing of .module files. So if this protection is missing (IIS, etc.) you are completely screwed anyway, because an attacker can look directly at the exact line of code and see if you've patched it or not. :P

As far as how far to take this, it depends to what extent we want to protect these files from wandering eyes.

CHANGELOG.txt will identify the Drupal version down to the point release.
INSTALL*.txt and UPGRADE*.txt will identify the Drupal version down to the major release.
MAINTAINERS.txt, COPYRIGHT.txt, and LICENSE.txt /could/ change each major release, as well.

I'd recommend at least COPYRIGHT.txt, README.txt, INSTALL.*.txt and UPGRADE.txt being added to the filter.

There's a patch at http://drupal.org/node/213921#comment-704649 which does the first two.

Er. Still CNW.

This one goes hog-wild and does all *.txt files in the root directory. It also includes a (probably stupidly overly permissive.. Morbus et al, help?) FilesMatch override in files/.htaccess so that you can attach a CHANGELOG.txt to your nodes.

Ok, well that's a bit too permissive. ;) Lets you view sites/default/files/.htaccess. :D

subscribe

Here. Let's try this.

Note: I have no idea what I'm doing. But it seems to work.

Bigger patch, because I couldn't fricking stand that long-ass line creating the .htaccess that was impossible to read, so now it's concatenated. :P

at least for apache 1.3, it has this in the default httpd.conf to protect .htaccess, etc, files:

# Also, folks tend to use names such as .htpasswd for password
# files, so this will protect those as well.
#
<Files ~ "^\.([Hh][Tt]|[Dd][Ss]_[Ss])">
    Order allow,deny
    Deny from all
    Satisfy All
</Files>

Hm. For that matter, earlmred brought up, why not just disallow /all/ .txt files everywhere in the source tree? And let the files/.htaccess file specifically allow it there.

Then I wouldn't need that god-awful hack. ;)

There! Let's try this.

margh. robots.txt.

Also, we're discussing in #drupal doing a whitelist, rather than a blacklist.

Also, we're discussing in #drupal doing a whitelist, rather than a blacklist.

This is good from a security standpoint. Start out with access restricted. Then open up only what you need.

But we can rename only core txt files to TXT.php instead code the htaccess. And just put a security tips/note, to contrib modules.

This is the most unfriendliest patch I have ever seen in my life (zing!) Earth to Drupal developers: people install things besides Drupal on their site. This patch, on disobey.com alone, would a) destroy the AmphetaDesk documentation which is meant to be public (README.txt, CHANGELOG.txt, etc.) and b) destroy all of the stored zines, started before Drupal was born, in http://www.disobey.com/detergent/, and those are just things off the top of my head. Hell, you know, some people store text files without the .txt, let's go one further and add in text/plain filtering for MIME types. Yeah!

If we really must, must, must, have protection (and it appears that the cacophony of MUSTS! constantly overrides the sanity of reason), I would suggest something along the lines of #26: rename the Drupal-related files to something Drupal specific, such that one can block only those. Alternatively, make a root level "docs", stick them all in there, and an .htaccess that Denies all access to that [as you've already suggested, webchick]. Ideally, though, this won't have any effect on installed third party modules.

(.htaccess)
"text/plain filtering for MIME types" don't all webserver support this.

I think that the best patch is really, to rename CHANGELOG.txt to CHANGELOG.TXT.php, and so one...

and add that code to the beginner of the file:

// No access from webserver
die( 'Restricted Access' );

Joomla made this security

Is quite easy, and don't create others problems, and work in every single server that suport Drupal (PHP).

@Morbus: So. You really liked the patch then? ;)

I don't know that it's a MUST, but it is a no-brainer thing that will greatly increase the security (or at least decrease the perceived LACK of security) for site admins out there, much like the patch that got in 6.x this time to not blurt out database details when the database is unreachable. Like, seriously now.

I like the /docs fix. It doesn't work for contrib, though. And currently, if a security release is put out and README.txt was changed between version 5.x-1.0 and 5.x-1.1 (security release), I can write a spider that'll go through my nice catalog of fingerprinted Drupal sites, and check for sites/all/modules/buggymodule/README.txt and give me a lovely list of targets. Particularly nice on those arbitrary code execution bugs.

Any other bright ideas? Or do we call contrib "not our problem?" If we do, then the /docs solution would be best. txt files are NOT php files.

@int: As stated in #13, this is a hack and not workable for contrib without a tremendous strain on the CVS admins, assuming we also want to protect contributed modules. If you're not being protected against viewing of .module via .htaccess (or equivalent in $web_server_of_choice), then viewing .txt files directly is absolutely the least of your problems.

@webchick

if there is a .htaccess file in a directory, does the FILESMATCH directive overwite one from a higher level directory, or just get added?

One thought:

create a documentation form (call it something very obvious like "documentation" or "readme") and but the core docs there.

Add either in the main .htaccess or in additional ones, block docs-type files (*.txt, README.*, CHANGELOG.*, etc) for /sites /modules /themes as well as for the new /documentation

This would generally protect contrib modules but would not arbitrarily kill other scripts installed in the same webroot (per Morbus).

webchick: that's exactly my point. This is /not/ a "no-brainer", and it's entirely about satisfying a perception ("perceived lack"). The real question is: how do we stop Drupal from telling people what version it is? And the proposed solution, to obscure text file access, is not a solution at all, and is often lamented against ("security by obscurity"). And, really now, people bring up the argument about "well, it's a small step that will force hackers to work a little harder". None of these statements contribute to the strength of the patch or even the /idea/ of what's being proposed.

"work a little harder", really? Check your webserver logs. How many IIS vulnerability URLs do you see? I'm still seeing tons. I'm still seeing phpBB requests (and I've never run phpBB). I still see Movable Type requests, for something I stopped using back in 2002. Arguing that this patch is a "no brainer", when it can a) disable legitimate parts of the site, and b) is disabled in an odd (often invisible) file, making "WHY WON'T THIS TEXT FILE WORK, ARGHH" problem solving even more difficult, is wrong. Crackers don't give a crap what version of the software you're running - they don't even care if you're running the software! They're just gonna run their little rootkits for weeks, then take a look at the list of matches they got. Detection is automated - obscuring the text files isn't useful - wait a little bit and the rootkits will be updated, the bots will be written, and all the damage the patch causes (by turning off every text file everywhere) will be in vein. And if we're talking about "real" hackers, making them "work a little harder" is a very ill-gained benefit for the destructive nature of what's being proposed.

@pwolanin: It almost seems to do "inheritance," from my testing.

So basically if .htaccess in the top-level directory says "don't allow .txt, .php, or .module files" and in files/ it says "no wait. do allow .txt files" it will disallow .php, .module but not .txt.

I was also very tired last night, so you may want to actually test it. ;)

A documentation form? You mean like hook_help()? The problem with any kind of in-Drupal solution is that if your database has exploded or you're in the middle of an upgrade, those docs are inaccessible to you.

Add either in the main .htaccess or in additional ones, block docs-type files (*.txt, README.*, CHANGELOG.*, etc) for /sites /modules /themes as well as for the new /documentation

This would generally protect contrib modules but would not arbitrarily kill other scripts installed in the same webroot (per Morbus).

That's a good idea. Although I think Morbus also is saying that sites/all/* shouldn't be affected, because he has modules that he's written expose their .txt files as public documentation.

To him I'd say, "Make a menu callback." ;)

Ok. Consensus reached in #drupal:
- Disallow the "root" .txt files that Drupal ships with.
- Disallow *.txt in sites/*

Moving to docs/* and disallowing from there is not a good fix, because as Morbus pointed out, some people (gasp!) might be using other things than Drupal in their site, and might already have a folder called docs/* and might not look kindly to all of a sudden their text files being unavailable. ;)

Anyone know how to write this patch? :)

um, after further study it doesn't seem like we can do what webchick and I suggested *except* by adding a .htaccess to /sites, and perhaps moving the docs to there too. To do it right would require access to the global httpd.conf file and the use of <Directory >

the Files and FilesMatch directives don't seem to allow any directory restriction.

http://httpd.apache.org/docs/1.3/sections.html

here a partial patch which adds two files - basically we'll need a CVS admin to move the following files to sites/default (or somewhere under /sites) also:

CHANGELOG.txt
INSTALL.mysql.txt
INSTALL.pgsql.txt
INSTALL.txt
UPGRADE.txt
MAINTAINERS.txt

I think sites/default is a reasonable place for them?

I'm agaist that we move the files to site/*.

We need some folder with the name make since with "docs", and that folder need to be in drupal root to we see it, when we descompress.

Maybe /DrupalDocs/

I would perfer to change the name txt to TXT.php. because all users see them when decompress. And work in server that don't have htaccess(because it slow performance).
And I think that we don't need to fix this in contrib. Because will imitate by default the contrib dev drupal way (in the next release), and some modules may have some txt files that want to be public and access directed.

I'm waiting to see some admin opinion..

As an alternative, and I'm not vested in this option I just had a thought...

Could we rename the files by appending "-DRUPAL" to the name (e.g., CHANGELOG.txt -> CHANGELOG-DRUPAL.txt)? Then you could block those specific file names. If someone has a file named INSTALL-DRUPAL.txt somewhere down in their directory structure, yes it would be blocked as well. I would avoid just matching against -DRUPAL.txt to avoid catching as many unintended files as possible.

If you don't go to rename (the best solution), I perfer the #39 solution.

Block this files in every single folder:
*-DRUPAL\.txt
*-drupal\.txt

I like #39 too.

webchick (#35) said:

- Disallow *.txt in sites/*

But need to allow *.txt in sites/*/files (or whatever the files folder is called ... which can't be predicted by .htaccess). And preventing all other .txt files under sites/ from being accessed seems a bit heavy-handed and restrictive.

I reckon I could quickly work out the major release version of any Drupal site just by looking at the HTML source for a few seconds. So don't think there's much point in trying to block all the .txt files.

My conclusions:

1. If doing this via .htaccess, suggest only hiding CHANGELOG.txt, and possibly README.txt, INSTALL.txt and UPDATE.txt, in the root folder only. (I'm thinking that maybe the last 3 could conceivably change in a minor release, so perhaps best to hide them as well.) This would at least keep out casual browsers like me. Small benefit perhaps, but then the "cost" it very small.
2. Renaming the files to end .php is growing on me. I don't understand the problems with doing it this way. Can someone explain? Surely it's then up to module developers if/when to adopt this convention and rename their text files.

This patch add the php tags and die(); to this files:
CHANGELOG.txt
INSTALL.mysql.txt
INSTALL.pgsql.txt
INSTALL.txt
UPGRADE.txt
MAINTAINERS.txt

And is needed that one admin rename that *.txt files to *.TXT.php

-DRUPAL.txt is far more palatable than .txt.php. One is a concession, the other is an outright lie.

I think that we (Drupal) are the last cms with this security bug.

#A Joomla, Mambo change txt to php;
#B Xoops, Dragonfly, PHPnuke, Xaraya have/docs /html (in /docs/ *.txt files, in /html/ core files and license) So txt files are out the CMS Root;

All of you are ok with #B way?

I would go for either #39 or #46 (#B). Definitely don't give them a php extension since these files aren't php files.

Changed the title to more correctly identify that we are only talking about the core-supplied .txt files.

#39 Simple and doesn't significantly change the experience for new users looking for the documentation files.

#46a Others have called it a hack to rename the files .php. I don't care for this option either. For most people who don't read the documentation on the site before they go playing around, it may not be obvious that these are files they should read.

#46b If we put the files into a zip file, can't I just grab that and get all of your documentation at once? This option seems like it will start another issue where we have to block zip files so no one can get the documentation...but we can't because of downloads...and so on. And for users, it's one more step to get to the information.

Of the main options right now, +1 for #39.

I think that security was discussed enough, and show lot of ways to fix this problems.

By .htaccess; #37 patch, or #39 way
By rename .txt to .TXT.php; #44 patch
By create to folders #46(#B)

Admin just have to choice the fix or wont't fix.

Restoring the title

ScoutBaker you understand wrong for #46b, the zip file that I was talking is the tar.gz/zip that allready exist. We move the docs out the drupal root (/html/)

Example:
drupal-6.0.tar.gz
-- /docs/
--- CHANGELOG.txt
--- INSTALL.mysql.txt
--- INSTALL.pgsql.txt
--- INSTALL.txt
--- UPGRADE.txt
--- MAINTAINERS.txt
-- /html/
--- /includes/*
--- /misc/*
--- /modules/*
--- ...
--- index.php
--- LICENSE.txt
--- update.php
--- robots.txt
---- ...

And the users just decompress the html folder to webserver and don't have the docs folder.

@int: if I understand you, you are advocating that the drupal package contain a docs folder which is to be stored outside of the web root? So that when the package is extracted into one's web root, you then mv the docs file up a directory? Or have I misunderstood?

OK. If I understand #51 correctly, it seems like a good bit of trouble, but ultimately, manually moving the "fingerprintable" files out of the web root is probably the best choice for the security-conscious web administrator. If it is a manual operation, though, it is unlikely to happen in most instances.

If we were going to go this route (and require manual intervention from an admin), then why not leave the files in the root directory and add a warning as a row in the table at admin/reports/status that checks to see if INSTALL.txt (among others) exists in the Drupal root directory? If so, we can gently suggest that the administrator move it.

If the admin is worried about fingerprinting they can move the files. Fortunately, this will be post-installation so the accessibility of INSTALL.txt won't be an issue, and presumably they'll know where they put them so that they can refer to them later. If they understand the risk and it doesn't bother them, then that's ok too.

Either way, I don't think this is clear-cut enough yet for it to be RTBC.

"If the admin is worried about fingerprinting they can move the files" We have to worried about newbies that install drupal and don't know this security issue.

But i like the your idea to put one warning in admin/reports/status if INSTALL.txt (among others) exists.

So, and now? we code your idea? We need the final solution, to we work on.. (admin request)

#46a - "it may not be obvious that these are files they should read", the linux files don't have the txt in the name, and people know that that file is to read. To CHANGELOG.php the users maybe think that this was a php page, but CHANGELOG.TXT.php is quite legibly.

I think renaming *.txt to *-DRUPAL.txt or some such seems to be the most reasonable proposal on the table. Also, then contrib modules can use this convention if they want to protect their documentation.

Moving the docs files our of the webroot would have been good if the project was originally set up that way, but I don't think it's worth a major reconfiguration of the CVS repo for this.

after long discussion on IRC, the patch is simple, but mean someone has to hack the core repo to rename all the *.txt files to *-DRUPAL.txt (except, of course robots.txt)

I was asked to comment here and put some of my CVS-fu on the problem.

A) renaming files in CVS isn't impossible. There's 2 ways to do it:

A.1) This takes almost no work:

mv CHANGELOG.txt CHANGELOG-DRUPAL.txt
cvs rm CHANGELOG.txt
cvs add CHANGELOG-DRUPAL.txt
cvs commit -m "Renamed CHANGELOG.txt to CHANGELOG-DRUPAL.txt"

After that, if you want to see the old revision history on the file, you need "cvs log CHANGELOG.txt". Not the end of the world, IMHO. Who *really* cares about the revision history on the CHANGELOG itself?

A.2) Run my cvs_rename script via the shell on cvs.d.o itself. This copies the ,v files around, strips off tags and branches, etc. That's how we moved stuff from modules/foo.module to modules/foo/foo.module. This would require either Dries spending the time on this, or agreeing to let me manipulate the core repo for this operation. If we're talking ~8 files, it'd probably be on the order of 1/2 hour of work. But, this way, you can do "cvs log CHANGELOG-DRUPAL.txt" and see the full history. Yippy.

B) If we don't want *.txt visible in the drupal webroot, this has implications for the d.o packaging script for D6 translations. :( See http://drupal.org/node/207401. If the de translation has a README.txt file, the packaging script currently turns that into README.de.txt, so that when you unpack it into your webroot, you don't clobber Drupal's README.txt. If we want everything to become -DRUPAL.txt, the packaging script should probably rename in the same way.

C) I'm not sure we can legally rename LICENSE.txt or COPYRIGHT.txt, but IANAL.

D) I think a better solution (which I proposed to the security team) is to work towards moving *all* of Drupal out of the webroot. This is a bigger change, but more useful. Won't happen until D7 at the earliest, and even then, maybe only a partial solution that doesn't help some shared hosting sites.

E) I don't think this is worth much effort. It's really all about perceived security, not real security. Hackers have all kinds of ways to try to fingerprint a site already. I'm not sure this is worth much trouble, and clearly, this issue has already sucked up many hours of Drupal ninja time/energy.

Whoops, didn't mean to clobber pwolanin's status change. I'm not sure this is RTBC, however, so I'll set his to needs review. ;)

I meant to set to CNR, must have slipped when picking in the drop-down.

Also, note that all the references to the old file names in the files themselves, and within Drupal, will need to be changed. Off the top of my head, COPYRIGHT.txt refers to LICENSE.txt, INSTALL.txt refers to INSTALL.mysql.txt and INSTALL.pgsql.txt, and there are probably others. Several Drupal error messages refer the user to INSTALL.txt for more information, as I recall.

doh- I knew that was too easy...

dww, we can resolve this problem without having this trouble.

"put one warning in admin/reports/status if INSTALL.txt (among others) exists."

And because the string freeze we can move this to Drupal 7?

Hm. That's not a bad idea at all.

Clarification, in case that was misread as sarcasm. :) I meant the idea itself sounds great. It's simple to implement and doesn't require any hacks.

@int, sorry i missed your comment, you made it between when i read the issue and the IRC discussion that led to my comment above. sure, a warning is fine, although slightly annoying from a usability standpoint (e.g. you just finish installing drupal and poof, you've got a big red warning to fix).

for 6 even? Checks via system_requirements()

It adds some new strings, and we're way past string freeze. I guess it would depend on what the core committers think.

we worked very hard not to have scary warnings on a fresh drupal install. i am not pleased about them returning. futher, it is a real bore to require people to start moving files to get rid of the warning.

i agree with morbus that we really can ignore this. if folks insist, i like the /docs directory.

@moshe - unless you are running on a server with suexec or some such, I think users will frequently still see the warning/error regarding settings.php or sites/default

However, we could append a note about this to the last screen in the installer?

So #51 is the best solution? (drupal 7)

moshe weitzman, when we install drupal you will get allways warnings, the cron never run, or the register_global is on, I think that the problem is that is lots of txt files, and the user may be just run "rm *.txt" and remove LICENSE.txt and roboot.txt. And at fresh install we(drupal) can offer to the user to remove the CHANGELOG.txt(and others) files for them.

#51: -1

Don't forget contrib projects, they have *.txt files, too.

Create the folder "documents" and not "docs" (Drupal use "modules" and not "mods", "includes" and not "inc").
Move all *.txt into it and protect the whole folder and not only *.txt files via .htaccess.

Contrib projects can use their own folder "documents" to protect their *.txt, *.doc or whatever files.

Ralf - you're confusing things. Whilst I agree with your point - there's no precedent to use abbreviations - the proper expansion is "documentation", NOT "documents", which carries an entirely different, and far more generic, meaning. Likewise, note that if we were to use "documentation" (or "docs" or any other subdir idea), it would NEVER affect contrib modules. Why? Because for "documentation" to be protected, we have to create a /new/ .htaccess file within that directory. This new file would only ever apply to that particular directory - it would never apply automatically to like-minded directories in contrib modules. Since our goals for security-minded patches, however misguided, should be to protect contrib as well, any attempt to do subdirectories will fail to protect contrib.

I still support only the -DRUPAL.txt approach.

@Morbus - any contrib module could emulate the code in the patch above in its .install in order to instruct the admin to protect its sensitive text (or other) files.

So we have:

Also protect contrib modules
By .htaccess protect renamed txt to*-DRUPAL.txt in all folders (We have to change documentation, work only with servers with htaccess on) Version: DRUPAL 6 or 7
By rename .txt to .TXT.php (We have to change documentation, work on all servers, users may be not know that this files is for read) Version: DRUPAL 6 or 7

Not protect contrib modules
By .htaccess protect only CHANGELOG.txt(and others) (work only with servers with htaccess on) Version: DRUPAL 6
By create folders to move docs out the core /html and /docs/ (easy, but lost "easy" CVS revisions) Version: DRUPAL 7
By drupal warning in admin/reports/status if INSTALL.txt (among others) exists (user may remove all txt files and he need reboot.txt) Version: DRUPAL 6 or 7
By drupal instalation/(admin/reports/status) a, offer the user to drupal auto remove the files. Version: DRUPAL 7

Won't Fix
Add one warning the INSTALL/README, to the user remove thats files (some users don't read this, almost all others CMS fix this issue) Version: DRUPAL 6

From #73, I'd say we can do the following in Drupal 6:

- "protect"/"hide" the files with .htaccess (although this only works for Apache, and some other servers parsing .htaccess)
- add a warning to the status report that these files allow fingerprinting (but this would only work for core files again)

I am not sure there is much to protect here though, there are characteristics of different versions on which we can fingerprint (at least major versions are easy, minor versions are not always so).

@Gabor - to protect the files with .htaccess in any reasonable way, we need to either rename or move them. Are you on board with that? Otherwise, the status report is a fairly simple approach we can take with no need to hack the core repo.

Why would we need to rename or move them? We can expand the protected regexp list at the top to include the txt files we know in core. (Sorry if this was already discussed but, this is a very long thread to read through, and I only found some short summaries). Well, there are contrib .txt files if that's the point, but those we could not protect without having a list of them or moving them around. That's definitely not for Drupal 6. Either the .htaccess protection or the status report page warning can only work for core files in D6, I guess.

Technically we could have a hook to gather a list of txt files to "protect" from the contrib modules, and warn if those are also not removed yet. However, I'd say the contrib txt files are even more important to be in place, since sometimes they provide important info as well, which is not otherwise documented. So I would not move to get people remove them. The same might apply to core txt files as well, depending on your inclination (ie. sysadmin == txt files; or web guy == web search based information gatherer, but that's a bit too broad conclusion to arrive to). Anyway, so even the core .txt files hold some value on a runtime site depending on experience, so I'd not force people to remove them.

So far it looks that we could protect the core .txt files on .htaccess aware servers, and continue thinking of better ways in Drupal 7.

damm.. 75 post to disscus about the .txt files.. I think I don't want to see how far could the converstion go when you start to talk about the introduced ".schema" file (this one includes database definition).. It's not in protected in the .htaccess neither.

@Gabor: the problem is that if the root .htaccess disables (for example) CHANGELOG.txt and README.txt, that removes access to any files named that anywhere in the site. That can break things if people install other things besides Drupal on the same site. See #27.

That's why the proposal is to rename the core files to *-DRUPAL.txt, and specifically exclude those in .htaccess, so that we're guaranteed not to break that which we don't own.

I don't personally care much about this (as per my comments in #56) I'm just summarizing the opinions/conclusions of others.

pwolanin, et. al: I do not support a patch that forcibly puts up a note to move those files out of the way.

Gabor: README.txt, CHANGELOG.txt, etc. are relatively well-known filenames. Protecting those files, as named, in the .htaccess would apply to the /entire/ directory and subdirectory structure, affecting things that are entirely unrelated to Drupal. It's not friendly, and would actively harm parts of my site, for instance.

Hm, my understanding was that there is a way to protect files by location as well, not only by recursively applying name matching. Unfortunately I could not find anything to restrict the match to locations, so looks like we move this issue further again with one version and do clever things about it in the next release.

Gabor - you can either use [Location], which is an Apache directive that can ONLY be used inside a .conf file, or you could do the same with RewriteRules to enforce it to a particular directory (say, / and /sites/*/). Naturally, [Location] isn't ideal, and Rewrites just add another layer of "this may not work" (after the ".htaccess's may not work either", of course).

ok, after further debate on IRC when we could all have been fixing real bugs, I'll suggest this very minimal change to INSTALL.txt as a way to address the original concern. The command suggested will do no harm, and a more detailed explanation seems like a distraction.

slight modification to add a link to a relevant handbook page as requested by Gabor

Patch looks reads good to me, save for the dangling final sentence. Fix that, and I'm OK with RTBC.

Also: "security, but if you wish you " --> "security but, if you wish, you"

The htaccess file is not protecting the .schema files with the database definition for modules..

@ilo: there are no .schema files anymore. hook_schema() implementations were moved back into the .install files. See http://drupal.org/node/150245. You must be using a very stale version of D6 core.

Yes.. I realized that I was using one of the beta releases. Sorry. I'm just slapping myself while I download the rc2.

fixed text as suggested.

there was a full stop instead of a comma. edit: don't credit me if this gets committed, and the rest looks fine.

Revised version attached, cleaned up dangling whitespace and missing comma. RTBC, IMO; thanks pwolanin.

Looks good to me now. So setting to RTBC.

Is is really intended to prevent users from being able to read their own CHANGELOG.txt?

yes - those files are expected to be viewed locally, not via a web browser.

chmod o-r CHANGELOG.txt will generally deny access via web browser, but still allow me to grab the file via FTP to read it (e.g. if I don't have gzip on my local computer)... otherwise I'd have to add "read" permissions to get a copy, then perhaps remove them again ...
Apologies if this bit of discussion has already been had in IRC...

??

Not entirely true. My own vhost setup, for example, creates files that are owned by "morbus:psacln", and the Apache webserver is also a member of the "psacln" (requiring g, not o). a-r is the only way to /ensure/, completely, that the webserver is unable to read it. It's the safest alternative.

How about chmod go-r CHANGELOG.txt then ? ;-)

No.

This is good text, and a good solution. But, I have a couple of content questions here:

• Is the intention that the administrator would repeat the chmoding for the other Drupal documentation files as well? The patch refers to "documentation files", and properly singles out CHANGELOG.txt as revealing the most information, but does not mention repeating the process for the other files. Maybe we should include a "Repeat this process for all of the documentation files in the Drupal root directory, substituting the name of each file for CHANGELOG.txt, as shown in the example.", or some such.
• "For more information" lines should refer to the actual subject matter, or else explain why you would need the additional information. Here, the line is for more information on changing file permissions, not for more information on version fingerprinting. That line probably needs a "For more information on making files non-readable, see...". Plus, the reference here doesn't help non-Linux users.
• I can think of at least two different meanings for "This will have little actual benefit to your site's security." and don't really know which one this patch intends. I think it means that whether these files are readable or not has no impact on the actual code-level security as implemented by a Drupal site, for either better or worse. It could mean, though, that we disagree with the "some administrators" who think these files should be non-readable. Plus, this sentence is an absolute, which is always troublesome. Really, I'm not convinced it adds much value. The paragraph conveys about as much information if you just have: "Some administrators suggest making the documentation files, especially CHANGELOG.txt, non-readable so that the exact version of Drupal you are running is concealed. To implement this suggestion, use the following command from a shell or system prompt (while in the installation directory)"

So, I would recommend something like:

+
+   Some administrators suggest making the documentation files, especially
+   CHANGELOG.txt, non-readable so that the exact version of Drupal you are
+   running is concealed. If you wish to implement this optional security 
+   measure, use the following command from a shell or system prompt
+   (while in the installation directory):
+
+          chmod a-r CHANGELOG.txt
+
+   Note that the example only affects CHANGELOG.txt. To completely hide
+   all documentation files from public view, repeat this command for each of
+   the Drupal documentation files in the installation directory, substituting the
+   name of each file for CHANGELOG.txt in the example.
+
+   For more information on setting file permissions, see "Modifying Linux, Unix,
+   and Mac file permissions" (http://drupal.org/node/202483) or "Modifying
+   Windows file permissions" (http://drupal.org/node/202491) in the online
+   handbook.

(Unfortunately, I'm not where I can roll a new patch but I will when I can, if no one beats me to it.)

Of course, if I am overthinking this, please set the status back.

Your revised sentence including "so that the exact version of Drupal you are running is concealed", with no further information, implies that the sentence is, in fact, true. But it's not - the exact version of Drupal you are running is NOT concealed by just changing these files. There are many other ways to find this information out, which is why "This will have little actual benefit to your site's security." exists - to indicate this is a "feel-good" measure that "some administrators" suggest.

Hmmm, yes, that's a very fair point and absolutely true.

Perhaps the first sentence needs to change to something like:

Some administrators suggest making the documentation files, especially CHANGELOG.txt, non-readable so that your exact version of Drupal is more difficult to determine.

I'd only accept that if "slightly" or "miniscule-y" or "takes 5 seconds longer" is added ;)

I think giving the impression that we think this is rock solid security practice could in itself lead to security scares down the line "Drupal core relies on this stupid measure, do they really think that'll work?" etc.. "slightly" sounds like a good addition.

If you are going to completely downplay the effectiveness of this step, then you eliminate the reason for this patch. If there's really no point in doing this, don't. If there is a point, at all, state the simple facts and don't be condescending to the readers of the file or the people suggesting the patch.

"If you are going to completely downplay the effectiveness of this step, then you eliminate the reason for this patch." -- now you're getting it! There's a lot to read through here, but ideally, this is about /perception/. The patch, in any of the forms presented here, really doesn't increase security - it just makes it slightly more difficult to detect the version of Drupal being run. Nearly everyone agrees on that point. But, numerous people seem to think that a spitting of the version number, of any software, is an indication of a security hole - of too much information. This patch merely offers a method of stopping that spitting via the CHANGELOG.txt, to ease those who do think it's a problem. However, it is in OUR best interest to tell them that they are NO MORE SECURE by doing this because, in fact, they aren't.

I think that even the people who are the most "meh" about this patch will confirm that it does "raise the bar" on the amount of effort required for determining *down to the minor version* what version of Drupal is running on a website.

Major versions have a lot of changes and are easy to pick out. This change has no bearing on fingerprinting that. But it won't advertise to the world that you're running Drupal 5.1 rather than 5.6.

A cracker with half a clue, of course, will figure this out by other means, or just have a collection of scripts that blindly try every known vulnerability against any website. It's just a barrier in front of the script kiddies.

I am usually very vehemently opposed to security by obscurity, but still:

Is a security update actually guaranteed to be always identifiable from the outside? If there is a release that fixes only the security issue, and it is some kind of XSS or SQL injection, then I fail to see how an attacker would find out if it is installed except by attempting the attack, or by reading the changelog.

We're not talking about whether someone is running 4.7, 5 or 6 RC2 - those are pretty obvious. It's a matter, for example, of whether you've installed 5.6. A cursory check of the bugs fixed in 5.6 gives me nothing that would be obviously revealed to anonymous users on a single page - and for everything else, the vulnerability can be tested more easily by attempting to exploit it directly.

On the other hand, /CHANGELOG.txt is an indicator so obvious that it can be parsed by a spider - just request it from every site on drupalsites.net and there's your list. Speaking as a late adopter of 5.6 (updating is kind of a hassle when using tarballs instead of CVS), it'd certainly be nice if my site didn't broadcast that fact on a known URL...

(Not to say that it will make the site more /secure/. But the version will be more difficult to determine, as advertised, and you're probably not going to do it in 5 seconds either.)

If you are going to completely downplay the effectiveness of this step, then you eliminate the reason for this patch. If there's really no point in doing this, don't. If there is a point, at all, state the simple facts and don't be condescending to the readers of the file or the people suggesting the patch.

Just to make clear, my post wasn't aimed at anyone in particular, and the bit in quotes was me feigning a 'drupal suxx' post, not my actual opinion on this (which is neutral).

I think your exact version of Drupal is slightly more difficult to determine. is both accurate and avoids condescension. So here's a patch incorporating the latest patch + the followups by Keith and Morbus.

@webchick, no, actually it's not. Script kiddies do not check versions. They run scripts that either work as they exploit successfully or they don't work so they fail. Script kiddies do not check, nor do they care. If they did, we wouldn't see log attacks for IIS on Apache servers and phpBB attacks on Drupal sites. 106 comments on something that will effectively do nothing (and absolutly nothing in IIS) but make the security through obscurity crowd happy, confuse the issue with some people having the false belief that this actually accomplishes something.

s/script kiddie/someone who wants to take down britneyspears.com specifically/;

Meant to set this to needs review.

@Morbus: I never expressed an opinion on whether we need this patch or not. I was actually agreeing with your earlier posting about the tone of the message and how it will be read.

@catch: Not a problem, and I wasn't trying to point a finger or be disrespectful. I just know what I thought when I read the text, and a couple other people mentioned similar things. I'm also hoping we can put this issue to rest since it's been dragging on for so long.

(Sorry -- I had a dental appointment today, and was out. Believe me, this would have been more fun.)

For what it's worth, I'm ok with the patch in #108. I'd rather "slightly" wasn't in there, as it is unnecessary. "[M]ore difficult" is still "more difficult" even if it is only "slightly more difficult". And, "some administrators" don't suggest doing this because doing so makes fingerprinting "slightly more difficult". They suggest it because every little bit helps.

But, that said, "slightly" doesn't bother me all that much, and certainly won't lead to any false impressions about this being a security panacea.

(I'm leaving it at CNR but will RTBC it if no one else does.)

Duplicate post - got caught in the site down.

Duplicate post - got caught in the site down.

As I said in #113, I'll RTBC the patch in #108.

I agree that it is not a good idea to dismiss this issue, although hiding, removing or making the text files unreadable does not make it harder for someone with a specific intent to identify the version of the Drupal being used on whatever site. Also, the number of people reading the INSTALL.txt is not too numerous, but now we have docs for those interested, so they can do this small change to improve their security a tiny bit.

I think that we need a better solution for Drupal 7.0.

May be one PHP auto-delete (as SMF delete install.php)
Or the warning in Admin..

But I think inspite of the user can delete this, allmost admin DON'T DO IT.. So we(drupal) need to do it for them.

All this sites are some of the most popular... about 70% of them have CHANGELOG.txt. A few sites are updated. And the sites that are updates, the administrator can past be weeks before update...

I think that is quite easy to break some of this sites: (the file CHANGELOG.txt were there at the time of this post)

This sites have CHANGELOG.txt:
http://buytaert.net/CHANGELOG.txt 5.7
http://drupal.org/CHANGELOG.txt 5.6
http://www.avrillavigne.com/CHANGELOG.txt 5.1
http://www.macworldexpo.com/CHANGELOG.txt 5.2
http://fosdem.org/2008/CHANGELOG.txt 5.6
http://www.eff.org/CHANGELOG.txt 5.2
http://harvardscience.harvard.edu/CHANGELOG.txt 5.5
http://endpoverty2015.org/CHANGELOG.txt 5.5
http://www.mylifetime.com/CHANGELOG.txt 5.1
http://exposure.cbc.ca/CHANGELOG.txt 5.1
http://www.warnerbrosrecords.com/CHANGELOG.txt 5.7
http://live.eclipse.org/CHANGELOG.txt 4.7.5
http://discoveryclub.de/CHANGELOG.txt 5.1
http://goingon.com/CHANGELOG.txt 4.6.2
http://community.menshealth.de/CHANGELOG.txt 5.1
http://uni.zeit.de/CHANGELOG.txt 5.7
http://www.greenpeace.org.uk/CHANGELOG.txt 5.3
http://amnesty.org/CHANGELOG.txt 5.3
http://opensource.org/CHANGELOG.txt 4.7.10
http://www.cdenv.be/CHANGELOG.txt 5.7
http://www.laatwatzien.be/CHANGELOG.txt 4.7.5

http://www.mtv.co.uk/CHANGELOG.txt 4.7.3
http://adsoftheworld.com/CHANGELOG.txt 5.6
http://www.votehillary.org/CHANGELOG.txt 5.7
http://teen.secondlife.com/CHANGELOG.txt 4.6.3
http://theworld.org/CHANGELOG.txt 4.7.11
http://www.fighthunger.org/CHANGELOG.txt 4.7.6

http://moby.com/CHANGELOG.txt 5.1

http://appel.nasa.gov/CHANGELOG.txt 4.6.3
http://16plus.be/CHANGELOG.txt 5.2
http://extensions.services.openoffice.org/CHANGELOG.txt 5.0

This is not a critical issue - as nearly everyone agrees, deleting the CHANGELOG, or any other text file, will do little to increase the security of the site. Anyways, I think you'll find far better "take home" value if you, int, go and email each of these sites and tell them that you a) helped bored malicious users compile a list of vulnerable versions, and b) they should delete their CHANGELOG.txt, not that it'll matter anymore cos you happily provided a list of their versions anyways.

Morbus Iff, I am here because I want to help drupal improve, and my point of view this is a serious problem.

And if I email them, isn't resolve this problem. Only resolve in some sites (and may be for only one/two new releases).

I want (if you agree) to drupal resolve the problem for them.
And I think that delete the some of the .txt files will became 99% more dificult to detect Drupal minimal version, and 70% hard to detect security bugs.
And (almost) all other CMS don't have this issue.

Deleting or removing this file does nothing to increase the security of a given Drupal site. All it does is promote the thought that someone may be more secure. In truth, sites are not exploited by looking at the CHANGELOG.txt as many sites that use version control do not end up updating the CHANGELOG.txt file.

A supplied list of sites that have left there CHANGELOG.txt file on doesn't actually prove your point.

http://en.wikipedia.org/wiki/Security_through_obscurity

This issue was marked fixed through documentation as noted in
http://drupal.org/node/79018#comment-709802

If you have an actual solution to propose, please open a new issue referencing this one and detail it.

@int - one can readily find sites that let anonymous users post PHP code. I think preventing that would be a much higher priority for 7.x

Hi all.. I was out of this topic but I can't stay in background after this..

Regarding #121 and #122, are the kind of responses anybody would expect to a 'supossed' security issue. I agree posting real live sites wont help (and is of course, a bad Idea), but it's a demonstration about the issue. I can't understand why anybody has suggested to "please, remove this information in the post" or anybody with enough permissions did that. Reminds me the times of Microsoft blading the sword of "unexpected features" to security or design flaws.

Still is not clear why are we discussing if we should put or not information about to hide this files, and later it's said this files are not a security problem. They could be.

In any case I've seen several approaches here.. You said that people should have the README files in text to be available in the shell while installing, because people read the files.. the put the 1st line in the document: "Please, remove these txt files". But.. what about fantastico and the other installation scripts allowing to have a drupal site in 2 clicks? do they read the txt files also? Other solution could be to put a reminder in the status report of the drupal, checking the presence of these 4 txt files. Or just perform a 6th installation step: removing and protecting files. And why not, when anybody tries to download drupal, drop them to a "security considerations about drupal you should know" before they download.. or what is better, while they download the file.

Refering the wikipedia as the mother of all solutions won't help also. If you look up there about security assessment you will find the the 1st step of any kind of them is discovering and recognition of the target. And I'm still trying to figure the way finding in google about drupal sites allowing to write php helps to drupal7, and why this is a high prio issue.. this is amazing.

We have scripts in the scripts folder of any drupal, why not a "secure_site.sh" or "secure_site.php" in the administration section.

Drupal has many other "design" problems exposed.. everything resides in the webroot folder. This is a really bad idea. The site modules can be mapped with a really simple script, sites list also.. I think this has higher prio than searching google. In fact, it can't be done with the current setup. I succeed in getting "sites" folder out of the webroot, but themes should be there to serve support files (images and css), and modules, core modules.. oh.. they also use ugly includes and requires, and serve support files, so they can't be removed..

After some thought and reading through this issue queue, the main issue seems to be a 'marketing' issue. No one will argue that the availability of a text file will make Drupal more or less secure.

Rather, some out there may 'feel' that Drupal isn't doing enough, because they didn't take this one small tiny little step that doesn't really do anything anyway. So maybe this needs to be turned into a marketing issue, or a message in some of the text files, that there is no benefit to hiding them, so they aren't hidden. Just to let people know that it's not "They didn't even think of that?!?!?" but rather, "THey have a really good understanding of real security issues." Also, this would help spread awareness of security through obscurity problems to everyone that installs Drupal and looks at it and thinks that "we" didn't try to make it more secure.

Secretly I'd love a "We think Drupal is so secure that we'll advertise what version we are running" but that may be taking it a little far.

No software is 100% security.
Security is given by the time that hacker can break it..
And if we can make more hard to hacker know the real subversion of the site, we are increase the time that one hacker have to break one site, so this is a security fix.

@int: we're not going to rehash this conversation *AGAIN*. Please re-read the earlier posts in this thread.

@all: This issue is closed and should stay that way. Therefore, I'm disabling comments on it.

Adding a note so that people will find it out from here as well. There are public and free tools to fingerprint web site building tools such as Drupal or Wordpress with their (in many cases exact) version. Such as this Firefox extension: https://addons.mozilla.org/en-US/firefox/addon/10493