Develop security policy and require OSUOSL and Drupal community members to agree and sign it

Title fixed.

Good idea. Please define 'community member' and 'volunteer'. For example, am I a 'volunteer' if I have contributed modules to Drupal.org? Am I a 'community member' ?

Please identify the people who will be subject to these requirements.

After the high-profile defacements of sites like joomla.org and gentoo.org, on the security mailing list we had a discussion about "what should we do if..?"

Here are my suggestions (sorry, it's a bit long...)

- We should setup a separate domain, like status.drupal.org, that points to a completely different server, and provides a Drupal install we can use to post general infrastructure announcements. Analogous to http://dreamhoststatus.com/. We'd need to publicize this so that people have it in their RSS readers and such. And we could use it for run-of-the mill downtime stuff so that it gets some use outside of the (hopefully extremely rare) security problems, so people would actually use it.

- In the event of a compromise:
1. Those active should /join #drupal-emergency (public channel), in order to hash through log files or whatever. #drupal-security can still be used for things that need to be kept private.
2. One of these active members immediately posts to the status blog:

10:15AM:
It has come to our attention that the drupal.org site has been compromised. Admins are currently working as fast as they can to bring it back online. If you'd like to help, please join #drupal-emergency.

3. Additional status messages are appended to the list, every 15 minutes.

10:30AM:
Here is a (sanitized) version of our log files just prior to the attack (attached). If you would like to help provide insight, please join #drupal-emergency, or comment here.

---

10:45AM:
Log files are still being analyzed, but we've determined that the database has not been touched, so all drupal.org user information is safe. We've also tracked the group responsible for doing this -- warez in the world is carmen sandiego -- and have reported the incident to authorities in California.

---

11:00AM:
Thanks to So and So, we've managed to limit this issue down to a bug in one of the custom modules used on drupal.org. The good news is that Drupal core does not have a security hole. We're currently fixing the problem and applying it to the live site.

---

11:15AM:
Still working on patching the bug. If you have familiarity with file injection vulnerabilities, we could use a hand in #drupal-emergency. Thanks!

---

11:30AM:
Good news! The hole has been fixed, the server's been patched, and all looks good. We'll post a more detailed, blow-by-blow description of what happened on the front page of drupal.org within the hour.

---

12:20PM:
Please see http://drupal.org/node/454234345 for the full announcement. Thanks for your support!

4. One of the security team member prepares a full-disclosure announcement and posts to the front page of drupal.org, that accomplishes the following things:

a) Sums up the problem: what happened, and when.
b) Outlines how it affects Drupal/Drupal.org users.
c) Describes in as much detail as possible the problem and the fix.
d) Talks about how we can be sure the error won't happen again.
e) Personally thanks those involved in helping to resolve it.

----
Title: Drupal.org website defacement

Body:
At approximately 10:15AM on August 21, 2009, the home page of drupal.org was replaced with an image of George Bush eating a mango. The authorities have been notified and are actively pursuing the attackers. The entry point was a custom module on drupal.org, and not anything in Drupal core itself, so no Drupal users are affected by this problem. The defacement was limited to placing a single index.html file in the directory; the database was not touched, and no drupal.org user information has been compromised.

The defacement occurred because we were using a module called foo_baz which connected the doohickey to the foohickey. In it was the following line of code:

$stuff = file_get_contents($insecure .'/foo.xml');

The attackers spoofed the $insecure variable, which allowed them to include a remote script which provided them with access to the server to write an index.html file to our web root. Apache then served this as the main page for the website, until 10:17AM when server admins were able to revert.

We've fixed the offending line by using the proper drupal_get_path() function:

$stuff = file_get_contents(drupal_get_path('module', 'foo_baz') .'/foo.xml');

To prevent such an issue from happening again, we've added a new rule to our security scanning tool, code-checker, to raise an error when it encounters insecure remote file inclusion such as this. We've also run the tool against the contributions repository, and have confirmed that no other scripts share this vulnerability.

We'd like to sincerely thank Foo, Bar, and Baz for their help in tracking down the issues, and So and So for their quick help in patching the server.

---

Basically, my feeling on this matter is that constant, consistent communication about what's going on is *equally important* as fixing the bug. Going one step further, and allowing/seeking people to actively get involved in helping, shows that we are truly doing as much as possible to get the problem resolved.

Angie

Looks great.

Should #drupal-infrastructure be used instead of #drupal-emergency? Or even plain old #drupal?

P.S. I am now concerned that someone will see this thread, think it is for real, and blog about drupal.org being hacked! Expect it on digg soon.

Good ideas. One possible revision/discussion point on webchick's post: I think you should use a lowest-common-denominator way of pushing out the blog page and corresponding feed. Meaning, don't use Drupal. While, of course, it can provide what we want, if there is a *core* vulnerability, someone might just go after this site first for kicks. I think a simple XML file with index.php and rss.php wrappers, with maybe a script to facilitate adding new posts to the XML file with a timestamp and such would be good. Of course, there could be a little CSS to pretty up the index.php page, but just simple stuff like the maintenance banner page. I guess I like to keep things simple.

Oh, and by the way, with regard to the issue title: having the OSL as an organization sign anything requires a very nasty mess of getting lawyers for the university that do not understand technical stuff like site outage details to approve this---a large time and money consuming process. I think we'd be better off with having an mutually agreed upon policy, without the legally binding parts.

@kbahey:

the home page of drupal.org was replaced with an image of George Bush eating a mango . . . The defacement occurred because we were using a module called foo_baz which connected the doohickey to the foohickey.

If someone seriously thinks that this post pertains to a real event, it will mean that the human race is officially turning back into apes. :P But just in case it needs clarification, "Dear blogger with the prominent forehead and hairy body: this scenario is a completely made-up, hypothetical scenario to use as an example of what might happen in the case of such an emergency. This is only a test. BEEEEEEEEEEEP."

#drupal's not good, because it'll be filled with people going, "OMG!! Did u know ur site's down?!" People concerned about fixing the problem need a quieter place where they can work. #drupal-infrastructure is fine, though.

@emsearcy: Sure, it could even be a hand-edited XML file. :) I was just thinking Drupal because it'd be nice for people who can't get on IRC to be able to supply feedback, and you can't post comments to a 'light' solution like you propose. But I guess an alternative here would be referring people to security [at] drupal [dot] org if they can't get on IRC for some reason.

Whatever 'process' we have, if it involves a technology component/solution. That solution should have at least two identified maintainers who check/test it 2-4 times a year. This is one of those things that if it is put into place and then people 'in the know' get busy/distracted or transition to something else tends to get forgotten or out dated.

So one of the checks could be a 'quarterly' reminder of the process to the infrastructure list to remind old hands and update any new ones.

I'd like to follow up a bit on webchick's proposal and the idea as a whole actually. I like it, but I think focus should be put on making it a very lite-weight solution. Having actually been in a couple situations where a server was compromised and 10-15 people were working on it, I have noticed several things:

A. Communication between team members is very important, communication with the outside world less so. For good and bad, it is often neglected. I like the idea of having a status page to tell people what is going on, but a middle-ground should be shot for. Posting every 15 min will be distracting and sometimes pointless. (and I can guarantee that it won't actually happen) Even keeping up with IRC is difficult when your trying to recover a server. I would let the team decide at the time how often to post to the status page and not put artificial restrictions on them.

B. Having a drupal-emergency or drupal-infra channel is a very good idea. Having whoever the security team is all having ops in it is an even better idea. The community should be allowed in during 'emergencies', but if someone starts to become distracting they need to be 'removed'.

C. I think it should be decided who the people who who will be the core team to investigate security issues. They should be defined somewhere (private) and have their contact info listed (privately). The last thing you need in these situations is wondering who is in charge/who needs to be contacted.

D. I agree entirely with the full disclosure part of this.

Just a couple of ideas...

Excellent suggestion for C!! B too.

For A, I was thinking one of the security team members who would be completely useless for performing server analysis, such as myself ;), would be responsible for communicating with the outside world. We definitely _do_not_ want to divert resources away from fixing/finding the problem, 100% agreed. I agree that it's also probably wise not to nail down a specific timeframe but just say "As often as possible." I do think that this communication is hugely important though; without it, it can come across like we're hiding something, or not viewing the problem as urgent, etc.

Basically, my feeling on this matter is that constant, consistent communication about what's going on is *equally important* as fixing the bug. Going one step further, and allowing/seeking people to actively get involved in helping, shows that we are truly doing as much as possible to get the problem resolved.

I couldn't have put it better myself.

9/10 of managing a security breach in an OSS project is NOT about fixing the bug. That is often pretty easy to track down and resolve. Far more difficult is managing the aftermath, and making sure that information is available to those hunting for it so as not to swamp communication channels with "OMG! WTF? HAXXORED!" messages.

Although some would say that managing perception and managing public opinion is unimportant and irrelevant, making sure that people know that 'experts are on top of the situation' via frequent pings is essential.

Also, I think we would do well to keep a privately funded Blackwater Security strike team on retainer.

Perhaps it would be appropriate to include some basic forensic advice in the proposed security policy, so that all members responding to an intrusion can be assumed to be aware of those procedures? Most of the people responding may be technical experts, but may not know some of the best practices for handling an intrusion. I certainly am no expert, but can think of a few important things to be aware of, e.g. to avoid destroying evidence of how the security breach happened. There must be some information out there on the basic first steps which should be taken in such a situation and could be included.