After what can only be described as the 'Extended Weekend from Hell', Drupal.org has been restored and is back in business.
First things first: what caused this outage, and why did it take so long to get the server back up? To phrase it simply and bluntly: misfortune, bad timing and miscommunication. Murphy's (annoying) law has been proven true once again.
At the moment, Drupal.org shares its resources with several other sites on a server kindly provided by Kjartan. Thursday evening, this server was hacked. One of the other sites on our server provided the hole through which the hackers entered; it appears someone wanted to turn us into a warez FTP, but completely messed it up instead. We discovered the intrusion quickly and were able to regain control of the server soon afterwards. However, the entire incident occurred only a few hours before a scheduled power outage at our current ISP; problems with remote administration and the lack of install media meant we were unable to fix the server remotely. Over the weekend we called to try and rectify the situation, but due to miscommunication with our ISP we had to wait until Monday morning before we could reinstall the OS and get the server purring again.
Still, it is not our intent to sling mud and point fingers. If anything, the whole experience has been a lesson for everyone involved.
Now, as you could read on the temporary page during the weekend, we were already planning to get another hosting, one dedicated to Drupal and related projects. Talks have been going on with Scott Kveton of the Open Source Lab for several weeks: they will provide everything we need for hosting and support, as long as we provide the machines. For this we planned a fundraising event which took place this weekend. The fundraiser was a great success, and during the next few weeks we will be gradually migrating to the new servers and new hosting facility at OSL!
As the amount you managed to raise for Drupal was above and beyond our expectations, we have asked the OSL to come up with a complete infrastructure plan for Drupal.org and its related services which will optimize reliability. Rest assured that your money will be well spent. Stay tuned for more information about this.
Comments
Total raised?
Actually, I think the downtime for Drupal paid off handsomely. How much was raised?
We'll follow up on that shortly.
We'll follow up on that shortly.
Was this other site which
Was this other site which provided the hole an unpatched drupal?
unlikely
the changes that "these" people run unpached drupal is next to nill, running apache/php with root access would be rather silly. most likely an ftp/ssh that was misused?
either way, i am sure dries (or another) will file an abuse at the (also hacked?) source ip address.
--
groets
bertb
--
groets
bert boerland
Restoring Drupal.org and Murphy's law
Talk about panic, lol.
I visited the site to download some modules and found it dysfunctional. I immediately began a quest to gather the nickles, dimes and quarters that have been collecting in the ash tray of my car, so that I could make a donation.
Get a good server. We are counting on you Drupal.org, and thank you for developing this fine product!
Jamie
http://www.jamiejamie.com
The word you want is
The word you want is "cracker", not "hacker". A hacker wouldn't want to setup a warez FTP site.
http://disobey.com/
http://gamegrene.com/
http://www.disobey.com/
Resistance is futile
I'm in the "context decides its meaning" camp... I think everyone understood the message. Geeks seem to have the most trouble with the idea that words can mean multiple things :P. And of course, it's pretty hard to resist the indoctrination by mass-media.
--
If you have a problem, please search before posting a question.
Wrong.
I hate to burst your bubble, but a cracker reverse engineers software. Nothing to do with [b]hack[/b]ing in to sites, it would infact, be a hacker as already stated.
Cracker vs Hacker
I'm in the "Who gives a flip?" camp. :P
Amen to that!
I don't care what you call them. You can call them "idiots" too, fine by me!
-- Kobus
what about
h@x0rZ vs hackers? I am sure eveyone knows what the first one means :)
---
if you dont like the choices being made for you, you should start making your own.
---
[Bèr Kessels | Drupal services www.webschuur.com]
my blog suffered a lonely
my blog suffered a lonely weekend :-)
Progression of donations
I happened to have a few web browser screen open when donations were coming in.
I summarized what I could find here: Drupal Open Source community exceeds target in fundraising for new server, which gives a glimpse of how things progressed, and the potential role of Slashdot posting a story on it.
With that much money, we could pay for the legal fees to make a Drupal Foundation a reality, we could run two servers as mirrors, separate MySQL from Apache/PHP, ...etc.
--
Drupal development and customization: 2bits.com
Personal: Baheyeldin.com
--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba
hardware
Keep in mind that Dries did the updates by hand manually. It will be interesting to see, and although slashdot probably had some impact, I suspect that moneywise we will find that the bulk of it is from within the Druapl community iteslf.
We asked for funds for hardware. We are probably obligated to spend it on hardware. People might get upset at anything else. :)
OSL is working up a hardware architecture for the setup, but servers have been ordered, they will take a few days to get shipped, not much we can do about that. Dries will post a more detailed update soon but remember, some folks have paying jobs they have to go do in the meantime or paying for rent and food gets hard to come by.
Remember, Friday it started, access to rebuild was only on Monday for a fairly complex server recovery with multiple sites, mail, listserv, cvs, automation scripts to recover. Plus, the request for donations was only put up Sunday and was more then anticipated.
So, discussions with OSL were on Monday noon (They are in Oregan, US so that's Monday PST (GMT-7/8)). Now it's Tuesday and the server is up. Folks need to sleep, go to work, say hi to significant others, etc. Stuff is happening. Now that it is happening, and the server is up. Hardware is on order and soon to be i the mail(well, shipping company). We can all take the time to implement the next hardware build out in a configuration that will last longer and provide more flexibility.
-sp
---------
Test site...always start with a test site.
Drupal Best Practices Guide
-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide
As someone who donated, I
As someone who donated, I would be just as happy if some of that money were also spent creating a drupal foundation. The way the donation request was worded, that seemed to be a good second use for the money once the hardware requirements were meant. I would be surprised if many -- heck, any -- people were upset if money went to do that as well.
-- Merlin
[Read my writing: ehalseymiles.com]
[Read my Coding blog: Angry Donuts]
infra
infrastructure has a broad definition:
i like:
The basic facilities, services, and installations needed for the functioning of a community or society
--
groets
bertb
--
groets
bert boerland
I really don't think anyone
I really don't think anyone would mind their money being used for non-hardware things. A Drupal Foundation would be a definite plus. Anything to benefit Drupal is a plus, not just hardware.
this is probably being considered
but it's worth saying. If at all possible, the mailing lists should be on a separate server from drupal.org so that *if* the server with drupal.org goes down--routine or unexpected maintenance--communication still exists for the Drupal community. As long as both these resources are hosted on the same server, another complete communication blackout for Drupal is always a risk.
Or, at least, a status page elsewhere
Perhaps a mirrored status page on some other sites with [separate servers] and the willingness to do so -- rather than a whole separate box for mlists? Maybe some of the companies using Drupal as their main offering? Or some of the core developers' sites [or do they all live on the same server?]. Again, likely already planned, just stating the obvious...
not a whole separate box
merely on separate boxes. for example, it might be useful to put cvs and mailing lists on one box, drupal installations on another.
drupal login
Were people who have the drupal login module installed affected by the outage? Seems like that would be better off on another box, as well...
Glad to see it back
When the site went down I did some searching and it seemed went down due to security problems with Comments. Either I read the dates wrong or it was misinformation.
Regardless, it's great to see the site back and the donation amount was 3x your initial asking. Awesome!
Drupal Foundation
I started in Finance.
My comment - any Foundation should be set up under the rules and jurisdiction of the country with whom those will run the foundation are based/familiar.
Anything else will enrich lawyers rather more than they deserve.
So prob looking at a Netherlands NFP or Trust type arrangement.
(Other Foundations in other jurisdictions can be set up if it becomes clear that people those jurisdictions are going to pony up sufficient donations to make it worthwhile.)
Congratulations
Well done for bringing back Drupal.org... we missed the site over the weekend. I';ve experienced similar... and you are spot on with saying that it has been a learning experience.
If, like me, this will allow you to put processes and procedures into place that will safe guard, to some extent, a similar experience re-occuring. Keep up the good work,
Jonathan
Jonathan Furness
teacher, developer, webmaster
http://www.jonathansblog.net
Good to see it online again
One of our drupal sites was hacked last week but we were able to restore it. It looked like a hacker from the Netherlands but that url was probably hacked itself. Does this point out some vulnerabilities though? Not sure if the site had the latest version but now it does.
Hope Drupal will get a real positive boost from this experience.
___________________________________
Building (drupal) webblog portals & communities
This points up
I've not looked at the installation instructions in a long long time, but it would seem helpful to have IN BIG CAPS strong recommendations for the admin to sign up with the email newsletter in order to get the latest update announcements. How many admins of Drupal-powered sites out there still have no idea of anything that's happened in the past 2 weeks?
===
Laura
pingV
_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet
Assume you've been attacked
All I can say is, assume you've been attacked. If you're not running the latest and you're on the internet with your site, they've found you and have at least tried to take over your machine. The bad part is, you won't notice. They're not going to send you an email or post an alert on your site -- but they're there.
- Robert Douglass
-----
If this helped you, please take the time to rate the value of this post: http://rate.affero.net/robertDouglass/
www.hornroller.com, www.robshouse.net
Easy to say ... if you already know that
...but many people do not do that. To a large extent, people are accustomed to software telling them when there's an important update. Or, as part of some registration process, they are automatically notified when a security issue is released.
And it might just slip people's minds. They come here, download, install, ask questions, fiddle for a while ... and then other things come up, and they don't have the time or ongoing developmental interest to keep coming back to Drupal.org every day.
And signing up for a security issues newsletter may not occur to them. Many may not know it's possible. So a simple urgent reminder in the installation instructions would give them all they need. Something like:
"Be sure to sign up for the security issues newsletter [link]. This email list is used only for announcements of new releases of core software. Do it now, before you forget."
===
Laura
pingV
_____ ____ ___ __ _ _
Laura Scott :: design » blog » tweet
hmmmm.. maybe....
Maybe we can work on some verbage (SHORT) to add to the Install.txt on this and submit a patch for it. Something like
-sp
---------
Test site...always start with a test site.
Drupal Best Practices Guide
-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide
Good idea
Good idea
Security
I'll add some Security Basics stuff to the Best Practices section this weekend and include that suggestion. Good call.
-sp
---------
Test site...always start with a test site.
Drupal Best Practices Guide
-Steven Peck
---------
Test site, always start with a test site.
Drupal Best Practices Guide