Drupal on multiple servers (securely)

By James Andres on 28 Oct 2005 at 21:45 UTC

Hey everyone!

I'm working on a very large Drupal project (www.projectopus.com, shameless self promo) and have come to the point where it is obvious that we need to get our system on multiple servers.

First, my question: Does anyone have experience with running drupal over multiple servers? what advice can you give me (us)?

Second, our setup (roughly):
Our site is a music portal, among other things, and it we are readily getting loaded up with more and more content: AWSOME!

However, the files directory (where all music, and other user uploads are stored) is getting REALLY large. Right now we have pretty good security in this system whereby all files are being transfered with an embedded music player and other indirect means. It is clear though that we need this data on a seperate server, or at some point in the near future we'll crash the system: NOT so awsome...

Thirdy, my idea for a new setup is as follows:

First server:
* Serves the main website, html, php, DB, images that are part of the main site
* contains super modified drupal installation that is smart and knows that downlod requests need to come off server #2

Second server:
* Has a small website of scripts that authenticate file transfer requests (serious development work here I think)
* Contains all user uploaded data (music, node attachments, et al.)

So, does anyone have experience with problems of this sort? Do you know what modules I'll be hacking to get this kind of functionality?

Comments

nfs

chx commented 28 October 2005 at 23:06

nfs
--
Read my developer blog on Drupal4hu.

--
Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile.

nfs is short term

James Andres commented 31 October 2005 at 17:58

Hey,

I've already got NFS going between the 2 servers; however, this is only a short term solution.

Here's why: 1) A user requests a file from www.site.com, 2) www.site.com pulls the file from nfs share which is on files.site.com, problem) the bandwidth to download the file is used 2x since it gets transfered from files.server and from www.server. This is a waste of money and bandwidth.

My plan is to have a system something like this (I have a pretty good idea of how to do it, but advice is welcome!!):

1) User requests a file from www.site.com
2) The file link on www.site.com is ACTUALLY a link to files.site.com
3) The user can keep browsing the www.site.com while downloading their file from files.site.com.
4) Now www.site only uses bandwidth for the website traffic and files.site.com only uses bandwidth for the files transfered. If there are TONS of people downloading files the www site is not bogged down. If there is tons of site traffic then the file server isn't bogged down.

James

Developer on Project Opus
w: www.projectopus.com
h: japhotogallery.webhop.org

Same problem here (more or less)

Geary commented 29 October 2005 at 00:28

I am facing the same problem with a couple of my church sites, KingsWay Community Church and Pressing In (both early works in progress). We have accounts on TextDrive and DreamHost. DreamHost has for our purposes unlimited storage (10GB, with 80MB added every week--if we upload less than 80MB/week we'll never run out). But MySQL runs much faster on TextDrive--our Drupal sites load 2-3 times faster there.

So if I can do it securely and without too much work, I want to host the sites on TextDrive and put our large .mp3 and .swf files on DreamHost.

I don't think nfs will help me, because this is shared hosting. But sftp seems like it ought to do the trick.

JLTB, are you on your own servers or shared hosting or what?

security is key

James Andres commented 31 October 2005 at 18:12

Hi Geary,

Security is one of the main reasons I'm asking for help with this.

We have 2 seperate severs (not shared hosting). They have to communicate over the internet (although they are probably in the same farm since I've benchmarked nfs speeds between the two at around 1.5 - 2.0 MB (byte) per second).

Basically, our site sells music. Some of the music is free, some is not free, but either way we can't have people just swiping the link and downloading music without paying.

Some of my ideas for security include:
1) temporary hash strings (this one seems the best to me). Something like http://files.site.com?secret=23rljdflj234jlc9p23d where this hash string is a special key that lets only 1 authenticated user download the file and only for a short period of time. However, this is still exploitable.

2) background datasharing. Behind the scenes the www.site could be sharing either DB access or some transfer method which would enable it to authenticate that a user is logged in and is who they say they are (IP based?). Again, exploitable ... especially by people who are on large 1 IP networks (ie: university dorm's).

Anyway, I don't really know very much about what I'm saying :-P (though I might be good at faking it).

For your needs Geary I think nfs should work good. NFS is pretty insecure but you can lock it down satisfactorily by limiting it to a 1-1 host configuration (for example: nfs only allowed from 204.23.56.2 and nowhere else) with a tight firewall. However, I don't know how much control you will have over that on a shared hosting plan.

James

Developer on Project Opus
w: www.projectopus.com
h: japhotogallery.webhop.org

two server drupal website plan (fileserver)

James Andres commented 1 November 2005 at 01:15

For those who may be stuck in a similar situation at some point, here's my plan thus far: Well it's a data transfer digi kinda.

1) Client clicks a link on 'www' server. This link is of the form files.com?file=35&user=SESSIONID (where file is an nid, and SESSIONID is the users session hash string).
2) 'files' server grabs the users IP address, and the SESSION hash.
3) 'files' server connects to 'www' server via an ssh tunneled mysql connection and finds out if there is a living session for a user in the last 10 minutes (ish) that corresponds to host=$ip AND session=$session.
4) If there is it returns the uid
5) Now 'files' server knows that a valid logged in user is here, and checks whether the user has access permissions to the NID they are asking for.
6) If so the files server grabs the file name (say files/blah.pdf) and sends it back via header calls and a readfile($filename). (see php.net header docs for more info).
7) done!

My plan is to wrap the 'www' side of things into a module called 'seperateserverdownload.mode' (working name ;-) and probably keep the 'files' server side as a bunch of PHP files. As it stands my 'files' server isn't drupal in anyway, so there is no need to have a module over there.

If you have any insight into what I'm doing here or would like to see any code just let me know.

James

Developer on Project Opus
w: www.projectopus.com
h: japhotogallery.webhop.org

NFS and fail over

sus commented 10 April 2007 at 16:28

We run a large site hosting music files and use this system

2 NFS servers - Running Debian linux, NFS server, drbd (file replication service) and heartbeat. Both have 1TB storage mounted at /data and 500GB storage mounted at /sites. nfs1 is the main server that provides the share. If the heartbeat service detects nfs1 is misbehaving, it mounts /data and /sites on nfs2 (which is kept up to date via drbd) and redirects all requests to nfs2 using the same ip address as nfs1. I know we are using double the disk space but high avaliabillity is a must for us. These servers have no public ip address only internal ones.

2 Database servers - Using the same system as above but without the drbd service. Insted, we just use MYSQL database replication to ensure sycronside databases. Again, if db1 fails, heartbeat redirects the ip address and starts serving databases from db2.

4 Web servers - all our web servers are configured exactly the same apart from host name and ip address. They all run apache2 and use nfs to mount the /data share at /var/data and the /sites share at /var/www. Because /data is outside of the the web root there is no access via browser. We use a simple script to copy a requested file to a download basket and use MD5 session Ids to ensure only one download. We then use a basic round robin style dns entry to stimulate load balancing (although we are looking at proper load balancing in the future)

This system has many advantages:

1 - centralised storage of data only accessible via internal network and fully redundant in case of server failure
2 - Fully redundant database in case of server faliure
3 - We can add web server resources by adding another web server and putting in a dns entry for it

Maybe a bit overkill for some sites but i don't want to travel 80 Miles to the data centre to replace a server whilst the website loses users. Using this system we have had 100% uptime even with 2 web server and a database server hardware failure.

Hi sus,That's good,You are

amitguptaadmin commented 13 May 2009 at 15:28

Hi sus,

That's good,

You are using a good environment.

But can you suggest me a solution.

Mine is also a large project.

My environment is solaris.

I am having the drupal, php, apache in two web server system in production environment (for load balancing).

If one system fails the other will be active.

I am having two database servers for database clustering.

The load balancer will forward the request to my webservers, webserber will be connnected to the database server

My question is that how i am going to share images between two servers.

I cannot have on more server, i mean one more system.

If I implement the your system, i have to have wo more NFS servers, where i have to store my data and site which will be fetched from my two webservers.

Can I implement the FRP service in my environment, so that if I submit image form from one server then it will store the copy of the image to other server as well, whether the image in created or modified.

Please suggest........

ALL of you Guys, u can also suggest me the solutions