I run a number of sites that feature lots of files such as images and videos. I have noticed some performance issues with increased traffic loads and also am running into backup issues and stuff like this. I am thinking it would be best to start thinking about abstracting the files directory to its own stand-alone server.
I was wondering if anyone had any experience in handling this in Drupal and could point me in the right direction.
My basic idea of this would be to have 2 servers. One main Drupal server with all my modules, sites and the databases, and then another file server that strictly handles content like images and videos.
I am thinking I would probably need to re-write a lot of the file architecture and allow it to pull files from different domains. This seems to be one of the main things Drupal won't let me do right now.
Does this seem like the way to go about something like this or am I totally off?
Any help is always appreciated,
Quinton
Comments
some thoughts...
Many high-volume sites use a lightweight web server (like httpd, or a minimal install of apache) with no dynamic capability (it saves memory and cpu cycles) to serve static files.
But before you try something drastic with rewriting image paths, what other optimization strategies have you tried?
Thanks for the reply. I am
Thanks for the reply. I am currently on a fairly nice Apache box with a Quad-Core 3210 Xeon, 4 GB ram and 4 SATA II Harddrives.
I run most of my high traffic pages on a static page cache (via boost.module) and this cuts down the the server load greatly.
My main problem comes when I have roughly 10+ video or image content embedded in my page (which is pretty much every article). It seems like I start hitting a bunch of requests and even static page caching can't deal with this. I have also tried caching my images on nyud.net which does work, but its reliability isn't the best.
Other than that I've made some minor tweaks to apache and php but I'm not sure how far these types of things will allow me to push it.
I'm not an expert on servers or anything, but I do think a lot of the basics have been covered.
Our server has also gone down a few times (for up to 30 min) after getting lots of traffic bursts like hitting the Digg homepage. I have seen cases where the server easily handles 1500 or so concurrent connections with just a static no-image article, but seen it go down with less connections and lots of media.
Hopefully this helps you out a bit more.
Appreciate the help :).
some things to try
The first thing you need to do is finding out which bottleneck (memory, CPU, disk) you're hitting, and what is hitting it (apache, or mysql). From the description of your problem and your use of large files, it's probably not a mysql bottleneck.
Watch the output of the 'top' command, and filter it so that you only see the apache processes (to do this, type 'u' and type the name of the user the apache processes are running at). While you're in top you can see your swap file usage.
My gut says it might be ram usage, but that should hardly be taken as advice. I'd have to know more about the server and its specific configuration to give more useful advice (other than general tips).
One more thing: have you installed an opcode cache, like apc?
Thank you very much for the
Thank you very much for the great replies :).
Using the top command it looks like majority of my requests are coming from user "nobody" under the command "httpd", but they're all fairly low cpu usage. I also get some requests from "marvquin" which is the username for one of my main cpanel packages. These request usually use a bit more cpu usage like 20% or so. This is also a lower traffic time so it's probably not as accurate.
I am noticing that although mysql is using fairly low cpu usage each time (1 - 5%) it is using roughly 3.9 %mem and 292:14.50 time while most other things are using 0.0 - 0.4 %mem and taking 0:36:00 or less time.
If mysql was my bottleneck would you recommend possibly throwing that on a separate box and keeping my web server / file system on the same one?
I have not installed on opcode cache. Would you advice something like this?
Thanks again for all your help!
yes
an opcode cache is a significant performance boost. Any should be fine: I would recommend apc as it can be installed directly through pecl. There should be a few installation guides on the Internet.
That's the usual course of action to take when mysql creates a bottleneck. But in your case, it may not be. Since you've described a loss of performance when your site is under heavy load and there are lots of images and videos on the homepage, I don't think that mysql is the bottleneck.
The most important thing to remember when trying to optimize Drupal performance is to try one thing at a time. Try the opcode cache first. If you can get access to a separate server, you might try temporarily hosting your frontpage images and videos on that, to see if it reduces the load on the main server.
Thanks for your help. I will
Thanks for your help. I will try the opcode cache and see how that goes. I have already tried hosting my images from certain articles on a separate server and I confirm that helped A LOT. Appreciate the help.