The goal of this thread is to nail down why Drupal and Dreamhost don't seem to get along all the time.
Since the issues appear as though they're only happening with Dreamhost accounts, I (personally) don't believe the "issues" themselves should remain in drupal.org's issue queue, inasmuch as it's not Drupal's responsibility to correct the issues imposed by outside parties / hosting providers (in addition to several other factors). That said, I'll be working on transferring over the content of the existing issues in the queue and then closing each one.
Examples of existing issues:
- #213260: Error 500: Premature end of script headers on Dreamhost
- #735048: "Error 500: Premature end of script headers" on admin pages
Anyway, hopefully we can figure this out because I think it's driving a lot of people crazy. Any feedback is welcome. :)
Comments
Same problem here. The errors
Same problem here. The errors seemed to have died down for a while, but they're back in the last few weeks.
Dreamhost support said it's
Dreamhost support said it's because my PHP process is exceeding some memory usage limit that they decline to disclose.
mod_security
Just saw this...
http://www.askapache.com/htaccess/mod_security-htaccess-tricks.html#mod_...
Anyone have any thoughts? (ie Would applying any of this to .htaccess help with the whole Happy Scrappy Dreamhost Error 500 Giver thing?)
---
Yo that's shizzle.
Actually I saw this one first...
(Saw this one first, which ultimately led me to the one above.)
http://wiki.dreamhost.com/Mod_security
---
Yo that's shizzle.
Not just Dreamhost !
I have been fighting the EXACT same problems with a GoDaddy shared hosting account.
Drupal 6.17
It's driving me insane. Been trying to figure this out for a week. Sometimes site will be zippy, and admin pages load fast with no errors. Other times (like 5 or 10 minutes later), 500 errors galore - and if the admin pages do load, they take a really long time. Of course GoDaddy says there is nothing wrong on their end, and it must be some "buggy code".
Pulling my hair out.
The cause could be different.
The cause could be different. Dreamhost is explicitly canceling processes that exceed an undisclosed memory limit. Is Godaddy doing the same?
Different
Actually, I'm not sure if they are killing processes or not, but I don't think so.
We were able to get Godaddy to admit that there was high server load, and they moved the site to a different server.
All is well..........for now.
Same problem for me...
...with GD Shared hosting. If I get it fixed I'll post of course. Problem has always been intermittent, but is almost permanent since moving to core 6.19.
Problem solved....
...unfortunately I don't know how. I did contact GD support, did suggest that maybe another user on the shared hosting was thrashing the memory on the box, bu they replied that no, nothing was wrong.
By a happy coincidence, all of a sudden my Drupal site is running significantly quicker, and most importantly of course not timing out with 500 errors. If I was to guess, I would say I was right, Godaddy moved the offender to another box, but just didn't want to let on that was the issue.
Shared Server or VPS?
Are the problems your are have with the Dreamhost memory limits on their shared severs, vps, or dedicated servers, or all three? And are the problems limited to the admin pages?
I am on a shared hosting
I am on a shared hosting account.
Same problem here. I even try
Same problem here.
I even try to move to a vps server to see if the issue's gone.
The speed is fast. Server load is low. No persistent script.
But still get several 500 error from the anonymous user everyday.
Similar problem with some Wordpress accounts on Dreamhost
Similar problem with some Wordpress accounts on Dreamhost
http://wordpress.org/support/topic/500-internal-server-errors-with-dream...
http://wordpress.org/support/topic/admin-sometimes-404-links-mainly-404-...
http://wordpress.org/support/topic/dreamhost-issues-with-wp-using-too-ma...
Here is a response I
Here is a response I received:
"I know you dont want to hear it, but it is indeed still your memory
usage:
2010-08-01 11:41:51 procwatch2 INFO: Process(pid=xxxxx, name='php5.cgi',
uid=xxxxxxxx(1234567), tty=None, cpu=0.0, rss=43888.0, vsize=155908.0):
kill for total RAM
You should try to focus on the following sites:"
I have removed my account details for privacy reasons but here is what this suggests to me... every site is combined underneath a single uid. RAM is calculated for each user, not each website. If this is the case, having a few websites hosted will pretty much render them useless.
I don't think there is a solution to this from Drupal's end. Even if the admin memory usage was reduced, you would still reach their memory limits after a few sites because it is RAM calculated per user. I personally am at breaking point and seriously considering alternate hosts.
That explains some stuff
Actually, that explains a lot of stuff. And if it's true (which I believe is highly likely) then I suppose the solution would be -- aside from dropping DH like a bad habit -- to set up each new site you deploy under a new username. Personally, I think that this would be the most secure way of doing things, but on that same note, I'm not sure which approach would be less of a pain in the neck: if you go one way, you get unpredictable and uncontrollable 500 errors (obviously unacceptable); and if you go the other way you can basically forget about using Drupal's multi-site functionality (which, to me, is equally unacceptable, if not moreso, especially when it comes to updates, ftp/ssh logins, etc).
And while I feel more-than-confident that DH will not address this issue, the fact of the matter is that it's DH's policy for each account to be entitled to unlimited users. Each additional user, however, still exists under the same "umbrella" user account, all of which are still on the same server, thereby putting the same load on the server no matter which way you slice it.
In DH's defense though, to my knowledge (don't quote me on this though), anyone -- even on a shared hosting account -- can "sublet" their own account space (aka anyone can essentially act as a reseller) and as long as the primary account is good standing, none of the account's (sub-) users should encounter any issues, nor have to worry (anymore than usual) about security breaches and other things of a similar nature.
So... I don't know. I guess it depends on your individual needs..?? Either way, @jasont28 thank you very much for sharing that info.
---
Yo that's shizzle.
Other hosts?
I too am experiencing this. I didn't have a problem with two sites per user until I upgraded the second site to Drupal6. The first site has always been on Drupal6, and is smaller and less active. The second site is complex with many modules in use. I also have a test version of the site, and had no problem until I upgraded the production site as well as the test site.
I would greatly appreciate knowing about other shared hosts that don't experience this problem. I do have one other site, on Hostmonster, which has never had the problem, although there is only one production site there, in addition to staging and development sites. I would move there, but don't want to just move the problem.
Some months ago the
Some months ago the "php_value memory_limit" directive worked well in .htaccess file, but now it seems not to be counted.
My .htaccess:
php_value memory_limit 128M
php_value max_execution_time 180
php_value max_input_time 180
php_value post_max_size 80M
But in phpinfo() in same directory, these settings are still in default & unchanged:
memory_limit 90M
max_input_time 60
Config: CGI / FastCGI.
Does anyone? (I have not tried to contact DH support yet).
Edit: Since known issue is php_value does not affect in CGI mode, adding ini_set() directly in PHP file works.
Just out of curiosity (not
Just out of curiosity (not trying to be sarcastic or anything), how come you don't just set those directives in the php.ini file?
---
Yo that's shizzle.
http://www.google.com/search?
http://www.google.com/search?q=dreamhost+"php.ini" suggests altering the php.ini on Dreamhost has no effect.
The only solution I have found
@jerdiggity Thanks for suggesting to create multiple users. I have created a user for every site run on the Dreamhost account and it has solved the problem. As mentioned, it creates a different problem where upgrading modules and the core is laborious. For now it works. In the future I think Dreamhost will figure out what people are doing to bypass their painful restrictions which will be the point at which I choose a different host. Restricting multiple Drupal sites to the same 150mb(estimate of limit) of memory is very irritating.
Hope this helps people with the same issue at the moment.
multiple users?
Does this mean to create these users under the same account in the DH panel? I have only one production Drupal site on my hosting instance but it is setup as a multi-homed Drupal instance. Today this issue hit me with a vengeance and DH support's answer (provided below) was similar to responses from others above about the process watcher script. My site has been live for just a little over a month and the only other production app on my system is a non-Drupal app.
I dug a little more and it appears that you process are getting
killed by our process watcher script, which ensures that no one user
overconsumes resources on our shared servers.
2010-10-19 13:35:48 procwatch2 INFO: Process(pid=10613, name='php5.cgi',
uid=(11058648), tty=None, cpu=5.4, rss=45476.0,
vsize=151884.0): kill for total RAM
Please review the following articles and see if there is anything you can
do to bring your usage down:
http://wiki.dreamhost.com/Slow_site_troubleshooting
http://wiki.dreamhost.com/Bots_spiders_and_crawlers
NOTE: On these two links provided by DH support above, I have already disallowed all crawlers via robots.txt and gone through all of the other troubleshooting in those documents.
Does this mean to create
EDIT: Almost 2.5 weeks later, and the errors appear to have disappeared!!! How do I know? I use Dreamhost's cron instead of poormanscron, and I have it email me whenever it encounters an error. The emails from the one Drupal site I converted to its own account has STOPPED sending cron errors!!
I'm also having a lot of DH problems today, so that's what I'm trying. I'm also on multi-site Drupal.
Here's what I did:
Create new account. Make sure it has shell access.
In Manage Domains in the Dreamhost panel, set that new account as the account for the Drupal domain you want to run under this account. You'll do this by clicking the Edit button, under the Web Hosting column, in this domain's row.
Use Putty (a SSH program) to SSH your domain, logging in as the new account. At the prompt, create a symbolic link to your master Drupal directory with ln -s /home/[mainaccount]/[drupalBaseDirectoryName] [drupalBaseDirectoryName] where:
Note that Dreamhost may have tried to be helpful and created [drupalBaseDirectoryName] in your new account's root directory. You'll have to remove it before you can create the symbolic link.
After all this is done, your domain should work as it did before. Apparently, all new accounts are members of the group that has broad permissions on your main account's files, so no filesystem permission changes should be required.
Thanks for your detailed
Thanks for your detailed explanation. Could you please also tell me whic command line you are using on dreamhost for cron ?
It's all configured through
It's all configured through the DH panel.
Update: Switch to HostGator
Update: Either Dreamhost realized the workaround below and started punishing users or this was only temporary as a fix. After a few months, and not adding any additional modules/server load I again began getting the errors.
I switched to HostGator and not only did the problems go away, my sites now run blazing fast. I only use DH for domain registration at this point.
Again, the workaround I shared below will not work indefinitely for your site. If you still want to use it feel free maybe you'll have better luck.
----------------------------------
This symlink method is golden and works great. It divides up the memory usage because each site is technically under a different user but each web directory is symlinked to your main drupal install. However, I actually did have to re-create file system permission settings. My file uploads for Imagecache module etc didn't work.
Imagecache and other modules which upload files to the drupal file system stopped working correctly and began throwing errors because the symlinked web directory for these sites were under different users.
1. First what I did was create a Unix Group under Users -> Unix Groups.
a) Make sure all your users you include in this group have shell access.
b) This is configured under Users -> Manage Users. Default configurations should work fine.
c) These users should be the users you created for your symlinked multi-site directories above also make sure to include your main account user
2. After you create and name your Unix group you will shell into your account through your main user: ie the user that set-up the drupal multi-site install. (I use Tunnelier, others like Putty...)
3. Next make a backup of your drupal install files just in case you screw this part up.
a)cp -rp [your-drupal-directory] [z_your-drupal-directory]
b) Now if you type "dir" you should see another directory labeled [z_your-drupal-directory]
4. Now you are going to type chgrp -R [Unix Group Name] [your-drupal-directory]
a) This changes the group assigned to your drupal directory and all folders and files within it to the unix group we created earlier.
5. Now grant each directory that needs it read write access
a) chmod -R g+rwxs [your-drupal-files-directory]
b) This grants full read write access recursively to your files directly and all sub-directories and files w/in the files directory
c) Do this for the files directory in each site directory ie: site2.com/files site3.com/files etc...
d) test out a file or image upload and you should be done!
This page really helped me understand how to do this.
http://wiki.dreamhost.com/Unix_Groups
Can't wait to try it, I've
Can't wait to try it, I've been lurking in the forums daily, for months, trying to find a solution to this problem!
I'm also experiencing this issue...
and so often that I too added a new user for one of my domains and upgraded to their VPS service since they kept telling me that my premature end of script errors were caused by exceeding my resources. The bad news is that I still get these errors. Not as often, but I still get them.
Now that months have passed, can the individuals above still say that their errors have not reoccurred?
Possible clue
I found this thread trying to figure out why I am getting premature end of headers... with a PHP application. (This app is not actually using drupal.)
however, this thread was infomrative and helped me examine some different possibilities, especially this fcgi / execCGI stuff.
Im on a Dreamhost VPS. Everything was running fine until I switched from the standard DH php.ini to my own custom PHP ini. I used the instructions below to create my fgc-wrapper et al. : http://wiki.dreamhost.com/PHP.ini
once I created the wrapper and my own customized php.ini, things *seemed* to work except for one particular PHP file. That file was getting this error. This file was unique because it was calling into image functions like imagecopy(). The others were not.
After pulling my hair out, I found that removing this statement from one of my own include files stopped the premature.. errors:
error_reporting (E_ALL | E_NOTICE | E_STRICT );
Once this was removed (actually it was removing E_STRICT that made the difference) I stopped getting this error. My application was working again. However, my logs started showing:
23:29:56] PHP Warning: imagecopy(): supplied argument is not a valid Image resource in /home/..../image.php
Interestingly, this E_STRICT was not a problem until I started using the fcgi-wrapper and my own php.ini.
The problem for me seems to have been the combination error level (strict) combined with calling a function in a module which resulted in a warning message. The function, warning aside, worked fine until I started using the fcgi/execCGI wrapper.
It might be worth looking into whether Drupal ends up in similar combination cases. Does it call into functions which in some pages/cases return warnings? Where such a case would fail with the premature end of headers error on DH, does it succeed but with warnings somewhere else? Does drupal use E_STRICT?
This is a shot in the dark, but thought it was worth passing along.
Creating multiple users
The symbolic link method listed above worked like a charm. I was able to keep my multisite install and manage memory without getting any "process killed" warnings, or 500/404 errors.
Perfect!
Update: Files not writeable
It looks like for file uploads and such, since DH FTP users only have access to their own folders and can't upload files across user folders that things like image uploads etc. don't work. Anyone know of a workaround for this?
Thanks!
I was having these same
I was having these same issues. I have a Dreamhost VPS (Dreamhost PS) and also dedicated MySQL memory resources. In Windows on Chrome I was getting this: Error 324 (net::ERR_EMPTY_RESPONSE): Unknown error.
On Safari I was getting a 500 error.
I ended up going into the php.ini and changing PHP script and memory usage limits. That ended up not fixing it but I'll leave the changes anyway. What stopped all these erorrs for me was just increasing the guaranteed memory on the PS. I had 300MB and doubled it to 600MB, and so far everything is pretty fast.
why dreamhost recommend on
why dreamhost recommend on drupal.org - neither one site on drupal does not work. let remove it from recommended.
http://drupal.org/hosting
Important update: In the last
Important update: In the last few weeks, Dreamhost changed how security works, so it is no longer possible to have separate accounts that point to the same Drupal install as I mentioned above. The support tech's response to my inquiry:
I contacted support because I noticed that the group that used to contain all of my accounts no longer contains any accounts. So a separate account could no longer access my central Drupal install unless I added undesirable world permissions to the site.
Note that this general trick
Note that this general trick still seems to help, you just have to move the entire website over to the separate users (tar/gzip it and scp it across to otherhuser@localhost). You just can't have the full ease of managing all your sites from your original username.