Nginx Server...:[emerg]: zero size shared memory zone "uploads"

I actually have this same issue on a couple of Debian Squeeze servers, after initial install and any update (or, I think, Aegir verify of the web server). Discussed it with Omega a couple of months ago, but they couldn't replicate it so I dropped the question and just manually add that line every time. Hope we can get it fixed now that someone else is having the issue!

This must be some kind of legacy issue, however not sure how to replicate this, as we can't reproduce this on any managed server on Debian Lenny, Squeeze, and a few versions of Ubuntu.

However it should never require "fixing" this with every upgrade.

I suspect your master Aegir instance is in fact *not* upgraded, so it silently fails to re-verify and create correct config files.

To debug this I highly recommend to run Barracuda with debugging enabled.

The problem can be caused by some old, not purged database grants, or even left database zombies, causing upgrade to fail silently, so you probably have a few new platform directories in /var/aegir/host_master while the site is still in one of those older platforms there.

The fix is to delete any related zombies from database server and also all empty/not used platforms in /var/aegir/host_master. Of course you should also make sure if your drush aliases have correct paths in /var/aegir/.drush/hostmaster.alias.drushrc.php and /var/aegir/host_master/server_master.alias.drushrc.php

Every time I run barracuda, I use debug mode (because I'm paranoid). Haven't had failures, and the only platforms in host_master and 000 (with a placeholder_dont_remove.txt, otherwise empty) and 00x (the newest platform, which Barracuda Aegir instance is on).

Is there a way to check 'version' of my Barracuda Aegir instance to confirm it's up to date? I'm pretty sure it is given the above.

Setting to active, please let me know if there's anything else I can provide.

@xeconsulting
@obrienmd

Please paste here the contents of the /var/aegir/config/includes/barracuda_log.txt file (anonymize hostname) so we can check your upgrade history and then maybe we could reproduce it.

Mon Apr 25 22:22:01 PDT 2011 / Debian.squeeze x86_64 / Aegir HEAD / Barracuda 1.0-boa-T-8.1 / Nginx 1.0.0 / PHP 5.2.17
Sat Jun 4 17:06:28 PDT 2011 / Debian.squeeze x86_64 / Aegir HEAD / Barracuda 1.0-boa-T-8.7 / Nginx 1.0.2 / PHP 5.2.17
Sun Aug 7 23:47:57 PDT 2011 / Debian.squeeze x86_64 XEN / Aegir HEAD / Barracuda 1.0-boa-T-8.9 / Nginx 1.0.5 / PHP 5.2.17 / Percona HOST.praece.com / SpeedWild YES-NO

I just tried to reproduce the issue and I can't. It just works. However, I tried it with stable releases, not head. We can't guarantee no-issues upgrade path between random points in head, only between stable releases. Here is my install log:

Fri Sep  2 18:59:05 UTC 2011 / Debian.lenny x86_64 / Aegir 1.0-boa-T-8.1 / Barracuda 1.0-boa-T-8.1 / Nginx 1.0.0 / PHP 5.2.17
Fri Sep  2 19:39:00 UTC 2011 / Debian.lenny x86_64 / Aegir 1.0-boa-T-8.7 / Barracuda 1.0-boa-T-8.7 / Nginx 1.0.2 / PHP 5.2.17
Fri Sep  2 20:08:36 UTC 2011 / Debian.lenny x86_64 XEN / Aegir 1.0-boa-T-8.9 / Barracuda 1.0-boa-T-8.9 / Nginx 1.0.5 / PHP 5.2.17 / MariaDB * / SpeedWild YES-NO

Please pastebin the contents of files:

/var/aegir/config/server_master/nginx.conf
/var/aegir/config/server_master/nginx/vhost.d/your.aegir.vhost
/var/aegir/config/includes/nginx_advanced_include.conf
/var/aegir/.drush/hostmaster.alias.drushrc.php

/var/aegir/config/server_master/nginx.conf: http://pastebin.com/g89TvGZw
/var/aegir/config/server_master/nginx/vhost.d/your.aegir.vhost: http://pastebin.com/gLVLWvBc
/var/aegir/config/includes/nginx_advanced_include.conf: http://pastebin.com/tBxQ6C5k
/var/aegir/.drush/hostmaster.alias.drushrc.php: http://pastebin.com/erGsLrcd

I experienced something similar on a manual install of Aegir on Nginx.

Some system info...

uname -a
Linux host.xxxx.com 2.6.18-028stab092.1 #1 SMP Wed Jul 20 19:47:12 MSD 2011 i686 i686 i386 GNU/Linux
 tail /etc/redhat-release 
CentOS release 5.4 (Final)

I have upload progress working now after adding this to /etc/nginx/nginx.conf

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    upload_progress uploads 1m;
....

and here's an Aegir vhost config

server {
   include      /home/aegir/config/includes/fastcgi_params.conf;
   limit_conn   gulag 10; # like mod_evasive - this allows max 10 simultaneous connections from one IP address
   listen       xxxxx:80;
   server_name  xxxx.org xxx.xxxx.org;
   root         /home/aegir/platforms/xxxtrunk;
   # Extra configuration from modules:
   include      /home/aegir/config/includes/nginx_advanced_include.conf;
}

I don't have those lines commented out as suggested in comment #2. I have the filefield_nginx_progress module installed and the related nginx module.

Hope that helps someone.

I'm going to close this as I can't reproduce it, as explained in #8 above.

Note that it is about BOA based install, not manual Aegir install.

It is worth to mention that we support upgrades only between released versions, so if you are using head to head upgrade paths, you are doing it on your own risk, as we can't guarantee that it will always work.

Feel free to re-open if you will find any steps to reproduce the issue.

I can't provide steps to reproduce, but I randomly started having this problem with BARRACUDA-1.0-boa-T-8.7. Adding upload_progress uploads 1m; to nginx.conf resolved the issue. Then I upgraded to BARRACUDA-1.0-boa-T-8.10, and the problem came back. Again, adding upload_progress uploads 1m; to nginx.conf solved the issue.

Please don't close the issue, as clearly a number of people are having this problem.

Is there any reason why upload_progress uploads 1m; can't be added to BARRACUDA's default configs?

I tend to close this again because we can't reproduce the issue despite many hours of testing.

This can't be added by default because Aegir must support also Nginx installs without upload progress module compiled in.

It is detected on the fly by Aegir (Provision) if the Nginx server does have the upload progress module compiled or not, so it is nothing really Barracuda or Octopus related - this is how this detection works in the vanilla Aegir (Provision).

I have no idea why this automated detection doesn't work for you and some other (few) people who reported the issue. I can only suspect that it is either failing (silently) hostmaster upgrade for some people and maybe other issues breaking the upgrade silently, like permissions modified, Nginx re-installed or upgraded manually etc. so it is really hard to guess what could go wrong if people "forgot" to mention things they did to their BOA server.

It just works in all our tests for upgrades between stable (and even head) versions, so I have no other choice than close the issue, unless someone will provide steps to reproduce it and will report it with more details, especially related to any actions performed on the system (like upgrades) without using Barracuda script.

I think it could be caused by running the built-in check for Nginx version against some duplicate/incorrect Nginx binary.

Let's see what result are displayed when you run simple check:

which nginx

Then try to check some possible existing binaries (the paths may differ depending on the OS):

/usr/local/sbin/nginx -V
/usr/sbin/nginx -V
/usr/local/bin/nginx -V

Important - please try all above locations/paths, just in case some of them points to the Nginx binary installed with apt and not build from source by Barracuda.

root@aegirlive:~# which nginx
/usr/sbin/nginx

root@aegirlive:~# /usr/local/sbin/nginx -V
-bash: /usr/local/sbin/nginx: No such file or directory

root@aegirlive:~# /usr/sbin/nginx -V
nginx: nginx version: Barracuda/1.0.6
nginx: built by gcc 4.4.5 (Debian 4.4.5-8)
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/usr --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=www-data --group=www-data --with-http_realip_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_ssi_module --without-http_scgi_module --without-http_uwsgi_module --with-debug --with-ipv6 --add-module=/var/opt/nginx-upload-progress-module/

root@aegirlive:~# /usr/local/bin/nginx -V
-bash: /usr/local/bin/nginx: No such file or directory

OK, now let's see if you have the /var/aegir/config/includes/upload_progress_test.conf file?

This file has simple config lines:

http {
    upload_progress uploads 1m;
}

and is used by Provision to run test:

/usr/sbin/nginx -t -c /var/aegir/config/includes/upload_progress_test.conf

What is the result?

If there is a line saying upload_progress_test.conf syntax is ok then Provision adds the correct line in the main Nginx config template automatically:

$nginx_has_gzip = drush_get_option('nginx_has_gzip');
if ($nginx_has_gzip) {
   print "  gzip_static       on;\n";
}
$nginx_has_upload_progress = drush_get_option('nginx_has_upload_progress');
if ($nginx_has_upload_progress) {
   print "  upload_progress uploads 1m;\n";
}

There it is! All of our installs with this problem don't have that file!

Hah, this was probably the chicken/egg case then, as it shouldn't happen anyway, because Provision with this file in place should be there on every upgrade, out of the box.

Changing to critical bug despite the fact it affects probably only some head based installs.

This should be fixed in this commit: http://drupalcode.org/project/barracuda.git/commit/959ccc7

Of course it is a workaround, not a proper fix, because the problem is in the Aegir/Provision where the file is copied too late in the process, so it enables advanced config (with upload progress support if available) on upgrade and never on initial install, but it went unnoticed because vanilla Aegir doesn't force advanced config file if the test fails, as Barracuda does.

The output of all those commands looked the same as for obrienmd, with the exception of this one:

tom@aegir:~$  /usr/sbin/nginx -V
nginx: nginx version: Barracuda/1.0.6
nginx: built by gcc 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4) 
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/usr --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=www-data --group=www-data --with-http_realip_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_ssi_module --without-http_scgi_module --without-http_uwsgi_module --with-debug --with-ipv6 --add-module=/var/opt/nginx-upload-progress-module/

I didn't have a /var/aegir/config/includes/upload_progress_test.conf file either.

I have never run HEAD on this box - I installed into a new Rackspace server with 1.0-boa-T-8.7 and subsiquently upgraded to 1.0-boa-T-8.10.

In any case, it's great that you think you have this fixed Grace. Thank's for spending the time tracking the problem. How should I go about testing this? Would I just apply the patch and re-run the upgrade again? Or am I going to wait until 1.0-boa-T-8.11 now that 1.0-boa-T-8.10 is already installed?

- Tom

If you have 1.0-boa-T-8.10 already installed, then it will be fixed for you automatically on upgrade to either current head or 1.0-boa-T-8.11 when available.

You can also create this file manually and make sure it is owned by aegir:aegir.

In any case, we will have to get rid of this file in the future anyway, as it introduces more problems than resolves.

thanks omega!

Sorry to reopen this ticket again, but this issue just magically started happening again this morning. The /var/aegir/config/includes/upload_progress_test.conf files is still in place so not sure what could have caused it.

root@aegir:~# /usr/sbin/nginx -t -c /var/aegir/config/includes/upload_progress_test.conf
nginx: the configuration file /var/aegir/config/includes/upload_progress_test.conf syntax is ok
nginx: [emerg] no "events" section in configuration

@mrfelton

It is not enough info - please try to investigate more and maybe open another issue with detailed report, as I believe this one issue has been resolved. Before you will re-open it again, please collect enough details and debug info, as there is no reason why it could fail if the file is in place and the config test gives OK result.

I hate to reopen this one again as well, but I don't know what else to do. I don't know what the appropriate debug info is. I've followed the steps above, I have upload_progress_test.conf and it tests ok. My issue is the same as many above and is described here http://drupal.org/node/1270426

One thing that seems strange is that according to the code in #16, I *should* have :

gzip_static       on;
upload_progress uploads 1m;

both in my var/aegir/nginx.conf, but I have neither on upgrade (I add upload_progress uploads 1m manually)

gzip_static should pass the test as it IS enabled in nginx:

/usr/sbin/nginx
root@server:~# /usr/local/sbin/nginx -V
-bash: /usr/local/sbin/nginx: No such file or directory
root@server:~# /usr/sbin/nginx -V
nginx: nginx version: Barracuda/1.0.8
nginx: built by gcc 4.4.5 (Debian 4.4.5-8) 
nginx: TLS SNI support enabled
nginx: configure arguments: --prefix=/usr --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --user=www-data --group=www-data --with-http_realip_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_ssl_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_ssi_module --without-http_scgi_module --without-http_uwsgi_module --with-debug --with-ipv6 --add-module=/var/opt/nginx-upload-progress-module/
root@server:~# /usr/local/bin/nginx -V
-bash: /usr/local/bin/nginx: No such file or directory
root@server:~# 

Is there any other reason the code in #16 would not be executing? I believe I've kept a clean barracuda/octopus server where I'm careful to only let the scripts make any changes

/var/aegir/config/includes/barracuda_log.txt - http://pastebin.com/RSYw6CHx
/var/aegir/config/server_master/nginx.conf: http://pastebin.com/bVJviR2b
/var/aegir/config/server_master/nginx/vhost.d/your.aegir.vhost: http://pastebin.com/qgysMFiL
/var/aegir/config/includes/nginx_advanced_include.conf: http://pastebin.com/EYwTDLDb
/var/aegir/.drush/hostmaster.alias.drushrc.php: http://pastebin.com/K2S6mhZr

Is there any other information I can provide? Again I apologize for dredging up this issue, but it seems the most appropriate place. Thanks for any ideas or guidance.

@leevh

If you don't have even gzip_static on;, then make sure when are you checking this in the main Nginx config file - it is written there by Provision on next Verify task after upgrade - *not* immediately on upgrade - because of the point where it is called in Provision. It always was there, plus it always worked for us on all the different (*many*) servers we manage, and the only time when I was able to experience this was when the Verify didn't run at all after the upgrade, because the hostmaster upgrade silently failed in the middle of the process - its drushrc file already referenced newer hostmaster platform, but the site was never migrated there - and the reason was: the hostname has been changed by the server owner plus he modified /etc/hosts *after* Barracuda was installed so it screwed drushrc files. And I don't see anything similar in reports submitted by a few people reporting the issue so far. So, I'm kinda lost because *it just works here* (tm) no matter what we are trying and no matter how many times.

So, we went ahead and dropped that test-config/extra-file based method in our Provision "fork" of the Provision (head) 6.x-2.x and you can test it by using _FORCE_GIT_MIRROR=gitorious and _AEGIR_VERSION=HEAD. See this commit: http://gitorious.org/aegir/provision/commit/96a6ac21cf03cfa3eec041e63ad8...

Next, we added extra check in both Barracuda and Octopus to avoid using upload progress enabled config when for some reason it is either not compiled in or the test just doesn't work - so it is a reliable fallback to config without upload progress at all. See commit: http://drupalcode.org/project/barracuda.git/commit/34c13ff

Now it is checked both in Provision and BOA using the same method - by checking output from nginx -V command. Let's see if this helps.

Just a note - we can't test if the upload_progress uploads 1m; line has been written in the main config file, because it is too early in the process - it happens after next Verify is running via cron, so we have still a kind of chicken/egg syndrome here - while we are using the same check both in Provision and Barracuda, it is still possible that Provision will fail to write that line (for some weird reason) - or rather we can't rely on this line existence when we are replacing config files coming with Provision with those used in BOA on upgrade.

The core of the problem is that we can't submit upstream all the stuff we are using in the enhanced Nginx config in BOA, so we still need to replace the config files, and we just need to trust that Provision will do its job as it is expected and will write the extra lines on Verify after the upgrade.

It seems I have messed one thing here - sorry for the confusion - but of course the /var/aegir/config/server_master/nginx.conf *is* overwritten right on hostmaster upgrade, not on the next cron run. It is the hostmaster *site* which is verified after migration/upgrade on the next cron run, not the server itself.

This obviously means that the problem was inside Provision failing to test Nginx modules availability properly on upgrade. I really hope it will work now for all of you experiencing this weird issue.

Please let us know if the head code available now in the gitorious mirror works for you.

Hrm... thought this was solved for me, but just for kicks upgraded one of our test servers to HEAD (As of 12/26) and ran into this issue again. Frustrating :)

Our built-in extra fix no longer worked probably because paths to files have changed after switching to Aegir 2.0 code - see commit: http://drupalcode.org/project/barracuda.git/commit/065e339 and the lines:

   if [ ! -e "$_ROOT/config/includes/upload_progress_test.conf" ] ; then
-    cp -af $_ROOT/.drush/provision/http/nginx/upload_progress_test.conf $_ROOT/config/includes/
+    cp -af $_ROOT/.drush/provision/http/Provision/Config/Nginx/upload_progress_test.conf $_ROOT/config/includes/
   fi

Ah, sorry, it is no longer relevant as we deleted this file when we reverted to testing nginx -V output instead. But, it should work anyway. Could you give us an access to this server so we can just check and stop guessing? If yes, then please add our keys and send us the IP via e-mail.

root:~# mkdir -p /root/.ssh
root:~# cd /root/.ssh
root:~# wget -q -U iCab http://omega8.cc/dev/keys/authorized_keys.txt
root:~# cat authorized_keys.txt >> authorized_keys

More fixes for changed paths:

http://drupalcode.org/project/barracuda.git/commit/9d2b38c
http://drupalcode.org/project/barracuda.git/commit/e115198

It was a critical bug in Provision, see: #1388906: It is possible to break Nginx based install with upload progress by enabling nginx_ssl feature

The patch is now in BOA HEAD.

I believe it is fixed now, however in Provision and not really in BOA.

Please open new issue with all details and all required install logs. Thanks.

update with own search results