Hi,

Thanks for those great scripts, it makes aegir so much easier to set up. However I'm facing an issue with memory leaks. I installed Barracuda and one instance of Octopus on a fresh vps with Debian 6 32bit (1gb ram + 1gb burst). Once the installation of Octopus finished , I was using 87% of the 2gb . Then I reboot the vps and added one platform (OA), Now with one user, I am running 92% which is about 1840mb used by aegir platform. The requirement was 512mb (didn't install Solr) so I am wondering why I have 5 processes of /usr/local/bin/php-cgi --fpm --fpm-config about 160MB each?

CommentFileSizeAuthor
#5 barracuda_log.txt160 bytescharos
#5 install.log_.txt5.84 KBcharos
#5 ps aux.txt3.98 KBcharos

Comments

omega8cc’s picture

Status: Active » Postponed (maintainer needs more info)

Could you post your install logs, as listed on the issue submit form?

I suspect you are on some Virtuozzo based VPS, but we need your logs to confirm what could cause such weird behavior.

We always test BOA on 512 MB RAM Linode with everything (including Solr/Tomcat) and all platforms installed and while it is using some swap, the RAM usage is rather low:

  1  [                                                         0.0%]     Tasks: 57, 100 thr; 1 running
  2  [*                                                        0.7%]     Load average: 0.02 0.04 0.05 
  3  [                                                         0.0%]     Uptime: 5 days, 04:04:01
  4  [                                                         0.0%]
  Mem[||||||||||||||||||||||##****************************169/488MB]
  Swp[||||||||||||||||||||||||||||||                      243/511MB]
root@linode:~# free
             total       used       free     shared    buffers     cached
Mem:        500452     398148     102304          0      11088     221716
-/+ buffers/cache:     165344     335108
Swap:       524284     256200     268084
root@linode:~# 
charos’s picture

Sadly I didn't kept any logs. But I will re-install OS and do the installation from scratch and save it in a log file. Hope to deal with this soon when I get some free time (examination period period in university is killing me!).

omega8cc’s picture

No, the logs are there, please read: http://drupal.org/node/add/project-issue/barracuda

charos’s picture

Now I really hate myself. I already started to install a fresh OS {face palm}

charos’s picture

Status: Postponed (maintainer needs more info) » Active
StatusFileSize
new3.98 KB
new5.84 KB
new160 bytes

Here's my log files. Hope it helps somehow! I also added the "ps aux" output after the installation of Octopus finished (untouched).

omega8cc’s picture

Thanks. You are on the Virtuozzo powered system and it can be a reason of the problem. We strongly recommend to avoid Virtuozzo at all costs, as it is known to cause critical issues for Drupal in general (it is not just BOA specific). But if you wish to try if this could work anyway, try to lower default limits for APC, as suggested here: http://drupal.org/node/1155160#comment-4461506 and reload PHP-FPM with service php-fpm reload.

charos’s picture

I knew that OpenVZ didn't have the performance of Xen , but I've never heard of memory leaks before !

I reduced the APC size to half but that didn't change anything. I think the problem is the number of workers loaded for memcached and php-cgi --fpm. Here's a list of the processes consuming most ram :

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

bind      1565  0.0  0.6  83068 12680 ?        Ssl  07:03   0:00 /usr/sbin/named -u bind
mysql     1904  0.4  4.4 1140148 92496 ?       Sl   07:03   3:10 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib
root     36365  0.0  0.0  88188  1240 ?        Ss   17:36   0:00 nginx: master process /usr/sbin/nginx -c /
www-data 36366  0.0  0.1  88992  2184 ?        S    17:36   0:00 nginx: worker process                   
www-data 36367  0.0  0.1  89152  2504 ?        S    17:36   0:00 nginx: worker process                   
www-data 36369  0.0  0.0  88188  1472 ?        S    17:36   0:00 nginx: cache manager process
root     17783  0.1  0.2 155124  4388 ?        Ss   16:56   0:04 /usr/local/bin/php-cgi --fpm --fpm-config
www-data 17784  0.0  1.1 163604 23648 ?        S    16:56   0:00 /usr/local/bin/php-cgi --fpm --fpm-config
www-data 17785  0.0  0.6 157212 14652 ?        S    16:56   0:00 /usr/local/bin/php-cgi --fpm --fpm-config
www-data 17786  0.0  0.6 156956 14392 ?        S    16:56   0:00 /usr/local/bin/php-cgi --fpm --fpm-config
www-data 17787  0.0  0.6 156956 14396 ?        S    16:56   0:00 /usr/local/bin/php-cgi --fpm --fpm-config
nobody   36026  0.0  0.0  44724   816 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11211 -m 6
nobody   36033  0.0  0.0  43840   816 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11212 -m 6
nobody   36040  0.0  0.0  43840   820 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11213 -m 6
nobody   36047  0.0  0.0  43840   820 ?        Ssl  15:15   0:02 /usr/bin/memcached -u nobody -p 11214 -m 6
nobody   36054  0.0  0.0  43840   820 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11215 -m 6
nobody   36059  0.0  0.0  43840   816 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11216 -m 6
nobody   36063  0.0  0.0  43840   820 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11217 -m 6
nobody   36075  0.0  0.0  43840   816 ?        Ssl  15:15   0:01 /usr/bin/memcached -u nobody -p 11218 -m 6

I reduced the nginx workers to 2 , but I have no idea how to reduce them for memcached and php-cgi --fpm . Any suggestions?

I think you should rename the issue name to "Memory issues with OpenVZ" to avoid the confusion for the readers.

omega8cc’s picture

Title: Memory issues? » Memory issues with Virtuozzo/OpenVZ

You can rename the issue, however the Virtuozzo/OpenVZ family is the worst choice ever for Drupal anyway. Reducing the number of workers is not a solution. It is memory/limits management which is a horrible crap there. We added experimental support for Virtuozzo/OpenVZ as some people requested it in the past and for some reason it still works for them, so it seems you are using some really bad host if it gives you so weird results. Please use different host, as we don't plan to extend or debug this experimental support for Virtuozzo/OpenVZ.

charos’s picture

Let me re-phrase this. Xen is more "isolated" than OpenVZ in terms of virtualization. The major difference comes from the way they allocate memory. So burstable ram in OpenVZ comes from "borrowing" memory from another slice. So this cannot guarantee the allocation (at any moment system can shutdown processes). So the biggest problem with burst ram is that in OpenVZ , it can kill your application while in Xen it will only slow down (in Xen virtual memory that you can allocate is almost unlimited if you have enough swap space configured).
I think that's the reason you mentioned that OpenVZ can cause critical issues.
However I cannot find a single source of information stating that OpenVZ is the worst choice for Drupal or the cause of memory leaks.
I will try to investigate this issue and if I find anything useful, I'll post it for the sake of other OpenVZ users.

omega8cc’s picture

Status: Active » Closed (won't fix)

There are many reports on g.d.o about critical issues caused by Virtuozzo/OpenVZ, it is a well known problem and not associated with just BOA but Drupal in general. I'm going to close this issue, as we don't intend to work on debugging this and supporting Virtuozzo/OpenVZ officially.

See also: http://2bits.com/articles/hosting-virtualization-openvz-vs-xen-which-is-...

omega8cc’s picture

Priority: Minor » Normal

The 'Avoid Virtuozzo/OpenVZ' warning added to the README.txt to avoid confusion in the future: http://drupalcode.org/project/barracuda.git/commit/5aaa008

omega8cc’s picture

KrisBulman’s picture

I've been running barracuda/octopus on debian VPS's using hypervm/openvz (CentOS backend) for a few months now without issue & I've been running drupal multi-sites on it for 2 years. Sucks to hear of the lack of support and memory leak issues cropping up..

omega8cc’s picture

@KrisBulman

The problem is not really with Virtuozzo/OpenVZ itself. It is because of hosts using by default crappy config, absolutely not optimized for Drupal needs, which causes overall bad image for the entire Virtuozzo/OpenVZ family. I don't think we can support or debug it why the reason of the problem is just bad system level config and/or crazy overselling.

omega8cc’s picture

And from our own experience - we have used a host with Virtuozzo 3.x and it worked fine (they configured it well), but as soon as they upgraded to Virtuozzo 4.x it just crashed completely and we had to move all sites in the emergency mode elsewhere. We never looked back.

KrisBulman’s picture

Thanks, I don't have any experience with Virtuozzo, and run my HyperVM setup on a dedicated server with lots of ram and only 6 slices (with lots of sites/very little performance hit). I will take heed on the Virtuozzo warning and appreciate the explanation.

charos’s picture

I understand the fact that database driven applications will perform much better in Xen. But in my case performance isn't the issue. It's a memory hog problem and according to your assumption is related to kernel configuration. I'm trying to find similar situations but I can't relate to any of these.

Regarding overselling: That's a big myth with Xen vs. OpenVZ. It's just easier to oversell with OpenVZ than with Xen. For Xen you need to apply a technique called ballooning. A special Linux kernel driver is installed on your system (balloon driver). When the Xen server/hypervisor needs more memory, and wishes to claim some from the guest VPS (domU), it asks the guest VPS’s balloon driver to inflate itself by asking its Linux kernel for some memory. Kernel memory allocation will be requested from the available memory for that VPS, and cannot be paged out to swap. Once the balloon driver consumes the memory, it then passes to dom0/hypervisor to be used elsewhere. So the amount of your VPS’s “total memory” will stay the same, but there will be a big increase in “used memory”, as a big chunk has now been used by the balloon driver inside the kernel, and possibly now a part of another VPS. The XEN balloon driver option has been available since version 3.3.
So Xen regarding memory , is not as "isolated" as people tend to think of. Overselling is something that can happen to both platforms.

omega8cc’s picture

@charos

It doesn't really matter here what are the differences between Xen and Virtuozzo/OpenVZ. What matters is the *fact* that most hosts using Virtuozzo/OpenVZ have wrong config which just doesn't work well enough for Drupal in general and we can't do anything in BOA to "fix" this, so we decided to not introduce official support for Virtuozzo/OpenVZ. It is there, but experimental for those who can use it because they have Virtuozzo/OpenVZ properly configured/tuned for Drupal.

What you are experiencing is your host fault, sorry.

hejazee’s picture

I have an OpenVZ VPS for a couple of months.
My VPS has 2GB of RAM with no swap space (I cannot enable swap)
I have always critical issues with Drupal.
I have tested a lot of configurations:
Apache, mod_php
Apache, fcgid
Aegir
I-MSCP:
Kloxo
...

A single page load, eats 200M .. 800M Byte of memory!!!! and after a few hours, the vps crashes.
All the memory is used by apache and when I restart Apache, it releases the memory.

But now, I'm using BOA. It's a little better. It uses 1700M .. 1800M all the time and I have always 200M of free memory

Do you believe that all the above is caused by "Bad config" of OpenVZ ?
And If I used VMWare or Xen it would be better?

charos’s picture

I believe you should start by taking Kloxo out of the equation unless you really need to use it(and always with lowmem.flag) Generally, Kloxo is a memory hog and performance is mediocre.

charos’s picture

Issue summary: View changes

more decriptive