New Relic Agent 3.0 integration, upgrade path and compatibility

We may want to add this in the hosted Aegir Root option at some point (but it is not considered yet), but not in any Standard or SSD option, so while we are not directly interested in this integration, any patches are welcome! :)

We may add this as an option, I think.

This would be a great service to add. Love Newrelic!

Status back to active.

Committed in http://drupalcode.org/project/barracuda.git/commit/64f3401

Thanks!

Is this the service, http://newrelic.com/ , that you're talking about and just added?

@wickwood

Yes.

Just three lines:

wget -q -U iCab http://files.aegir.cc/versions/BOA.sh.txt
bash BOA.sh.txt
boa in-head public server.mydomain.org my@email o1 max newrelickey

Could this be applied to an existing instance using:
octopus up-stable 01 newrelickey

Didn't see any mention in the documentation re upgrading server to include the monitoring for all apps.

No, we need to add this as an option to barracuda up-* command.

You could also add it by running Barracuda script directly and entering your key as _NEWRELIC_KEY.

Feel free to open a new issue.

But this only applies to fresh installs is that correct?

No. You can add New Relic integration by running BARRACUDA.sh.txt script directly on upgrade, since barracuda up* command doesn't support it yet.

Oh ok, so to confirm the command, you'd run:
bash BARRACUDA.sh.txt _NEWRELIC_abc123def456etcetcetc
or just:
bash BARRACUDA.sh.txt _abc123def456etcetcetc

Tx

No. BARRACUDA.sh.txt doesn't accept arguments. You must edit the script. There is _NEWRELIC_KEY config variable inside.

Checked the BARRACUDA.sh.txt config file and a lot has changed since the last time we updated in a good way!

Just one thing that came up, we already had server monitoring so the install script didn't create the /etc/newrelic/newrelic.cfg config file nor initialize the individual app monitoring so I had to remove both packages first:
newrelic-php5 newrelic-sysmond

I re-ran the install script and everything's working as expected.
Thanks for clarifying this!

Working to deploy with 2.0.4 -- on an upgrader - can you please advise where the BARRACUDA.sh.txt so can set the key.

If there was no New Relic installed before, you should add _NEWRELIC_KEY=yourkeyhere line to your /root/.barracuda.cnf file before running barracuda up-stable. There is no need to use BARRACUDA.sh.txt directly. You may want to read the script comments in its config section, which is in the project's repository main directory, to learn more about various options used and available.

Interesting issue has come about since the 2.0.4 upgrade.

Now when I use drush I'm getting this:
PHP Warning: PHP Startup: Unable to load dynamic library '/opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so' - /opt/local/lib/php/extensions/no-debug-non-zts-20090626/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0

and Newrelic has stopped monitoring obviously.

Server is Debian Squeeze, 3GB Xen latest stable BOA.

Thanks in advance but I'm picking this is something silly like a newer version of the agent is interfering with the old perhaps??

Yeah, but we have introduced New Relic Agent 3.0 compatibility and the upgrade path worked before the release. No idea yet why now it fails - and we were also hit by this on upgrade :/

That's ok. So is there a recommended work around without doing more damage. I try not to get creative on boa installs anymore as its far safer to ask first! :)

After system upgrade, new PHP Agent has been installed, however extensions in /usr/lib/newrelic-php5/agent/x64 directory were not updated, and worse yet, newrelic.so symlinks pointing there have been removed. We have tried to run manuall install again, but discovered that also the /usr/lib/newrelic-php5/newrelic-install.sh script (referenced in the /usr/bin/newrelic-install wrapper) has been deleted, so manual install is now broken completely. It is a New Relic package what breaks the upgrade horribly and we have notified them about the problem already.

The only solution was to uninstall New Relic support, purge it completely and then run system upgrade again:

$ aptitude -f -y remove newrelic-php5 newrelic-daemon newrelic-php5-common newrelic-sysmond
$ aptitude -f -y purge newrelic-php5 newrelic-daemon newrelic-php5-common newrelic-sysmond
$ apt-get autoremove -y
$ mv /etc/newrelic /tmp/
$ barracuda up-stable

Thanks #25 worked perfectly. :)

It's been a few hours since the fix process was applied, but just hit a 404 doing some testing on a client site and got this emailed to me:

### PHP: CRASH at 10:59:27 [1] discovered on 121119-110001 for o1.aegir.sitename.com
### SYS: Nov 19 10:59:27 hostname kernel: [2632527.817428] php-fpm[48859]: segfault at 40000000010 ip 00007ff55ea6b797 sp 00007fffa5d77d20 error 4 in newrelic-20090626.so[7ff55ea28000+23c000]
### NGX: "123.123.123.123, 127.0.0.1" o1.aegir.sitename.com [19/Nov/2012:10:59:27 +1300] "GET /hosting/tasks/2582/list HTTP/1.0" 502 568 1046 713 "https://o1.aegir.sitename.com/hosting/c/clientsite.com" "Mozilla/5.0 (Macintosh\; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11" 0.093 "-"
### PTH: 
### VHT: 

Is this referencing the correct new newrelic.so?
Your advice would be much appreciated.

Yes, new newrelic.so causes random segfaults *both* for PHP 5.2.17 and 5.3.18, so I guess they should release some update soon :/

Some other behaviour noted was several apps don't appear to be sending data therefore not showing up on the new relic dash.
So the per app agent is very limited currently and may not provide accurate information at all.

Has the issue been raised with New Relic?

No, we didn't experience any such issues (besides the random segfaults). The App sends the data only when the site has visits and it always comes with some delay. if you have any issues, I would suggest to contact New Relic, as it is already outside of the scope of this project.

Just a note: there is a new release, as expected, which fixes several problems, including those causing segfaults: https://newrelic.com/docs/releases/php

newrelic-daemon 3.1.5.111
newrelic-php5 3.1.5.111
newrelic-php5-common 3.1.5.111

Fixed an agent segmentation fault that occurred during Slow SQL processing at the end of a transaction.
Fixed a daemon segmentation fault that occurred if the agent communication with the daemon was interrupted for any reason.
Fixed a memory leak in the agent when the maximum number of Slow SQL statements for a transaction was reached.
Fixed an error that was preventing the daemon from re-spawning a copy of itself if the worker daemon died due to a segmentation fault.
Updated the newrelic_set_appname API call to no longer send the transaction thus far to the daemon unless you give it an argument explicitly requesting this behavior. This considerably reduces the overhead of changing application names on the fly.
Updated the newrelic_end_transaction API call to allow you to decide whether or not the transaction should be sent to the daemon. It is by default.
Fixed an error in the installer script where the license key was not being inserted into the INI file correctly.
Functions with custom instrumentation (newrelic.transaction_tracer.custom) will now produce metrics that can be used in custom dashboards.
If your application produces a Content-Length header, automatic RUM ("auto-RUM") will no longer inject the RUM header and footer, which would invalidate the content length calculated by your application.
The NR_INSTALL_KEY environment variable is now correctly obeyed.
The daemon is now compiled to support showing a backtrace in the event of a segmentation violation.

You will need to uninstall, purge it and run barracuda up-stable to avoid issues, since we didn't automate it yet to avoid broken upgrade path.

I followed your instructions but I'm still seeing Segfaults. Any special instructions for Octopus?

### PHP: CRASH at 17:11:43 [1] discovered on 121123-171201 for mysite.com
### SYS: Nov 23 17:11:43 server2 kernel: [11312450.789945] php-cgi[36508]: segfault at 40000000010 ip 00007fc382e9fa77 sp 00007fffe649c0f0 error 4 in newrelic-20060613.so[7fc382e5c000+23e000]
### NGX: "204.93.223.151" mysite.com [23/Nov/2012:17:11:43 +0000] "HEAD / HTTP/1.0" 502 0 219 145 "-" "NewRelicPinger/1.0 (38103)" 0.140 "-"
### PTH: /data/disk/o1/static/mysite.com-2012-03-03-2051/sites/mysite.com
### VHT: /data/disk/o1/config/server_master/nginx/vhost.d/mysite.com

There are no instructions for octopus, you should run only barracuda up-stable after uninstalling and purging old Agent.

The version check should result with:

server:~# newrelic-daemon -v
New Relic local collector daemon version 3.1.5.111 ("beatific")
(C) Copyright 2009-2012 New Relic Inc. All rights reserved.
server:~# 

I just tested this on our Linode test server and it fixed also #1842736: New Relic prevents Drush prompts from showing, appears to hang

But of course it doesn't mean that there are no further bugs in this new Agent version.

Also, this issue is not fixed yet, since we have to introduce automated forced re-install when there is newer Agent available.

Not sure why but after removing newrelic the daemon was still running. Then I noticed that newrelic-sysmond was not being removed.

dpkg: warning: while removing newrelic-sysmond, directory '/etc/newrelic' not empty so not removed

After moving those files to my home directory just to be sure I didn't lose them and re-running the remove and purge commands I successfully removed newrelic. Then I ran the barracuda up-stable command and NewRelic is monitoring my server again. There are now no segfaults but also no individual site monitoring of my octopus sites. In New Relic I now have one App being monitored called "PHP Application". Anyone else having these results?

Deleting /etc/newrelic is safe, because Barracuda stores your New Relic key in its /root/.barracuda.cnf config file, but I just tested it and it works as designed:

Maybe you need to restart again *both* New Relic Agent and php53-fpm/php-fpm services?

I've restarted the agents and php services and still no data. Nothing in the New Relic logs which is odd. After looking and the phpinfo() I see that the newrelic.license is showing ***INVALID***. My license key is valid and correct in the config files.

So maybe you need to contact New Relic then? It works for me:

Just wanted to close the loop on my issue. I was able to resolve this with Newrelic. I'm still not sure how it happened but deleting the existing license key and replacing it with the one copied out of my Newrelic admin corrected this issue. The key was incorrect in both /opt/etc/php.ini and /opt/local/etc/php53.ini. Very odd indeed.

I'm seeing PHP Segfaults again.

Yeah, I have noticed that. It is better than before, but it still happens. But have you tried to re-install it again? There is newer release available.

PHP Agent 3.1.5.120
This release is a bug-fix release only and adds no new features to the agent. The following bugs introduced in 3.0/3.1 have been fixed:

* Fixed two segmentation violations that would occur after calling the newrelic_set_appname() API call. One had the potential to affect all users, the other was restricted to customers who used prepared SQL statements both before and after the call to newrelic_set_appname().
* Fixed a problem with the Solaris init scripts which made the incorrect assumption about the process ID of init.
* Fixed a small memory leak that affected CodeIgniter, CakePHP and Drupal applications.

I just did a reinstall. Currently running 3.1.5.120. We will see how this goes.

Just had another one.

### PHP: CRASH at 04:11:41 [1] discovered on 121219-041202 for mysite.com
### SYS: Dec 19 04:11:41 server2 kernel: [2067779.103331] php-cgi[11418]: segfault at 40000000010 ip 00007f6bbf8324cf sp 00007fff2d1f84b0 error 4 in newrelic-20060613.so[7f6bbf7ec000+23e000]
### NGX: "50.18.57.7" mysite.com [19/Dec/2012:04:11:41 +0000] "HEAD / HTTP/1.0" 502 0 219 145 "-" "NewRelicPinger/1.0 (38103)" 0.183 "-"
### PTH: /data/disk/o1/static/mysite.com-2012-03-03-2051/sites/mysite.com
### VHT: /data/disk/o1/config/server_master/nginx/vhost.d/mysite.com

I can only guess they test just PHP 5.3 then? But if there will be another crash with php-fpm (5.3) and not only php-cgi (5.2), then they have still some work to do.

#43 I'm seeing this too same agent version.

Not sure if it's related, but several sites are not displaying in NR dash, including an entire platform of Ubercart sites. Co-incidence?

I had the same issue. Check your php.ini for the correct New Relic key. Mine has been wrong. Not sure why this is happening but every time I un-install NR and BOA reinstalls it, it gets the key wrong even though it is right in the .barracuda file.

@kepford - do you have also the /etc/newrelic/newrelic.cfg file with correct license key?

We now force New Relic full re-install on every upgrade: http://drupalcode.org/project/barracuda.git/commit/031c36b