OK, so for the paste several days I have been trying to figure out what is going on here and have tried to resolve things with no success. I'm at my wits end and really needs some help!! It's a very weird problem (to me at least), so I will do my best to provide as much documentation as to what is going on with my BOA as possible.

My Setup and Background
I'm working on an Ubuntu 10.04 LTS 64bit Linode VPS with 80GB total storage (67 GB used, 13 GB Available, most space used is from site backups), 2 GB RAM. As per the README that comes with the BOA scripts, using http://gist.gifthub.com, here is a link to the
BOA files for before upgrading to latest version of BOA head (3/31/2012)
and a link to
BOA files after upgrading to latest version of BOA head (4/27/2012):
/var/aegir/config/includes/barracuda_log.txt
/data/disk/user/log/octopus_log.txt
/var/aegir/install.log (remove the password)
/root/.barracuda.cnf
/root/.USER.octopus.cnf

Also here are links to:
Barracuda and Octopus Script Output
Barracuda Output when MariaDB was down and log file it referred to

1. The Problems
There are several problems I have with my BOA at the moment and I suspect they are all related.

1a) The principle problem is that when trying to add a node to a nodequeue on any website on any platform that uses the latest version of Nodequeue (7.x-2.0-beta1) the following error occurs on production sites:

The website encountered an error while retrieving http://www.WEBSITE.com/nodequeue/breaking_news/add-node/4dee6d97b0341/225?tab=1&destination=NODE-NAME%23overlay%3Dnode%2F225%2Fnodequeue&token=kRn3IfLNpqeoFgfZ9jIw1Ppvn4vJ_IBqtwOqthwee6E. 
It may be down for maintenance or configured incorrectly.
Here are some suggestions:
Reload this webpage later.
HTTP Error 500 (Internal Server Error): An unexpected condition was encountered while the server was attempting to fulfill the request.

and on development sites we get this error:

Additional uncaught exception thrown while handling exception.

Original

PDOException: SQLSTATE[HY000]: General error: 2013 Lost connection to MySQL server during query: INSERT INTO {nodequeue_nodes} (sqid, qid, nid, position, timestamp) VALUES (:sqid, :qid, :nid, IFNULL((SELECT MAX(position)+1 FROM (SELECT * from {nodequeue_nodes} WHERE sqid = :sqid) as nn), 1), :time); Array ( [:sqid] => 4dee6d97b0341 [:qid] => breaking_news [:nid] => 225 [:time] => 1335129673 ) in nodequeue_subqueue_add() (line 1044 of /data/disk/USER/static/PLATFORM/sites/all/modules/nodequeue/nodequeue.module).

Additional

PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away: SELECT * FROM {system} WHERE type = 'theme' OR (type = 'module' AND status = 1) ORDER BY weight ASC, name ASC; Array ( ) in system_list() (line 165 of /data/disk/USER/static/PLATFORM/includes/module.inc).

Uncaught exception thrown in session handler.

PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in _drupal_session_write() (line 206 of /data/disk/USER/static/PLATFORM/includes/session.inc).

What's really weird about this is that latest version of Nodequeue (7.x-2.0-beta1) came out in September 2011 and I have been using this module on several websites, some just built or rebuilt in Feb and March of 2012, without any issue at all.

So I don't think the problem is with the module, but rather with the something with my BOA setup which was last upgraded to BOA HEAD on March 31st. I don't think any of the websites using this version of the module attempted to add anything to a nodequeue since March 31st.

We have had no other problems adding nodes or doing other stuff on any of these websites that have this error and websites using the previous version of the module are are uneffected.

I tried cloning one of these sites and completely uninstalling and reinstalling the nodequeue 7.x-2.0-beta1 module, but this did not have any effect.

1b) Neither Chive or SQLBuddy are no longer installed.
Or at least they cannot be found when I go to the domains where they were previously installed.

Since this error was related to the nodequeue databases I thought I would investigate what those databases looked like. But then I discovered that neither Chive or SQLBuddy were installed on my BOA system anymore. (I don't use these except when I have problem to fix that requires them, so I don't know how long they have been missing.)

Both of these use to work just fine, but neither exist now. I doubled checked my DNS, which hasn't changed for these services since I originally set up BOA about a year ago.

So I decided to upgrade to the latest HEAD for BOA and ...

1c) Upon Upgrading to the Latest BOA HEAD, things got worse…

I thought upgrading the latest BOA HEAD might bring back my Chive and SQLBuddy, and maybe even fix this other error, but it did not. Things just got worse.

I discovered that when I ran the latest version of BOA HEAD that even though I have touched the following files to prevent my custom settings from being overridden by Barracuda, the corresponding configuration files are always reset.
/etc/mysql/custom.my.cnf
/opt/etc/custom.php.ini
/usr/local/lib/custom.php.ini
/opt/local/etc/custom.php53.ini

but it gets mush worse than that…
After upgrading to the latest version of BOA HEAD, if I try to add a node to a nodequeue on a site using Nodequeue 7.x-2.0-beta1, not only do I get the errors that I reported above, but the entire MariaDB server shuts down!

So not only is the site that had the error ceasing to function, but all sites including the BOA hostmaster sites are going down because the database is gone completely.

I used ps aux | grep mysql both before and after triggering the error and I can confirm that the Database server is indeed gone after the error.

Plus, if after upgrading to the latest BOA HEAD I cannot reboot by Linode and have the MariaDB server start on it's own. This is regardless of whether I have made any changes to the following config files or not, or whether the database server was running or not before reboot.
/etc/mysql/my.cnf
/opt/etc/php.ini
/usr/local/lib/php.ini
/opt/local/etc/php53.ini

I found a suggestion on this webpage: http://madhavvyas.blogspot.com/2011/04/sqlstatehy000-general-error-2006-...
to change max_allowed_packet to 100M in the MySQL (MariaSQL) configuration when this type of error occurs, but it made no difference for me both before and after updating to the latest version of BOA HEAD.

Also, I tried to run the Barracuda script again when the MariaDB was down and discovered that it will not complete and gives you an error becuase MariaDB is down. The error refers to a log file for more information, but I found nothing in that file related to the database being down.

2. So...My Questions
2a) Of course the first question is, does anyone have and idea how I stop this error from occurring and stop the MariaSQL server from going down and taking the whole system with it after I upgrade to the latest version of BOA?

2b) How do I get Chive and SQLBuddy reinstalled on the system?

2b) What is the best way to review the Syslogs and which ones should I be looking at find information about this problem? I ask because I can't find the errors related to the crash of everything when adding the node to the nodequeue. I'm a noob when it comes to this Sys Admin stuff, if you couldn't tell already. ;-)

2c) How does one restart services like MariaDB on the BOA system? Again I ask because typical MySQL commands don't seem to work.

THANKS for ALL Your Help in Advance!!!! I need it badly!!!!
Steve

P.S, If you are wondering why I'm on BOA HEAD and not the stable version of BOA, it's because I had to switch to HEAD when the last release of stable BOA had not been tested on Ubuntu. So I've been waiting for the next stable BOA to be released so that I can get off this track.
P.P.S. Again, my apologies to anyone who got my first version of this post when I accidentally pasted in the wrong version and hit save button instead of the preview button. It's been a hell of a week here.

Comments

wickwood’s picture

Issue summary: View changes

I didn't mean to post the orginal message yet, I hit save instead of preview button.

Anonymous’s picture

See this post:
http://drupal.org/node/1315594#comment-5653282

This guy had a similar problem and ended-up breaking that line of code in half (line 1042 in ver 6.x, line 1044 in ver 7.x) in nodequeue.module as a work around. I haven't tested it but could worth a long shot.

Peace,
Michael Clendening

omega8cc’s picture

Please open separate issues for every problem, as it is too hard to follow such giant report/request. Thanks.

wickwood’s picture

I'm sorry it's so long, but I don't know if I can summarize it any smaller and/or break it up and not leave anything out that would be pertinent.

That's why I tried to summarize the questions at the bottom of the post.

But I will I try to do so tomorrow because I really need some help with this. I would do now, but I was just about to go to bed and don't have the mental capacity to even try to break this down any further. In fact it took me several days to write this up.

Steve

omega8cc’s picture

Status: Active » Postponed (maintainer needs more info)

We require separate issue per problem to be able to provide good support.

Also, please read all config options. We no longer use control files. There are new options in the config file to avoid overwriting custom configuration. But this really require separate issue per problem.

wickwood’s picture

I totally understand, and after working this morning on sepearating the issues I think I have done a better job of explaining each issue I'm dealing with. Also, I re-read the change log for instructions on config files options, but I don't see what I'm missing or doing wrong, so I posted that again as separate issue, too.

So this post has been broken down now into 4 seperate issues:

  1. Adding a node to nodequeue causes error with older version of BOA and shuts down databases for all sites with new version of BOA.
  2. Chive or SQLBuddy are no longer installed. How to reinstall?
  3. How does one restart services like MariaDB on the BOA system?
  4. php.ini and my.cnf Config Files Get Overridden upon Upgrade

Thanks again for ANY and ALL Help! I truly appreciate it!
Steve

P.S. Also, My apologies if I came across as whining with my comments last night, too. After rereading my comments back, I think they kind of sound that way.

omega8cc’s picture

Status: Postponed (maintainer needs more info) » Closed (duplicate)

Thank you. Closing this one only as a duplicate.

omega8cc’s picture

Issue summary: View changes

I finished editing my post, added a PS, added links to gist files.