SQL check ERROR and too many SQL check CLEAN notices

You can comment out one line in the /var/xdrago/minute.sh script:

perl /var/xdrago/firewall/check/sqlcheck

This will stop automated SQL repair, but then you may want to check your logs for crashed tables manually. It shouldn't happen, unless your VPS crashes, is powered off instead of gracefully shut down etc.

You could also limit the number of syslog lines checked by changing default 200 to maybe 10 in the /var/xdrago/firewall/check/sqlcheck script:

local(@MYARR) = `tail --lines=200 /var/log/syslog 2>&1`;

Note that all those changes will be overwritten on every Barracuda upgrade.

I'm gettings the same issue. ~10 emails in an hour or so. Not daily though, so maybe its just working as expected. Saving my butt from imminent disaster. Usually the first has a subject of "SQL check ERROR" then all the rest are "SQL check CLEAN"

This happened last week one time then again this last night. I have had 2.0.2 installed for a few weeks now at least.

I can confirm that all the tables in question are MyISAM.

Last night was a little different as mysql stopped accepting connections throwing the error "Too many connections" I had to kill it and restart it to get it going again. Which is most likely what caused the table errors. I tried to restart it normally but I couldn't find a way to communicate to it. Sending SIGTERM did seem to work though, but maybe I sent it to the wrong process as mysql using that odd child process thing.

It seems to be a different table each time that is crashing.

From here I'm going to convert the rest of my imported databases to InnoDB. I'd like to figure out what is causing the connection issue as well though.

Any suggestions or questions to help the maintainers?

P.S. Thanks for the awesome system omega8cc and any others. Your hard work is much appreciated.

I run risk of revealing my ignorance on this subject, but I'll ask the question anyway:

Could the frequent SQL errors be the result of not having enough RAM to provide sufficient memory to keep the tables there?

It seems to me that I recall having a problem with db tables crashing on an old server and it was traced back to not enough memory.

(I'm moving this issue to Active. Though it may not be specifically a BOA problem, it can be reproduced. Also, this is a support request, not a bug report.)

Tuning your database configuration and debugging your specific issues is beyond the scope of this project. It provides some good defaults, but if you are experiencing crashes, then either something is wrong with your hardware or parent system. Low memory is not enough to cause db crash. Low memory will cause heavy swap use and will slow down things but will not crash the server. For example, this is "normal" under Virtuozzo & family, because they kill processes and don't care about data integrity.

Not sure what you mean by "it can be reproduced"?

Also, we can't really help if you don't attach your install/config files, as listed here. Please attach them for review, if possible.

Thank you for explaining how memory should not be the cause here. This confirms what I feared, that I don't know much about this subject.

The only reason for saying it can be reproduced was that the behavior of multiple 'SQL check ERROR' and 'SQL check CLEAN' messages was happening to multiple people. I realize that's not the same as reproducing the conditions that cause this problem and does not imply that the same cause is in play for everyone.

You've given me some direction for where to go with this and that's sufficient. Thanks again.

@JimSmith

The only method to really "reproduce" this problem is to use Virtuozzo/OpenVZ based VPS, or use some VPS provider known to freeze VMs in the read-only state often and causing all associated damage to the databases.

I tend to believe that most people reporting frequent crashes are either victims of Virtuozzo/OpenVZ or hosts with unreliable resources management or simply "cloud done wrong" - like at VPS.net, where you can expect that the parent system will either crash your VPS or kill mysql without any clean shutdown. It must lead to broken tables, so BOA auto-healing discovers that and runs repair tasks until there are no errors in the syslog. Without this auto-healing in place, you could never realize what happened to your db server and why some tables are broken (and when that happened).

I hope this helps.

I understand. Fortunately, I learned many months ago the trouble caused when trying to use VPS.net, so that's not the issue here. My sites are now on Linode.

However, this issue #1553158: Adding a node to nodequeue causes error with older version of BOA and shuts down databases for all sites with new version of BOA confirms that it is possible to cause the db server crash because of some issues in the drupal contrib code (probably).

Closing for now. Feel free to re-open when there will be any new related report/issue, but remember that any such issues are with 99,9% probability not related to BOA.

For reference, similar issue (however not a db crash related): #1354276: MYSQL Server has gone away = Max_allowed_packets

I've seen this too and I think it (or at least the symptom) is a barracuda problem. The script that triggers the check searches in the last 200 lines of the log for the error string. So the first time it spots that, it triggers the repairs correctly, however, it will then retrigger every time the script is run afterwards until the error line falls off the last 200 lines of the log, resulting in a large number of 'CLEAN' reports. Shortening the number of lines it looks at will reduce the problem (as was observed above), but could also mean that it misses legitimate errors.

The right fix for this is to filter on the log entry time, not a fixed number of lines, and store the time of the last check (which it's already doing) so it knows where to start next time. The log checker is also looking at an unfiltered view of syslog, which may contain the error string from some other source (e.g. firewall or cron messages), so it should be pre-filtered with grep first to only look at mysql entries. Lots of possible ways of doing that such as this one. Generally searching by date in logs isn't especially easy with normal shell tools, so perl or php methods might be appropriate.

@Synchro - and the patch? :)

This has been fixed thanks to much smarter check - since it runs every minute, it extracts only related errors and then compares timestamps to automatically skip all errors older than 1 minute - http://drupalcode.org/project/barracuda.git/commit/c0fa44c