Problem/Motivation
When running Drupal on MySQL or Maria DB you can run in several errors that corrupt the database connection. Some examples for these kind of errors on the database layer are
1100 - Table 'xyz' was not locked with LOCK TABLES1153 – Got a packet bigger than ‘max_allowed_packet’ bytes- ...
If such an error happens, the MySQL server invalidates the connection and answers all further request through this connection by 2006 - MySQL server had gone away.
Just have a look at https://www.google.com/search?q=site%3Adrupal.org+2006+MySQL+server+had+... to see that the drupal project is affected by that.
Again, 2006 - MySQL server had gone away is not the error that causes an issue, it's the symptom you face after a different error happened. The causal error might be software bug in core or a contributed module or an inappropriate setup of the environment (for example the MySQL setup).
Normally you'll find details about the causal error in the logs. But if you use dblog, which is the most common in shared hosting environments, you won't find anything in the log because the log entry is not written due to the already invalidated connection.
If you also turned off error reporting on the site, all you get is a WSOD.
Proposed resolution
The solution to this problem is easy. The 2006 - MySQL server had gone away error causes an Exception while wrinting to the log. If such an exception occurs we can simply open a second connection to the database and write the log entry again.
If you read through the comments, you'll notice that @Damien Tournoud already agreed on the proposed solution throughout the years for the different drupal versions:
Good idea, let's see how we can implement that in the new database layer.
Still applies to 8.x, which uses the default connection for watchdog messages.
Remaining tasks
Review and commit the existing patch.
User interface changes
None.
API changes
None.
The original issue description as posted for Drupal 6
If you read through different drupal bug reports you often see a quote like
Table 'watchdog' was not locked with LOCK TABLES
This could happen if you're using MySQL and a module requests a table lock and run into an error afterwards. In this case drupal is unable to store the error because it could not access the watchdog table.
I attached a patch that opens a dedicated database connection to write watchdog entries like I already proposed in these bug reports:
#298678: Impossible to lock two MySQL tables
#195812: db_error() doesn't work on default connection
Here are some issues related to this bug report:
#286176: Views causing error "Table 'watchdog' was not locked with LOCK TABLES'
#286380: Got past mem problem only to hit a timeout problem
#174705: Modify node type configuration storage to use a serialized collection stored in one variable rather than long variable names
#99718: user warning: Duplicate entry - work around available
Beta phase evaluation
| Issue category | Bug because we don't display the right error and we display errors on the site while nothing is in the log |
|---|---|
| Issue priority | Major because ... Critical/Not critical because ... |
| Prioritized changes | The main goal of this issue is a bug fix in the database error reporting. |
| Comment | File | Size | Author |
|---|---|---|---|
| #63 | 298768-63.patch | 2.72 KB | aerozeppelin |
| #44 | 298768_44.patch | 6.18 KB | stefan.r |
| #27 | 298768_failing_test.patch | 1.79 KB | mkalkbrenner |
Comments
Comment #1
mkalkbrennerDrupal 7 seems to have the same Problem.
Comment #2
mkalkbrennerComment #3
damien tournoud commentedGood idea, let's see how we can implement that in the new database layer.
Comment #4
mkalkbrennerBut don't forget the "backport" to drupal 6 and drupal 5 using my patch please.
Comment #5
Anonymous (not verified) commentedThe last submitted patch failed testing.
Comment #6
dpearcefl commentedThis issue will not go anywhere without confirmation that this issue still exists in current Drupal 7.x and without a patch against a current version.
Comment #7
mkalkbrennerIt was Damien who moved the issue forward to D7 three years ago. My patch was for D5 / D6.
I don't have time to create a patch for D7 at the moment or to confirm, that the issue still exists in D7.
But it definitely exists in D6 which is still supported!
Comment #8
dpearcefl commentedWhat version of D6 still has this issue?
BTW: Your last patch failed. Without a valid patch this issue won't go anywhere.
Comment #9
mkalkbrennerMy patch from 3 years ago could still be applied against 6.22 and 6.x-dev:
Comment #10
dpearcefl commentedPlease attached your patch and mark this ticket as "needs review". The automated patch review system will tell you if the patch is acceptable.
Comment #11
mkalkbrennerComment #12
dpearcefl commentedIf this is against Drupal 6.22 (which it should be), the version needs to be changed on the issue.
Comment #13
mkalkbrennerIt's against 6.22 and of course 6.x-dev
Comment #14
dpearcefl commentedYour patch has been "ignored". Try changing the version to 6.x-dev and resubmitting the patch.
Comment #15
mkalkbrennerComment #16
damien tournoud commentedStill applies to 8.x, which uses the default connection for watchdog messages.
Comment #17
dpearcefl commentedLet's not confuse the system. If this patch is needed and applied cleanly to 8.x-dev, lets open a new issue in that queue.
Comment #18
hefox commentedPatches go into the highest version of drupal they are applicable to first, tmk, which would be 8.x-dev. This is the appropriate issue for it.
Comment #19
kscheirer#15: dedicated_watchdog_connection-298768.patch queued for re-testing.
Comment #20
kscheirerRetesting against latest HEAD since it has been over a year.
Comment #22
mkalkbrennerHere's a initial patch for drupal 8.
Comment #23
mkalkbrennerAnd here's an alternative patch that does "lazy opening" of the dedicated connection.
Comment #26
mkalkbrennerComment #27
mkalkbrennerOK, before I upload an improved patch, here's a test only patch that should fail.
Comment #28
mkalkbrennerAnd here's the real patch including the test.
Comment #29
mkalkbrennerfixed white spaces in #28
Comment #31
mkalkbrennerComment #34
dawehnerI think it would be great to describe the error in the issue summary. ...
Basically something like this?
Comment #36
mkalkbrennerComment #37
mkalkbrennerComment #38
mkalkbrenneradded verbose comments as requested by @dawehner
Comment #39
mkalkbrennerComment #40
fabianx commentedRTBC, this looks great how simple after all it is to setup a new DB connection! :)
Issue summary is good, needs a beta evaluation.
--
I wonder if simply doing what you do in the test would not be easier? (closeConnection, openConnection)
Though that would restore DB access for all of Drupal, which could lead to severe errors in itself as it might have been missing out on data, then trying to write it to cache.
So this is fine as is - also while dependency injection is nicer, in this case it is fine to go via the lowest subsystem level possible as in the mean time e.g. the container might be corrupted already, or other things have happened.
I love how dependency-injection overall makes this issue (quite) easy and clean.
Thanks!
Comment #41
stefan.r commentedMariaDB, run into
logging
logging
errors
requests
Comment #42
mkalkbrennerJust fixed the comments. Thanks @stefan.r.
Comment #43
stefan.r commentedWe forgot a second "request" :)
Gave the code a quick look if you think this looks good now, I think this can go back to RTBC.
Comment #44
stefan.r commentedFixing a 81 character line
Comment #45
mkalkbrennerComment #46
mkalkbrennerBack to RTBC because we just fixed typos within comments.
Comment #47
hchonovI just wanted to test the patch with some real use case and excahged
Database::closeConnection()with
My point was to actually break the connection. Now the test fails locally with
Am I missing something or doing the test wrong?
Comment #48
mkalkbrenner@hchonov:
if you read through the already referenced follow-up issue #972528: dblog fails to log MAX_ALLOWED_PACKET errors because they're longer than MAX_ALLOWED_PACKET it's obvious what you're missing in your "test".
The large packet breaks the default connection. That immediately causes a log entry that opens a dedicated connection. But this one breaks as well because it logs the large packet.
So, my test is OK and the issue is still RTBC here.
But the follow-up #972528: dblog fails to log MAX_ALLOWED_PACKET errors because they're longer than MAX_ALLOWED_PACKET which targets a MySQL specific problem will turn your "test" green, too.
Comment #49
hchonov@mkalkbrenner:
only uncaught exceptions get logged. That is why I do the test in a try catch block, so that the exception does not reach the ExceptionLoggingSubscriber.
Comment #50
mkalkbrenner@hchonov:
You're right that my explanation was not 100% correct.
Nevertheless the answer could be found in #972528: dblog fails to log MAX_ALLOWED_PACKET errors because they're longer than MAX_ALLOWED_PACKET in LargeQueryTest:
You leave the test with a corrupted default connection. So my assumption is that the logging using a dedicated connection works but
$wid = db_query("SELECT MAX(wid) FROM {watchdog} WHERE message = 'testConnectionFailureLogging'")->fetchField();leads to error 2006 until you addDatabase::closeConnection();.Comment #51
hchonov@mkalkbrenner
you are right. In this case the patch and the test look good to me.
@mkalkbrenner++ very good work!
Comment #53
stefan.r commentedRequeueing as tests failed with
copy(/var/lib/drupaltestbot/sites/default/files/checkout/sites/simpletest/996777/settings.php): failed to open stream: No such file or directorycopy('/var/lib/drupaltestbot/sites/default/files/checkout/sites/default/default.settings.php', '/var/lib/drupaltestbot/sites/default/files/checkout/sites/simpletest/996777/settings.php')Comment #55
mkalkbrennerthanks @stefan.r
Comment #56
alexpottThis change is really going to help people what has gone wrong. Nice work on a nearly 7 year old bug.
I agree that there is no disruption with this change and it is tested. Committed 2c0f5e1 and pushed to 8.0.x. Thanks!
Minor fix on commit.
Comment #58
mkalkbrennerThanks :-)
Would be nice if someone can do a review on the related issue #972528: dblog fails to log MAX_ALLOWED_PACKET errors because they're longer than MAX_ALLOWED_PACKET now.
Comment #59
David_Rothstein commentedLooks like this issue affects Drupal 7 too? (In fact, was originally filed for Drupal 6 :)
Comment #63
aerozeppelin commentedRerolled patch to D7.
Comment #65
aerozeppelin commentedComment #68
dagmarThis is the code that was committed to 8.x http://cgit.drupalcode.org/drupal/commit/?id=2c0f5e1
@aerozeppelin it seems you rerolled a different patch.
Comment #69
kscheireraerozepellin's patch in #63 is for D7, @dagmar are you saying it doesn't accomplish the same thing?