DatabaseConnection_sqlite->dbNextId() always returns NULL, breaking batch api, installation

Yikes. This is another one of our new features and it would be wonderful for this to work in the first alpha.

Confirmed for php-5.3.1-3 on apache 2.2.14-2.
There is also another error:
[Wed Jan 13 19:58:00 2010] [error] [client 127.0.0.1] PHP Fatal error: Call to undefined function drupal_exit() in /srv/http/includes/install.inc on line 884

I just reproduced this error on a fresh checkout. PHP 5.3.0, apache 2.2.13. However, no drupal_exit() error, so I doubt there is a connection with http://drupal.org/node/637146

PHP Fatal error:  Class 'MergeQuery' not found in C:\www\head\includes\database\database.inc on line 740
Stack trace:
 1. {main}() C:\www\head\index.php:0
 2. drupal_bootstrap() C:\www\head\index.php:21
 3. _drupal_bootstrap_page_cache() C:\www\head\includes\bootstrap.inc:1519
 4. drupal_bootstrap() C:\www\head\includes\bootstrap.inc:1586
 5. _drupal_bootstrap_variables() C:\www\head\includes\bootstrap.inc:1527
 6. variable_initialize() C:\www\head\includes\bootstrap.inc:1648
 7. cache_set() C:\www\head\includes\bootstrap.inc:724
 8. DrupalDatabaseCache->set() C:\www\head\includes\cache.inc:140
 9. db_merge() C:\www\head\includes\cache.inc:418
10. DatabaseConnection->merge() C:\www\head\includes\database\database.inc:2203

BTW The registry table is completely empty so autoloading won't work.

Below is an entire install, until the point of failiure.

[14-Jan-2010 18:08:36] bootstrap configuration
[14-Jan-2010 18:08:36] _cache_get_object  is cachebin
[14-Jan-2010 18:08:36] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:08:40] bootstrap configuration
[14-Jan-2010 18:08:40] _cache_get_object  is cachebin
[14-Jan-2010 18:08:40] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:08:40] bootstrap configuration
[14-Jan-2010 18:08:40] _cache_get_object  is cachebin
[14-Jan-2010 18:08:40] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:08:40] bootstrap configuration
[14-Jan-2010 18:08:40] _cache_get_object  is cachebin
[14-Jan-2010 18:08:40] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:08:52] bootstrap configuration
[14-Jan-2010 18:08:52] _cache_get_object  is cachebin
[14-Jan-2010 18:08:52] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:08:53] _cache_get_object  is cachebin
[14-Jan-2010 18:08:53] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:08:53] Driver class DatabaseConnection_sqlite  selected
[14-Jan-2010 18:09:01] bootstrap pagecache
[14-Jan-2010 18:09:01] bootstrap database
[14-Jan-2010 18:09:01] registered autoload
[14-Jan-2010 18:09:01] bootstrap variables
[14-Jan-2010 18:09:01] variable_initialize setting cache
[14-Jan-2010 18:09:01] bootstrap session
[14-Jan-2010 18:09:01] bootstrap page header
[14-Jan-2010 18:09:01] bootstrap language
[14-Jan-2010 18:09:01] bootstrap full
[14-Jan-2010 18:09:03] bootstrap configuration
[14-Jan-2010 18:09:03] _cache_get_object  is cachebin
[14-Jan-2010 18:09:03] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:09:03] Driver class DatabaseConnection_sqlite  selected
[14-Jan-2010 18:09:03] bootstrap pagecache
[14-Jan-2010 18:09:03] bootstrap database
[14-Jan-2010 18:09:03] registered autoload
[14-Jan-2010 18:09:03] bootstrap variables
[14-Jan-2010 18:09:03] variable_initialize setting cache
[14-Jan-2010 18:09:03] bootstrap session
[14-Jan-2010 18:09:03] bootstrap page header
[14-Jan-2010 18:09:03] bootstrap language
[14-Jan-2010 18:09:03] bootstrap full
[14-Jan-2010 18:09:04] bootstrap configuration
[14-Jan-2010 18:09:04] bootstrap pagecache
[14-Jan-2010 18:09:04] bootstrap database
[14-Jan-2010 18:09:04] registered autoload
[14-Jan-2010 18:09:04] bootstrap variables
[14-Jan-2010 18:09:04] _cache_get_object  is cachebin
[14-Jan-2010 18:09:04] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 18:09:04] Driver class DatabaseConnection_sqlite  selected
[14-Jan-2010 18:09:04] variable_initialize setting cache
[14-Jan-2010 18:09:04] PHP Fatal error:  Class 'MergeQuery' not found in C:\www\head\includes\database\database.inc on line 740
[14-Jan-2010 18:09:04] PHP Stack trace:
[14-Jan-2010 18:09:04] PHP   1. {main}() C:\www\head\index.php:0
[14-Jan-2010 18:09:04] PHP   2. drupal_bootstrap() C:\www\head\index.php:21
[14-Jan-2010 18:09:04] PHP   3. _drupal_bootstrap_page_cache() C:\www\head\includes\bootstrap.inc:1523
[14-Jan-2010 18:09:04] PHP   4. drupal_bootstrap() C:\www\head\includes\bootstrap.inc:1596
[14-Jan-2010 18:09:04] PHP   5. _drupal_bootstrap_variables() C:\www\head\includes\bootstrap.inc:1533
[14-Jan-2010 18:09:04] PHP   6. variable_initialize() C:\www\head\includes\bootstrap.inc:1660
[14-Jan-2010 18:09:04] PHP   7. cache_set() C:\www\head\includes\bootstrap.inc:726
[14-Jan-2010 18:09:04] PHP   8. DrupalDatabaseCache->set() C:\www\head\includes\cache.inc:142
[14-Jan-2010 18:09:04] PHP   9. db_merge() C:\www\head\includes\cache.inc:420
[14-Jan-2010 18:09:04] PHP  10. DatabaseConnection->merge() C:\www\head\includes\database\database.inc:2204

DrupalDatabaseCache->set does a merge.

This is all before the registry update ever runs.

I wonder how many loops through the batch is happening.

The batch system cannot find a batch on SQLite:

[14-Jan-2010 19:03:20] full
[14-Jan-2010 19:03:20] _batch_page
[14-Jan-2010 19:03:20] _batch_page request id found
[14-Jan-2010 19:03:20] _batch_page NO active batch found

// HERE A NEW REQUEST STARTS

[14-Jan-2010 19:03:20] configuration
[14-Jan-2010 19:03:20] pagecache
[14-Jan-2010 19:03:20] database
[14-Jan-2010 19:03:20] registered autoload
[14-Jan-2010 19:03:20] variables
[14-Jan-2010 19:03:20] _cache_get_object  is cachebin
[14-Jan-2010 19:03:20] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 19:03:20] Driver class DatabaseConnection_sqlite  selected
[14-Jan-2010 19:03:20] variable_initialize setting cache
[14-Jan-2010 19:03:20] PHP Fatal error:  Class 'MergeQuery' not found in C:\www\head\includes\database\database.inc on line 740

Compare with MySQL where a batch IS found:

[14-Jan-2010 19:05:24] full
[14-Jan-2010 19:05:24] _batch_page
[14-Jan-2010 19:05:24] _batch_page request id found
[14-Jan-2010 19:05:24] _batch_page active batch found
[14-Jan-2010 19:05:24] _batch_page op = start 
[14-Jan-2010 19:05:24] _batch_page start
[14-Jan-2010 19:05:24] configuration
[14-Jan-2010 19:05:24] _cache_get_object  is cachebin
[14-Jan-2010 19:05:24] _cache_get_object doesn't exist, default DrupalDatabaseCache
[14-Jan-2010 19:05:24] Driver class DatabaseConnection_mysql  selected

In case you wonder, that's because $_REQUEST['id'] is an empty string.

Re #6, the relevant code might be here (from the batch API update patch):

+    // Assign an arbitrary id: don't rely on a serial column in the 'batch'
+    // table, since non-progressive batches skip database storage completely.
+    $batch['id'] = db_next_id();

Possible that SQLite has a problem with using db_next_id() here?

As $batch['id'] = db_next_id();

It seems that the db_next_id implementation for SQLite doesn't work properly during install (maybe never).

retitling.

Ouch. This is an example of our lack of unit testing for other DB engines :(

cross posted.

Indeed the function is broken:

  public function nextId($existing_id = 0) {
    $transaction = $this->startTransaction();
    // We can safely use literal queries here instead of the slower query
    // builder because if a given database breaks here then it can simply
    // override nextId. However, this is unlikely as we deal with short strings
    // and integers and no known databases require special handling for those
    // simple cases. If another transaction wants to write the same row, it will
    // wait until this transaction commits.
    $stmt = $this->query('UPDATE {sequences} SET value = GREATEST(value, :existing_id) + 1', array(
      ':existing_id' => $existing_id,
    ));
    if (!$stmt->rowCount()) {
      $this->query('INSERT INTO {sequences} (value) VALUES (:existing_id + 1)', array(
        ':existing_id' => $existing_id,
      ));
    }
    // The transaction gets committed when the transaction object gets destroyed
    // because it gets out of scope.
    return $new_value;
  }

Notice $new_value here is undefined!

Installation completed succesfully with this patch :-)

Still needs testbot love.

very true Heine

rerolled

hold it for a second.

OK, SQLite transactions lock the table on the first write at latest.

Note to selves:

check all the code for undefined variables. easy way to prevent critical bugs like this.

The cause lies in #633678: Make sequence API work on non-MySQL databases btw, edited to add: or even earlier issues as that commit merely moved the parent implementation.

Wow, folks! Well done!! That has to be among the highest "fewest lines of code changed : gnarliest bug to debug" ratio ever!

Since this *only* changed sqlite files and those files couldn't possibly be more jacked up right now than they currently are, I'm going to break my "no commits 24 hours before release" rule and commit this to HEAD.

Hi,

I think this happens on MySQL also... I can report exactly the same errors. Please see my post here:

http://drupal.org/node/686196

I will check with my hosts but I'm sure I'm using Mysql and not sqlite.

EDIT: Infact, same problem and the patch is already committed. I don't think I should mark my post as a duplicate, as I am on Mysql (verified) not sqlite. Can anyone help there?! Thanks.

OK thanks mfer. I'll continue to de-bug as much as I can and leave feedback there. Seems I'm not the only one having this problem.

Same problem here and I'm running a VM with Ubuntu Linux, Apache, MySQL and PHP 5

If you try to use Drupal on MySQL, you don't have the _same_ problem. This is SQLite specific. The error you get is a symptom of another problem. Please file a new issue.

Fortina, can you please check http://drupal.org/node/686196 to see if it is the issue you are having - I'm trying to find others with this problem so we can define that its not just my host or a mistake I'm making!

To clarify - this issue is regarding SQLITE, not MYSQL. Thanks,

Adam

I am facing the same problem on PostgreSQL. I hope that the solution for SQLite would work for PostgreSQL as well.

I have tested this with Alpha release and I can confirm that it is working fine.