Impossible to update D6 -> D7

I have the same problem and am unable to upgrade (trying to test a patch for upgrade path).

The problem is coming from database.inc line 356.

catch (PDOException $e) {
      if (!function_exists('module_implements')) {
        _db_need_install();
      }

I can't get to update.php because this PDOException always redirects me to install.php.

There's also the issue that users who've started with D6 may never have seen their settings.php before, so it will be a shock to upgrade. Opened a new issue for that #304163: Allow update.php to regenerate settings.php.

I tried adding a redirect to update.php whenever the old $db_url connection string was detected. However, it then quickly turned out that the real culprit is within drupal_is_denied(), which is called during DRUPAL_BOOTSTRAP_ACCESS while running update.php. This specific function queries a list of blocked IPs from the database, which isn't updated at that point. That is, the {blocked_ips} table doesn't exist, because it has yet to be added as part of the update. And here I'm stuck: I have no idea how to circumvent this little chicken and egg problem.

EDIT: I believe this has formerly been solved by running some preliminary updates (aka. update_fix_d6_requirements()), my guess is that something similar might have to happen for D7, right?

I too cannot upgrade from D6 to D7.

Here is some more info on this.

First obstacle is this code in includes/bootstrap.inc.

    case DRUPAL_BOOTSTRAP_ACCESS:
      // Deny access to blocked IP addresses - t() is not yet available.
      if (drupal_is_denied(ip_address())) {
        header($_SERVER['SERVER_PROTOCOL'] . ' 403 Forbidden');
        print 'Sorry, ' . check_plain(ip_address()) . ' has been banned.';
        exit();
      }
      break;

Drupal goes into checking whether access is denied to the IP address of the user, and the blocked_ips table does not exist.

So, it goes here:

    public function query($query, Array $args = array(), $options = array()) {

      // Use default values if not already set.
      $options += $this->defaultOptions();

      try {
        // We allow either a pre-bound statement object or a literal string.
        // In either case, we want to end up with an executed statement object.
        if ($query instanceof DatabaseStatement) {
          $stmt = $query;
          $stmt->execute(NULL, $options);
        }
        else {
          $stmt = $this->prepareQuery($query);
          $stmt->execute($args, $options);
        }

        // Depending on the type of query we may need to return a different value.
        // See DatabaseConnection::defaultOptions() for a description of each value.
        switch ($options['return']) {
          case Database::RETURN_STATEMENT:
            return $stmt;
          case Database::RETURN_AFFECTED:
            return $stmt->rowCount();
          case Database::RETURN_INSERT_ID:
            return $this->lastInsertId();
          case Database::RETURN_NULL:
            return;
          default:
            throw new PDOException('Invalid return directive: ' . $options['return']);
        }
      }

The try fails, so it goes into this catch:

      catch (PDOException $e) {
        if (!function_exists('module_implements')) {
          _db_need_install();
        }

At this stage, it cannot find the module_implements, and hence redirects to install.php.

If I comment the chunk in bootstrap.inc like this:

    case DRUPAL_BOOTSTRAP_ACCESS:
      // Deny access to blocked IP addresses - t() is not yet available.
      /*
      if (drupal_is_denied(ip_address())) {
        header($_SERVER['SERVER_PROTOCOL'] . ' 403 Forbidden');
        print 'Sorry, ' . check_plain(ip_address()) . ' has been banned.';
        exit();
      }
      */

      break;

I am able to get to update.php?op=info, but it gives a WSOD and I can't proceed further.

xdebug also reaches a place where there is a throw new PDOException('... and can't proceed any further.

Hope this helps whoever can take it further ...

Here is a bit more narrowing down where this is:

When update.php?op=info is run, we end up here in that file:

if (db_table_exists('batch')) {
  return;
}

Which starts calling the PDO stuff to do the query.

Then this function gets called because of the autoload

function drupal_autoload_class($class) {
  return _registry_check_code('class', $class);
}

Which does a db_query() on SELECT filename FROM {registry} WHERE name = :name AND type = :type.

This goes on and does a prepare, and then goes to this code:

$stmt->execute($args, $options);

$args have the following:

$args[':name']       = (string) 'DatabaseSchema_mysql';
$args[':type']       = (string) 'class';

And $options have the following:

$options['target']   = (string) 'default';
$options['fetch']    = (int) 5;
$options['return']   = (int) 1;
$options['throw_exception'] = (bool) 1;

Then we call this, and that is where the exception happens:

return parent::execute($args);

So, we end up here:

throw new PDOException($query_string . " - \n" . print_r($args,1) . $e->getMessage());

In includes/database/database.inc around line 365.

Xdebug complains of:

response xmlns:xdebug=http://xdebug.org/dbgp/xdebug xmlns=urn:debugg
error code=5 : Command not available (Is used for async commands
message
command is not available

And hence the WSOD.

The underlying reason is that the registry table does not exist, but we are trying to query it. Of course, we will create it in the update.php, but we can't do so. Catch 22!

Well, then the registry table needs to be moved to update.php but also -- why do wind up in autoload class? We should not. Manual loading would be good...

So it seems like we need an update_fix_d7_requirements() which creates the blocked_ips and registry tables. It'd be nice not to have to do that, but it's consistent with the 5-6 upgrade (which adds block_cache as soon as you hit update.php and probably some others too).

As chx says in #7, install.php already does a manual load of the required DB files. update.php should do the same. It's one of those autoloading edge cases that is, fortunately, very easy to bypass. Whether we want to move that utility function to bootstrap.inc or something I am not sure yet, but we certainly can.

Here is a start at a patch:

• Fixes the issue noted by @Lynn by removing the function_exists('module_implements') checks (it's not clear to me why that would ever indicate that Drupal is not installed) and replacing them with a single, consolidated check that hopefully makes more sense. That way at least you don't get mistakenly redirected to install.php when you should be getting a WSOD :)
• Took @Crell's advice to try to get the database working. It improved things, but we're not all the way there yet... see the error messages I've copied below.
• The {blocked_ips} issue is a bit more complicated. First, drupal_is_denied() was actually designed to fail (somewhat) gracefully when the {blocked_ips} table didn't exist; it returns FALSE and generates an error which is then suppressed by the call to ini_set('display_errors', FALSE); in update.php. I guess the PDO error handling changed that, so in the attached patch I have update.php wrap it in a try-catch block so the errors can once again be correctly suppressed. However, there is maybe a bit of a security issue here since until you've fully updated your site to Drupal 7, any IP addresses which you had previously banned in Drupal 6 can now reach your site without being blocked! For now, I did not try to deal with that; we would probably need to put fallback code in drupal_is_denied() itself to deal with that correctly.

So, here is where things stand after the attached patch is applied.... we get further along in the update process and hit some new errors (all WSODs, it seems).

For a privileged user accessing update.php:

( ! ) PDOException: SELECT filename FROM {registry} WHERE name = :name AND type = :type - Array ( [:name] => DeleteQuery_mysql [:type] => class ) SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal.registry' doesn't exist in /home/droth/web/drupal/includes/database/database.inc on line 363
Call Stack
#	Time	Memory	Function	Location
1	0.0018	251672	{main}( )	../update.php:0
2	0.2001	12272112	update_info_page( )	../update.php:712
3	0.2001	12272112	_drupal_flush_css_js( )	../update.php:414
4	0.2001	12272112	variable_set( )	../common.inc:3601
5	0.2005	12273544	cache_clear_all( )	../bootstrap.inc:580
6	0.2005	12273544	db_delete( )	../cache.inc:201
7	0.2005	12273544	DatabaseConnection->delete( )	../database.inc:1325
8	0.2005	12273544	class_exists ( )	../database.inc:468
9	0.2005	12273656	drupal_autoload_class( )	../bootstrap.inc:0
10	0.2005	12273656	_registry_check_code( )	../bootstrap.inc:1439
11	0.2005	12273656	db_query( )	../bootstrap.inc:1449
12	0.2006	12273656	DatabaseConnection->query( )	../database.inc:1193

This time we wind up in the autoload class because the function DeleteQuery_mysql() doesn't exist anywhere in Drupal, so it doesn't matter that we've manually loaded all the files. I tried fixing this by adding spl_autoload_unregister() calls within update.php so that the autoloading never happens. If you add that to the patch, then you wind up here:

( ! ) PDOException: INSERT INTO {watchdog} (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - Array ( [0] => 1 [1] => php [2] => %message in %file on line %line. [3] => a:4:{s:6:"%error";s:6:"notice";s:8:"%message";s:36:"Undefined property: stdClass::$base";s:5:"%file";s:47:"/home/droth/web/drupal/modules/node/node.module";s:5:"%line";i:595;} [4] => 3 [5] => [6] => http://localhost/drupal/update.php?op=info [7] => [8] => 127.0.0.1 [9] => 1224050471 ) SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'link' cannot be null in /home/droth/web/drupal/includes/database/database.inc on line 363
Call Stack
#	Time	Memory	Function	Location
1	0.0026	252672	{main}( )	../update.php:0
2	0.1987	12298276	theme( )	../update.php:742
3	0.1987	12299360	call_user_func_array ( )	../theme.inc:607
4	0.1987	12299360	theme_update_page( )	../theme.inc:0
5	0.1988	12299360	template_preprocess_maintenance_page( )	../theme.maintenance.inc:171
6	0.1988	12299360	theme_get_setting( )	../theme.maintenance.inc:208
7	0.1988	12299360	theme_get_settings( )	../theme.inc:922
8	0.1988	12299424	node_get_types( )	../theme.inc:883
9	0.1988	12299424	_node_types_build( )	../node.module:446
10	0.1994	12302444	drupal_error_handler( )	../common.inc:0
11	0.1994	12303752	watchdog( )	../common.inc:628
12	0.1996	12306848	module_invoke( )	../bootstrap.inc:883
13	0.1996	12306992	call_user_func_array ( )	../module.inc:453
14	0.1996	12307192	dblog_watchdog( )	../module.inc:0
15	0.1996	12308080	db_query( )	../dblog.module:137
16	0.1997	12311924	DatabaseConnection->query( )	../database.inc:1193

Which is where I got stuck. (Note that in the case of an unprivileged user trying to access update.php, you wind up with the above error too.)

Finally, it's not clear to me if this is a separate issue or not, but above we've been talking about accessing update.php... what about if a regular user comes along and tries to access the main site while the upgrade is happening? We want them to see the maintenance page, but right now they would get a WSOD due to the {blocked_ips} issue:

( ! ) PDOException: SELECT 1 FROM {blocked_ips} WHERE ip = :ip - Array ( [:ip] => 127.0.0.1 ) SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupal.blocked_ips' doesn't exist in /home/droth/web/drupal/includes/database/database.inc on line 363
Call Stack
#	Time	Memory	Function	Location
1	0.0004	63596	{main}( )	../index.php:0
2	0.0092	355548	drupal_bootstrap( )	../index.php:21
3	0.0227	764112	_drupal_bootstrap( )	../bootstrap.inc:1027
4	0.0227	764356	drupal_is_denied( )	../bootstrap.inc:1083
5	0.0227	764456	db_query( )	../bootstrap.inc:981
6	0.0251	806752	DatabaseConnection->query( )	../database.inc:1193

(And if you remove that, they get other errors instead.) It seems like the ideal solution would therefore be to move a lot of this stuff into the main bootstrap rather than update.php -- i.e., have a function that somehow detects that they still have a Drupal 6 database and, if so, does the necessary loading to prevent things from failing?

OK, each bug fixed revealed a new one!... but this patch should get them all. It is now possible to run update.php and successfully make it all the way through.

The other problems discussed above could be split off into separate issues, so I think this one is ready for review.

I did a code review and it looks good. Needs testing.

Code looks good, ran a 6-7 update (put the d6 database name into an existing Drupal 7 settings.php). Works great too.

Here is another +1.

I applied the patch and was able to update a Drupal 6.6 site to HEAD without any issues.

This needs to go in.

Sounds like you are missing port?

Mine looks like this:

$databases = array (
  'default' =>
  array (
    'default' =>
    array (
      'driver' => 'mysql',
      'database' => 'dbname',
      'username' => 'dbuser',
      'password' => 'dbpass',
      'host' => 'localhost',
      'port' => '',
    ),
  ),
);

Also, in my case, I used a Drupal 6 with only core, but lots of generated content (comments, nodes, taxonomy, users).

I was LOGGED OUT when I did the update, and had a working settings.php from before. All I did was change the free access to update to TRUE, and hit update.php, and it all went well.

After the update, I logged in, and all was well.

So, check your settings.php first.

If that does not make a difference, the other differences that I have are: I did not have to convert settings.php manually, and I was logged off, it that makes any difference.

@17, The port array key is not in the default.settings.php comments. That shouldn't be the problem...?

So I installed a fresh DRUPAL-6, upgraded to CVS HEAD, applied patch from #11. I modified my settings.php to the correct format and the update worked. I'll try with the old $db_url in settings.php

The port is optional. The auto-generated array includes it because the internal data structure it comes from has it, but it should not be needed unless you're running on a non-standard port.

Here is a reroll to get rid of offsets introduced lately.

It still works for me, and I am able to update from d6 to d7 without issues.

That can only be because you have not had search module on. Testing more.

webchick's dump works for me.

So the issue was , if you had search module enabled the when the maintenance theme ran theme_get_settings you had a user_access running which tried to query tables which did not exist. FAIL.

Ah, I get it... in order to see the last bug, you needed both search module enabled and to not be logged in as user #1; that's why most of us didn't see it.

I started to look into the issue of parsing $db_url and no, it doesn't seem like it will be too hard, but I won't be able to get to posting a patch until tomorrow. It actually occurred to me that once we go through the trouble of parsing it, we could probably just use it to populate the $databases array directly rather than printing anything to the screen that needs to be copy-pasted -- in other words (for now at least) make Drupal 7 completely backwards-compatible with the old $db_url format? More coming tomorrow...

Please make the settings.php issue separate! And commit this :)

Why a separate issue? Because it's a highly controversial idea to add backwards compatiblity to Drupal -- although this is something different than we had BC before because those things that we broke happily were developer and not enduser facing.

Here is the other issue for migrating the database settings http://drupal.org/node/326424

The patch at hand works and I agree with chx, it needs to be de-coupled from any migration.

OK, here is a version with the $db_url parsing so that Drupal 6 settings.php files don't have to be edited. I've tested this on:
(a) a new installation
(b) updating from Drupal 6 with all core modules enabled and logged in as user #1, and
(c) updating from Drupal 6 with all core modules enabled and using $update_free_access = TRUE.

It seems to work fine, although I haven't tested the more obscure possibilities for the $db_url format.

I agree that if this is considered controversial, it's better to commit the patch from #22 now and have this one be a follow-up patch. Although I thought there was some precedent for this kind of thing (didn't Drupal used to have an entire "Legacy" module)? So it might be reasonable to move the function I've introduced here into an includes/legacy.inc or something like that so that it can be included only when necessary...

Oops, crosspost... well, I can always copy my code over to the other issue, depending on what is eventually decided ;)

The last patch did not apply.

I fixed it and now it applies AND it does make update.php work.

So a +1 on it going in.

Patch no longer works.

Specifically, this patch that Dries committed http://drupal.org/node/298600#comment-1085318 made update.php redirect to install.php once again, so we are back to square one.

It is no longer possible to update from D6 to D7.

kbahey, remember Steven's advice about microwaves?
You just needed to reroll against HEAD -- only some whitespace changes failed. My patch never stopped working.

@chx,

No, it did not work. And I tried the microwave too ... not derailing anything ...

Here is what I did:

$ cvs -q up -dP
$ dbclean d7; mysqldump d6 | mysql d7
$ wget http://drupal.org/files/issues/update_d6_d7_1.patch
$ patch -p0 < update_d6_d7_1.patch

Then I change settings.php to do update free access, and hit update.php.

I get this:

An unrecoverable error has occurred. You can find the error message below. It is advised to copy it to the clipboard for reference.
Please continue to the <b>error page</b>
An error occurred. http://d7.../update.php?id=2&op=do (no information available).

I hit the error page link, which is /update.php?id=2&op=finished

And I get this:

Fatal error: Exception thrown without a stack frame in Unknown on line 0 

Then visiting the home page or /user gives this error:

PDOException: SELECT filename FROM {registry} WHERE name = :name AND type = :type - 
Array
(
    [:name] => MergeQuery_mysql
    [:type] => class
)
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'k7.registry' doesn't exist in ../HEAD/drupal/includes/database/database.inc on line 425

Call Stack:
    0.0000      95760   1. {main}() ../HEAD/drupal/index.php:0
    0.0004     126904   2. drupal_bootstrap() ../HEAD/drupal/index.php:21
    0.0031     451824   3. _drupal_bootstrap() ../HEAD/drupal/includes/bootstrap.inc:1040
    0.0031     451824   4. variable_init() ../HEAD/drupal/includes/bootstrap.inc:1110
    0.0039     493784   5. cache_set() ../HEAD/drupal/includes/bootstrap.inc:552
    0.0039     508744   6. db_merge() ../HEAD/drupal/includes/cache.inc:123
    0.0039     508856   7. DatabaseConnection->merge() ../HEAD/drupal/includes/database/database.inc:1461
    0.0040     508856   8. class_exists() ../HEAD/drupal/includes/database/database.inc:490
    0.0040     509320   9. drupal_autoload_class() ../HEAD/drupal/includes/bootstrap.inc:0
    0.0040     509320  10. _registry_check_code() ../HEAD/drupal/includes/bootstrap.inc:1443
    0.0040     509672  11. db_query() ../HEAD/drupal/includes/bootstrap.inc:1453
    0.0040     512024  12. DatabaseConnection->query() ../HEAD/drupal/includes/database/database.inc:1363

So, how can the patch be working for updates?

Well, a) you derail stuff because you continued David's patch b) you reported that it redirected to install.php which it never did. I am well aware of the error (it's btw. the create registry statement, the key is too long) but that's NOT related to this issue which is about making the update.php NOT redirect to install.php. That's TWO installs (edit: issues), one fix the registry update (trivial!) two fix the _drupal_log_error function so that it behaves for install and update. I am working on that right now. For a taste of things to come:

function _drupal_log_error($type, $message, $backtrace, $fatal) {
  $caller = _drupal_get_last_caller($backtrace);

  if (defined('MAINTENANCE_MODE')) {
    $t = get_t();
    $message = $t('@type: %message in %function (line %line of %file).', array('@type' => $type, '%message' => $message, '%function' => $caller['function'], '%line' => $caller['line'], '%file' => $caller['file']));
    drupal_maintenance_theme();
    print theme('maintenance_page', $message);
    error_log($message);
    if ($fatal) {
      exit;
    }
    return;
  }

does not work yet but I am on it.

a) you derail stuff because you continued David's patch

Don't know what "continued David's patch" means ...

b) you reported that it redirected to install.php which it never did.

Right now, without any patches, and using up to the minute HEAD, if you start with a D6 database, and try to visit any page, be it front, /user or update.php you get redirected to install.php. This is what I reported, and I am still getting it.

After the Halloween scare above, chx and myself worked on this patch.

I can confirm that it does work for me, and update.php works as expected.

Here is a reroll, with a field length change for system.install so hook_update_xxx() matches the schema for the registry table that prevented it from working previously.

More testers with more data needed.

Related issues are #304163: Allow update.php to regenerate settings.php and #328781: Fix horrible things in the error reporting.

Read through the patch, everything looks sane, removes the weird function_exists() check and it worked great for me.

I tested:
A clean D7 install
Upgrading a reasonably clean D6 install to D7
Visiting update.php when there's no settings.php
Visiting install.php when Drupal's already installed.
Visiting update.php when there's no updates to run.

Seems pretty solid to me, could do with one further review.

I tried the patch from #38 and it doesn't work for me. It fails on system_update_7007(). Let me see if I can look into it.

Confirmed that it fails on the same dataset that was successful a day ago.

Must be something that was committed today that broke it?

Ah, yup, it looks like it's due to #257910: Performance Issue during Indexing - search_dataset.sid_type unique key should be an Index and the problem with update.php has already been noted there (with a patch to fix it). If I apply the patch here along with the patch from comment #17 in the other issue, then it seems to work again. So if both get committed we are OK.

I have no idea why Drupal was reporting to me that the problem is in update #7007 when actually it was in update #7012, but I guess that's part of the overall brokenness of the error handling right now?

The patch here looks good to me, although I'm not sure what's up with this part:

 // Some unavoidable errors happen because the database is not yet up-to-date.
 // Our custom error handler is not yet installed, so we just suppress them.
 ini_set('display_errors', FALSE);
+ini_set('display_errors', TRUE);

Was that extra line some debug code that got left in?

still not working for me!

Here is the patch and I foudn the same debug piece that David did, sry. Ain't easy to get this flying. There are problems in DBTNG that caused things to fail.

minus the error reporting patch.

Ah, yes, so in the course of a single day not one but two new problems appeared!.... Sorry, I (again) forgot to test with the search module enabled, so I didn't see the second one.

I tested the patch in #46 under a few scenarios, and it seems to work fine as long as you start with D6 search module installed. If you don't start with search module installed, you get an error, but that is the issue taken care of by #257910: Performance Issue during Indexing - search_dataset.sid_type unique key should be an Index so I think we are OK here.

I think this is ready but I can't "legally" set to RTBC since a lot of the code was written by me, and it could probably use one more review/test anyway. However, if this doesn't get committed soon, I predict it will be a very short amount of time before it finds itself broken again... :)

#46 fails on 7012 for me.

Sorry. 7010 runs fine, so must be 7011 (not 7012) that fails.

Patch in #46 works like a charm :)

When #46 is applied with http://drupal.org/node/257910#comment-1089685, the combo works well.

see this error #569410: Can't update D6 -> D7 table not found (drupal_blocked_ips)