db_index_exists() missing, module updates cannot handle indexes properly

sammys - January 19, 2009 - 04:11
Project:Drupal
Version:7.x-dev
Component:database system
Category:bug report
Priority:critical
Assigned:sammys
Status:closed
Issue tags:Needs tests, PostgreSQL Code Sprint
Description

When updating sites from version A to version D there is a chance an index was created in version B and removed again in version D. PostgreSQL throws an error if a non-existent index is used in a DROP INDEX. This can be avoided by checking the index exists. PostgreSQL 8.2 introduced an IF EXISTS clause to DROP INDEX.

Ref:
DROP INDEX (8.2)
DROP INDEX (8.1)

Patch to follow.

#1

sammys - January 19, 2009 - 04:13

#2

sammys - January 19, 2009 - 06:38
Status:active» needs review

Turns out db_drop_primary_key() and db_drop_unique_key() are also affected. I've added _db_index_exists() and used that in all three functions.

We need some reviewing of the patch since it's larger than trivial.

AttachmentSizeStatusTest resultOperations
360854_200901191536+1100-D6.patch1.74 KBIgnoredNoneNone

#3

sammys - January 22, 2009 - 09:49

Patched code didn't work. Fixed it and rerolled.

AttachmentSizeStatusTest resultOperations
360854_200901222043-D6.patch1.53 KBIgnoredNoneNone

#4

webchick - January 25, 2009 - 13:16
Version:6.9» 7.x-dev
Status:needs review» needs work

Ran across the need for this today in #363262: Missing index on url_alias table.

1. Needs to be fixed in D7 first.
2. We already have db_column_exists() and db_table_exists(), so there's no reason to prefix this function with an underscore.
3. The function is missing PHPDoc.

#5

Crell - January 26, 2009 - 07:21

webchick asked for my input here. I've no problem with an indexExists() method on the schema API, with a procedural wrapper. It should follow the same logical design as the rest of schema, of course, but I like having a rich set of primitives. :-)

#6

sammys - January 26, 2009 - 12:33

Here is a rerolled patch with underscore removed and docs added.

Crell: Are you adding indexExists() or shall I assist?

AttachmentSizeStatusTest resultOperations
360854_200901262326+1100-D6.patch1.85 KBIgnoredNoneNone

#7

sammys - January 26, 2009 - 12:37

Tagging

#8

Crell - January 26, 2009 - 16:18

You can go ahead. :-) A D7 implementation, however, must put the actual logic back in the schema class, and then provide a DB-specific implementation in each DB's schema class. Then add a wrapper function that mirrors the other functions in database.inc.

#9

sammys - January 29, 2009 - 00:59

Ok... i'll play with it over the weekend. Been waiting for another opportunity to get my hands dirty on your new DB stuff :)

#10

sammys - January 29, 2009 - 04:08

Reposting the patch with the leading underscore removed on the calls to db_index_exists().

AttachmentSizeStatusTest resultOperations
360854_200901291505+1100-D6.patch1.84 KBIgnoredNoneNone

#11

Damien Tournoud - January 29, 2009 - 05:21

I fail to see how this is database specific. Isn't MySQL also returning some kind of error when you try to DROP a non existing index?

Plus, I'm -1 on this approach. Every inconsistency should throw a warning during the update process. It's the job of the administrator to choose to ignore them or not.

#12

sammys - February 2, 2009 - 08:57

Damien,

I agree some feedback is required when the index you want to drop doesn't exist. It could mean duplicate indexes or that some column change operations will fail. In the case of duplicate indexes, it is a performance hit and doesn't break the system. In the case of failing column operations, the failed column operation (such as allowing dupes in a column with unique constraint) would fail showing an error.

To me, the issue is really about where to put the responsibility of error/warning reporting. Update code is the only part knowing the operations being performed and having the ability to report the condition appropriately.

Perhaps we don't bypass reporting of the failed index removal and make it a warning instead. This will allow an administrator to still see there was something fishy. An error on a following operation can also be associated with the warning without looking at code.

How does that sound?

#13

sammys - February 3, 2009 - 08:50

I did a little dev on the D7 implementation of db_index_exists(). From my reading MySQL associates indexes with tables and PostgreSQL have them as independent entities. This will mean we have a function prototype of:

db_index_exists($table, $name)

Most of it is implemented. Couldn't easily find information for sqlite. Can anyone offer some suggestions on this?

#14

sammys - March 5, 2009 - 16:38
Status:needs work» needs review

Here's a preliminary patch. I haven't tested the code yet.

AttachmentSizeStatusTest resultOperations
360854_200903051131-0500.patch4.12 KBIdleUnable to apply patch 360854_200903051131-0500.patchView details

#15

sammys - March 22, 2009 - 02:57
Title:DROP INDEX throws an error if index doesn't exist on PostgreSQL» Adding db_index_exists() to DBTNG and updates handling missing indexes properly

Changing the title to match what's now going on in this issue. Still waiting on a review.

#16

Crell - March 23, 2009 - 14:42
Status:needs review» needs work

db_result() is deprecated. Use ->fetchField() instead.

#17

sammys - April 4, 2009 - 07:40
Status:needs work» needs review

Rerolled using ->fetchField().

AttachmentSizeStatusTest resultOperations
360854_200904041839+1100.patch4.17 KBIdleUnable to apply patch 360854_200904041839+1100.patchView details

#18

catch - May 6, 2009 - 11:09

Seems like this should add to the database tests. Just checking for an existing and non-existing index ought to work no?

#19

Dries - May 9, 2009 - 18:31
Status:needs review» needs work

The code looks good to me, but as catch suggests, it needs tests.

#20

sammys - May 14, 2009 - 10:15

I'll get to it as soon as I can.

#21

markus_petrux - July 15, 2009 - 19:23

Bug in latest patch: $table needs to be escaped somehow when enclosed between {}. Otherwise table prefix substitution will fail.

Fix:

<?php
-  return db_query("SHOW INDEX FROM {$table} WHERE key_name = '$name'")->fetchField();
+  return
db_query("SHOW INDEX FROM {" . $table . "} WHERE key_name = '$name'")->fetchField();
?>

#22

Josh Waihi - July 31, 2009 - 02:37

subscribing and promoting to PostgreSQL Sprint

#23

markus_petrux - August 30, 2009 - 13:02

I implemented this method to check for existence of indexes in CCK, which is available in CCK 2.5. Reference: #231453: Allow indexing columns.

But it has been reported (see #562260: content_db_index_exists has wrong syntax for postgres) that the following query doesn't work in PostgreSQL.

<?php
db_query
("SELECT COUNT(indexname) FROM pg_indexes WHERE indexname = '$name'")
?>

That it should look like this:

<?php
db_query
("SELECT COUNT(indexname) FROM pg_indexes WHERE indexname = '" . $table . '_' . $name . "_idx'")
?>

Is it possible that the contents of the indexname column varies depending on PostgreSQL versions? I've been unable to find information about how PostgreSQL stores data in this column, so I cannot really tell. :(

I hope the experience collected by the use of such a feature in CCK could potentially help provide a more reliable method for D7.

Does anyone knows more about the format of the indexname column in pg_indexes table throughout the different PostgreSQL versions supported by Drupal itself?

#24

markus_petrux - August 30, 2009 - 13:52

LOL... it was sooo much easy. The fact is that the Schema API for PostgreSQL is building the index names like that, so the statement to check if an index exists should be built using the same pattern for index names as the one used when creating the index. The following syntax is then correct:

<?php
db_query
("SELECT COUNT(indexname) FROM pg_indexes WHERE indexname = '" . $table . '_' . $name . "_idx'")
?>

And the one implemented in the patch in #17 is not, which is like this:

<?php
db_query
("SELECT COUNT(indexname) FROM pg_indexes WHERE indexname = '$name'")
?>

For references, see methods dropIndex() and _createIndexSql() in class DatabaseSchema_pgsql, implemented in file includes/database/pgsql/schema.inc

I hope that helps here. ;-)

#25

Josh Waihi - August 31, 2009 - 01:19
Priority:normal» critical
Status:needs work» postponed

I think this will be easily done with #302327: Support cross-schema/database prefixing like we claim to in. So lets wait for that. I'm sure this will go in after the freeze too.

#26

sun - December 8, 2009 - 03:14
Title:Adding db_index_exists() to DBTNG and updates handling missing indexes properly» db_index_exists() missing, module updates cannot handle indexes properly
Status:postponed» active

Absolutely required. Also for the module update numbering scheme we teach in http://api.drupal.org/api/function/hook_update_N/7

#27

markus_petrux - December 8, 2009 - 07:05

If that helps, here's the code that's using CCK for a few months already:

<?php
/**
* Checks if an index exists.
*
* @todo: May we remove this funcion when implemented by Drupal core itself?
* @link http ://drupal.org/node/360854
* @link http ://dev.mysql.com/doc/refman/5.0/en/extended-show.html
*
* @param $table
*   Name of the table.
* @param $name
*   Name of the index.
* @return
*   TRUE if the table exists. Otherwise FALSE.
*/
function content_db_index_exists($table, $name) {
  global
$db_type;
  if (
$db_type == 'mysql' || $db_type == 'mysqli') {
    if (
version_compare(db_version(), '5.0.3') < 0) {
     
// Earlier versions of MySQL don't support a WHERE clause for SHOW.
     
$result = db_query('SHOW INDEX FROM {'. $table .'}');
      while (
$row = db_fetch_array($result)) {
        if (
$row['Key_name'] == $name) {
          return
TRUE;
        }
      }
      return
FALSE;
    }
    return (bool)
db_result(db_query("SHOW INDEX FROM {". $table ."} WHERE key_name = '$name'"));
  }
  elseif (
$db_type == 'pgsql') {
   
// Note that the index names in Schema API for PostgreSQL are prefixed by
    // the table name and suffixed by '_idx'.
   
return (bool)db_result(db_query("SELECT COUNT(indexname) FROM pg_indexes WHERE indexname = '{". $table ."}_{$name}_idx'"));
  }
  return
FALSE;
}
?>

#28

moshe weitzman - December 8, 2009 - 14:32

use schema api instead of querying db directly

#29

Crell - December 15, 2009 - 06:26
Status:active» needs review

Chasing HEAD from #17, tidied the code to use the DB API properly (internal calls should always use $connection->query(), never db_query()), and adds unit tests.

AttachmentSizeStatusTest resultOperations
db_index_exists.patch11.21 KBIdlePassed on all environments.View details

#30

Dries - December 15, 2009 - 08:28

+++ modules/simpletest/tests/schema.test 2009-12-14 02:47:45 +0000
@@ -1,4 +1,4 @@
-<?php
+g<?php

Looks wrong. I'll correct that manually before committing. Interestingly, all tests pass.

#31

Dries - December 15, 2009 - 08:31
Status:needs review» fixed

Fixed, and committed to CVS HEAD. Thanks!

#32

Crell - December 15, 2009 - 16:05

Weird, but thanks. :-)

#33

System Message - December 29, 2009 - 16:10
Status:fixed» closed

Automatically closed -- issue fixed for 2 weeks with no activity.

 
 

Drupal is a registered trademark of Dries Buytaert.