Sooo, you are updating Drupal and got some duplicate entry? Use this!

Keep your errors close, you'll need them. You can always see them in the logs if you are at Drupal 6 www.test.com/?q=admin/reports/dblog

First of all, these querys should return nothing:

SELECT format, module, delta, COUNT(*) FROM filters GROUP BY format, module, delta HAVING COUNT(*) > 1;
SELECT language, lid, plural, COUNT(*) FROM locales_target GROUP BY language, lid, plural HAVING COUNT(*) > 1;
SELECT nid, uid, hostname, COUNT(*) FROM poll_votes GROUP BY nid, uid, hostname HAVING COUNT(*) > 1;
SELECT uid, fid, COUNT(*) FROM profile_values GROUP BY uid, fid HAVING COUNT(*) > 1

If so, use this template to remove duplicates :

ALTER IGNORE TABLE _TABLE_NAME_ ADD UNIQUE INDEX(_WHAT YOU SEE between "GROUP BY" and "HAVING" here!_)

Let say you had when updating :

Update #6043
Failed: ALTER TABLE {filters} ADD UNIQUE KEY fmd (format, module, delta)

Then you should use this to remove duplicate :

ALTER IGNORE TABLE filters ADD UNIQUE INDEX(format, module, delta)

And then do the query again to update manually (and remove the brackets):

ALTER TABLE filters ADD UNIQUE KEY fmd (format, module, delta)

Now, why these duplicate happens? No ideas. Happy upgrading!

References

- "Unable to upgrade from 5.9 to 6.3!!" http://drupal.org/node/287752#comment-942513
- "MySQL: Remove duplicate entries" http://free.netartmedia.net/Databases/Databases8.html

Comments

alexandreracine’s picture

I'll just add this so this post can be counted for something that is solved. If I don't do this, this will change the statistics of Dries ;)

Alexandre Racine

www.alexandreracine.com - mon site perso
www.salsamontreal.com La référence salsa à Montréal

deverman’s picture

We ran into the following failed attempts when trying to upgrade from drupal 5.7 to drupal 6.8:

Update #6043
Failed: ALTER TABLE {flood} ADD INDEX allow (event, hostname, timestamp) <-- index already exists?
Failed: ALTER TABLE {history} ADD INDEX nid (nid) <-- index already exists?
Failed: ALTER TABLE {blocks} CHANGE theme `theme` VARCHAR(64) NOT NULL DEFAULT '', ADD UNIQUE KEY tmd (theme, module, delta), ADD INDEX list (theme, status, region, weight, module) <-- which part failed?
Failed: ALTER TABLE {blocks_roles} ADD INDEX rid (rid) <-- index exists?
Failed: ALTER TABLE {filters} DROP INDEX weight <-- index does NOT exist?
Failed: ALTER TABLE {filters} ADD UNIQUE KEY fmd (format, module, delta) <-- Are there filters we are not carrying forward with which we can drop data from?
Failed: ALTER TABLE {filters} ADD INDEX list (format, weight, module, delta) <-- index exists?
Failed: ALTER TABLE {node} DROP INDEX status <-- index does NOT exist?
Failed: ALTER TABLE {node} DROP KEY nid_vid <-- key does NOT exist?
Failed: ALTER TABLE {users} ADD INDEX mail <-- index already exists?
Failed: ALTER TABLE {users_roles} ADD INDEX rid (rid) <-- index already exists?
Failed: ALTER TABLE {contact} ADD INDEX list (weight, category) <-- index already exists?
Failed: ALTER TABLE {locales_target} ADD PRIMARY KEY (language, lid, plural) <-- does the locales_target table exist?
Failed: ALTER TABLE {profile_values} DROP INDEX uid <-- does the locales_target table exist?
Failed: ALTER TABLE {profile_values} ADD PRIMARY KEY (uid, fid)
Failed: ALTER TABLE {term_data} DROP INDEX vid
Failed: ALTER TABLE {term_data} ADD INDEX vid_name (vid, name)
Failed: ALTER TABLE {term_data} ADD INDEX taxonomy_tree (vid, weight, name)
Failed: ALTER TABLE {term_node} DROP PRIMARY KEY
Failed: ALTER TABLE {term_node} DROP INDEX tid
Failed: ALTER TABLE {term_node} ADD PRIMARY KEY (tid, vid)
Failed: ALTER TABLE {term_relation} DROP INDEX tid1
Failed: ALTER TABLE {term_relation} ADD UNIQUE KEY tid1_tid2 (tid1, tid2)
Failed: ALTER TABLE {term_synonym} DROP INDEX name
Failed: ALTER TABLE {term_synonym} ADD INDEX name_tid (name, tid)
Failed: ALTER TABLE {vocabulary} ADD INDEX list (weight, name)
Failed: ALTER TABLE {vocabulary_node_types} ADD INDEX vid (vid)

So we tried the suggested queries and here are the results we got:

SELECT format, module, delta, COUNT(*) FROM filters GROUP BY format, module, delta HAVING COUNT(*) > 1; - result - 0 rows
SELECT language, lid, plural, COUNT(*) FROM locales_target GROUP BY language, lid, plural HAVING COUNT(*) > 1; - result - unknown column language
SELECT nid, uid, hostname, COUNT(*) FROM poll_votes GROUP BY nid, uid, hostname HAVING COUNT(*) > 1; - result - poll_votes does not exist
SELECT uid, fid, COUNT(*) FROM profile_values GROUP BY uid, fid HAVING COUNT(*) > 1; - result - over 90,000 rows

I think select 2 and 3 results are because we didn't use a language module or the poll module in 5.7. I don't understand the result of the fourth select statement.

We tried the suggested remedy above but couldn't get it to work. I didn't see any other comments on this post to confirm if this is really a correct remedy.

From what we can tell a majority of the errors should be from duplicated keys but I think there maybe different reasons for some errors.

I know that we are not going from the latest version of drupal 5 but I read this thread and the poster tried upgrading to the latest version of drupal 5 and it had no effect:

http://drupal.org/node/357283

I am pretty sure we will be in the same boat. I can't believe how hard and tedious this drupal 5->6 upgrade is. I think a new version of drupal 6 needs to be released that better addresses these upgrade problems.

Some detailed upgrade advice would be greatly appreciated.

gabble’s picture

As soon as I:

ALTER IGNORE TABLE blocks ADD UNIQUE INDEX(theme, module, delta);

I get:

ERROR 1071 (42000): Specified key was too long; max key length is 1000 bytes

Any ideas? :-(

alexandreracine’s picture

The only thing I can tell you is that there is something really wrong here. If your key is more then 1000 bytes, your table might not be in good shape. Either some module did a mess to that key or you have some corruption. Easy answer, you'll have to pay someone to do it or make a lot of trial and errors to fix this.

You do have backups do you?

Since this is the blocks table, I would do something like this:

First try
1-Before the upgrade, while in drupal 5, delete all blocks that can be deleted.
2-deactivate all modules
3-upgrade

Second try
1-In phpmyadmin, search for a too long string in the table "blocks" and delete that entry.
2-rince and repeat
3-try to upgrade

Third try
1-Before the upgrade, while in drupal 5, delete the blocks table (in phpmyadmin).
2-see if drupal still work
3-if yes, try to add a new block
4-if it works, upgrade
5-re-add all blocks somehow...

If you don't know phpmyadmin, it's time to learn.

Good luck

gabble’s picture

This one trick helped:

Before the upgrade, while in drupal 5, delete all blocks that can be deleted.

Thanks a lot!

awolfey’s picture

I had this error with the blocks table when trying your techniques:

#1071 - Specified key was too long; max key length is 1000 bytes

So first I did:

ALTER TABLE blocks CHANGE theme `theme` VARCHAR(64) NOT NULL DEFAULT ''

then

ALTER IGNORE TABLE blocks ADD UNIQUE INDEX(theme, module, delta)

and I was then able to execute the original failed update:

ALTER TABLE blocks CHANGE theme `theme` VARCHAR(64) NOT NULL DEFAULT '', ADD UNIQUE KEY tmd (theme, module, delta), ADD INDEX list (theme, status, region, weight, module)