Hi,

I've been having a problem where I cant create more aliases than the amount set in pathauto_max_bulk_update. Doing some digging in the code, I [think] I've found a problem with the SQL in the node_pathauto_bulkupdate() function defined in pathauto_node.inc.

The line reads like this:
$query = "SELECT nid, type, title, uid, created, src, dst, vid FROM {node} LEFT JOIN {url_alias} ON CONCAT('node/', CAST(nid AS CHAR)) = src WHERE src IS NULL ". $type_where;

The problem is that the CAST(nid AS CHAR) part of the SQL is truncating the node id to 1 character, so the two tables never link and the same rows are returned every time the bulk update is run.

Eg: If the nodes id is 1234, the above SQL essentially comes out joining on 'node/1' = 'node/1234'.

By changing the type to VARCHAR(10) I have managed to get my bulk updates to work as they should.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

greggles’s picture

I'm not sure exactly what change you're asking for - could you provide a patch http://drupal.org/patch ?

DrDeth’s picture

Status: Active » Needs review
FileSize
702 bytes

Sure thing.

I've attached a patch for the file.

Hope this helps.

greggles’s picture

This seems fine to me. Which database do you use?

mysql> select CONCAT('node/', CAST(nid AS CHAR)) from node limit 100;
+------------------------------------+
| CONCAT('node/', CAST(nid AS CHAR)) |
+------------------------------------+
| node/25                            | 
| node/26                            | 
| node/27                            | 
| node/28                            | 
| node/29                            | 
| node/30                            | 
+------------------------------------+

I assume you use PostgreSQL and I'm hesitant to fix this since this has basically no reviews and the original code was reviewed pretty heavily in #212327: cast the concatenated join value to be able to use the url_alias index.

Dave Reid’s picture

Title: node_pathauto_bulkupdate() - SQL Broken » Bulk updated and PostgreSQL incompatibility (CAST as CHAR)
Version: 5.x-2.3 » 6.x-1.x-dev
Issue tags: +PostgreSQL

Marked #328178: Postgresql 7.4 Bulk Update as a duplicate of this issue.

Josh Waihi’s picture

Dave Reid’s picture

Issue tags: +bulk

Tagging all the bulk alias issues for #713238: RFC: Pathauto Bulk module.

JoelKleier’s picture

FileSize
924 bytes

I can verify this is an issue for PostgreSQL db's -- we fixed our install with the attached patch, but the patch in #5 looks like it may be more db agnostic.

Josh Waihi’s picture

Status: Needs review » Reviewed & tested by the community

I'm RTBC'ing #5 on the bases that this problem is a postgres problem and that the changes fix postgres and don't break mysql.

Dave Reid’s picture

Status: Reviewed & tested by the community » Needs review

Does this patch need to apply to the pathauto_taxonomy.inc and pathauto_user.inc? They seem to have the same things going on:

  $query = "SELECT tid, vid, name, src, dst FROM {term_data} LEFT JOIN {url_alias} ON CONCAT('taxonomy/term/', CAST(tid AS CHAR)) = src WHERE src IS NULL AND vid <> %d ". $vid_where;
  $query = "SELECT tid, vid, name, src, dst FROM {term_data} LEFT JOIN {url_alias} ON CONCAT('forum/', CAST(tid AS CHAR)) = src WHERE vid = %d AND src IS NULL";
  $query = "SELECT uid, name, src, dst FROM {users} LEFT JOIN {url_alias} ON CONCAT('user/', CAST(uid AS CHAR)) = src WHERE uid > 0 AND src IS NULL";
  $query = "SELECT uid, name, src, dst FROM {users} LEFT JOIN {url_alias} ON CONCAT('blog/', CAST(uid AS CHAR)) = src WHERE uid > 0 AND src IS NULL";
  $query = "SELECT uid, name, src, dst FROM {users} LEFT JOIN {url_alias} ON CONCAT(CONCAT('user/', CAST(uid AS CHAR)), '/track') = src WHERE uid > 0 AND src IS NULL";

If so, we'll need an updated patch.

Status: Needs review » Needs work

The last submitted patch, batch-update-fix-6.x.patch, failed testing.

Josh Waihi’s picture

Status: Needs work » Needs review
FileSize
5.04 KB

yes.

progga’s picture

#11 has worked fine for me on Postgres 8.3. I have only used it for creating url aliases for nodes. Thanks for the patch.

Status: Needs review » Needs work

The last submitted patch, pathauto-nodebulkupdate-for-pgsql_3.patch, failed testing.

jamespharaoh’s picture

FileSize
5.32 KB

Casting to char(99) should work in both databases. Patch against HEAD attached.

tuffnatty’s picture

Status: Needs work » Needs review
FileSize
4.1 KB

Rebuilt the last patch against HEAD. Also changed CHAR(99) to CHAR(20) although I don't know if it will give us any significant speedup.

AlexisWilke’s picture

tuffnatty,

In PostgreSQL, the fastest is TEXT, not CHAR(<whatnot>);. It is specified in the docs, check out the first Tip on that page:

http://www.postgresql.org/docs/8.0/interactive/datatype-character.html

It would be good to always use TEXT and not bother with CHAR() at all. It has generated many problems in Core (aggregator URL too long and watchdog strings too long for two that I recall.)

Thank you.
Alexis

Dave Reid’s picture

Title: Bulk updated and PostgreSQL incompatibility (CAST as CHAR) » Provide a wrapper for cross-db and cross-field-type concatenation (PosgreSQL, MSSQL)
Version: 6.x-1.x-dev » 7.x-1.x-dev
Assigned: Unassigned » Dave Reid
FileSize
6.58 KB

How about a function that will return the proper concatenation depending on the database type being used? This seems like a much better and future-proof method.

Dave Reid’s picture

I've tested #17 with success using MySQL 5.1.37 and PostgreSQL 8.4.4. Still needs testing with lower versions of PostgreSQL and also with SQLite in D7.

Dave Reid’s picture

Tested and it works great with sqlite with Drupal 7. Still could use someone testing on PostgreSQL below 8.3. Please report back your results/success.

Dave Reid’s picture

Version: 7.x-1.x-dev » 6.x-2.x-dev
FileSize
6.9 KB

Since I've tested this on all possible database types for D7, I'm committing #18 to this branch. The following patch still needs testing as per #19.
http://drupal.org/cvs?commit=377544

Damien Tournoud’s picture

That doesn't look correct. On MS SQL, '+' is both the concatenation operator and the addition operator, and doesn't do automatic casting. You need an explicit cast here if you want to be able to use non-text columns in the concat.

That's one of the rational behind *not* using || in Drupal 7, see #809698: Refrain from using the || operator for concatenation.

Damien Tournoud’s picture

Version: 6.x-2.x-dev » 7.x-1.x-dev
Status: Needs review » Needs work

Let's get back to the old code.

Our own PostgreSQL's CONCAT() function has been fixed, and now properly accept any non array type as arguments. This abstraction is thus completely unecessary, and actually breaks Drupal on SQL Server (because using || *requires* a cast, while our CONCAT() function doesn't).

Dave Reid’s picture

Status: Needs work » Needs review
FileSize
3.5 KB

Agreed, especially since we removed support for feed aliases, so now we never need to concat more than two strings.

Dave Reid’s picture

So we can remove this in D7, but we still have to do something about this in the D6 versions, as this is affecting critical functionality.

Dave Reid’s picture

Version: 7.x-1.x-dev » 6.x-2.x-dev
Priority: Normal » Major
Dave Reid’s picture

Here's the patch for 6.x-2.x.

Dave Reid’s picture

Version: 6.x-2.x-dev » 6.x-1.x-dev
Status: Needs review » Patch (to be ported)

In #26 we removed the support for MSSQL for 6.x since I don't think I can support it. I tested it with both MySQL and PostgreSQL.

Committed to 6.x-2.x: http://drupalcode.org/project/pathauto.git/commit/4aff481

Dave Reid’s picture

Title: Provide a wrapper for cross-db and cross-field-type concatenation (PosgreSQL, MSSQL) » Provide a wrapper for cross-db and cross-field-type concatenation (PosgreSQL)
Status: Patch (to be ported) » Fixed

Committed to 6.x-1.x as well as fixed a small bug in the committed code with the forum aliases in 6.x-2.x
http://drupalcode.org/project/pathauto.git/commit/7af542f
http://drupalcode.org/project/pathauto.git/commit/084dbd4

Status: Fixed » Closed (fixed)
Issue tags: -PostgreSQL, -bulk

Automatically closed -- issue fixed for 2 weeks with no activity.