The "double pipe" operator is standard ANSI SQL, but it is not that well supported across the board:

  • It doesn't seem to be supported in every platforms by Oracle.
  • It is not supported at all by SQL Server, which uses + for this. We can easily replace || by +, but that breaks automatic casting to string, when mixing the types of columns in a expression (like 'The age of ' || name || ' is ' || age || '.' we use in DatabaseAnsiSyntaxTestCase::testFieldConcat()).
  • It is not supported at all by Drizzle, which doesn't have the PIPES_AS_CONCAT SQL Mode that MySQL supports.

For all those reasons, I suggest we strongly advice against using the double pipe operator, and standardize on CONCAT(anyvalue, anyvalue) (with two parameters only).

Comments

damien tournoud’s picture

Title: Consider refraining to use the || operator for concatenation » Refrain from using the || operator for concatenation
Crell’s picture

If CONCAT() is part of ANSI SQL, sounds fine to me. It should be added to the handbook page.

Perhaps we should consider adding a link from the main docblock for the DB layer to the handbook page when it's more fleshed out.

lambic’s picture

As far as I know, all Oracle platforms support ||

CONCAT() doesn't exist in PostgreSQL, but you could write your own.

hshana’s picture

|| is the ANSI SQL standard and is supported by everything but MSSQL. (Shocker, I know.) http://en.wikibooks.org/wiki/SQL_Dialects_Reference/Functions_and_expres...

damien tournoud’s picture

Status: Active » Needs review
StatusFileSize
new2.42 KB

Other then documentation, I think we need something like this.

Crell’s picture

Hm. What about hshana's point in #4? I'd rather go with "ANSI and supported by the core DBs and some others" than "not-ANSI and supported by the core DB and some others", for different "others".

damien tournoud’s picture

@Crell: CONCAT() is easily supported in all the databases. || is not supported (and *impossible to support*) in a least two databases: SQL Server and Drizzle.

Sounds like a no-brainer to me.

bojanz’s picture

What about #3?

Since CONCAT() doesn't work in PostgreSQL, we'd need to create our own (like here)

damien tournoud’s picture

@bojanz: CONCAT() has been the standard way of concatenating strings for years in any version of Drupal < 7. We already have a compatibility CONCAT() function in PostgreSQL and SQLite.

bojanz’s picture

Good to know, thanks for clearing it up.

Crell’s picture

OK, that makes sense. Thanks, DamZ. +1 on #5 then.

Do we know that any other databases we may want to support can be made to handle CONCAT() the way we do with Postgres?

chx’s picture

Status: Needs review » Reviewed & tested by the community

Wow. just wow. Dont we all love standards...

Crell’s picture

SQL--

That's all I have to say.

tstoeckler’s picture

StatusFileSize
new2.41 KB

Two instances of "ANSI" left in the function descriptions.
Since test bot will test RTBC patches, leaving there.

dries’s picture

Status: Reviewed & tested by the community » Fixed

Committed to CVS HEAD. Thanks.

damien tournoud’s picture

Status: Fixed » Needs work
Issue tags: +Needs documentation

Can we get some help from the documentation team in documenting that?

catch’s picture

Priority: Critical » Normal
josh waihi’s picture

Priority: Normal » Critical

Just installed HEAD on PostgreSQL 8.3 and test failed:

PDOException: SQLSTATE[42883]: Undefined function: 7 ERROR: function concat(bigint, unknown) does not exist LINE 1: SELECT CONCAT($1, CONCAT(name, CONCAT($2, CONCAT(age, $3))))... ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts.: SELECT CONCAT(:a1, CONCAT(name, CONCAT(:a2, CONCAT(age, :a3)))) FROM {test} WHERE age = :age; Array ( [:a1] => The age of [:a2] => is [:a3] => . [:age] => 25 )  in DrupalTestCase->run() (line 442 of /home/josh/projects/drupal7/modules/simpletest/drupal_web_test_case.php).

Here is our existing code:

if (!db_query("SELECT COUNT(*) FROM pg_proc WHERE proname = 'concat'")->fetchField()) {
   db_query('CREATE OR REPLACE FUNCTION "concat"(text, text) RETURNS text AS
      \'SELECT $1 || $2;\'
      LANGUAGE \'sql\''
   );
}

The problem is for every argument data type you can pass to CONCAT you need to specify another function. Ugly.

chx’s picture

Hardly. You just need to add casts.

damien tournoud’s picture

This bug was known, the new test only exposed it.

This should do it:

CREATE OR REPLACE FUNCTION "concat"(anynonarray, anynonarray) RETURNS text AS
      'SELECT CAST($1 AS text) || CAST($2 AS text);'
      LANGUAGE 'sql'

CREATE OR REPLACE FUNCTION "concat"(text, anynonarray) RETURNS text AS
      'SELECT $1 || CAST($2 AS text);'
      LANGUAGE 'sql'

CREATE OR REPLACE FUNCTION "concat"(anynonarray, text) RETURNS text AS
      'SELECT CAST($1 AS text) || $1;'
      LANGUAGE 'sql'

(yes, we need three functions, because all anynonarray must be of the same type when a polymorphic functions is called in PostgreSQL)

dave reid’s picture

I have given up on using CONCAT() in SQL. I honestly think we need some kind of $query->concat() function that will return the proper concatenation SQL for each database driver. See #319070: Provide a wrapper for cross-db and cross-field-type concatenation (PosgreSQL).

josh waihi’s picture

Last I checked, Crell was against smaller functions like that, though personally, I'm for it.

bellHead’s picture

How many of the situations where CONCAT() is used can't be designed around? Looking at the bug issues involving it, I get the feeling that most can.

Most of the CAST() issues are related to the use of CONACT() too, so trying to move away from its use would reduce or eliminate two of the big areas of database compatibility problems.

I think that any documentation on this issue should suggest in strong terms you need to think hard, at least twice, before deciding that CONCAT() and therefore probably CAST() are part of the solution to your problem.

damien tournoud’s picture

I think we can design a type-insensitive CONCAT() in all the databases we support.

chx’s picture

Document that SQL strings cant use concat, add a concat to mapConditionOperator and live happily ever after.

Crell’s picture

I don't think mapConditionOperator can handle concat in its current form. That's not really what it's there for.

damien tournoud’s picture

What we need is being able to replace a || b by something as complex as CAST(a AS text) + CAST(b AS text). Could we use mapConditionOperator() for that?

josh waihi’s picture

regex for drivers who don't support ||?

if (strpos($sql, '||')) {
    $sql = preg_replace('/([^,\s]+)?\s\|\|?\s([^,\s]+)/', 'CONCAT($1, $2)', $sql);
}
josh waihi’s picture

regex for drivers who don't support ||?

if (strpos($sql, '||')) {
    $sql = preg_replace('/([^,\s]+)?\s\|\|?\s([^,\s]+)/', 'CONCAT($1, $2)', $sql);
}
chx’s picture

string parsing is not a solution. Damien is right -- we might need a separate concat method :(

damien tournoud’s picture

Josh, do you really believe that you can parse a random SQL expression with a regexp? Remember that there might be *anything* on both sides of the operator, and that you have to take into account precedence rules.

josh waihi’s picture

No, I don't think REGEX are a good solution for SQL. But I knew it would stir people up! ;)

DamZ solution seems reasonable to me. 3 CONCAT functions. If it makes 'Basic SQL syntax tests" pass without an exception then I'm happy.

Crell’s picture

So to summarize then:

1) Document that || is verbotten.

2) We need a new patch that makes all 3 variants of Concat() for Postgres. Who's doing that one?

3) Do we need anything for SQLite?

josh waihi’s picture

Status: Needs work » Needs review
StatusFileSize
new1.49 KB

Attached is PostgreSQL funcs. Have tested and they work.

chx’s picture

MySQL CONCAT takes any number of parameters -- but then I guess we can document that plz do not use such CONCAT (good luck with people keeping that one...)

damien tournoud’s picture

@chx: I already documented (as a stub) that only CONCAT(string1, string2) is supported in the list of functions and operators wie support.

Stevel’s picture

Status: Needs review » Needs work

The concat(text, text) function is already created several lines before the place where this patch is applied, so we are creating/replacing the function twice for the (text, text) arguments.

Also, in HEAD the patch applied with an offset of 32 lines.

damien tournoud’s picture

// PostgreSQL requires the function to be defined *foreach*
// different argument variation the function can handle.

I guess this foreach needs to read "for each" too.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new1.9 KB

My opinion in the patch name but meh.

damien tournoud’s picture

Status: Needs review » Reviewed & tested by the community

Tested #40 on Postgresql 8.3, works as intended (which is weird because generally on PostgreSQL nothing works the way you would expect them to).

damien tournoud’s picture

Note: SQLite is not affected, so with this patch the "Basic SQL" test passes on every database we support.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Agreed, they're not critical. Where is that major priority anyway? :(

Committed to HEAD all the same, though.

yesct’s picture

:)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

mustanggb’s picture

Priority: Critical » Major