I'm trying to make a view's query use GROUP BY to eliminate duplicates on a field other than the base table's primary id.

As far as I can tell, this can't be done in the UI, so I looked at using hook_views_query_alter().

I can do this:

  $query->groupby[] = 'node_node_data_field_foo';

but then the resulting SQL query has a GROUP BY clause that lists EVERY field. This is wrong.

The problem is in this code in query.inc:

      if (!empty($field['count'])) {
        $string = "COUNT($string)";
        $has_aggregate = TRUE;
      }
      else if (!empty($field['aggregate'])) {
        $has_aggregate = TRUE;
      }
      elseif ($this->distinct && !in_array($fieldname, $this->groupby)) {
        $string = $GLOBALS['db_type'] == 'pgsql' ? "FIRST($string)" : $string;
      }
      else {
        $non_aggregates[] = $fieldname;
      }
      if ($field['alias']) {
        $string .= " AS $field[alias]";
      }
      $fields .= $string;

      if ($get_count_optimized) {
        // We only want the first field in this case.
        break;
      }
    }

    if ($has_aggregate || $this->groupby) {
      $groupby = "GROUP BY " . implode(', ', array_unique(array_merge($this->groupby, $non_aggregates))) . "\n";

      if ($this->having) {
        $having = $this->condition_sql('having');
      }

What is happening is that we're getting ALL the fields in the GROUP BY because they are all in the $non_aggregates array. I can't figure out why they're all getting in there, because the complex if/elseif/else block above isn't commented at all, so I can't figure out what case each condition is trying to account for. The final else is the problem, basically.

Comments

dawehner’s picture

I would recommend to try out views3 and see whether it's fixed there already.

joachim’s picture

Not tested it yet, but this code looks like the same sort of problem might be present:

    if ($this->has_aggregate || $this->groupby) {
      $groupby = "GROUP BY " . implode(', ', array_unique(array_merge($this->groupby, $non_aggregates))) . "\n";
      if ($this->having) {
        $having = $this->condition_sql('having');
      }
    }
joachim’s picture

Version: 6.x-2.x-dev » 6.x-3.x-dev

Still a problem in 3.x:

function views_query_alter_views_query_alter(&$view, &$query) {
  dsm($query);
  
  $query->groupby[] = 'node_title';
}

produces this query:

SELECT node.nid AS nid,
node.title AS node_title
FROM node node
GROUP BY node_title, nid

achton’s picture

Subscribe.

Luciuz’s picture

Subscribe.

joachim’s picture

BTW, the use case for this was this:

I have performance nodes which nodereference both a parent event node, and a venue node.

On the event node, I want to list the venues. In most cases, there are several performances of an event, but most if not all are all in the same venue: eg a play showing 5 nights a week in the same theatre; but could equally be a play that tours several theatres over the course of a month.

So the view takes the nid argument from the current event node, gets performance nodes with that value in the event noderef field, and adds a noderef relationship to get the venue nodes details such as title, google map link, etc -- which intentionally produces multiple rows for the single event node the argument gets.

Obviously, in those multiple rows I get a ton of duplicate venues, and the problem is to eliminate these. The 'distinct' UI option only affects the base field, the nid, which is the nid of the event node.

I've solved it with theming, but it would be cleaner to get the groupby working properly -- and have wider uses too.

achton’s picture

@joachim: from my research in the issue queue, I think this problem is really hard to solve (for the devs).

I am also trying to solve it somehow, and I think the workaround in #903768: groupby on cck fields only works when all values are the same is my best bet, but I haven't tested it yet. Your insight into the problem is obviously better than mine, so could you perhaps cast a glance at that workaround and see if it works for you? If it does, I'd say this is a duplicate issue, and we probably won't get a better response than the one given over there.

There are more related issues here:

A similar issue is #1000004: View with Argument and Filter using OR operator only appliers the argument to the first group, where the workaround also is to use hook_views_query_alter().

The whole thing might be founded in CCK, actually, see #695298: Allow CCK to work with GROUP BY support in Views.

joachim’s picture

I think #903768: groupby on cck fields only works when all values are the same is *perhaps* the same issue, but if it is, it's not expressed correctly.

The problem is not that "groupby on cck fields only works when all values are the same", it is that groupby adds *all fields in the query* to the GROUP BY clause, thus rendering the GROUP BY pointless.

And I don't think it's impossible to fix -- at least I am not convinced it is impossible.

The first step would be to take that big chunk of code I posted -- the monster if/elseif/elseif/elseif... block and *document it* -- that is, write a comment for EACH clause explaining what it's catching and what it's doing with it. Then we stand some chance of figuring out which of those should apply when we're adding a single groupby to the query.

merlinofchaos’s picture

Status: Active » Closed (works as designed)

This is not wrong. According to the ANSI SQL spec, if GROUP BY is invoked, any item in the SELECT portion of the query must either 1) be an aggregate function or 2) appear in the GROUP BY.

MySQL does not enforce this rule. PostgreSQL does. Since we support postgres, we must enforce this rule.

What this means is that Views is behaving properly, and you (because you probably use MySQL) are expecting it to produce a non ANSI SQL compatible query.

It's made worse because CCK adds a bunch of additional fields whether or not you want them -- and there's a patch that's been sitting in the CCK queue for a long time to try and deal with that.

joachim’s picture

O_o

Further evidence that PostgreSQL is crack...

joachim’s picture

Category: bug » feature
Status: Closed (works as designed) » Active

Okay then the way to do it is with a more complex model for how to add a DISTINCT to the query:

http://www.postgresql.org/docs/9.0/interactive/queries-select-lists.html...

After the select list has been processed, the result table can optionally be subject to the elimination of duplicate rows. The DISTINCT key word is written directly after SELECT to specify this:

SELECT DISTINCT select_list ...

-- we need the ability for the DISTINCT to affect fields other than the view's base.

Though I still maintain PostgreSQL is absolute tosh which we shouldn't bother with :/

joachim’s picture

Title: query object doesn't handle group by properly » allow DISTINCT on other fields

... and change the title accordingly.

merlinofchaos’s picture

Unfortunately, SQL databases don't really do DISTINCT on fields, it turns out. You get DISTINCT rows. :/ And while you can say DISTINCT(node.nid) it doesn't really work that way.

Iritscen’s picture

So... is there a way to make this happen without relying on DISTINCT? I have a whole bunch of records from which I need to weed out duplicates. I'm open to alternative approaches, but I'm too much of a novice to know what they are. Is there a way to use a theme to perform a "distinct" filter?

joachim’s picture

> Is there a way to use a theme to perform a "distinct" filter?

Absolutely. Loop over your rows in the theme before they are rendered and remove whatever you like.

mototribe’s picture

Version: 6.x-3.x-dev » 7.x-3.0-beta3

Subscribe

dawehner’s picture

Version: 7.x-3.0-beta3 » 6.x-3.x-dev

Please please don't move something when you subscribe. BTW. in 7.x-3.x this is not possible, because DBTNG does not support it at all.

merlinofchaos’s picture

Status: Active » Closed (won't fix)

And if we can't do it in 7, we're not doing it in 6.

sydkoh’s picture

So basically Group by with aggregation functions don't work in D6???