I'm trying to make a view's query use GROUP BY to eliminate duplicates on a field other than the base table's primary id.
As far as I can tell, this can't be done in the UI, so I looked at using hook_views_query_alter().
I can do this:
$query->groupby[] = 'node_node_data_field_foo';
but then the resulting SQL query has a GROUP BY clause that lists EVERY field. This is wrong.
The problem is in this code in query.inc:
if (!empty($field['count'])) {
$string = "COUNT($string)";
$has_aggregate = TRUE;
}
else if (!empty($field['aggregate'])) {
$has_aggregate = TRUE;
}
elseif ($this->distinct && !in_array($fieldname, $this->groupby)) {
$string = $GLOBALS['db_type'] == 'pgsql' ? "FIRST($string)" : $string;
}
else {
$non_aggregates[] = $fieldname;
}
if ($field['alias']) {
$string .= " AS $field[alias]";
}
$fields .= $string;
if ($get_count_optimized) {
// We only want the first field in this case.
break;
}
}
if ($has_aggregate || $this->groupby) {
$groupby = "GROUP BY " . implode(', ', array_unique(array_merge($this->groupby, $non_aggregates))) . "\n";
if ($this->having) {
$having = $this->condition_sql('having');
}
What is happening is that we're getting ALL the fields in the GROUP BY because they are all in the $non_aggregates array. I can't figure out why they're all getting in there, because the complex if/elseif/else block above isn't commented at all, so I can't figure out what case each condition is trying to account for. The final else is the problem, basically.
Comments
Comment #1
dawehnerI would recommend to try out views3 and see whether it's fixed there already.
Comment #2
joachim commentedNot tested it yet, but this code looks like the same sort of problem might be present:
Comment #3
joachim commentedStill a problem in 3.x:
produces this query:
SELECT node.nid AS nid,
node.title AS node_title
FROM node node
GROUP BY node_title, nid
Comment #4
achtonSubscribe.
Comment #5
Luciuz commentedSubscribe.
Comment #6
joachim commentedBTW, the use case for this was this:
I have performance nodes which nodereference both a parent event node, and a venue node.
On the event node, I want to list the venues. In most cases, there are several performances of an event, but most if not all are all in the same venue: eg a play showing 5 nights a week in the same theatre; but could equally be a play that tours several theatres over the course of a month.
So the view takes the nid argument from the current event node, gets performance nodes with that value in the event noderef field, and adds a noderef relationship to get the venue nodes details such as title, google map link, etc -- which intentionally produces multiple rows for the single event node the argument gets.
Obviously, in those multiple rows I get a ton of duplicate venues, and the problem is to eliminate these. The 'distinct' UI option only affects the base field, the nid, which is the nid of the event node.
I've solved it with theming, but it would be cleaner to get the groupby working properly -- and have wider uses too.
Comment #7
achton@joachim: from my research in the issue queue, I think this problem is really hard to solve (for the devs).
I am also trying to solve it somehow, and I think the workaround in #903768: groupby on cck fields only works when all values are the same is my best bet, but I haven't tested it yet. Your insight into the problem is obviously better than mine, so could you perhaps cast a glance at that workaround and see if it works for you? If it does, I'd say this is a duplicate issue, and we probably won't get a better response than the one given over there.
There are more related issues here:
A similar issue is #1000004: View with Argument and Filter using OR operator only appliers the argument to the first group, where the workaround also is to use
hook_views_query_alter().The whole thing might be founded in CCK, actually, see #695298: Allow CCK to work with GROUP BY support in Views.
Comment #8
joachim commentedI think #903768: groupby on cck fields only works when all values are the same is *perhaps* the same issue, but if it is, it's not expressed correctly.
The problem is not that "groupby on cck fields only works when all values are the same", it is that groupby adds *all fields in the query* to the GROUP BY clause, thus rendering the GROUP BY pointless.
And I don't think it's impossible to fix -- at least I am not convinced it is impossible.
The first step would be to take that big chunk of code I posted -- the monster if/elseif/elseif/elseif... block and *document it* -- that is, write a comment for EACH clause explaining what it's catching and what it's doing with it. Then we stand some chance of figuring out which of those should apply when we're adding a single groupby to the query.
Comment #9
merlinofchaos commentedThis is not wrong. According to the ANSI SQL spec, if GROUP BY is invoked, any item in the SELECT portion of the query must either 1) be an aggregate function or 2) appear in the GROUP BY.
MySQL does not enforce this rule. PostgreSQL does. Since we support postgres, we must enforce this rule.
What this means is that Views is behaving properly, and you (because you probably use MySQL) are expecting it to produce a non ANSI SQL compatible query.
It's made worse because CCK adds a bunch of additional fields whether or not you want them -- and there's a patch that's been sitting in the CCK queue for a long time to try and deal with that.
Comment #10
joachim commentedO_o
Further evidence that PostgreSQL is crack...
Comment #11
joachim commentedOkay then the way to do it is with a more complex model for how to add a DISTINCT to the query:
http://www.postgresql.org/docs/9.0/interactive/queries-select-lists.html...
-- we need the ability for the DISTINCT to affect fields other than the view's base.
Though I still maintain PostgreSQL is absolute tosh which we shouldn't bother with :/
Comment #12
joachim commented... and change the title accordingly.
Comment #13
merlinofchaos commentedUnfortunately, SQL databases don't really do DISTINCT on fields, it turns out. You get DISTINCT rows. :/ And while you can say DISTINCT(node.nid) it doesn't really work that way.
Comment #14
Iritscen commentedSo... is there a way to make this happen without relying on DISTINCT? I have a whole bunch of records from which I need to weed out duplicates. I'm open to alternative approaches, but I'm too much of a novice to know what they are. Is there a way to use a theme to perform a "distinct" filter?
Comment #15
joachim commented> Is there a way to use a theme to perform a "distinct" filter?
Absolutely. Loop over your rows in the theme before they are rendered and remove whatever you like.
Comment #16
mototribe commentedSubscribe
Comment #17
dawehnerPlease please don't move something when you subscribe. BTW. in 7.x-3.x this is not possible, because DBTNG does not support it at all.
Comment #18
merlinofchaos commentedAnd if we can't do it in 7, we're not doing it in 6.
Comment #19
sydkoh commentedSo basically Group by with aggregation functions don't work in D6???