Inconsistent behavior of data facets producing unlimited filtering, inconsistent counts, or gap mismatches [#1302844]

Comment	File	Size	Author
#42	broken-facet.png	49.48 KB	acbramley
#35	solr3.5-date.jpg	24.06 KB	nick_vh
#35	solr1.4-date.jpg	43.78 KB	nick_vh
#30	facetapi-1302844-30.patch	479 bytes	cpliakas
#24	no-filter.png	110.69 KB	acbramley
#24	double-dates.png	59.38 KB	acbramley
#20	1302844-apachesolr.patch	1.98 KB	nick_vh
#16	facetapi_dump.txt	1.26 KB	acbramley
#14	apachesolr-1302844-14.patch	1.34 KB	cpliakas
#8	date-facet.png	35.35 KB	acbramley
#7	1302844.patch	508 bytes	nick_vh
#5	1302844.patch	2.23 KB	nick_vh
#4	Search \| Site-Install.jpg	153.88 KB	nick_vh
#4	1302844.patch	920 bytes	nick_vh
#2	broken-date-facet.png	58.11 KB	acbramley
	Search \| Site-Install.jpg	15.67 KB	nick_vh

Comment #1

cpliakas commented 7 October 2011 at 19:03

Issue tags:

+Facet API integration

Adding Facet API Integration tag.

Log in or register to post comments

Comment #2

acbramley commented 26 October 2011 at 21:31

Status	File	Size
new	broken-date-facet.png	58.11 KB

I can reproduce this using a custom date field facet.

Log in or register to post comments

Comment #3

acbramley commented 6 November 2011 at 22:01

Priority:

Normal

» Major

Has there been any progress on this? We desperately need this fixed.

Log in or register to post comments

Comment #4

nick_vh

he/him

Ghent

commented 7 November 2011 at 10:52

Status	File	Size
new	1302844.patch	920 bytes
new	Search \| Site-Install.jpg	153.88 KB

I've looked into this issue and found out I need more information from the facetapi guys.

These values come from facetapi.callbacks.inc

/**
 * Maps date ranges to human readable dates.
 *
 * @param $ranges
 *   An array containing the date ranges.
 *
 * @return
 *   An array mapping the ranges to nice display dates.
 */
function facetapi_map_date(array $values) {
  $map = array();
  foreach ($values as $value) {
    $range = explode(' TO ', trim($value, '{[]}'));
    if (2 == count($range)) {
      $gap = facetapi_get_date_gap($range[0], $range[1]);
      $map[$value] = facetapi_format_date($range[0], $gap);
    }
  }
  return $map;
}

My question to the facetapi maintainers is : What is the expected format so the last value of a range is seen as a value of that range itself and not the whole range all over again?

On another note I adjusted some of the processing regarding the date ranges. When we had ranges of dates it took the previous date and added the next date as its end range. This had as effect that you were querying solr on more then 1 day (28/09/2011 till 30/09/2011)while you knew there was only a result in 28/09/2011 and 29/09/2011 and 30/09/2011 were completely irrelevant.

Log in or register to post comments

Comment #5

nick_vh

he/him

Ghent

commented 7 November 2011 at 11:25

Issue tags:

-Apache Solr Search

Status	File	Size
new	1302844.patch	2.23 KB

Further research :

case (gmdate('Ym', $start_time) != gmdate('Ym', $end_time)):
  return FACETAPI_DATE_MONTH;

Prooves that facetapi wants another syntax in the case of data from the last day of the month. It should detect it is only a day instead of a month?

Changing this seems to fix everything magically. I just need confirmation from you guys that this does not break anything in other implementations.

case (gmdate('Ym', $start_time) != gmdate('Ym', $end_time) && $time_diff != 86400):
  return FACETAPI_DATE_MONTH;

Log in or register to post comments

Comment #6

nick_vh

he/him

Ghent

commented 7 November 2011 at 11:15

Project:	Apache Solr Search	» Facet API
Component:	Code	» Code (functionality)
Status:	Active	» Needs review
Issue tags:		+Apache Solr Search

Adding Solr Tag

Log in or register to post comments

Comment #7

nick_vh

he/him

Ghent

commented 7 November 2011 at 11:27

Issue tags:

+Apache Solr Search

Status	File	Size
new	1302844.patch	508 bytes

And better/cleaner patch

Log in or register to post comments

Comment #8

acbramley commented 7 November 2011 at 20:46

Status	File	Size
new	date-facet.png	35.35 KB

After applying your patch I'm still getting the same problems in some cases :( What's really weird is I get a piece of content returned with a date of 1st December 2008 and my facet is showing November 2008 as an option (see screenshot).

Log in or register to post comments

Comment #9

acbramley commented 7 November 2011 at 23:52

What version of facetapi is your patch in #4 for? Neither the --dev or beta7 have the file query_type_date.inc under plugins/facetapi

Log in or register to post comments

Comment #10

acbramley commented 7 November 2011 at 23:55

Aaaah never mind that patch was for solr.

Log in or register to post comments

Comment #11

cpliakas commented 8 November 2011 at 02:01

The patch in #7 seem to be logical, I just want to wrap my head around it to make sure it doesn't cause any regressions. I think it would be really helpful to write a unit test using a dummy adapter the test the dat range thing. I think that will be critical to make sure that dates remain solid going forward.

Log in or register to post comments

Comment #12

acbramley commented 8 November 2011 at 02:22

@cpliakas the patch in #7 did not fix the broken facets for me.

Log in or register to post comments

Comment #13

cpliakas commented 8 November 2011 at 13:00

@Zombienaute, thanks for testing.

Log in or register to post comments

Comment #14

cpliakas commented 8 November 2011 at 13:22

Status:

Needs review

» Needs work

Status	File	Size
new	apachesolr-1302844-14.patch	1.34 KB

So I tested this out on my local instance and I cannot replicate this issue. I did a diff between the 7.x-1.0-beta7 and 7.x-1.0-beta10 release of Apache Solr, and this is the only change I found. Not sure why this would effect anything, but apply it and see if it works anyways.

A coupe of questions...

1) Has the content been reindexed lately?
2) This use to show up when we weren't handling tomezones correctly. Everything has to be normalized to GMT in the index and on display.
3) What versions of Facet API and Apache Solr are being used?

~Chris

Log in or register to post comments

Comment #15

cpliakas commented 8 November 2011 at 13:35

Status:

Needs work

» Needs review

What I am very interested in is the dsm() of the $build variable returned by ApacheSolrFacetapiDate::build(). That will ve very instrumental in narrowing down the root cause of the issue.

Log in or register to post comments

Comment #16

acbramley commented 8 November 2011 at 20:56

Status	File	Size
new	facetapi_dump.txt	1.26 KB

1) Currently developing this site so must've reindexed the content at least a dozen times in the past week with no change (index has over 1000 pieces of content in it).

2) The field in question is a Date (ISO Format) field using no time zone conversion because it only collects Year, Month, and Day with no end date and something in the date module doesn't let you save a date field with time zone conversion when it only collects those attributes.

3) I'm running beta10 of solr and the latest dev release of FacetAPI.

The attached file contains a dump of $build for 1 search (it seems to be called multiple times).

Log in or register to post comments

Comment #17

cpliakas commented 11 November 2011 at 21:03

Status:

Needs review

» Needs work

Marking as needs works since we haven't pinpointed this yet.

Log in or register to post comments

Comment #18

acbramley commented 16 November 2011 at 01:33

Any progress on this? We desperately need this fixed.

Log in or register to post comments

Comment #19

cpliakas commented 16 November 2011 at 14:40

Unfortunately I cannot replicate, so it is really hard to debug. This issue will be resolved as quickly as there are contributions. I would also take a look at the Apache Solr module, as that is where the data calculated are being calculated. Keep in mind that Facet API doesn't perform the facet calculations, it simply takes the data returned from the server and displays it via the widgets. Judging from the output provided above, Facet API is displaying exactly what is returned by the backend.

Log in or register to post comments

Comment #20

nick_vh

he/him

Ghent

commented 16 November 2011 at 18:00

Status	File	Size
new	1302844-apachesolr.patch	1.98 KB

Zombienaute, please try following patch.

I was easily able to replicate and the following patch for apache solr search integration tries to fix it.
The reason it broke is that it did not have any mechanism to stop the date facetting when facetapi decided to stop facetting. Eg.: There is a limit in date filtering and it is set to minute. When this function did not return a new "gap (say, second)" it tried to query solr and break.

Anyway, let me know if it fixes this problem for you.
I also added some new logic so it would be more corresponding to the the actual filter .

+        $from = $value;
+        $to = facetapi_isodate(strtotime($value . $gap));
+        $new_value = '[' . $from . ' TO ' . $to . ']';

This only filters for items within the real gap range, it used to filter for the current to the next item gap range.
ATTENTION: Patch is for apachesolr search integration. The patch in #4 is probably still valid as it solved a use case for me

Log in or register to post comments

Comment #21

acbramley commented 16 November 2011 at 21:45

That seemed to fix it, there's still some strange things going on but I think they're unrelated. Cheers!

Log in or register to post comments

Comment #22

nick_vh

he/him

Ghent

commented 16 November 2011 at 21:50

Status:

Needs work

» Needs review

I guess we should commit #7 first to facetapi and then move this issue over to apachesolr search integration again

Log in or register to post comments

Comment #23

cpliakas commented 16 November 2011 at 22:07

Give me a day or two to wrap my head around all this.

Thanks for everyone's work on this,
Chris

Log in or register to post comments

Comment #24

acbramley commented 16 November 2011 at 22:10

Status:

Needs review

» Needs work

Status	File	Size
new	double-dates.png	59.38 KB
new	no-filter.png	110.69 KB

Sorry, the patch in #20 caused another couple of issues:

1) Dates that aren't part of the returned results are showing (see double-dates.png attachment, there's a filter labelled November 15 when the only result is one with November 16 as the date)
2) Drilling down to last facet loses all filtering (see no-filter.png, filtered down to 12:00AM, now all results are showing)

I've tested removing the patch and these issues go away, so setting back to needs work.

Log in or register to post comments

Comment #25

nick_vh

he/him

Ghent

commented 21 November 2011 at 10:39

I'm a bit in the dark here on how to solve this because the facetapi_get_timestamp_gap has a minimum gap of seconds and it seems like we can't change this. Maybe Chris can give us some guidelines on how to resolve the problem of keeping the facets available if the maximum gap is different then the default?

$gap =  facetapi_get_timestamp_gap($start_time, $end_time);
$next_gap = facetapi_get_next_date_gap($gap, FACETAPI_DATE_MINUTE);
if ($next_gap == $gap) {
  $next_gap = NULL;
  return;
}
...

For this code we need to find a correct return value so the items that are selected stay in the date block and don't dissapear if there is nothing to facet for anymore.

Log in or register to post comments

Comment #26

acbramley commented 21 November 2011 at 23:09

Further to this:

Clicking the incorrect month (i.e content is set to 1st December and clicking the November facet) returns a Fatal error:

Fatal error: Maximum function nesting level of '100' reached, aborting! in /home/adamb/projects/mohpub/sites/all/modules/contrib/facetapi/plugins/facetapi/adapter.inc on line 1236

Log in or register to post comments

Comment #27

acbramley commented 21 November 2011 at 23:15

Applying #7 seems to have stopped the fatals at least but still getting the extra facets that shouldn't be there.

Log in or register to post comments

Comment #28

cpliakas commented 3 December 2011 at 02:45

So I was able to replicate after generating 10,000 nodes. Debugging now and testing patches.

Log in or register to post comments

Comment #29

cpliakas commented 3 December 2011 at 02:57

So this is interesting... looking at the facet.date.include documentation for Solr it looks like both the upper and lower bounds are inclusive. This might mean that if you are filtering for January 01 to February 01, then dates from February 01 are returned. Trying out some different variations to get around this.

Log in or register to post comments

Comment #30

cpliakas commented 3 December 2011 at 04:19

Status:

Needs work

» Needs review

Status	File	Size
new	facetapi-1302844-30.patch	479 bytes

Disregard comment in #29, doesn't look like that has anything to do with this.

After some investigation, it looks like the issue here is in the gap calculation for date ranges on the last day of the month. When evaluating the date range "2011-01-31 TO 2011-02-01", the gap should be "DAY". Due to a logic error, it is evaluating to "MONTH" because the code checks for differences in the numeric representation of the month as opposed to the time. The attached patch calculates the number of seconds in a given month which fixes this issue.

Log in or register to post comments

Comment #31

acbramley commented 4 December 2011 at 20:46

Component:	Code (functionality)	» Code
Status:	Needs review	» Needs work

Tested this patch and still getting the same issue of 2 months showing with 1 result.

Log in or register to post comments

Comment #32

cpliakas commented 4 December 2011 at 20:50

Is the date field you are working with a date field or one of the core node created / published properties?

Log in or register to post comments

Comment #33

acbramley commented 4 December 2011 at 20:52

A date field I've created using "Date (ISO format)" The field only uses Year Month Day components and no end date.

Log in or register to post comments

Comment #34

cpliakas commented 4 December 2011 at 21:05

Useful information. Will try to replicate that setup.

Thanks,
Chris

Log in or register to post comments

Comment #35

nick_vh

he/him

Ghent

commented 10 December 2011 at 12:55

Status:

Needs review

» Needs work

Status	File	Size
new	solr1.4-date.jpg	43.78 KB
new	solr3.5-date.jpg	24.06 KB

Some more information regarding this unlimited date issue

I was testing a patch regarding dates for apachesolr and I figured that this problem appears in Solr 3.x but it does NOT in Solr 1.4
For everyone involved in this issue I would recommend you to take a Solr 3.5 instance and test with this.

My guess is that Solr 3.5 offers a much wider range of dates to be filtered on and that somehow we misinterpret the range? When I can (Monday) I will make a dump of the whole process for the 1.4 and for the 3.4. This should get us closer to a resolution and maybe the patch will be in apachesolr, maybe in facetapi.

Log in or register to post comments

Comment #36

nick_vh

he/him

Ghent

commented 10 December 2011 at 12:54

Status:

Needs work

» Needs review

Patch #20 on apachesolr.module and patch #30 on facetapi seem to fix this issue for me (Solr 1.4 and solr 3.4)

Log in or register to post comments

Comment #37

cpliakas commented 10 December 2011 at 16:53

Status:

Needs work

» Needs review

Thanks, Nick. Still trying to replicate. I think the patch in #30 is still valid and a good change anyways, so could you possibly confirm that the changes you mentioned in #36 are compatible with it?

Thanks,
Chris

Log in or register to post comments

Comment #38

nick_vh

he/him

Ghent

commented 10 December 2011 at 17:01

Not sure if I understand your comments but the issues disappear when I apply both patches (#20 and #30) for Solr 3.4.
I would commit #30 for facetapi and then we can continue this issue in the apachesolr issue queue to see if this needs further work or committed as-is

Log in or register to post comments

Comment #39

cpliakas commented 11 December 2011 at 17:12

You don't understand my comment because after re-reading #37 doesn't make sense :-P. I misread your post. Disregard, and I will commit #30.

Thanks,
Chris

Log in or register to post comments

Comment #40

acbramley commented 11 December 2011 at 20:46

I'm using Solr 3.1 and the problem doesn't seem to be fixed for me.

Log in or register to post comments

Comment #41

nick_vh

he/him

Ghent

commented 12 December 2011 at 10:16

@Zombienaute
Can you list all the specs of your setup again please, so we are able to replicate? Maybe even a dump of the data to fill the fields with? If we are not able to replicate, we are not able to fix your issue.

A summary of the information I obtained from the issue

Config

A date field using "Date (ISO format)" The field only uses Year Month Day components and no end date.
?Solr 3.4?

Issues
?

Data
See http://drupal.org/files/facetapi_dump.txt

Log in or register to post comments

Comment #42

acbramley commented 12 December 2011 at 20:51

Status	File	Size
new	broken-facet.png	49.48 KB

Yup that's the correct config (I was using Solr 3.1 but just upgraded yesterday to 3.4)

The issues are that I'm getting 2 days/months when there are single result searches. For example, if I facet down to a piece of content that has a date of 1st of February 2011 then the facet will show go:

(-) 2011
January, 2011
February, 2011

You can see this in the screenshot.

I'm running apachesolr beta11 with patch #20 and facetapi 1.x-dev from 18th November with patch #30 applied.

Log in or register to post comments

Comment #43

cpliakas commented 14 December 2011 at 20:52

Thanks for the screenshot. That seems a little weird. This is a tough issue to replicate, and it seems the back and forth is unproductive. I we can't obtain a copy of the site then we are debugging in the dark, so we either need some help with the leg work in identifying / fixing the issue or need to be able to spin up a local copy of the site to replicate.

I am going to commit #30 because I think it it a positive change regardless.