Allow "fuzzy" granularity

Marked #259702: Representing inexact historic dates as a duplicate, in which whereisian had posted

I'm not sure if this is a feature or support request.

I work with humanities researchers and deal with a lot of dates represented in inexact terms. Sometimes they are centuries, quarter centuries, decades, year range, etc.

With dates2, I can now represent historic dates more easily, but I am still unable to have one field handle several levels of granularity. For instance, some nodes will a precision to the day, other to the year, others to the century. When I try and enter a year range with a granularity of y-m-d, the dates are stored with a seemingly random value.

The work around seems to be using the a precise date range to represent an unknown period. So to represent a century, you'd enter say 1800-jan-01 to 1899-jan-31. Is it possible to allow some date fields to be blank?

Circa dates can also not currently be expressed. It occurs frequently that only one date of a range is known, such as an unknown birth date, but known death date. Would a checkbox to represent a 'circa' date be implemented?

Any comments are appreciated.

Cheers

There have been a number of request for this, but, as you noted, it's not that easy to do. It's not at the top of my todo list, but if someone needs it and creates a patch, I'll look at it.

Marked #262646: Empty/Incomplete dates as a duplicate of this request.

I'm going to mark #148118: 'variable' granularity with '00' padding as a duplicate of this as well.

Marked #271680: Dynamic granularity as a duplicate, in which phayes suggests "having the date system parse the input to determine granularitry. It would require an extra column that defines the granularity in the database structure."

Should be version 6, not version 7 :)

The need for "fuzzy" granularity has come up several times for me recently. Would love to see this feature. Thanks to all the contributors to this module; wish I had the skills necessary to write this patch.

this is an interesting feature request. i don't have time to work on it at the moment, but i'd like to see some more use cases for this to get a better sense of the full requirements. the cases reported in the requests above are a good starting point, but i'd like to hear more about edge cases like the need for historical "circa" dates.

Marked #134820: Allow variable granularity on date/times as a duplicate, and changed version per KarenS in that issue.

subscribe

Marked #311540: Allow year OR year+month OR year+month+day as a duplicate of this feature request.

michaek:

this is an interesting feature request. i don't have time to work on it at the moment, but i'd like to see some more use cases for this to get a better sense of the full requirements.

I want to create a chronicle of past events.
So let's say, an association celebrates bicentenary. Documentation was not good all time. So it is not exactly know, on which day the association was found; only the year is known:
1800: foundation of the association
The next known event is the first mention in a local newspaper, here the exactly publishing date is known:
1800-07-23: mentioned in local newspaper
The managing committee changed later, but the exact day is unknown, it's only proved, that this has happened at the yearly association meeting in december:
1801-12: new managing committee

So, maybe we get something like:

• 1800: foundation of the association
• 1800-07-23: mentioned in local newspaper
• 1801: relocation of the association office
• 1801-12: new managing committee
• 1801-12-20: associations Christmas ceremony

And here you can see the way, the dates should be sorted.
Year only should be interpreted (internally!) as: 1801-00-00
Year and month as: 1801-12-00

Maybe that's a (probably dirty) way of implementing this?
Fill "missing" date fields with "00", but don't show it in the ouput.

Subscribing - I need this too!

I need this too.

In addition to the issue of uncertain historical dates previously raised, I need to account for variable periodicity in certain types of nodes. For example, one of my clients has a series of papers whose publication schedule has varied over the decades; some are month-year like "December 1991" and others have a full date like "13 September 2001." There is currently no way to use the same date field to represent both as publication dates; the former gets recorded as "1 December 1995" which is not accurate, and which can lead to some confusion (e.g. a paper dated 1 April reporting on a conference that didn't even start till 4 April). And of course, many papers were in fact published on the first of the month, so we cannot assume any instance as being auto-inserted.

There is even more potential for variability in cases where granularity is set for hours or minutes. I set up each event at a conference as a node so that related video, lecturer bio, papers, transcripts, etc. can be attached easily. But the exact time of a presentation might not be known until just a few weeks before the conference. They are recorded as occurring at midnight in the meantime. We can mask this at the theme level, as no conference events occur after 9pm, but the same could be said of sites storing showtimes, tour itineraries, and any number of other things that occur throughout the 24-hr day.

Marked #342844: Partial dates as a duplicate of this issue.

Cross-referencing two possibly related feature requests:

• #104287: Time without a date feature
• #219794: Allow dates without years.

I need partial dates as well, and may be able to contribute to implementing it.

My project needs to store dates for old artifacts, mostly photos and documents. In many cases only the year, or month and year, are known. On the other extreme, we also have modern "artifacts" that have precise to-the-second dates.

From a coding perspective, it seems that we need another field that stores either (a) some indicator of precision (circa,century,decade,year,month,day,hour,minute,second) or (b) a second, partial representation of the date. For example, ISO 8601 allows for leaving off values that are not known. (c) a second date field that marks the other end of the range. I lean towards (a) or (b), as (c) could be implemented, if clumsily, with two date fields.

For my purposes, it seems that the interface should allow the user to enter as much information as they have, and derive the precision from this. So if someone enters only "1943", then this is stored as having year accuracy, "July 1943" has month accuracy, etc. (Not sure how would tell decade or century accuracy, however. E.g. '1900' or '2000' would be ambiguous.)

I looked into the module code a few weeks ago and the situation didn't look good: many separate functions that pass around only fully qualified dates. I'll take another look. I'd love to hear from anyone with more CCK coding experience as I've only done 'normal' modules before.

I've had some time to work on this and have a rough version working. Attached is a patch file for date 6.x-2.0-rc6 that demonstrates what is working so far. I'm no CCK coding expert so right now the goal is merely something that works. If anyone has time, I would love feedback on the direction this is going. This is only for evaluation, do not use on production sites.

The first issue is how to indicate that you want to accept fuzzy dates. For now, I've accomplished this by recognizing a special 'fuzzy flag' in custom date formatting: Set the 'Custom Input Format' and 'Custom Display Format' to 'F d, Y - H:i \X'. The '\X' at the end is the signal that fuzzy versions should be accepted.

So there are three levels of granularity accepted: Day, e.g. "March 14, 1945"; Month, e.g. "March, 1945"; and Year, e.g. "1945"

What do you think is the ideal way to do this? Should we add an option under Granularity that is "Fuzzy". However, there may be problems in 'fuzzy parsing' of arbitrary date input formats. If the input is simply "05/05", it will be tricky to get the system to recognize that those must be month and year. It would be even more difficult to recognize levels of granularity not expressed by the PHP date format, e.g. centuries, decades, seasons, quarters, weeks, etc...

The second big issue is how to store the granularity/precision within the database. For now, I'm using a clever hack: the precision is encoded within the seconds field. So the seconds field is ignored, and instead is used as an indicator of precision. I'm using this rather arbitrary mapping: 59=minute, 58=hour, 57=day, 56=week, 55=month, 54=season, 53=quarter, 52=year. So, for example, the date "March 1945" is actually stored in the database as "1945-03-01T00:00:55". When this date is displayed, the "55" at the end tells the display code to only show the year and month, so only "March 1945" is shown. The nice thing is that the data is still valid dates and can be re-used or exported without special treatment.

Another possibility, perhaps more elegant, would be to store a fuzzy date as a date-range. However, this would require more complex mucking with the field widget which is beyond my drupal skillz at the moment.

As an aside, my project has a lot of data in "seasonal" dates, e.g. "Summer 1945." I'm also working on getting the module to recognize these sorts of fuzzy dates. It seems that this would also be useful for business data where something may be dated in quarters, e.g. "Q3 2008".

Updating status.

Oops, wrong version.

Subscribed

Hi Stan,

thanks for starting to work on this issue; for the moment just a question: What about a fourth level of granularity: something like "3rd november" (independent of a year) or even a Computus (calculation of the date of Easter in the Christian calendar, meaning: dates calculated in relation to other dates, like solstices and equinoxes)? Is this too complex?

Thanks & greetings, -asb

What do you think is the ideal way to do this? Should we add an option under Granularity that is "Fuzzy". However, there may be problems in 'fuzzy parsing' of arbitrary date input formats. If the input is simply "05/05", it will be tricky to get the system to recognize that those must be month and year. It would be even more difficult to recognize levels of granularity not expressed by the PHP date format, e.g. centuries, decades, seasons, quarters, weeks, etc...

I was going to suggest making granularity as precise as what the user entered. E.g., if the user enters May 2005, then only make it as precise as that. Not sure there's a need for more granularity. However, you still need to save as a date code of some format, and I'm not sure there's a way to specify "no value" instead of, say, "00" in a field.

Well, it may be possible if you want to roll your own date comparison system (would be easy if this was object oriented--argh, Drupal!) with separate fields for each portion of the date, and each field can have a value or an accepted "no value" value.

The second big issue is how to store the granularity/precision within the database. For now, I'm using a clever hack: the precision is encoded within the seconds field. So the seconds field is ignored, and instead is used as an indicator of precision. I'm using this rather arbitrary mapping: 59=minute, 58=hour, 57=day, 56=week, 55=month, 54=season, 53=quarter, 52=year. So, for example, the date "March 1945" is actually stored in the database as "1945-03-01T00:00:55". When this date is displayed, the "55" at the end tells the display code to only show the year and month, so only "March 1945" is shown. The nice thing is that the data is still valid dates and can be re-used or exported without special treatment.

Sounds OK. If you get to the second, who would use fuzzy granularity? Only problem I see is that when sorting by date, fuzzy granularity items will probably be sorted at the beginning of the time period they describe. E.g., if I specify June 2009, it will probably be sorted as if it's 06-01-2009 00:00:00. So that seconds field stuffing could cause arbitrary sorting if we rely on PHP's date handling features.

Should there be a separate field to specify degree of granularity? Or use the "roll your own" date system mentioned above?

(removed my comment that probably was erroneous)

We will have to require that fuzzy dates be stored as ISO dates, the only type that will handle this properly. ISO dates have a built-in way of handling fuzziness -- you just use zeros for any unused date parts, so a date with only a year and a month will be 2009-01-00T00:00:00, and year only is 2009-00-00T00:00:00. This makes it very clear how precise the date is and the values will also sort correctly, plus when there is a standard for things like this, we use it. So I don't want to use a non-standard system of adding arbitrary numbers to indicate fuzziness.

The other issue is how to display the value. This will probably require a function that can interpret the 'accuracy' level of the stored date value and only display the right date parts. So if the format code is m/d/Y H:i and the date only has a month and a year, it would display m/Y. Code to do things like limit a date format to a requested granularity is already in the Date API, so we can already do that part.

Interpreting something like 05/05 is a very big problem indeed, one of the primary reasons why we can't do fuzzy dates now. We can't gloss over that issue, it must be solved in whatever solution we come up with. There is no way we can come up with a system that could accurately guess what that should mean in any situation in any country (depending on the country and situation it could mean YY/MM or MM/DD or DD/MM or MM/YY). So there is no way we can just accept anything the user inputs and assume we know what we have.

I think we will have to tell the user what the expected format is, and then accept blank values for part of the date, in that format. That way we can be sure what 05/05 means. So I would have a date that says the format is 'MM/DD/YYYY HH:MM' but will accept something like 05/00/2008 for a month and year in a text field, or with a select list a blank value for the everything but the month and year. Then the field definition would have to have a flag to indicate that fuzziness is allowed so it doesn't set a validation error or blank out the results because it isn't a valid date. You can't use the Date Popup calendar for fuzzy dates because the javascript code will choke on anything other than a valid date, and we won't allow repeating dates for fuzzy dates because the RRULE code and date repeat logic expects valid dates and won't work right without them. And I don't think we'll allow timezones for fuzzy dates because applying timezone conversions to fuzzy dates wouldn't make any sense and would probably bomb in all kinds of ways.

So this could be a new kind of date field rather than trying to mix this into the way existing fields work. It would be an ISO date with just two possible widgets -- textfield and select -- and no timezone conversion. Since it's a different field, it could (and maybe should) be created as a separate field module rather than trying to shoehorn this into the existing date fields. That would simplify the logic quite a bit.

ISO Dates are certainly the way to go--the only date standard that can encode precision. One minor point: from wikipedia and w3c, it seems that partial dates are represented in ISO 8601 by leaving out the unknown parts, not by filling with zeros. (Presumably because zeros would be ambiguous in the time field, e.g. in the ISO date "1997-07-16T19:00:00", one couldn't be sure if minutes and seconds were unknown or actual zeros.) So a date with only a year and a month will be merely "2009-01".

I'm a newcomer to the Date Module codebase, but it was clear that a date was converted/processed multiple times in the normal flows of saving or displaying. I believe that every date gets converted to a PHP Date object at some point, and AFAIK, these objects have no inherent way to represent fuzzy dates. My point is that it would be a significant re-write of this module to support fuzzy dates without a hack like my patch uses. It seems this require modifying the flow to keep the date represented as an ISO string throughout all processing, or creating a "sidecar" data object alongise the php object to record date precision.

Karen, you certainly know the code better than I. If you want to undertake such a re-working, or if you see a better way of doing it, I'm happy to help out in either case.

_{(As an aside, my project is also a tool for Elders; cool to find ElderWeb.)}

Leaving out the values is an option if we make this into a different field type, it is not an option in the current fields. I have read in other places that it is also valid to fill zeros in for the unknown values. Either way, we can tell how precise the value is, so either will work.

The current code does lots of date validation, but it *does* accept ISO dates with missing values (you can do year-only or year-and-month-only dates now and they work). So long as we have at least a year in an ISO date, and the missing parts are padded with zeros, the date will work in the current code. I don't think anyone is suggesting that the fuzzy date would not have at least a year (i.e. it looks like the 'missing' parts will be the less precise parts of the date), so this is at least theoretically possible.

However, as I noted above, we may really want to create a new date field that doesn't use all the date validation and timezone conversion done in the current code. It might be the easiest way to do this and the code would probably be easier to follow.

I need to mull this over a bit...

Interpreting something like 05/05 is a very big problem indeed, one of the primary reasons why we can't do fuzzy dates now. We can't gloss over that issue, it must be solved in whatever solution we come up with. There is no way we can come up with a system that could accurately guess what that should mean in any situation in any country (depending on the country and situation it could mean YY/MM or MM/DD or DD/MM or MM/YY). So there is no way we can just accept anything the user inputs and assume we know what we have.

For the sake of intuitiveness, should fuzzy date/times have separated fields for each time unit in order of increasing granularity? Year first, seconds last. That way there's no guessing or working through the American date representation convention where order of display does not correspond to order of granularity. Also no need to use 00. Just leave undefined granularity blank.

A good approach for parsing fuzzy dates may be to use successive calls to the date_limit_format() function in date_api.module. This function is passed a granularity and a php date format string, and returns a new format string limited to that granularity. The date field would have a specified format (as now) but allow the user to enter a fuzzy string; this string would be parsed by successive calls to date_limit_format(), moving from more precise to less precise until it finds a format which fits the user's string.

For example, if the format for the date field is "F d, Y - H:i" and the user enters "July, 1942", the system will try increasingly fuzzy formats:
"F d, Y - H:i"
"F d, Y - H"
"F d, Y"
"F, Y"
...At this point it will get a match, and record that the date has only 'month' granularity. This would be converted to an ISO date of "1942-07" (or "1942-07-00T00:00:00" if we use the zero format.)

This approach solves KarenS's "05/05" problem in an at least predictable way. If the format is "n/j/y" (American date format) and the user enters "03/05", it will be parsed as March 2005 (and not May 2003). It is then up to the site administrator to provide explanation to users as to what format is being used for parsing.

Hi,

Has any more progress been made on this? It seems that if the current version of Date allowed for the entry of a zero value for day and/or month, it would address most people's needs. Internally a sort would detect the granularity from the presence of these zeros.

But of course I'm not a programmer. Obviously this is an easier said than done case.

Hi wanderingStan. I'd love to give you some feedback on your patch. But I've never installed a patch before. So I have a couple questions.

I am on a Mac and attempting to do this via terminal. I THINK the file I want to patch is the date.module inside date, which is in the date top directory. This is where I put date_fuzzy.patch

So in terminal (on the Mac) I change to the directory:
JW-Pwrbook:~/Desktop/date jw$ cd /Users/jw/Desktop/date/date
Then I type:
patch < date_fuzzy.patch

I get a bunch of errors.

So then I said, "Lets install it into the base date folder. I did the same procedure to a fresh download of date 6.x-2.0-rc6 and placed date_fuzzy.patch in there. This time I get:

can't find file to patch at input line 4
Perhaps you should have used the -p or --strip option?
The text leading up to this was:
--------------------------
|diff -urp d:\Temp\date/date/date.module .\date/date/date.module
|--- d:\Temp\date/date/date.module 2008-12-02 10:32:35.000000000 -0700
|+++ .\date/date/date.module 2008-12-18 17:14:40.359375000 -0700
--------------------------
File to patch:

Am I patching wrong? Does date_fuzzy.patch need to be renamed? I'd love to test this and help if I can, but I haven't got past square one.

well I got patching sorted out. I needed to put date.module and patch inside the same directory and run the patch.

I went through the process. replaced the module with the patched one and could not get year or month/year to work with my day.month.year granularity.

Is there anymore development going on in this regard?

Interesting issue.

One problem with the ISO format is that it can't handle the kind of phrases that often come up in conversation, like "in the 1980s", "in the 1800s", "in the first half of the 20th century", "one cold winter around the turn of the century".

Regexes, however, can. The above all become something like:
198./.*/.*
18../.*/.*
19[0-4]./.*/.*
(199.|200.)/(11|12|1|2)/.*

Just a thought. Can't see anything such vagueness would be useful for, really, other than stuff like timelines. And for most cases, simple date ranges do them just fine already.

subscribe

Dewi: From my reading of the ISO spec, some of the phrases you mention *can* be represented. The crux is this detail

For reduced accuracy, any number of values may be dropped from any of the date and time representations, but in the order from the least to the most significant.

So, for example, "in the 1980's" would be "198", and "in the 1800s" would be "18".

But this still doesn't help all issues. I have a lot of date data expressed in strings like "Spring 1947", and I know others have dates in terms like "2007 Q4".

I agree that we should stick to the ISO spec as far as it will go, but I would like the potential for other more esoteric date representations.

subscribe - #12 gives a good idea of my requirements

This would be perfect for what I want to do - I am constantly getting more information on dates of events that can start off as just a year and end up with an exact date. Same as #12

Wow, this request has been going on for almost a year now. Is there any progress on this?

Will this likely be implemented for Drupal 6 in the next time?
If not, are there any suggestions, how you might get this (or similar) on a different way?

subscribing

Subscribing

Subscribing

subscribing

This is not really a working patch, nor has anything been reviewed, so it is not ready to be included in the code.

subscribing

2KarenS: In your opinion can it be realized by custom module instead?

I see it as 'fuzzy date' widget that stores 'from' and 'to' dates of selected period and then displays it in suitable manner. Am I on the the right track?

What no2e said is already how DATE and DATETIME column values are handled in MySQL (allowing zeroes in any particular date sub-field to act as a wildcard: e.g., '2009-12-00' means December 2009 as the entire month, without specifying a specific day).

If Drupal were MySQL-only, this would therefore not be too difficult. But since it supports multiple databases (PostgreSQL in 6.x, any PDO-compatible database in the upcoming Drupal 7, if I understand correctly), we cannot rely on this.

Can we not say that this feature requires MySql until it can be developed further?

@#49: That would be fantastic.

subscribe

I also need this to work for birthdates and deathdates that are not always known exactly.

+1 for mysql only

Would it even be acceptable to have a Module be database-specific? Is it not a requirement that all Drupal modules use the Drupal database libraries (if they do database access at all), which currently allow either MySQL or PostgreSQL, and may or may not allow database-specific features?

As I understand it, Drupal 7 will use PDO, which will allow many more databases, including Microsoft SQL Server. I know for a fact that MS SQL Server, at least as of 2005 (2008 may have changed this), does not allow zeroes in date parts (it does in time parts, of course, but only as actual zero values).

Subscribed.

Subscribing. Need the functionality described in #12.

I have books and articles in something of a library. Books have years, whereas articles have years, months and days.

Any progress on this?

Thanks

subscribing

This would be very handy indeed. I have a content type for music releases (albums, DVDs, EPs, ...) which should also be able to contain upcoming releases. More often than not, only a month and/or a year is known in that case. I am tempted to investigate the applicability of wanderingstan's patch, despite his express warning not to use it in production, because having to fill in a bogus date is hardly user friendly.

Subscribing!

Subscribing

Subscribing
Similar to #57 but for Book Publication data, often Month and Year only

Subscribing.

I have a custom node type for records that date back to the early 1900's; except some of the records are incomplete and only have their given year. Anything other than the full date DD/MM/YY throws an error on submit.

Subscribing. This would be great in conjunction with the timeline module.

Looks like it may be a while before a solution is found. Meanwhile, has anyone come up with a good workaround with this or any other modules?

How is this a support request?

Subscribing

Subscribe.
Going to store values in the database with 00 in a bulk operation, if that value is null. Going to look into displaying it so it cuts off the day or day/month. Do you foresee any issues by doing this (00 for null) with views integration? I'm side stepping the validation issue here.

Right now this stored in the database
1866-12-00T00:00:00
Outputs this
Saturday, December 1, 1866

Would like it to output
December, 1866

Subscribing - Same problem as everyone else. #12 describes it nicely. Will try the patch in #19