Create a CCK date field mapper

I need this feature. I'm not sure if I have the time this week, but I would like to give it a try. I will contact you by email for advice.

Here's my starter at it.

As it stands the code works with the timestamps created by the parsers [1] and with textual dates in feeds [2].

I'll keep working on it soon -
* ensuring the text parsing (see below)
* looking into timezones
* multiple dates for a single field (this would also be the case with iCal start and end times)
* other possible arrays
- that is unless someone else wants to ;-)

[1] The parsers are inconsistent about what information they create the timestamp from:

parser_simplepie - uses get_date() - which returns in order of preference:
ATOM_10: published - updated
ATOM_03: issued - created - modified
*: pubDate - (DC11):date - (DC10):date,

parser_common_syndication - returns in order of preference
ATOM: published
RSS1.0/RDF: (DC):date - pubDate
RSS0.91/2.0: pubDate

[2] This is not rigorously tested (yet). List follows of RSS and ATOM date formats that are in the documentation (not counting other extensions and badly formed feeds). iCal also needs to be added:

== RSS and Atom Dates ==

(in RSS item / Atom entry - not at channel level)

=== RSS 0.90 ===

Has no date format

=== RSS 0.91+ ===

* pubDate (RFC 822) - Publication date

=== RSS 1.0 + ===

No date format in specification http://purl.org/rss/1.0/spec

Official Module Dublin Core http://purl.org/rss/1.0/modules/dc/ has
* dc:date (W3CDTF) - Publication date

Unofficial Modules:
* Aggregation http://web.resource.org/rss/1.0/modules/aggregation/
* ag:timestamp (ISO 8601)
* Audio http://web.resource.org/rss/1.0/modules/audio/
* audio:year (YYYY)
* Context http://nurture.nature.com/rss/modules/mod_context.html
* various not 100% clear from above
* Qualified Dublin Core http://web.resource.org/rss/1.0/modules/dcterms/
* dcterms:* (see below)
* e-mail http://web.resource.org/rss/1.0/modules/email/
* email:date
* event http://web.resource.org/rss/1.0/modules/event/
* ev:startdate (W3CDTF) Start Date (timezone can be implied by ev:location!)
* ev:enddate (W3CDTF) End date, optional
* prism http://nurture.nature.com/rss/modules/mod_prism.html
* prism:coverDate (W3CDTF)
* prism:coverDisplayDate
* prism:creationDate
* prism:embargoDate
* prism:expirationDate
* prism:modificationDate
* prism:publicationDate
* prism:receptionDate
* RSS 091 http://web.resource.org/rss/1.0/modules/rss091/
* rss091:pubDate (RFC 822)
* rss091:lastBuildDate
* Service Status http://web.resource.org/rss/1.0/modules/servicestatus/
* ss:lastChecked (W3CDTF)
* ss:lastSeen
* Streaming http://web.resource.org/rss/1.0/modules/streaming/
* str:live.scheduledStartTime (W3CDTF)
* str:live.scheduledEndTime (W3CDTF)

=== RSS 2.0 ===

As RSS 0.91

Could be extended, probable ns dc: and dcterms: (below)

=== Atom 1.0 ===

* published (RFC3339) http://atompub.org/rfc4287.html#element.published time early in lifecycle of entry
* updated (RFC3339) http://atompub.org/rfc4287.html#element.updated most recent significant modification

=== Dublin Core ===

As in RSS 1.0 (possible in 2.0 and atom)

If added with name spaces as:
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:dcterms="http://purl.org/dc/terms/"

Recommended(!) to be using W3CDTF

* dc:date

* dcterms:created
* dcterms:valid (could be a range)
* dcterms:available
* dcterms:issued
* dcterms:modified
* dcterms:dateAccepted
* dcterms:dateCopyrighted
* dcterms:dateSubmitted

=== SSE ===

xmlns:sx="http://www.microsoft.com/schemas/rss/sse"
For RSS http://msdn2.microsoft.com/en-us/xml/bb190613.aspx

Elements:
• sx:history
• sx:update
• sx:conflict
Can all have _attribute_ when. Could be nested.

I guess these might actually end up in an array return rather than as text?

=== Formats ===

RFC822 (RFC1123 RFC2822)
W3CDTF http://www.w3.org/TR/NOTE-datetime
ISO 8601 http://en.wikipedia.org/wiki/ISO_8601
RFC3339 http://tools.ietf.org/html/rfc3339 ISO 8601 subset

Works for me!

Thanx for that feature. Saves me a lot of time!

Alex

This is great.

Have you started on iCal yet (see http://drupal.org/node/214688)? If not, perhaps I can find some time to take a look at it over the next weekend.

Right, I'd not seen the feedapi_mapper_content_is_cck_type() before (should cvs up more often :-/ )
The date mapper does do a check for type though:

  $field = content_fields($field_name);
  if (! ($field['type'] == 'date' || $field['type'] == 'datestamp')) {
    // if not a date just return
    return;
  }

It is actually useful doing it this way because then the $field is available to the rest of the mapper (see further down the mapper code).

The code in the feedapi_mapper_content seems to cycle through all the possible fields for the content_type, not just the one for the field (which is already known). Could it not become simpler and return array or false for the field at hand like above? So it would become something like:

function feedapi_mapper_content_cck_field($field_name, $field_types) {
  $field = content_fields($field_name);
  if (in_array($field['type'], $field_types)) {
    return $field;
  }
  else {
    return FALSE;
  }
}

Or have I missed something about content_type?

Great job, ekes! I'm mapping ISO dates with the greatest of ease now and I feel a bit guilty. The only issue that I've noticed so far is that there is no option to map to a Date To field. Not sure if that falls within the scope of date_feedapi_mapper(), though.

The reason for the convoluted dbdate thing was the // TODO Timezone bit. dbdate as I understand it does timezone handling.

There certainly is Timezone information in some iCal feeds (sadly in a standard and a non-standard way even from Apple themselves :( I'm guessing if people are interested in attaching timezones to other feeds to correct the times displayed/stored that it will be needed then too.

I have say I've not worked through the implications timezones on pulling feeds yet. I'm just going to start trying to pull in some iCal feeds and I'll see.

Version 2 of the Date API changes the functions for creating and accessing dates. Looks like the integration of Timezone handling is easier. But, it does seem, at the moment however, to drop some strtotime in the date parsing which certainly would be needed for some of the formats above.

I'm not sure if this want mapper wants to work with Version 2 only or both?

Timezones:

Many of the dates with RSS/Atom feeds are pretty timezone unaware. For the moment I'm just assuming passing them to the date_api and it will assume they are local time without additional info. I could see a point in the future that it would be desirable for users to enter the Timezone associated with each feed - I'm planning for it, not planning to do it at the moment.

With iCal the Timezone can be fed with the *TIME element or it seems can be set using an (non-documented as far as I can tell, but very commonly used) X-WR-TIMEZONE for the feed. This should be passed to the date_api

The Date fields themselves can be timezone correcting or not. It depends how they are set up, and as far as I can see not the concern here as long as the information is passed to date_api.

I've already written - in testing code - a sub_field for from/single date mapper, a sub_field to date. These accept text dates, but they should accept iCal arrays too which include the timezone separately.

Date api 2:

It seems that v2 is to replace v1 Even though Calendar for example is still being worked on to use v2 I'm more tempted now to move onto just working for v2

Taking a first look at integrating feedapi_mapper_date_2_timezones.patch with the from and to date stuff and I notice it is using the PHP > 5.1.2 date_parse which also doesn't have an equivalent in the date_php4 includes.

I'll experiment with it but I'm pretty sure if the formatting is one recognised the DateAPI date_make_date is good. It's just for some of those other wierdness that a strtotime (or equivalent #233431: Has strtotime functionality gone from api date object creation?) will be needed.

It seems the way date2 has gone is to using the PHP 5 datetime functions and objects, and recreating them for PHP 4. Seems to me that date_create does everything needed with the companion the date_*_get.

This isn't fully tested as I'm having some difficulties retaining the timezone on with cvs version of date2 for hand submitted forms.

I've added 'to' and 'from' date support, and handling for the full iCal DTSTART and DTEND arrays (which can have timezones in them).

Subscribing.

subscribing

Hello,

I've been assigned by Alex to look into this addition. There seem to be some remaining issues with date 2 that cause some unreliability. However, I believe these changes mean that the date mapper is storing the dates properly, even if the date module sometimes garbles them on display and edit. It seems that the date module expects dates to be stored by UTC in all cases, with the timezone and offset optional. The new version should handle that adequately. One thing to pay attention to is that a bug in date_api_ical.inc's date_ical_date function basically sets the date to null half the time. The line:
$date = date_timezone_set($date, timezone_open($to_tz));
should be:
date_timezone_set($date, timezone_open($to_tz));
This is because date_timezone_set returns null on success and true on false...not a date object in either case. However, using this change to date_api_ical.inc, try out the new feedapi_mapper_date.inc file.

David

subscribing

Subscribing, I haven't had time to try this out, but I think the bug in #25 is fixed in the latest rc. If there are any other bugs related to getting this working, be sure to post them in the Date issue queue with a link back to this issue.

subscribing

I was able to import modules using the ical parser here http://drupal.org/node/214688 and the feed api with the comment #25 code in D6.

My set-up is as follows
Drupal Core 6.5
cck 2.0 rc 10
date 2.0 rc3
feedapi 1.3

Subscribing.

Is there a 6.x patch?

I'm trying to see if I can get the FeedAPI ical solution working in D6, but I can't get the files in #25 to download and open in my Windows environment. Can someone post them as patches or text files instead?

Never mind, I figured out a way to get the gz files working and I'm trying to piece together the right code from the above comments and patches.

I needed the following patches for other parts of feedapi to get this working without errors: #324001: Undefined property: stdClass::$feed in feedapi_node.module line 84, #325012: Fix E_ALL and E_STRICT warnings.

Here is the Date mapper I got working. It works with the code I submitted at #214688: Create an iCal parser (parser_ical) to create the iCal parser.

You need to map raw->DTSTART to the From field and raw->DTEND to the To field. I tried to get rid of things like raw->DTEND->datetime: and raw->DTEND->tz: since those sub-fields don't have all the information needed. You have to map the top-level ical date element to the Date module date to get the right results. For some reason, they still show up as available mapping fields, which is kind of confusing.

Oops, I need to make a small change in the way timezones are handled, use this file instead.

Some updated code, as noted in the linked issue. This works with a better method I found for the iCal parser.

Hi All,

I just tried installing the parser_ical.module and this patch. I have events from civicrm showing up in my calendar but they show up on the day that the feed is refreshed rather then the day the event is scheduled for. I know one step I'm missing is mapping the ical dates to the node dates but I have no mappers available. Has anyone run into this issue before? I'm sure I'm missing something obvious here :(

Best Regards,

See instructions on using the ical parser to create CCK date fields at http://drupal.org/node/214688#comment-1113271.

Thanks Karen. I was able to get it working. Much appreciated!

Hi folks, I needed this but *not* for importing iCal dates--I got caught up here (lines 35/36)

      $feed_element = unserialize($feed_element);
      $date = date_ical_date($feed_element);

Because my $feed_element was an ISO8601 formatted date, rather than an array, the code would produce nothing for me and I'd get a bad date (today's) in place of the date I was interested in.

So, I've tried this code below and it works for me, but I'd like to have some input on whether this is an appropriate way to make this more general and not dangerous--I have no idea if there are also other formats that I should account for as well:

      if (is_array($feed_element)) {
        $feed_element = unserialize($feed_element);
        $date = date_ical_date($feed_element);
      } else {
        $date = date_create($feed_element);
      }

Perhaps there's a safer way to test if something is an iCal date or not?

patch attached, this should be applied to the patch in comment http://drupal.org/node/215979#comment-1079581 (thank you KarenS!).

How can you have an ISO8601 formatted date in an ical feed? That's not in the specs. The assumption in this patch is that you're using it to parse feeds that comply with the ISO standards.

I'm just trying to understand the use case or an example of why and when other dates would be in there or why or how you would be using this for anything other than an ical feed. I'm totally open to making this as flexible as possible, but I'm confused about what you're doing.

Oh, are you using this to map non-ical feeds? Maybe you said that :)

OK, I get the idea now. Let me think about what approach would work for both, yours is probably fine, except that you're not accounting for the timezone (the ical array has timezone info). The ISO dates have an offset, not a timezone, so they're hard to interpret. You'll end up with a date that might or might not have the right timezone.

There is no safe way to convert UTC offsets to time zones, that's the whole problem with using offsets. You can make some guesses about timezone based on location, but you can never be sure if you have the right one because there are a number of timezones that could be using the same offset at any point in time. To get even close you have to know the location and whether or not the time is in daylight savings time, and even then you can't be sure.

But thinking about this more, I guess what we could do with those dates is use the offset to set the imported date back to UTC, which would be the right value to store in the database in most (but not all) cases, and then see if we need to make other adjustments to the value based on the settings of the date field it's being imported into. For instance, dates that don't use timezones need the date converted to the site's timezone before storing it to match the way it would have been created in the UI.

Hmm... that could work.

I reworked the code to take the above issue into account and create something that will work for all dates. The changes for the feed element mapper are here, the ical parser changes are posted on the other issue.

Wow, I finally checked this and you've gone ahead and made the code more general AND more robust. Kudos. I'll give it a shot ASAP with my ISO8601 dates...

EDIT: Oh, and FYI, what I am importing is an RSS feed with this namespace: http://purl.org/rss/1.0/modules/event/

Thanks again for your work, all of you (and KarenS in particular, your code saved my butt), this is a lifesaver!

DD

Just glancing at my code for manipulating the ISO 8601 dates I can see a number of errors (can you tell I didn't test that part?). I'll try to work on this tomorrow unless someone else gets to it. You can see what needs to be done, but dividing by 100 to get the hours isn't going to produce the right results, we should go the other direction and compute either minutes or seconds.

Does anyone have this working with the Date Repeat API? In other words, are you able to generate nodes that understand the iCal RRULE?

My Calendar view works great with mapped iCal events, but I can only see the event the first day it starts. It is not showing up when it is supposed to repeat.

Any ideas/suggestions?

#51, there is already an issue on the Date queue about doing something with repeating imported dates. That needs a more complicated fix, but right now we need to just get the basic stuff working. The biggest problem with that is that we will have to put some outside limit on the date range we create or you could be creating hundreds of nodes to match a repeat rule that has no end date.

@alex, I can't replicate your errors using that feed and the latest Date module code. I did find and commit some fixes to the Date API ical parser, but it was mostly doing more error trapping of values that aren't in the expected format. Nothing I fixed should have had any affect on the error you saw.

Anyway, I still need to fix the problems I noted above for the rss feed, but I am able to import the feed you linked to with no errors and the mapper shows me values that look right.

I'll do some more work on this and then post an updated patch to go with the latest Date module changes.

Doing some more work on this and I think we need to adapt the ical parser to attach the RRULE and EXCEPTIONS to each date to be able to do anything at all with that information. If we do that, and figure out a way to create an outside range for dates that should be created from the RRULE, we could have the date feed element mapper create repeats as multiple values in the field to match the way the current date repeat works.

The biggest problem I have is that the feed element mapper assumes you can map an individual part of a feed to a CCK field and have it work correctly, but with dates we need a lot of information from different feed elements (DTSTART, DTEND, DURATION, RRULE, EXCEPTIONS, etc) to do anything intelligent, so we may have to map the whole feed to the date field instead of mapping a single part of it.

@alex, does that make sense? Do you have any ideas on how to get more information to the field mapper?

One idea is to adapt the API of the mapper so you map a single element but you also get passed the whole raw value of the feed as another parameter in case you need other information from the feed.

Here's a patch for feedmapper that would add feed information to what is passed to the feed element. With this information I could create the right values for the date field.

Then what I'd like to do is move the date field mapper into the date module because I can keep it current better there and make sure it is set up correctly for whatever version of Date is being used. This is what we've done with other things like Token integration. It looks like you already have a hook set up for that, so you'd just need to remove the version stored here so it won't interfere with the one I want to create.

My remaining problem is that we really need to map the whole date field as a single unit, not the individual parts, to get this working right. Is there any way so that all the parts don't show up as available targets for mapping? I see there is a way I can not show options for From date and To date, but both of them still show up on the mapper, so people will be confused if I do that.

We don't need the patch to FeedAPI if we change the way this works to pass more info to the date field. I posted a patch for the parser at #342303: Handle events measured in days and I've re-worked the date mapper code to use it and I can even get repeating dates working with the RRULE.

This is not finished code, it needs more testing, but I've got to pull away from this project for the next couple weeks so I'm posting what I have so far. As soon as we're comfortable that it's working correctly, I'll add it to the Date module and you can remove it here. But I'm going to leave it here for now so we don't lose the context of the discussion about how it is supposed to work.

BTW, I've posted some fixes to the Date iCal parser code to better handle that feed from upcoming.org and I can get a nice feed now that seems to import correctly into my date fields.

I have a question about the feedapi_mapper_date.inc patches. In #46, KarenS posted a patch against a revision that included adding some custom mappers in the date_feedapi_mapper function. This added the ability to choose "Simple date", "From date" or "To date" as part of the same date field.

In the latest #58 patch to feedapi_mapper_date.inc, these custom mappers are gone.

Is this supposed to be the case? Are those custom mappers supposed to be gone now?

You can't map to those individual date parts and have things work right, that's the point I was trying to make. The whole batch of information about the date is needed at the top level of the date field to be able to do things like figure out whether it is an 'all day' date or set the duration between the from and to date or see if it is a date with repeating rules that needs to be multiplied into several values. Mapping part of an ical date to just the 'From' or just the 'To' date won't work right. So now you map the top-level ical 'Date' field to the top-level date field in your content type and the mapper figures out what to do from there.

@alex, I'm waiting for the Parser_ical patch to land because this new code won't work at all unless that gets patched. When it does, I can commit the mapper code to the Date module.

Thanks for the reply KarenS and all the work. I have mine working as I wanted it to.

Look forward to hearing you speak at DIWD.

FYI - I tried parser_ical + Date mapper with the patches from KarenS to import the attached ical feed with RRULES and I got an unexpected result. It did seem to create one event for each repeat, and it populated the RRULE column, but with an incorrect RRULE. Instead of the RRULE from the file, I get an RRULE that repeats daily and has as its end date the time that I imported the feed. Actually I think it's creating the same exact (incorrect) RRULE for every repeating event. I'll look into it a bit more and report any other findings, but let me know if you think it's user error :-)

Here's a patch to fix the issue I reported in #65. It contains two changes in feedapi_mapper_date_ical():

1. Removed the call to date_ical_parse_rrule() because the RRULE is already in parsed form.

2. As a stopgap, bumped $max_repeats up to 50 from its previous value of 5. This is because I had a problem with monthly events that were created for June 2008 only appearing until October 2008 - which doesn't do me any good for this month!

There are several patches here. Which one to use if I just want to map the publish date to a date field?

Yes, this is confusing! I eventually got the right patch put in the right place -- I think. I'm still not sure because, although I don't have any errors I'm not getting a proper mapping of dates. Using http://www.mozilla.org/projects/calendar/caldata/USHolidays.ics, the dates that display are today's date.

I have been doing some heavy debugging to try and get this working successfully with our exposed Zimbra iCals. This is a summary of what is working, what problems Zimbra has caused me (and perhaps others that use Zimbra) and what issues still seem to exist.

What is working

-- Mapping Zimbra iCal feeds to a CCK type and have them visible in a views calendar and other views blocks

-- Getting repeating events to show up as expected (with an upper bound of 50 repeats)

Zimbra Problems

-- In my Zimbra implementation, the exposed iCal is not using UTC for the RRULE UNTIL portion per RFC 2445

-- Zimbra doesn't seem to allow for the time to be part of the UNTIL portion. (eg FREQ=DAILY;UNTIL=20081120;INTERVAL=1) This causes problems with the calculating of repeating dates. Zimbra interprets this as an inclusive UNTIL date and the event repeats on this last day within Zimbra's own calendar. In the date_repeat_add_dates() function of date_repeat_calc.inc, it formats this UNTIL date as 2008-11-20 00:00:00. This formatted version is compared to $current_day which has a time attached. (2008-11-20 08:30:00). Unless the event takes place at 00:00:00, the comparison 2008-11-20 00:00:00 > 2008-11-20 08:30:00 always fails, and the last day is not included in the repeats.

-- Padding the events UNTIL datetime is required to interpret repeating events properly

Possible Issues

-- In the _date_repeat_calc() function of date_repeat_calc.inc, the $until_date is adjusted to local timezone with a call to date_ical_date(). Then this $until_date is sent to date_timezone_set where it gets adjusted again. In my case, CST -6, the $until_date being adjusted pushes the date back 6 hours so 2008-11-20 00:00:00 becomes 2008-11-19 18:00:00 which causes the ending repeat date to be incorrect.

Hopefully this post will help anyone who may be experiencing problems with Zimbra's exposed iCal feeds.

Feel free to comment on anything or ask additional questions regarding this post.

Has anyone been able to successfully pull events from google calendar? I have gotten as far as importing the events but the date still displays as the feed refresh time. I am using newest dev versions and most recent patch.

I too am not able to map to date fields properly .... Feed refresh works after applying the patch, but all the date fields take on today's date.

I tried changing the import format based on a wing and a prayer, but no luck (pretty much as expected - didn't think that would affect the feed to cck field mapping anyhow).

Can anyone comment on how far we are from getting a updated release of feedapi_mapper that works with date fields?

Found another small "bug" - set the feed item content type to use a date field in the feed as the create date for the item nodes, here's what happens:

Feed date value: 2008-04-01 .... (April 1, 2008)

Newly created node create-date comes out as: 03/31/2008 - 23:00

I have time-zone conversion turned OFF for all my date fields, and the user-configurable time zones are turned off for the site too.

Any ideas what's going on here?

BTW, I'm using the CSV parser, so I didn't need to worry about patches to the iCal parser; I don't even have it enabled.

FeedAPI and Element Mapper are awesome, but for now, I can't handle even the simplest of date field mappings ...... never-mind the added complications that iCal, recurring/repeating and To dates will bring when I move on to trying to use the iCal parser.

Am I right in assuming that we just need to wait for KarenS to look at this again?

Sorry, could someone make a list of the patches to review?
I'm interested to this feature too..

Two days now trying to get this to work - no joy. I'm experiencing the same problems as listed above. In short, I need to create a Calendar feed for CiviCRM events; this took about 5 seconds to do in D5 but is looking pretty impossible in D6.

It would be most helpful if somebody could post:

1 - Whether importing an iCal event feed into Calendar is actually possible right now

2 - Which module versions to use (D6) and which patch(es) to apply, if any, and in which order. Following the instructions and applying the "latest patch" to feedapi_mapper_date.inc as described here results in "malformed patch at line 169" and the file remains unchanged - it's still the September 8 version.

3 - How to set up the target Date field(s) in CCK and how to apply - the instruction simply to have "a Date field that can be used for the iCal Date" isn't enough. I'm assuming it should be two "Date Fields with Custom Input Format" and the options left blank. Is that correct?

You've all (especially Karen) obviously put in a lot of hours making this work (in certain cases, anyway) and for that, I am eternally grateful! But it's a shame that more of us can't share in your PHP goodness... on account of poor documentation!

Again, setting up Calendar to read a CiviCRM iCal feed should not be so "Rube Goldberg-esque," especially when it works easily in D5!

If anyone has this working and would be kind enough to let the rest of us know how they did it, that would be spectacular. In the meantime, let me once again express my appreciation for what the Drupal community is doing - you are all amazing... thanks in advance!

Thanks for this addition.

I agree with the other posters when saying that it's very confusing to figure out what the actual patch for feedapi_mapper_date.inc should be based on this thread.

Nevertheless, I was able to cut and paste (AKA "hack") my way to getting an updated feedapi_mapper_date.inc file to work with and to use to debug (it's based on post #66).

I know everyone would like to see a silver bullet patch, and so would I, but I'm not there yet AND I'll have to learn how to create patches first. ;)

-----------------------

Ok, so the "patched" code takes two main execution paths based on these lines of code (let's call them ICAL, and SIMPLE):

    // A date created by the iCal parser will have an array of values
      // and can be used to create complex, multiple values.
      // A date created by an RSS feed may have only a simple date.
  
      if (is_array($feed_element)) {
         return feedapi_mapper_date_ical($node, $field, $feed_element);
       }
       else {
         return feedapi_mapper_date_simple($node, $field, $feed_element);
      }      

I know most people are having problems with the ICAL piece, but I focused on SIMPLE initially; admittedly, for selfish reasons, and to "keep it simple" so that I could see if I could get it working with a basic inbound date format (my inbound data fields look like: YYYY-MM-DD (i.e. IS08601)).

So, with a little debugging with good ol' dsm(), here are some key issues I uncovered:

1. the code calls date_format(), and from what I was able to find out here: http://api.freestylesystems.co.uk/api/function/date_format_date (I could have looked at the source code for date_api too), that call should be date_format_date() instead (woah slow down ;) , don't go off and edit a change on that one function call and expect it to work just yet - this is a process, and I'm not all the way through it yet).
2. the CCK date field seems to expect a different format for the array that gets built and then shoved into $node before it gets returned, than what appears in the patch code ... Let me illustrate:

the patch implies the following format for the $items array (=$node->$field_name) that will eventually get passed through to the CCK date field in your newly created nodes:

  $format = date_type_format($field['type']);
+
+  if ($formatted = date_format($date, $format)) {
+    // Create node field information.
+    $items = isset($node->$field_name) ? $node->$field_name : array();
+    $items[0]['value'] = $formatted;
+    $items[0]['value2'] = $items[0]['value'];
+    $items[0]['timezone'] = $to_tz;
+    date_timezone_set($date, timezone_open($to_tz));
+    $items[0]['offset'] = date_offset_get($date);
+    $items[0]['offset2'] = $items[0]['offset'];
+    $node->$field_name = $items;
+  }
+  return $node;

But, again thanks to devel module, I was able to determine that CCK date fields expect a different array structure as compared to what the above code implies (see below). Yes, I hard-coded my timezone for debugging purposes and because the date-data in my feed (CSV file) has no timezone variability (recall date format in feed = YYYY-MM-DD)

I hacked the above code to look like this instead:

    $items[0]['value'] = $feed_element.'T08:00:00';
    $items[0]['value2'] = $items[0]['value'];
    $items[0]['rrule'] = NULL;
    //date_timezone_set($date, timezone_open($to_tz));
    //$items[0]['offset'] = date_offset_get($date);
    //$items[0]['offset2'] = $items[0]['offset'];
    $items[0]['timezone'] = 'America/Vancouver';
    $items[0]['timezone_db'] = 'UTC';
    $items[0]['date_type'] = 'date   | (Callback) date();';

It seems that offset, offset2 and have been replaced by timezone and timezone_db, rrule was added for dealing with repeated dates in ICAL (I'm assuming that that's what that's for), and date_type is needed to specify a callback for some reason (I'll dig up why later).

Note,

• I'm using Drupal 6.9 and with date module (6.x-2.x-dev), the latest version, as of today found here: http://drupal.org/project/date).
• $feed_element in my case was just equal to the string YYYY-MM-DD, so I just appended T08:00:00 to it.
• this part of the if statement: $formatted = date_format($date, $format) was not used in my hack so item[0]['value']=item[0]['value2'] take the value YYYY-MM-DDT08:00:00 instead of what was being passed in the original code
• reading ahead in the code for the function return feedapi_mapper_date_ical($node, $field, $feed_element) BOTH code execution paths ICAL and SIMPLE suffer from ailments [1] and [2], so a patch that takes into account the above MAY solve the problem people are having with mapping ICAL feeds too.

Once I made these hacks, and also commented out all the timezone manipulation lines of code, IT WORKED!! The CCK date fields in the newly created nodes are correctly reflecting the dates that appear in the feed. That's significant progress!

BUT, to get it to work, I by-passed the use of date_format_date() and all timezone related API calls. So this is NOT a generic solution, AND I haven't tackled the ICAL code execution path .... If I find time, I will try to do so since we'd all benefit from getting ICAL based-feeds to work with feedAPI and feed_element_mapper in D6.

Incidentally, I also discovered what I think is another bug too (which I'll report in detail separately), where it seems that changing the value of a field or two in the source feed (not just dates, any field), causes FeedAPI Node processor to create NEW nodes upon subsequent refresh instead of updating the existing ones. (At least this is the behaviour I expected). I have to dig deeper before making sense of this one.

Anyhow, I hope this helps. At least I know I'm on the right path and getting some where. I'll report back as I dig deeper.

Below, I've cut and paste the content of the feedapi_mapper_date.inc file that I used when it worked, so that it's here for all to see:

<?php
// $Id: feedapi_mapper_date.inc,v 1.1.2.2 2008/02/28 22:04:44 alexb Exp $
 
/**
 * Implementation of hook_feedapi_mapper for date field from content.module (CCK),
 *
 * @param string $op
 * @param Drupal node $node
 * @param string $field_name
 * @param string, number or array of string or number $feed_element
 * @param string or number as id $sub_field
 *
 */
 
 // Patched manually to overcome references to date functions that no longer exist in the date module
 
function date_feedapi_mapper($op, $node, $field_name, $feed_element = array(), $sub_field = '') {
  if (!$field = feedapi_mapper_content_is_cck_type($field_name, array('date', 'datestamp', 'datetime'))) {
     // if not a date just return
     return;
  }

  switch ($op) {
    case 'describe':
      // Describe what we are doing in this mapper. This shows up as help text on the mapping page.
      return t('Maps a date to a date CCK field.');
    
    case 'list':
      // Here we are being asked to list sub fields we would like to map to.
      // In this case, we only map to the CCK field or not, so we return just TRUE.
      return TRUE;
      
    case 'map':
      // Here is where the actual mapping happens.
      include_once(drupal_get_path('module', 'date_api') .'/date_api_ical.inc');

      /*
      dsm($field_name);
      dsm($feed_element);
      */
     
      // A date created by the iCal parser will have an array of values
      // and can be used to create complex, multiple values.
      // A date created by an RSS feed may have only a simple date.
  
      if (is_array($feed_element)) {
         return feedapi_mapper_date_ical($node, $field, $feed_element);
       }
       else {
         return feedapi_mapper_date_simple($node, $field, $feed_element);
      }      
  }
}

/**
 * Create a date from a simple string date value.
 */
function feedapi_mapper_date_simple($node, $field, $feed_element) {
  
  $field_name = $field['field_name'];
  
  /*
  // ISO 8601 dates will have +/-05:00 offset adjustments.
  if (strstr($feed_element, '+')) {
    //dsm($feed_element);
    $date_info = explode('+', $feed_element);
    list($hours, $minutes) = explode(':', $date_info[1]);
    $timezone = 'UTC';
    $date = date_make_date(trim($date_info[0]), 'UTC');
    date_modify($date, '-'. trim($hours) .' hours');
    date_modify($date, '-'. trim($minutes) .' minutes');
  }

  elseif (strstr($feed_element, '-')) {
    dsm($feed_element);
    $date_info = explode('-', $feed_element);
    list($hours, $minutes) = explode(':', $date_info[1]);
    $timezone = 'UTC';
    $date = date_make_date(trim($date_info[0]), 'UTC');
    date_modify($date, '+'. trim($hours) .' hours');
    date_modify($date, '+'. trim($minutes) .' minutes');
  }
  // Not an iCal date and not a ISO 8601 date, just use the value.
  else {
    $date = date_make_date($feed_element);
    //dsm($date);
    $timezone = date_default_timezone_name();
  }
  */
  //dsm($feed_element);
  //$date = date_make_date($feed_element);   
  
  //dsm($date);

  //$db_tz = date_get_timezone_db($field['tz_handling'], $timezone);
  //$to_tz = date_get_timezone($field['tz_handling'], $timezone);
  //date_timezone_set($date, timezone_open($db_tz));
  //$format = date_type_format($field['type']);
  //dsm($format);
  
  if ($feed_element) {
    // Create node field information.
    $items = isset($node->$field_name) ? $node->$field_name : array();
    //dsm($items);
    //dsm($formatted);
    $items[0]['value'] = $feed_element.'T08:00:00';
    $items[0]['value2'] = $items[0]['value'];
    $items[0]['rrule'] = NULL;
    //date_timezone_set($date, timezone_open($to_tz));
    //$items[0]['offset'] = date_offset_get($date);
    //$items[0]['offset2'] = $items[0]['offset'];
    $items[0]['timezone'] = 'America/Vancouver';
    $items[0]['timezone_db'] = 'UTC';
    $items[0]['date_type'] = 'date   | (Callback) date();';
    $node->$field_name = $items;
  } 
  
  return $node;
}


/**
 * Create a date from an iCal feed.
 */
function feedapi_mapper_date_ical($node, $field, $feed_element) {
  $field_name = $field['field_name'];
  
  dsm($node);
  
  $timezone = $feed_element['DTSTART']['tz'];
  if (empty($timezone)) $timezone = date_default_timezone_name();
  $date = date_make_date($feed_element['DTSTART']['datetime'], $timezone);
        
  $db_tz = date_get_timezone_db($field['tz_handling'], $timezone);
  $to_tz = date_get_timezone($field['tz_handling'], $timezone);
  date_timezone_set($date, timezone_open($db_tz));
  $format = date_type_format($field['type']);
      
  if ($formatted = date_format($date, $format)) {
    // Create node field information.
    $items = isset($node->$field_name) ? $node->$field_name : array();
    $items[0]['value'] = $formatted;
    
    $date2 = date_make_date($feed_element['DTEND']['datetime'], $timezone);
    date_timezone_set($date2, timezone_open($db_tz));
    $formatted2 = date_format($date2, $format);
    $items[0]['value2'] = $formatted2;
    $items[0]['timezone'] = $to_tz;
    date_timezone_set($date, timezone_open($to_tz));
    $items[0]['offset'] = date_offset_get($date);
    $items[0]['offset2'] = $items[0]['offset'];
    $node->$field_name = $items;
        
    if (array_key_exists('RRULE', $feed_element) && !empty($feed_element['RRULE']) && module_exists('date_repeat')) {
          
      include_once('./'. drupal_get_path('module', 'date_repeat') .'/date_repeat_calc.inc');
      include_once('./'. drupal_get_path('module', 'date') .'/date_repeat.inc');

      // Explode the RRULE into parts so we can analyze it.
      $rrule = $feed_element['RRULE'] . (!empty($feed_element['EXDATE']) ? "/n". $feed_element['EXDATE'] : "");
      $form_values = date_ical_parse_rrule($field, $rrule);
          
      // Set an maximum value for repeating dates.
      // One year, 5 repeats for now, could be adjusted or made configurable later.
      $max = date_now();
      $max_repeats = 5;
      
      date_modify($max, '+1 year');
      $until = date_format($max, 'Y-m-d H:i:s');
      if (empty($form_values['UNTIL']) || $until < $form_values['UNTIL']['datetime']) {
        $form_values['UNTIL'] = array('datetime' => $until, 'tz' => 'UTC');


      }
      $form_values['COUNT'] = $max_repeats;
          
      // Reconstruct the RRULE with our changes and build the repeats.
      $rrule = date_api_ical_build_rrule($form_values);
      $node_field = $items[0];
      $values = date_repeat_build_dates($rrule, $form_values, $field, $node_field);
      $node->$field_name = $values;

    }
  }
  return $node;
}

Thanks for the detailed post, Ben A - unfortunately, I get WSOD when I try to map. Not soup yet - as I'm sure you're aware. Good to see you taking a stab, though - thanks much! I'll keep checking in...

(Oh... if I could only turn the clock back to D5 where you could enter a simple iCal feed into the calendar... those were the "good old days," eh? *sigh*...)

Just a point of clarfication before someone calls me out on it. ;)

Both date_format and date_format_date are valid function calls. date_format() is a PHP >=5.2 function call (http://php.net/manual/en/function.date-format.php), and date_format_date is a Drupal date_api call (again, here's the reference that I was able to track down: (http://api.freestylesystems.co.uk/api/function/date_format_date).

subscribing, thanks

subscribing

I've been trying to make sure that the mapper will work for various kinds of date feeds, ical feeds are only one of them. The ical feed has several different elements that are all needed to correctly create the resulting date, so you can't do a one-to-one mapping of the ical feed 'fields' and the node 'fields'. My earlier attempt at a date mapper tried to account for this by combining the ical parts into something that will match the date fields.

However, in #350405: CSV Format - delimiting double quotes are not discarded I'm trying to work with another pattern, the csv output created by something like an export from Outlook or Yahoo calendar. In that case there are several different 'fields' in the output that need to be combined into a single date field, totally different that the ical feed. These csv feeds create several fields: 'Start date', 'Start time', 'End date', 'End time' and 'All day' and they are missing a timezone. So this combination needs to be mapped in a totally different way than the ical combination.

A third pattern would be a normal rss feed, which might be a single, complete, date field in the format 'YYYYMMDDTHHMMSS -05:00'. That would map one-to-one with a date field.

So I'm trying to figure out how to configure the mapper so it will work correctly for all those situations.

Some of the conclusions in #76 are incorrect, db_timezone is not a replacement for offset and offset2, not all dates have offset or offset2 (only dates that use a 'date' timezone), so not seeing those fields did not mean they are no longer used, it is because the field definition was different. But that is a clue to the timezone issue because the code was trying to convert the date to the timezone indicated in the ical rule and then save it to a date that uses a site timezone.

I do think the idea of splitting the code by date type is the right idea, all these kinds of dates need totally different treatment. The tough thing is to identify which treatment to use where.

I'm playing around with this more to see if I can come up with a good, generic solution for all these kinds of situations.

Trying out Karen's patch from #58, I dates start getting added to the created nodes, but like others have mentioned, it is today's date, not the one from the feed I'm pulling in (in the format

2009-01-30T00:00:00Z

).

Trying to debug this a little, it doesn't seem as though the 'map' case in the new feedapi_mapper_date.inc file is actually being called, despite the information being mapped from the feed element into a date (datestamp) field. That could be the cause of my missing date, but I will need to debug a little further to find out why this isn't being called.

I'm working a new approach that can be more flexible. It still needs a bit more work, but is starting to come together. It's a totally different approach to the earlier patch, but I've been able to do things like bring in the Yahoo calendar csv file mentioned above that has all the date parts in separate fields. Once I get this working, the 'date_constructor' would go back into the Date API somewhere, since I keep re-creating these kinds of things for every application.

I'm just posting the progress, it's not ready to commit yet.

patch works fine
I tested it on my local setup fixes http://drupal.org/node/323286

Dates coming good compared with real feed.. not very sure about time zones yet

just changing status to - to be ported as KarenS is still working on it

If the current code is broken, I guess it can't be any worse. It's a much better approach, I'm just not convinced I've worked through all the different paths that need to be tested to be sure it creates the right results in every case.

But it's very cool to be able to do things like bring in the events from a Yahoo calendar csv file -- I did get that working for my own application at least.

I haven't had time to get back to this, but I can post an additional patch for whatever changes are needed if you want to commit this much.

Downloaded the latest from CVS but getting errors on refresh:

date_modify() expects parameter 1 to be DateTime, boolean given in /var/www/html6/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_date.inc on line 77

and after I commented the above out I got this:

Call to undefined function dsm() in /var/www/html6/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_date.inc on line 290

this sort of works, I'm trying to map two fields onto one cck date with to and from. It only seems to pull the to date and maps it onto the from date which is wrong according to my mapping plan. Maybe this doesn't support mapping of two fields onto one CCK date. I'll experiment a bit more and get back to you.

Ok this is what my tests conclude, hopefully you can see where the code is tripping up.

Raw CSV values: Start Date, End Date
Map to single CCK field with To and From dates.

Map CSV End Date to CCK Start Date (date and time) - Works
Map CSV Start Date to CCK Start Date (date and time) - Works
Map CSV Start Date to CCK End Date (date and time) - Doesn't Work
Map CSV End Date to CCK End Date (date and time) - Doesn't Work

So it looks like the issue lies with setting and End Date for CCK fields.

This does the trick for me, YMMV.

The trouble was that for each date field, a complete date object was constructed and put into the node, consisting of both a start and end time. So you parse the first date field you come across (start date) and insert it into the node, giving you a start date and no end date, then you parse the next date field (end date) and insert that into the node, overwriting the start date with nothing (at some stage the null value is interpreted to mean now). Karen appears to have worked around this by just putting anything into the start date which obviously works fine if you only have a single date value to import into the field.

Here I check for the existance of $node->{$field_name}[0]['value'] and $node->{$field_name}[0]['value2'], and put those into the date object before overwriting one or the other with the imported value. A bit kludgy, but it works.

I rerolled Matthew Davidson's patch to apply to the HEAD.
Now i'm testing it if it does really solve the problems in #90

I could not find the patch in #94 helpful. For me, it simply did not do the job.
I created a patch for handling correctly the To date as well.

Here's the current code I'm using. I incorporated Aron's patch from #95 and have been doing a bunch of testing of various kinds of ical feeds, including feeds with RRULEs, all day dates in feeds, etc. I haven't tested this with every possible type of feed but I'm pretty sure it's an improvement on the current code and seems to be working fairly well for ical imports. I found a few small updates that needed to be made to the Date API to better handle things like the 'all day' dates and those changes are in Date's -dev code.

I can't spend any more time on it right now, but wanted to post it so others can give it a try.

Bah! I left debugging crap in there again, another try...

FWIW, I added some test files to the Date module to test ical and csv imports of date values with FeedAPI. The csv test is in date/tests/Yahoo.csv, which is a file in the format of a csv export from Yahoo calendar or Microsoft Outlook. There is a file called rrules.ics which tests a bunch of complicated repeating dates. Another file is USHolidays.ics which tests 'All day' dates in the format that is used by Microsoft and Apple reporting the End date as the following day. I'm running all of these through my Date feed mapper to make sure it is pulling the right values into the created nodes.

If anyone wants to provide additional generic test files for importing dates in other formats (XML or whatever), I'll add them to the test directory so they can be tested, too.

Here's the latest patch. I can get this working correctly to import RRULES and all day dates in ical files and csv values from my Yahoo.csv example using the test files I noted above. I don't have any test file for a straight XML or RSS feed, so I don't know about that.

I should note that I have various PHP notices when doing this, but the import works correctly. The notices are not created by or controlled by this patch, so I am ignoring them for now.

In general it's a great patch!
I found an annoying issue although. Date field allows you to set the input format to whatever format you want. I set it to U - means unix timestamp because i wanted to use a unix timestamp value for importing. Unfortunately it failed. The reason is that line:

$date = date_create($feed_element, timezone_open($to_tz));

So this patch works if you want to map the textual representation of the stuff. If you use some other wild (unix timestamp for example) input format, it won't work.

I searched for a solution, but it seems that we can only solve this elegantly in PHP5.3.0 (or later) http://php.net/manual/en/function.date-parse-from-format.php
So I'd say that the patch is good enough now.

It was really tough figuring out how to import dates that could be in any possible format -- 05/15/2009 08:30 PM or 2009-05-15 20:30 or May 15, 2009 or RSS format or iCal format or Unix or whatever, and on top of that they could be whole datetime values or date-only or time-only. I'm not sure how to do any more without making some of these other formats fail.

We can check for a numeric value, but that will screw up a value like 20090515. We can try the loose regex over the value first to pull out anything like an ISO date, and if that fails check for a numeric value to treat as a timestamp, and then if that fails try date_create()/strtotime().

I'm open for ideas.

BTW, the Date API has a something that will do something like parse_from_format, see date_convert_from_custom(), but both of those assume you know ahead of time what format the date will be in, and with a feed, you don't.

I came up with an idea to allow the timestamp entry to work by adding a couple additional options to the mapper. Try this patch.

One more time, got a little to quick on the copy/paste for the new function, it should refer to start_stamp and end_stamp instead of start_datetime and end_datetime.

Thanks for all your efforts here Karen! With the patch from #106, I am getting a whitescreen when trying to update the items in my feed though, and nodes are created without any data at all. There are watchdog entries about the date being in the wrong format.

The dates in the feed I am trying are created by the RDF module, are in the form 2009-03-20T09:42:00Z, and are in the first element of an array instead of being a simple text string:

            [xcal_dtstart] => Array
                (
                    [0] => 2009-02-28T07:00:00Z
                )

Checking for an array and taking the first element of it should fix that for now (though should we be able to handle multiple dates in an array?). Also, the current regex is picking up the dashes in there and treating them as an offset, but changing it to the following should force it to look only at the end of the string for that offset:

preg_match('/(([\+\-])(\d{2})?:?(\d{2})?)$/', $feed_element, $matches);

Sorry for the lack of a patch. I am still trying to figure out why it's giving me a whitescreen. With these changes, there are no longer any errors in the watchdog about incorrect date formats, but it is still giving the white screen and creating nodes without any data (or even a node type) in them.

After further testing, I realised that the errors I was seeing were happening because there was no return $node; in the code. So, with $node now being returned, along with the slight changes I suggested above, here is an updated patch (based on #106) which now seems to be working for me.

One issue remains, and that is to do with multiple values...

Is this patch now at a level where it can support ICal feeds without repeating events? If so might it be rolled as another beta release at least? I don't have access on my environment to patch but have a need to import ICal feeds from CiviCRM and would be more than happy to test that integration.

I know from the CiviCRM forums that there are others also eager to get this sorted out as putting an organization's events into their calendar is a fairly basic requirement.

The iCal feeds part of the patch should be working fine, with and without repeating rules. The patch changes being debated above are other kinds of feeds.

I have tried the latest patch and mapping seems to work for the largest part, but I still get a white screen when updating feed items. I am importing a Google agenda iCal file (http://www.google.com/calendar/ical/ontdekhetonzichtbareheelal%40gmail.c... it's a public agenda). When I map raw->all_day to the All Day part of my Date field I get the white screen, when I turn that specific mapping off it works perfectly, so the problem seems to be with mapping to all day (or perhaps with mapping from a Google iCal to all day).

Another thing I noticed is that I am unable to map to the Date field at all (it isn't available as mapper) if I set the Widget Type of the field to Select List with Repeat Options. I haven't tried them all, but Select List does work, so I'm sticking with that for the moment.

Hope that helps you to make this mapper even more solid; as it is though it is already of great use to me, so thanks for all the effort!

When mapping ical items you just map the top-level ical VEVENT item to the matching date item (also called VEVENT to make it easy to see). You do *not* map all the individual parts of the ical element, like 'all_day', just the one top-level element. This is because the ical element has lots of interacting parts and the mapper can't do anything meaningful with the date without the whole thing.

It's confusing and it would be nice if there was a way to hide all those other parts that you don't need in that situation, but that's the way it works now.

There is no VEVENT item to map from. Could that have to do with Google's iCal format or are my settings wrong? Perhaps someone could try importing the same iCal (see #111 for url).

I've been testing out the timezone handling a bit more and noticed that the part I updated didn't handle the timezones properly, so here's an improved version that seems to do the job for me (note I'm only changing the handling in the feedapi_mapper_date_datetime function).

One thing that strikes me as strange though, and I'm not sure if it's intended or not, is that the timezone for each of the created items is set to that of the current user (the one that refreshes the feed). Should that be the case?

I have no VEVENT item in my iCal feed either (not from Google Calendar). Here's a sample of what I'm seeing (the second "raw" item looks like another event, should not be bundled with the listed event). I tried mapping DTSTART + DTEND separately as well as mapping "options" to VEVENT, no luck either way.

Array
(
[title] => E. Coli
[description] => A Darwin
[options] => Array
(
[original_author] =>
[location] =>
[original_url] =>
[timestamp] => 1235598710
[guid] => 8e0265b1-f752-41a5-918a-1b2095850f1f
[tags] =>
[DTSTART] => 2008-09-03 11:00:00;tz=America/New_York
[DTEND] => 2008-09-03 11:59:59;tz=America/New_York
[DTSTAMP] => 2009-02-25 21:51:50;tz=UTC
)

[raw] => Array
(
[SUMMARY] => Distribution of the Sample Mean
[UID] => 8e0265b1-f752-41a5-918a-1b2095850f1f
[DESCRIPTION] => K Koenig
[LOCATION] => SLH E
[ORGANIZER] => CN=Eileen%20Cahill
[COMMENT] => Activity (EBPM)
)

)

EDIT: I took a look at the raw iCal file, and the two events that are being confused aren't adjacent (it also looks like the guid for the "options" part is incorrect). Here's a bit of the .ics file...

BEGIN:VCALENDAR
PRODID:-//SakaiProject//iCal4j 1.0//EN
VERSION:2.0
CALSCALE:GREGORIAN
BEGIN:VTIMEZONE

... etc ...

BEGIN:DAYLIGHT
TZOFFSETFROM:-0400
TZOFFSETTO:-0400
TZNAME:EPT
DTSTART:19450814T190000
RDATE:19450814T190000
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20090226T003725Z
DTSTART;TZID=America/New_York;VALUE=DATE-TIME:20080827T110000
DTEND;TZID=America/New_York;VALUE=DATE-TIME:20080827T115959
SUMMARY:E. Coli
UID:d8f44d87-0098-47d6-000b-e58d1eaafbb7
DESCRIPTION:A Darwin
LOCATION:Schwartz E
ORGANIZER:CN=Carolyn%20Holmes
COMMENT:Activity (Host Defense)
END:VEVENT

... multiple VEVENTS ...

BEGIN:VEVENT
DTSTAMP:20090226T003725Z
DTSTART;TZID=America/New_York;VALUE=DATE-TIME:20080903T110000
DTEND;TZID=America/New_York;VALUE=DATE-TIME:20080903T115959
SUMMARY:Distribution of the Sample Mean
UID:8e0265b1-f752-41a5-918a-1b2095850f1f
DESCRIPTION:K Koenig
LOCATION:SLH E
ORGANIZER:CN=Eileen%20Cahill
COMMENT:Activity (EBPM)
END:VEVENT

... more events ...

END:VCALENDAR

Hi folks, thank you so much for working on this patch, unfortunately, I need to be able to map a date into a cck date feild and the experts tell me this patch is how its going to happen, but alas I have not figured out CVS or patching with software and this thread has thoroughly confused me. I tried manually adding and deleting lines from the patches to get me where I need to be but I just get a white screen when I referesh my feed. Would it be possible for someone to send me or post a patched version of the include? This patch is the only thing keeping my drupal weather system from operating.

I imported the feed from #111 and the 'feed item example' on the Map tab looks like this, you can clearly see that there is a VEVENT in it even though #113 says there is not. If you are not seeing something like this, it would be a problem with the parser, not a problem with the mapper, but I suspect you just don't have the right code for the parser since it works fine for me. Once you have the right info in your feed item example, you should be able to map the result without any problems.

Array
(
    [title] => Discovery Truck Groningen
    [description] => Aanwezig:
Gergö
Vincent
    [options] => Array
        (
            [original_author] => 
            [location] => 
            [guid] => gjkq62bmliqatkednnc4qr8ib4@google.com
            [original_url] => 
            [timestamp] => 1235904931
            [tags] => 
            [VEVENT] => Array
                (
                    [DATE] => Array
                        (
                            [DTSTART] => Array
                                (
                                    [datetime] => 2009-02-23
                                    [all_day] => 1
                                    [tz] => 
                                )

                            [DTEND] => Array
                                (
                                    [datetime] => 2009-02-23
                                    [all_day] => 1
                                    [tz] => 
                                )

                        )

                    [DTSTAMP] => Array
                        (
                            [datetime] => 2009-03-01 10:55:31
                            [all_day] => 
                            [tz] => UTC
                        )

                    [UID] => trq0pq5ehles0v2019l7t67s38@google.com
                    [CLASS] => PUBLIC
                    [CREATED] => Array
                        (
                            [datetime] => 2009-01-23 14:17:18
                            [all_day] => 
                            [tz] => UTC
                        )

                    [DESCRIPTION] => Aanwezig:
Gergö
Vincent
                    [LAST-MODIFIED] => Array
                        (
                            [datetime] => 2009-02-02 10:34:48
                            [all_day] => 
                            [tz] => UTC
                        )

                    [] => 
                    [SEQUENCE] => 
                    [STATUS] => CONFIRMED
                    [SUMMARY] => Discovery Truck Groningen
                    [TRANSP] => TRANSPARENT
                    [all_day] => 1
                    [LOCATION] => Franeker, Nederland
                )

        )

)

I just found a bug in the main mapper code that might be a factor here -- if you create a Repeating date it will not show up as a field available for mapping because CCK fields that do their own multiple values handling are added to the form in a different way. I posted a patch for that problem at #387408: Mapper misses CCK custom elements.

OK. First I am using this version of the feedapi_mapper_date.inc (http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/feedapi_map...). So I created an ical feed cck type and the new ical feed node using the ical feed from google calendar in #111. Then I tried to map the items as Karen describes in #112. Here is a picture of what I did: http://picasaweb.google.com/ghankstef/Misc#5308401580835395522.. No love. Node title and description import fine but no date and no location mapped. What am I doing wrong. If you help me I promise to make a screencast on how to do this.

Thanks

Did you apply the patch in #116 to the cvs version? It won't work without the patch.

#116 does not have a patch.....which patch do we apply to the CVS version?
F

I agree this is confusing, having just wandered in off the street also. I believe that you need to go to the revisions log for feedapi_mapper_date.inc and download the latest revision, rather than try to figure out what the heck patch to apply to what version (and how).

Here's the link to the current revision, as of today: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/feedapi_map....

Here's the link to the log: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/feedapi_map...

This is helpful info. Unfortunately, the suggestions for patching (in Windows) involve installing and learning how to use tools that turn out either not to work (UnxUtils throws a runtime error for me, Tortoise SVN will only patch on an SVN repository), or are massive overkill for the task at hand (cygWin, Eclipse) - or both.

I would humbly suggest that when you find yourself instructing users on how to apply a patch (#120), it's time to put up a new revision.

I withdraw my comment about when to commit - not my place to say.

Re. cygwin - that's exactly what I'm afraid of! ;-) I'm downloading it now. We shall see...

I did succeed in getting unxutils patch to work though. I had to change the patch file's line endings to cr-lf.

In my latest test (with the feedapi_mapper_date.inc patch #9 applied), my feed is creating completely empty nodes.

Before applying the patch it was sort of working: it had been creating nodes with several fields populated, though the from and to dates were set to the vevent's dtstamp, rather than its dtstart and dtend. So every node said March 9th (All Day) for start and end date/times.

I'm mapping "raw:" to "Map to field_date (date): iCal VEVENT". Is this correct?

thanks everyone for edumacating me ;-) However, as clarity is setting in, I am still confused by #120, the reference to the patch in #116 is a mistake, and I just want to know now which actual patch KarenS was referring to? was it:
a) 215979_rdf_date_support.patch attached to #114
b) feedapi_mapper.module.patch from a different thread mentioned in #118
c) or something else?

Lastly, some unsolicited opinions from the point of view of a dedicated drupal user, this module provides the *ONLY* solution for ingesting this data into my drupal 6, I am not here because I like to tinker, I am here because this patch is the only thing standing between a working drupal application and one that will work some day. I am building this application to manage Forest Fire data, and the Forest fires will start soon in my area. While I have no problems learning a new tool when needed, I dont think patching contribed modules is for amatures and should not be treated as such, it is a highly complex process that regular users are not likely tro understand or perform successfully, but they NEED modules to work, either because they upgraded to a buggy module or becuase like this module there is no real way to do it otherwise. The way contrib mod developers and debuggers and patchers toss jargon, patches and lines of code around in bug threads, well the discussion threads like this are nearly proprietary to drupal developers who understand the jargon. But as you all know, users who NEED modules will actually come here and try to learn the seemingly crazy process and jargon, just to get the module to work again or to get it working for the first time. Regardless, I appreciate all help and effort to get this module working and to set me straight on patching, just try and remember that there are people in here depending on the patches who dont know how to apply them. Hopefully they are as eager as I to learn. helpful posts like #122 and #123 should be automatically posted as sticky in every bug thread so that people can get a jump start on patching. These threads are often the first drupal.org experience users have when they arrive looking for a solution to a bug in the drupal they are using but did not build. Thank all, The Drupal Community rocks. period.

Franco

OK, I have checked out the latest version via CVS like this:

 cvs -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal-contrib checkout -d feedapi_mapper -r DRUPAL-6--1 contributions/modules/feedapi_mapper

Then I found the latest patch in this queue for the date mapper (feedapi_mapper_date.inc) which as far as I can tell is #108 applied by saving the patch file to the same director as feedapi_mapper_date.inc and running

patch < feedapi_mapper_date.inc__9.patch 

Then I went back to me ical feed node and deleted all feed items and refreshed the feed. I still get this date_create error:

warning: date_create() [function.date-create]: Failed to parse time string (2009-03-11 00:30:00;tz=UTC) at position 19 (;): Unexpected character in sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_date.inc on line 95.

So in other words it doesn't like the semicolon

line 35 is as follows

    $value = date_create($feed_element, timezone_open($to_tz));

and lives in the function feedapi_mapper_date_datetime

I am going to try to sort this out and if I can create a new patch

here is my patch to fix the error in #128. Take it with a grain of salt as its the first one I submitted to drupal.org but at least I can import/refresh feeds with date_create errors now

patch created like this:

 cvs diff -up > feedapi_mapper_date.inc__10.patch 

ghankstef:

which element are you mapping. If it's coming from the iCal feed you can (should?) map the whole VEVENT. If you are doing this what feed are you mapping, because it all sounds a bit odd.

generally:

I'd support closing this thread and creating new specific ones. While I've been able to use the patches I've just not been able to keep track of the discussion.

@ekes Mapping this feed: http://www.google.com/calendar/ical/i93ecasoiselnqqvdfmblqkvuk%40group.c...

Feed example array comes in like this:

Array
(
    [title] => "Great Conversations" with Seymour Hersh, Larry Jacobs, and Walter Mondale
    [description] => Larry Jacobs, Mondale Chair for Political Studies and director of the Center of the Study of Politics and Governance at the Humphrey Institute, and Walter Mondale, former Vice President of the ...
    [options] => Array
        (
            [original_author] => 
            [location] => 
            [original_url] => 
            [timestamp] => 1237208248
            [guid] => 9vvve3e3le0pkeicne3ukm0tno@google.com
            [tags] => 
            [CREATED] => 2009-03-02 02:06:18;tz=UTC
            [LAST-MODIFIED] => 2009-03-02 02:06:19;tz=UTC
            [DTSTART] => 2009-03-12 20:00:00;tz=UTC
            [DTEND] => 2009-03-12 21:30:00;tz=UTC
            [DTSTAMP] => 2009-03-16 12:57:28;tz=UTC
        )

    [raw] => Array
        (
            [UID] => 9vvve3e3le0pkeicne3ukm0tno@google.com
            [CLASS] => PUBLIC
            [DESCRIPTION] => Donald Moynihan, associate professor at the La Follette School of Public Affairs, will discuss his latest book The Dynamics of Performance Management: Constructing Information and Reform on ...
            [LOCATION] => Wilkins Room
            [SEQUENCE] => 
            [STATUS] => CONFIRMED
            [SUMMARY] => "The Dynamics of Performance Management"
            [TRANSP] => OPAQUE
            [all_day] => 
        )

)

To be honest I am not sure where to map VEVENT to. I am now mapping, options->DTSTART: to field_event(date): Start Date (date and time) and options->DTEND: to field_event(date): End Date (date and time) - I see I can map to field_event(date): VEVENT but get this error when trying to map
options to field_event(date) to VEVENT: warning: date_create() [function.date-create]: Failed to parse time string (2 00:00:00) at position 0 (2): Unexpected character in sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_date.inc on line 397. Not clear on how to map VEVENT

Try this with the latest or -dev release of Date, the latest release of parser_ical 6.x-1.1 (just moved it out of -dev), and the latest feedapi_mapper with the __9 patch. Mapping the

''options->VEVENT''
to
''map to field_**** (date**): iCal VEVENT''

Rather than each individual part. If the VEVENT has got a start and end time, and the date field can too, it will get both. I've just done this with the feed mentioned above and it's gone in fine.

I tried the instructions given in #133, but when I try to refresh the iCal feed I get the warning messages:

# warning: array_key_exists() [function.array-key-exists]: The second argument should be either an array or an object in /etc/drupal/6/sites/default/modules/feedapi_mapper/mappers/feedapi_mapper_date.inc on line 161.
# warning: array_key_exists() [function.array-key-exists]: The second argument should be either an array or an object in /etc/drupal/6/sites/default/modules/feedapi_mapper/mappers/feedapi_mapper_date.inc on line 168.

...repeated lots of times, probably once per feed item. Any idea what's going on there? I'm using Date 2.0, iCal Parser 1.1, Feed API 1.6 and FeedAPI Mapper from CVS with the _9 patch applied.

can confirm that following the instructions of #133 works for me

am importing an iCal feed generated via Kerio Mail Server (6.6.2) all works fine, my only niggle is that multiple day events (using the all day tag) only show the first day and dont span multiple days, if i edit an event manually to span the multiple days it displays over multiple days (only problem i have 225 events of varying lengths and some not all day events)

not an experienced coder but learning fast, especially as the people i work for are seriously considering moving to drupal (from a commercial and expensive solution) and have already relaunched some of our smaller sites within drupal successfully, any assistance or nudges in the right direction greatly appreciated.

Tried adding a simple var_dump() in the date mapper; $feed_element is always either the empty string or NULL. Does this mean that the iCal parser is broken? I'm pretty sure I'm using the latest version.

larsl:

latest or -dev release of Date accoriding to ekes - you said you where using date 2.0 - try 6.x-2.x-dev

OK, ignore my last two messages. The iCal parser produces a field called options->VEVENT-> in addition to options->VEVENT and I was trying to map that instead. Keyboard/brain interface error on my side. When I map the proper fields it works great. I had some problems with the date filtering in the calendar view, but updating the Date module to the most recent dev release fixed that as well.

Hi there-- great work!

The first feed I tried this out on (using the #133 tip) was http://www.meetup.com/dcbloggers/calendar/ical/The+Washington+Blogger+Me...

It seems to work, except that:

- The iCal specified UTC (good, expected)
- My Site is configured to use America/New_York (OK)
-The CCK field is configured to display using the site's timezone
- The imported date doesn't seem to be timezone-aware, it displays the UTC time (23:00) instead of the local time (19:00)

I'd like to help debug this, if you can point me in the right direction.

Hi, I'm totally lost here. I'm following the steps from http://drupal.org/node/341716, but when trying to apply the patch I get a big fail (using UnxUtls, converted to CRLF):

...\sites\all\modules\feedapi_mapper\mappers> patch < feedapi_mapper_date.inc__9.patch
patching file `feedapi_mapper_date.inc'
Hunk #1 FAILED at 29.
Hunk #2 FAILED at 48.
Hunk #3 FAILED at 63.
Hunk #4 FAILED at 84.
Hunk #5 FAILED at 180.
Hunk #6 FAILED at 286.
Hunk #7 FAILED at 295.
Hunk #8 FAILED at 306.
Hunk #9 FAILED at 325.
Hunk #10 FAILED at 361.
Hunk #11 FAILED at 393.
Hunk #12 FAILED at 401.
12 out of 12 hunks FAILED -- saving rejects to feedapi_mapper_date.inc.rej

I downloaded the most current version (?) of FeedAPI Mapper from here, instead of the cvs version recommended here.

Actually comparing the files, the dates on the feedapi_mapper_date.inc seem to indicate the cvs version is older by a couple months, but the "new" version is missing a lot of code; this is apparently what the patch is looking for. I'm hesitant to download the cvs file into a potentially "different" module just to get the patch to work.

So...what am I missing? I bet it's something obvious too.

Hi folks!
Today I'm testing the new beta version of feedapi_mapper for drupal 6 (feedapi_mapper-6.x-1.0-beta5).

One thing that i have to say is, that the feedapi_mapper_date.inc should really be patched with ghankstef's patch 10...
...otherwise the file is not really usable (you would have to comment out the dsm() call and modify the UTC handling for datefields since the call to date_modify fails in the unpatched version and to handle a problem where date's got overwritten when 2 feed-elements were mapped to the same date... not to speak of the newly implemented timestamp handling).
Great work!, hopefully this patch is included in beta6 (if there is another beta version)

@ghankstef:
When handling non standard-dates you are using a needle to build a substring of the feed_element here

	$needle = ';';
	$pos = strripos($feed_element, $needle);
	$feed_element = substr($feed_element, 0, $pos);
	$value = date_create($feed_element, timezone_open($to_tz));

The problem i did have with this code is, that if the needle is not present in the feed_element, the mapping fails... shouldn't we use something like this?

	if ($pos > 0) $feed_element = substr($feed_element, 0, $pos);

@ #140 all i did was download the latest version of the file from cvs, patched it and overwrote the file provided in the release version and this worked for me

I've tried to update the tutorial Create a feed of iCal events using Date, FeedAPI, and iCal parser to reflect the current state of the date field mapper and its usage.

I did **not** correct the screenshot showing erroneous mapping settings!

I would appreciate it if someone could review it and correct any errors that I may have introduced, as well as correct that screenshot.

CVS version and patch __10 from #130 is working great for me. I'm importing 8 Google Calendar feeds into a single content type, using a taxonomy term each to categorize them.

I currently have 77k records in my "content_field_event_date2" table and performance is still good for "day" views filtered by one of 8 taxonomy terms.

I would suggest some type of interface for setting the maximum imported repeats though... I changed mine to 1000 and set my execution limit to 600 seconds to do a one-time import of my data. An event that happens 3x weekly is 156/year, so the hard-coded limit of 52 is rather low. Some of my calendars have 4-5 events every day for 3 years. I'm not sure what a better solution would be, or where the UI would go, but I think a cron-based expansion would be slick. Maybe even a "dry-run" mode for the repeat insertion functions to allow a confirmation before creating too many repeats.

I'm too busy to get to it right now, so it's just a suggestion.

I tried the patch in no. 130 with the mapper beta 7 and it seems to work. The issue I had was that empty dates in my file where filed up by today's date, even though the cck field was setup to have blank as a default, it's a bit annoying cause now I have a lot of nodes to clean up as I need this to work very quickly. I still wanted to report it cause it seems like a bug to me.

Patchak

@patchak: the reason that empty input results in a default/current datetime for the field, is that the date_constructor() used in the mapper has this default behaviour for each individual field of the timestamp (date, time, timezone, etc). If you want blank input to be explicitly skipped, you can add this explicitly in the mapper function:

function feedapi_mapper_date_part(&$node, $field, $feed_element, $sub_field) {
  // Skip empty feed elements 
  if (empty($feed_element)) {
    return $node;
  }
  // End: Skip empty feed elements 

  static $date;
  $field_name = $field['field_name'];
...

or even do this one level higher up, for all sub-field types:

    case 'map':
      // Skip empty feed elements 
      if (empty($feed_element)) {
        return $node;
      }
      // End: Skip empty feed elements 

      include_once(drupal_get_path('module', 'date_api') .'/date_api_ical.inc');
      switch ($sub_field) {
...

Lyricnz, this solved my issue thanks a lot!!

#148: sure, feel free to reroll! :) Navigating the patches on this particular issue makes my head spin!

Should this patch work on feedapi.mapper.inc revision 1.1.2.3.2.3 (beta 7), or only revision 1.1.2.3.2.2 (beta 5)?

I think it's beta 7, it woked for me, except for that empty dates issue.

#151: We had this discussion previously, but didn't seem to get anywhere: the module maintainer needs a clean patch from this (very long) issue, but I couldn't figure out where the code (that you gave me) came from, or how it related to the patches in this issue. Setting status back to "needs work" so a clean patch can be built for this issue.

#145, 151: patch __10 will report "hunk 12 failed" on beta 7, however note that the relevant section already appears to have been changed according to the patch. ie., don't worry about it.

@lyricnz : the patch we are referring to is the one in number 130 afaik... it's used to patched the beta 7 version of the date mapper... I hope this is the info you needed.

So I guess if anyone would want to insert your changes in comment 146, he would have to use patch in no. 130 as a start.

Patchak

This patch applies cleanly and successfully maps ical item dates for me.

It also adds some much needed validation checking to the date parser to prevent bad feed items from being created when the date information on the feed is incomplete or invalid.

Thanks for your diligent efforts!

How do I implement this patch? I've looked at and followed this: Applying patches on Mac OS X with no luck. I get "permission denied".

How do I implement this patch? I've looked at and followed this: Applying patches on Mac OS X with no luck. I get "permission denied".

Perhaps, for some reason, you don't have write access to the file that you're applying the patch to.

There is no VEVENT item to map from. Could that have to do with Google's iCal format or are my settings wrong? Perhaps someone could try importing the same iCal (see #111 for url).

I beat my head against this for a while. My Google calendar URL had a space in it. When I saved the feed to a webserver, renamed it, and used that URL, the VEVENT and other mapping fields appeared.

I'll file a bug report if necessary.

As posted on http://drupal.org/node/735416

warning: Invalid argument supplied for foreach() in /httpdocs/sites/all/modules/parser_ical/parser_ical.module on line 101.

All required modules enabled. Followed tutorial at: http://mustardseedmedia.com/podcast/episode34 (as indicated at http://drupal.org/project/parser_ical).

Unfortunately mapping is not possible, since

Problem seems to be here: foreach ($ical['VEVENT'] as $event) {

Same procedure and code on the same server works fine for other websites.

Testing this. Subscribing.

By the way, should this issue really be closed?

Note for anyone looking for "Create a CCK Date feed mapper" as suggested in the help section on iCal import, you can download the module at http://drupal.org/project/feedapi_mapper and continue with the help text.