Has someone already written an iCal processor? If not, how much work would it be?

I am using the calendar module for displaying event type nodes. The calendar module does have an iCal component, but it does not create event nodes.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alex_b’s picture

Title: Has someone written an iCal processor » Create an iCal parser (parser_ical)
Category: support » feature

Here is the code that we could use for building an iCal parser.

In Date module the iCal logic lives in its own inc file:
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/date/date_a...

The hard work is done. What's missing is a parser that integrates this iCal parser with feedapi: parser_ical.module !!!

The parser should create parsed items the feedapi way (see parser_simplepie or parser_common_syndication) - the rest can be handled by a Feed Element Mapper mapper implementation for date fields (see feature request here: http://drupal.org/node/215979).

This is not rocket science, it'll take a developer one or two days to get this up. I am unfortunately swamped with work at the moment, but I would love to see this feature happen and I am ready to give advice to whoever steps up to this task.

gustav’s picture

Is anyone working on this already? If not, perhaps I can find some time to take a look at it over the next weekend.

ekes’s picture

Assigned: Unassigned » ekes

I've made something of a start on this - using code from date_ical and the present parsers. It will need some little workrounds to make it work with the date_ical parse to start off with. So I'm claiming this -- *but* if anyone wants to join in, take over, or what ever I'm kinda busy. I just really want it for the continued development of http://drupal.org/node/215979 ;-)

alex_b’s picture

hooray :)

ekes’s picture

OK, I'm not getting so much time, so this is slow, so I'm going for the release early, release often, and be damned option.

Attached is 'working'.

It has no requirements testing, it has little validity testing, and it's got no install script.

It reuses the parser_common_syndication functions for getting the feed, because they are good, but so has an install issue (you need parser_common_syndications tables already).

It uses the date_api_ical functions for parsing, but has to patch the upload() function as that include the functionality for collecting the feed (which is being done with above). I've mailed karens about the possibility of accepting a patch to separate out the fetching of the ical file and the parsing it but heard nothing yet.

ekes’s picture

OK, I'm not getting so much time, so this is slow, so I'm going for the release early, release often, and be damned option.

Attached is 'working'.

It has no requirements testing, it has little validity testing, and it's got no install script.

It reuses the parser_common_syndication functions for getting the feed, because they are good, but so has an install issue (you need parser_common_syndications tables already).

It uses the date_api_ical functions for parsing, but has to patch the upload() function as that include the functionality for collecting the feed (which is being done with above). I've mailed karens about the possibility of accepting a patch to separate out the fetching of the ical file and the parsing it but heard nothing yet.

Aron Novak’s picture

First, thanks for working on an iCal parser, it's a really exciting use-case of FeedAPI!
Well, I suggest you the following:
- You can safely delete the block after // Do the autodiscovery at this level, pass back the real data // Maybe it's HTML. If it's not HTML, not worth to take a look into the downloaded string
- a simple processor, like this, does not need any tables. You can make it even more simple. The caching is just an option.
- i recommend you to simply copy more code from parser common syndication. The problem is that FeedAPI is not dependent on any parsers, they are totally interchangeable.

Can you supply here a "feed url" suitable for ical parser? I would really like to try out the code :)
I attached a simplified version of the module. I haven't tried it out, but it contains my ideas in code-form. :)

ekes’s picture

Cool I'll have a go with it - I'm just looking at the date_mapper again so I'm going to try with some of the iCal output.

I've been using the index of feeds at:

http://icalshare.com/

to come up with interesting and different iCal feeds.

alex_b’s picture

Status: Active » Needs work
FileSize
10.83 KB

Hey,

Looks like a good start. I created a patch from ekes and aron's posts here. (for adding new files cvs add all new directories and files followed by a cvs diff -u -N from the feedapi directory, let's use a single patch on this queue for our discussion)

It doesn't work though :( :

When I use webcal:// as schema, there is an error that the feed couldn't be retrieved, when I use http://

I get this error:

Call to undefined function date_make_date()
http://skitch.com/alexbarth/8we7/mozilla-firefox

(feed http://ical.mac.com/ghodder/1st%20Year%20multimedia%20timetable.ics )

In regards to patches to date module files, please go ahead and just post them on the date module issue tracker and link to them from here. I am sure Karen will review them. For simplicity, I think the way you took by copying the code into the ical parser module and clearly marking it as patched is a good method for now.

One request: Can we call this module parser_ical ? I don't know where the common is coming from.

Alex

ekes’s picture

Call to undefined function date_make_date()

date_make_date is a date_api version 2 function

ekes’s picture

Can we call this module parser_ical ? I don't know where the common is coming from.

It was only there because it was based on parser_common_syndication, so sure. But, on that point...

- a simple processor, like this, does not need any tables. You can make it even more simple. The caching is just an option.
- i recommend you to simply copy more code from parser common syndication. The problem is that FeedAPI is not dependent on any parsers, they are totally interchangeable.

This isn't a big issue, but I feel I should mention as it feels big in my underlying design brain.

While I agree that the feedapi itself does not want to be dependent on any particular parser. If there are parsers that are coming collectively with feedapi (or in the future just for feedapi) it seems wrong to reuse code by cutting and pasting. There is some good code in the feedapi_common_syndication.

It would be nice to have a common part of the parsers to reuse. Be it just a set of functions and tables.

But like I say it's not that important.

Leeteq’s picture

Subscribing.

yngvewb’s picture

Also subscribing.

David Goode’s picture

Component: Miscellaneous » Code
Assigned: ekes » Unassigned
FileSize
10 KB
10 KB

Here is a minor patch to the parser_common_ical module, but more important a patch and new info file that make it work with Drupal 6, in anticipation of rolling out parser_common_ical with FeedAPI's Drupal 6 version when ready.

David

Mike Sances’s picture

Subscribing

geodaniel’s picture

subscribing

matt_p’s picture

Hi.
I have managed to add events using FeedAPI Node and the iCal parser.
Am I correct in thinking that the start date of an event is not updated with the DTSTART value in the ical feed? How would I go about getting FeedAPI Node to store the start time in a CCK field?

Thanks

KarenS’s picture

Subscribing, I haven't had time to try this out, but I'm obviously interested :)

tyromind’s picture

subscribed

carnive’s picture

http://groups.drupal.org/node/13720

This just came out, not sure if it's useful for you guys or not. Author created a module to take a GCal XML feed and display it in a block.

johnmunro’s picture

Subscribing

fm’s picture

subscribing

johnmunro’s picture

A couple mods to work with current D6.4 & & FeedAPI 1.3 Date 2rc3 to remove errors on refresh.

Changes to parser_common_ical.module following the .toD6 patch

As _parser_common_ical_feedapi_parse is being used for both compatible AND parse, it needs the same data type for the URL:
Line 31 of function parser_common_ical_feedapi_feed
- $url = $args[1]->url;
+ $url = is_object($args[1]) ? $args[1] : FALSE;

To stop errors, $headers must be declared as an array on line 147 in
function _parser_common_ical_download($url) {
$method = 'GET';
$follow = 3;
$data = NULL;
+ $headers = array();

Renee S’s picture

Ok, I've tried the D6 version and nothing happens... it just doesn't work; nothing happens and when I refresh the node it forgets the URL. Could one of y'all who got it working post what you did?? :)

johnmunro’s picture

After applying the patches, I think I just followed the instructions in the README.TXT

fm’s picture

Has the iCal parser been incorporated in to a Dev version yet?

fm’s picture

I've pieced together the module and the patches from above, but haven't gotten an iCal feed to work.

I tried a number of feeds. The following URLs are representative of those that I tried:

webcal://flagfootball.meetup.com/100/calendar/ical/Philadelphia+Flag+Foo...
http://flagfootball.meetup.com/100/calendar/ical/Philadelphia+Flag+Footb...

What is the acceptable URL formats for this? webcal? http?

I keep receiving a "Could not refresh feed" message when I refresh the feed. There is no indication of a problem in the watchdog log. But then I could never get the Common Syndication Parser to work. I could live without the Common Syndication Parser because SimplePie Parser worked, but I'd LOVE to have the iCal parser ... that would rock.

happysnowmantech’s picture

subscribing

Renee S’s picture

I found a few bugs in the 6.x module that David G posted. I didn't get it working mostly because the stupid FeedAPI refused to recognize when I'd deleted a feed and re-added it so it and would come back with "0" items, so it was a HUGE pain to test ... I think that's a bug right there in FeedAPI!

The first thing was that the get_url function was grabbing the feed twice for some reason; it wouldn't recognize the BEGIN because it would see BEGINBEGIN.

I came sooo close, though. I would love to have this functionality. The first person to get it working can have my cat.

Shiny’s picture

subscribing

alex_b’s picture

Version: 5.x-1.x-dev » 6.x-1.x-dev

updating version

KarenS’s picture

I was going to try this out to see if I can get the FeedAPI ical solution working for D6, but I'm unable to get either of the files in #14 to download and open in my Windows environment. Are those the current working files? Can someone post a patch or text files instead of the tar files?

KarenS’s picture

Never mind, I figured out a way to get the gz files working and I'm trying to piece together the right code from the above comments and patches.

KarenS’s picture

I'm getting some of this to work now, just a few general comments:

1) _parser_common_ical_patched_date_ical_parse() in the patch is no longer needed, it's available in the Date API as date_ical_parse().
2) I'm renaming the module to parser_ical instead of parser_common_ical, as noted somewhere above
3) I'm adding a dependency on Date API, which eliminates the need for the redundant function above and for hook_requirements.
4) There is no such ical item as 'CREATED', the right value for the created date is 'DTSTAMP'.
5) Practically all of the ical items are optional, so we have to check isset() on all of them to avoid undefined index errors in D6.

I'm still plugging through it, I'll post an updated patch when I get a little further.

Renee S’s picture

This is wonderful, KarenS, I have a huge project that needs this and my skills are limited to tracking down bugs - I can tell what's not working, but not always how to fix it :) Let me know if you need a guinea pig or tester.

KarenS’s picture

Status: Needs work » Needs review
FileSize
4.51 KB
207 bytes

I needed the following patches for other parts of feedapi to get this working without errors: #324001: Undefined property: stdClass::$feed in feedapi_node.module line 84, #325012: Fix E_ALL and E_STRICT warnings.

Here is the iCal Parser module I got working. It works with the code I submitted at #215979: Create a CCK date field mapper to create nodes that have date fields from iCal input.

KarenS’s picture

I found a way to get rid of the confusing extra date part elements to make the mapping easier. There are some changes in both this module and the one at #215979: Create a CCK date field mapper.

And here are some screen shots.

Renee S’s picture

Works for me. Thank you!!

redndahead’s picture

Any chance of getting instructions on how to use this module to pull in the groups.drupal.org ical feed? I'm trying to review this and I'm haven't been able to pull in a feed.

redndahead’s picture

I got it figured out. I missed the part about the feedapi_mapper module and that was the sticking point. It's working for me. I've had a couple of small errors come up here and there, but I haven't figured out where the blame lies.

bigjim’s picture

Karen

I think line 105 is incorrect from the module code in comment #37. The CATEGORIES element of the $event array is a string, hence you need explode() in lieu of implode(). It could be my feed (an ical feed from civicrm on the same Drupal install)
if I'm correct it should be

$item->options->tags = isset($event['CATEGORIES']) ? explode(',',$event['CATEGORIES']) : array();

I'm running Drupal 6.6, civicrm 2.1.1, feedapi 6.x.1.4

gustav’s picture

FileSize
53.25 KB
109 KB

Karen, thank you very much for putting this together. Unfortunately I can not get it to work. Here is what I did:
- Downloaded latest 6-1 code for feedapi and feedapi_mapper modules
- Overwrote feedapi_mapper/mappers/feedapi_mapper_date.inc with the code from http://drupal.org/node/215979 #36
- Created new module directory parser_ical and put your code from this thread into it. feed I took parser_ical.info from #36 and parser_ical_module from #37
- Went to admin/build/modules and enabled FeedAPI, FeedAPI Node, FeedAPI Mapper, Common Syndication parser, iCal parser
- Went to admin/content/node-type/feed and chose iCal parse, see screenshot.
- Went to admin/content admin/content/node-type/feed/map and could only see what is shown in the screenshot map.jpg. No ical fields to map.
What did I do wrong?

By the way: the parser_ical module does require the FeedAPI Mapper module, doesn't it? This should be added to the dependencies in the .info file.

KarenS’s picture

One problem I see in the screenshot in #42 is that you don't have a date field in the node you're mapping things to. Can you paste an export of the content type you're using for your imported nodes? They need to have a date field that will hold the values you're importing. You will probably need the date to allow for both a 'from' and an optional 'to' date, and set it up to have a date-specific timezone, since you don't know what you'll be getting in the ical feed and it could include a from and to date and a timezone.

I don't have more time to work on this right now, but that would be a starting point.

KarenS’s picture

Also I found that for some reason I had to create a new content type for the ical parser -- it wouldn't work right to just change the parser on the default content type. So I created a new content type called 'iCal feed' and set that new type up to use the iCal parser, then created my feed with it.

I don't know enough about Feed API to guess why that was, but see if that works for you.

KarenS’s picture

To (maybe) help matters, here are the steps I followed. This is copied from the Advanced help in the latest -dev version of the Date module, along with screenshots you can see if the Advanced help module is installed. Also make sure you have the latest -dev code for the Date module, too.

  1. Decide what content type will be used for nodes imported from your iCal feed. It can be a separate content type for each feed, or they can be combined into other content types. Make sure the content type you select has a date field that can be used for the iCal date, plus fields for any other iCal information that will be imported.

  2. Create a new content type for iCal feeds.

  3. Scroll down to the bottom of the content type edit page and look for the place to set up Feed information. Set it up to be a Feed type that uses the iCal parser.

  4. Go to 'Create Content' and create a new node using the iCal feed content type. When you create the node, you will find a place to input the url of the feed and select the content type for the resulting nodes.

  5. Once the feed node has been created, you will see a tab at the top where you can 'map' its data to the fields in your target content type.

  6. Scroll down and look for options->DTSTART: and options->DTEND and map them to the 'From date' and 'To date' fields. Then map any other information from the feed to right fields in the target node.

  7. Once the fields have been mapped, use the 'Remove items' tab at the top of the page to eliminate nodes that may have been created before the fields were mapped, then use the 'Refresh' tab to re-import them.

  8. You can see in the Content screen in the administration area that the nodes have been imported.

gustav’s picture

Karen, thank you very much for all your help. This helped me to get things working.

My problem was that I tried to set up the mapping for the feed content type on admin/content/node-type/feed/map rather than for the individual feed node.

I don't know the FeedAPI framework. Would it be easy to move the setting up of the mapping of the date fields to admin/content/node-type/feed/map so that one only has to set it up once per feed content type rather than once for each feed node?

Setting it up individually for each node also has the difficulty that if one ticks the "Refresh feed on creation" checkbox on the node add form then the ical event nodes are created before the mapping is set up and are thus created without dates.

I agree with jalama that the fix from #41 is needed.

mckien’s picture

This is really good stuff (thanks, KarenS!), and has given me just what I need to be able to use our corporate calendars on Drupal. The problem I'm having now is with all-day appointments. They are showing up in the Calendar view either on the day after they actually occur, or on both the day they occur and the next day. Here's what is in the ICS file, for example:

BEGIN:VEVENT
UID:84404659-870e-4fa9-a4c7-285810d1065e
SUMMARY:THANKSGIVING DAY
ORGANIZER;SENT-BY="mailto:someone@corporate.com":mailto:corporateevents@corporate.com
X-MS-OLK-SENDER;CN=Someone:mailto:someone@corporate.com
DTSTART;VALUE=DATE:20081127
DTEND;VALUE=DATE:20081128
STATUS:CONFIRMED
CLASS:PUBLIC
X-MICROSOFT-CDO-ALLDAYEVENT:TRUE
X-MICROSOFT-CDO-INTENDEDSTATUS:FREE
TRANSP:OPAQUE
X-MICROSOFT-DISALLOW-COUNTER:TRUE
DTSTAMP:20081105T162136Z
SEQUENCE:0
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER;RELATED=START:-PT1080M
DESCRIPTION:Reminder
END:VALARM
END:VEVENT

So I see the end date is indeed the day after it should be, which I'm sure is the problem. However, there is also an indicator that it is an all-day appointment which I'm pretty sure is not being used. Tried not mapping the end date, and also mapping the end date to the start date (since most of our appointments for this calendar are all day), but doing either of those makes the dates not show up on the Calendar view at all.

I think I'm totally up-to-date on the iCal parser, the Date module (latest dev version) and the Calendar module (also latest dev version).

Any suggestions on what I could do to make this work?

KarenS’s picture

The only thing in that ical file that tells me this is an all-day event is the proprietary 'X-MICROSOFT-CDO-ALLDAYEVENT:TRUE', which is not an ical standard. Microsoft not using the standard, what a surprise! I guess we could write in something to handle this, but I really hate to do something so hackish :(

I don't see any other way to do it, though, since the ical file incorrectly says it ends a day later.

This would really be an issue for the Date ical parser, though. I'm pretty sure we can fix it there and not have to make changes in this module, but I'd have to think about that. Let me mull that over...

KarenS’s picture

#46 - I'm no expert on Feed API, but I think it has to work the way it does. You can't map the content type because the content type could be used to create feeds that need different mapping. You have to map the individual feeds to account for the values that specific feed is returning and the place you want that information to go. You could use the same ical feed content type to create several feeds coming from different places and create nodes either in a single content type or in different content types.

You can't do any mapping until it pulls back the feed so it has feed data to use in the mapper, so the two step process that will create fields without dates before they are mapped is unavoidable, but that will only happen the first time you use it. Once the fields are mapped you don't do that again.

gustav’s picture

Karen, thanks for that attempt at explaining this to me. But I still don't understand. Why, on the page admin/content/node-type/ical/map can I map the author fields but not the date fields? Should what you just explained about the date fields apply to the author fields as well?

KarenS’s picture

I can't tell you anything else. It works the way I did it, that's all I know.

KarenS’s picture

FileSize
4.72 KB

Here's an updated ical parser module that will allow mapping of other kinds of dates in the date mapper. It also adds in the fix in #41, but does not yet do anything about #47.

alex_b’s picture

Reviewing now, 2 questions:

1) PATCHED FUNCTIONS: _parser_ical_download() is much shorter than _parser_common_syndication_download() - what's the story here?

2) How do you guys feel about rolling the iCal parser into its own project? I'd be very much in favor of this in order to keep projects small and issue queues clean. Anybody wants to maintain this project? If not, I would step up.

alex_b’s picture

Status: Needs review » Reviewed & tested by the community

This is looking pretty good. Ready to be rolled into a project I would say. I will wait for some responses on
#53 - 2) for that.

For convenience and for the meantime I committed the code to my sandbox:
http://cvs.drupal.org/viewvc.py/drupal/contributions/sandbox/alex_b/pars...

There were some minor issues I fixed:

* Minor code style issues
* Cleaned up comment for _parser_ical_download()
* Catch 'webcal' URL scheme and substitute it by 'http'
* Remove FEEDAPI_FEED_NOT_UPDATED comment.

This is really great work. Thank you everybody. I can't wait to use this on a project :) I'm sure there is going to be an opportunity soon.

ekes’s picture

Also tested (with patch) with D5 and FeedAPI 5.x-1.4 working nicely - thanks all :-)

I could make the time to do a proper job of maintaining this one as a standalone, if people want?

ekes’s picture

Status: Reviewed & tested by the community » Fixed

Project started:- http://drupal.org/project/parser_ical
\o/ really nice thanks everyone that took part. I've got this running on a live site already. Sorry anyone I missed off the commit message.

Setting this issue fixed. New issues can go in the module's queue.

kleinmp’s picture

I put a page in the Drupal 'HowTos' using the directions from #45.
http://drupal.org/node/341716

ekes’s picture

Excellent.

I've updated it with the link to the project and added a link from the project page to the HowTo.

alex_b’s picture

It is really great to see the iCal parser come true. Thank you to everybody who helped building it. And a special BIG THANKS for ekes for taking on maintainership.

germs12’s picture

Component: Code » Code feedapi_node

First let me say Thank you to everyone who has worked on this, what a great tool.

Next, I have a question: If I'm using the CCK to build my content type for pulling in iCal feeds which Date type do I use? It appears that CCK allows for Date, Datestamp, and Datetime. All of which are different from the standard that the iCal implementation uses for the DTSTART.

I can successfully pull the feed into Drupal but I can't get the date part to correctly populate. Has anyone done this yet?

Thanks in advance!

germs12’s picture

First let me say Thank you to everyone who has worked on this, what a great tool.

Next, I have a question: If I'm using the CCK to build my content type for pulling in iCal feeds which Date type do I use? It appears that CCK allows for Date, Datestamp, and Datetime. All of which are different from the standard that the iCal implementation uses for the DTSTART.

I can successfully pull the feed into Drupal but I can't get the date part to correctly populate. Has anyone done this yet?

Thanks in advance!

ekes’s picture

germs12 I think what your looking for is in the 'how to' mentioned in the comment above #57 It's not the parsing the ical but the mapping. The sections of the 'how to' about the feedapi_mapper and especially the proposed date field mapper are I think want you want to be looking at.

If I misunderstand you then please open a new ticket on the appropriate module.

ekes’s picture

Status: Fixed » Closed (fixed)

I'll actually close this one to encourage the start of new issues / threads.