Internet Archive + Embedded Video field

This is great- thanks! Now we can use emfield to pull all the DrupalCon Boston sessions! If Aaron doesn't object I'll add this to the dev version so that it will be included in the next major release...

I tried to use this to pull the videos from DrupalCon Boston 2008, and I get the following error:

ERROR: Could not find MP4 video for DrupalconBoston2008-DesigningForDrupal in archive.orgmetadata. The emfield Internet Archive module may need to be updated.

Alex: The Archive very recently switched from 256kbps mp4s to 512kbps. This inc is looking for the 512 version, and I bet those videos were posted long enough ago that only the 256 ones are there.

Hmmm...most of the Drupalcon videos I'm seeing up there are not fully "derived" by the archive--no thumbnail images, for example. And are the videos split into multiple pieces? This inc is assuming that you've uploaded a single video, and that the archive has been able to make derivatives from it.

Maybe this is a little restrictive. but since the archive doesn't really put any limits on what you can do as far as # of files, formats, etc., I think the inc has to make some assumptions.

Is there a way that we could provide for both possibilities (similar to how bliptv.inc works)? If not, this .inc is going to end up taking a lot of work for site developers...

I will see if I can get it to grab whatever mp4 it finds.

OK, made a small change so it will detect any mp4 (just the first one it finds), rather than being picky:
http://www.wccatv.com/emfield_archive

Version "0.1."

Now it will work with older Archive videos, which derived a lower-bitrate mp4.

I'm not sure why the people uploading Drupalcon Boston vids didn't have the Archive automatically derive thumbnails and flvs from them. Maybe this has something to do with uploading via Ourmedia rather than straight to the Archive?

Hi Alex,
It was great to see you at drupalcamp in Philly.

I have a question - probably for either you or Mike:
Are there any plans to support embedded video (and audio?) from archive.org in the 6.x branch of emfield?

We're trying to decide where to host audio and video for a non-profit cultural organization - and then embed that content in the cultural org's 6.x drupal site. --- I'm interested in archive.org because they host both audio and video, and seem to be a good fit with for this particular content (and it would be all in one place, etc). But is there any possibility of embedding that in the drupal site using emfield?

Thanks in advance for your advice :)
Scott

Yes, this will definitely be supported in the not-too-distant future. Anyone could really port the code, since the changes to the provider files aren't that substantial- want to give it a shot? Either way, given that the drupalcon videos are hosted at archive.org, this seems like a "must have" provider.

Hi Alex,
Here's a stab, attached... I compared to youtube.inc and didn't find much to change much at all (I did change the function names, add appropriate comments etc)...

So I can enable 'Allow content from The Internet Archive' in the Embedded Media Field configuration screen, and select it in the Providers Supported form in the Embedded Video field settings for a new field.

But I must have missed something - because while I can get blip.tv videos to work just fine (as a comparative test), I can't get archive.org ones to display (example: http://www.archive.org/details/videojam_521). I also tried the archive.org embed code from that video, and that doesn't seem to work either.

Can you take a look at this inc file to see what I must have overlooked? - Do you have any pointers?

I took a quick look at the inc file and noted one quick issue, you're missing function emvideo_archive_emfield_subtheme().

This inc file needs to be majorly cleaned up before it's committed as well- someone definitely needs to run this through coder...

What will it take to get this officially included in both the 5.x and 6.x versions of emfield?

I need a list of specific issues to fix so that I could hire a PHP developer to fix all of them. I can't tell a developer that the module needs to be "majorly cleaned up."

If someone following this is interested in completing the project for hire, send me a private message.

If I was to assign the task to one of our devs it would cost between $500 and $1,000, depending on the features we built. We also may finish it up at some point in the future, but at the moment all of our volunteer hours are being spent on core testing/patches, the media module, and the Summer of Code. If you're interested shoot me an email.

Alex,

I still need a list of specific issues that need to be fixed so I can compare your rate to my usual
PHP developer's estimate.

I think there might be some miscommunication. I'm only asking for the Internet Archive provider to be cleaned up enough to be included in the emfield module for Drupal 5.x and 6.x. You seem to be proposing new features. I don't need any new features. The existing code is good enough that we use it on Drupal 5.x on a regular basis. I expect it would take an hour at most to finish the port to Drupal 6 and review it enough that you feel comfortable distributing it.

Let's talk about money offline. We shouldn't negotiate price in a public forum.

Nick

How did other contributed provider modules get integrated into emfield? What process did they follow?

Are there issue nodes documenting this somewhere?

Hi Alex & Nick,

I may be able to contribute a bit to this as well. I'm writing an application for a small arts grant this weekend which includes some web development expenses, and archive.org integration via emfield is a part of our proposal – if we get the grant we can help contribute some funds.

One question – is there already audio integration with archive.org in the Drupal 5.x version? We would need both audio and video. In either case, can you estimate the costs of also including integration with archive.org audio with emfield's embedded audio field?

The grant application is due this coming week, so any cost estimates (just so I know what kind of universe of expenses we're talking about) would be helpful for applying :)

@Nick, I really can't go through an pull out all of the issues I found with the module, but there were many. First and foremost, the last time I tried out the Drupal 5.x version (in order to pull the DrupalCon Boston 2008 videos into Zivtech's "Drupal Video Planet") it did not work for the majority of videos. If I recall correctly, some of that had to do with the fact that a listed video might not yet have the correct format available, or it didn't have the format the provider was looking for (divx vs mpeg4). It's possible that's been resolved, but I doubt it, and there was also a big problem with grabbing thumbnails (again, I can't recall exactly what). It also now appears that archive.org has a new player (at least I don't recall seeing it before), so the provider will have to utilize that as well.

Take a look, for example, at this video: http://www.archive.org/details/DrupalconBoston2008-HowIbmIsUsingDrupal
It lists 3 ogg videos, 3 512k MPEG4 videos, 3 MPEG4 videos, and zero Divx videos. So which is the correct one to pull for the user? There's no way to tell, afaik, so you would have to give the user options to select which format and which size they wanted (or, preferably, let them order there choices in the settings).

This is not a question of "getting it into the official emfield releases". If it works, under the circumstances that I/aaron/other maintainers can forsee, then it will be committed. In general we're trying to strip out providers that we don't think are getting used, but this will definitely get included when it is ready. What we will not do is include a new provider that is not ready-for-primetime (even though there are a few in there that fit that definition- we're going to be fixing them up or pulling them out over the next few weeks).

As I said, we (Zivtech) will definitely finish this at some point in the future, since it's a great add-on for the module and, because of the DrupalCon videos, would be a great for the community at large. But, at the moment all of our community time is spoken for by various additional projects. My estimate was that it would take somewhere between 4 hours and 2 days of work to complete (4-14 hours).

@scottrigsby - you might want to consider the new Media module for providers that can give you video as well as audio. This is probably a great reason to push for -enzo-'s single emfield, but unless someone takes care of the upgrade path for that, it will not happen. We assigned our best developer (my partner Jody Lynn) for the last upgrade path--which allowed us to upgrade our site to Drupal 6.x--and it took at least a day's worth of work and troubleshooting (and, I believe still missed some parts- i.e. the fields were inactivated, which there is now a patch for), so we won't be going through that again unless a client pays us to do so. Otherwise, if you don't go in either direction, you'll need two separate providers, and will need two separate fields.

I'm changing the title to reflect the true state of this provider.

@ Alex – just to clarify, it sounds like you're saying there are a few possible routes to take... but one way is to also add archive.org as a provider for Embedded Audio Field (in emfield/contrib/emaudio/providers/archive.inc), separately from the provider .inc file for the Embedded Video Field.

I may be misunderstanding, but wouldn't this still be necessary with or without the single emfield direction? I read the discussion from #172007: ONE field for Video, Images, Audio (Unified Media Field) (though haven't worked with the patch yet), and it sounds like we'd still need an archive.inc file inside both audio and video provider directories. This does sound like a great option! But the two issues are separate, aren't they?
Also, the Media module looks like it makes use of Emfield, but even still - for adding archive.org audio integration we'd still need to add to the Embedded Audio module, right?

oops, I must have started replying before you changed the status - changing back

@scottrigby - there are, in fact, a couple of routes to take, but only one of which would work with the current emfield suite- maintaining separate providers for audio and video. Even though the "ONE field" patch does keep them separate, there's no over-arching reason that I can see to do so, though it would likely take a bit more logic or shifted logic in the way that -enzo- implemented this to make it work.

Things are still a bit up in the air in regards to the Media module, but the idea is that each provider would be self contained and maintained as a separate project (e.g. http://drupal.org/project/media_youtube ). I'll have to let aaron speak to the differences between the emfield provider vs. PHP Stream wrapper...

@Alex - would it be helpful to first get a proof of concept of archive.org audio working as a separate file in the audio providers directory (assuming it could be combined later, once the upgrade path for one field patch is more developed)? Do you have an idea how much work this initial emaudio/providers/archive.inc addition would take? This is for a cultural project planned to launch in a few months (the overall cultural project is what the grant application is for, with a smaller portion for development), and I'd like to go with archive.org because it's more appropriate to the project than the other options (like podomatic etc).

If you have some idea of the approx amount of work for an initial version for the archive.org audio, I can take that into account when writing this grant application. There's no guarantee we'll get the grant, but if so, that could help pay for more archive.org / emfield integration. Do you think the estimate would be similar for audio as the video integration (between $500-1,000)?

@scottrigby, it's hard to say without a lot more research. Ideally it shouldn't be much work at all, a few settings and the setting for posting it to the site. However, emaudio is truly still an experimental module, mostly due to the fact that most audio providers handle services in an odd way, forcing you to grab entire playlists rather then episodes. Even archive.org wants you to pull playlists, rather than single instances of audio. This is very different from images and video, where you are given a single instance and then need to create a mapped FeedAPI instance to grab playlists.

But, if pulling playlists is not an issue for you, then I doubt it would take more than an hour or two to create the emaudio archive.org provider once the emvideo provider has been completed.

@Alex

I tested out the DrupalCon videos that you suggested wouldn't work with the module. It works well. There is a way to know the correct video, and this module finds it and embeds it correctly. It only shows one of the three videos on the page, but none of the other providers support grouping multiple videos together this way so I don't think this is necessarily a bug.

Are there any other outstanding issues? How did you arrive at 4-14 hours work?

Can we move the discussion about emfield audio to another thread?

@Nick - it does seem to make sense to move the audio discussion (whenever we start it back up) to a new issue... at least until the point when there's work toward merging the audio & video provider .inc files like Alex suggested above ;)

@Alex - thanks for the info & advice. I'll let you know if the grant is successful.

Just in case you can't follow this thread:

This module was written for wccatv.com six months ago and has been in use since then. We'd like to contribute it to the emfield project. I have a small budget to make that happen.

There was some miscommunication -- developers thought the module still needed to be "majorly cleaned up" but that doesn't seem to be the case. There are no known bugs. At this point, we've spent way more time dealing with BS on this message board than we spent actually creating the module! I still don't have a clear answer as to how to get this integrated! Is there an official document describing how Drupal community members can contribute? In the future, how could we streamline this process?

@Nick Nasser, please don't post e-mails I send you in private without my permission. It's rude and unprofessional.

Anyway, if you're looking to get this done in some other way (totally your call- other firms may have much faster/better ways of working on projects), I'd suggest posting in the bounty forum. But yes, it will have to be majorly cleaned up before it could be committed to the main release. We did a lot of cleanup during the 5.x->6.x update, and we intend to try and keep things as close to the Drupal Coding Standards as we can. Either way, please do post any patches you make- every step will get us a bit closer to the finalized provider.

And, again, as I said above we'll probably finish this at some point in the future on our own, or aaron might need it for something and make it happen. We just don't have extra community hours to spare right now.

Alex,

If I wanted to hire someone else to do the review, how would I go about doing that?

Nick- I also forgot to mention: developers who are volunteering their time generally don't appreciate it when you call their responses to your questions "BS".

Sorry if I offended you by helping maintain this module you obviously use.

@Alex

I'm sorry I offended you. You understand why I'm frustrated, right? After all this time, I can't even put together a checklist of what needs to happen to get this integrated. That's all I want to know, the criteria.

@Nick Nasser. Yes, as someone who did a fair share of volunteer coordination, I can understand why it's frustrating when people volunteering to help you aren't appreciative enough of the work they've done for free for you. I'm going to leave this post open, but I think this part of the conversation has gone on for far too long.

If anyone wants to submit patches, please feel free. When the code is cleaned up, commented correctly, and has at least a few api options (at the very least the ability to specify a preferred format and size), I promise you it will be committed.

@Alex

Please, forgive my rudeness. I didn't mean to offend anyone.

Just wanted to include the 0.2 version of archive.inc_.txt in this thread.

Rough instructions if you want to play with this:

1. Install the emfield module on your site.

2. Download archive.inc_.txt, rename it to archive.inc, and upload it to yourwebsite.com/sites/all/modules/emfield/contrib/video_cck/providers

3. Go to the "Content Types" admin screen for your site at yourwebsite.com/admin/content/types

4. "Edit" your content type and add an Embedded Video field.

5. Configure the field to work with The Internet Archive.

6. Now when you create a node of this content type, just paste in the URL of a page at archive.org in the "URL" text field. For example, http://www.archive.org/details/videojam_521

7. When you save the node, you'll see a Flash-based player with your video, as well as a link to download the mp4 version, a link back to the original page at archive.org, and whatever description was entered. (Note that you have to add your own Title.)

many thanks to all drupalers. . .

been in the playground for about a year, has provided many publishing solutions that I have been searching for far too long.

archive.org is top of list for video serving, universal access to human knowledge. . . nice

have been working with various versions of drupal and the myriad video solutions that would integrate archive video's into cms. . .

current test site with 5.15 and emfield 5.1.3 with archive.inc loaded, links load but video does not feed into player
http://www.totalvision.tv/test/2004.05.10.02

blip version loads fine
http://www.totalvision.tv/test/2003.12.15.01

emmfield and feed api screencast at drupaltherapy looked to be perfect solution, looking to load both feeds http://www.totalvision.tv/320.240.xml - http://www.totalvision.tv/640.480.xml which are fed to itunes and miro

not a coder yet, have been tempted to dive in but feel time may be better served as tester?
was glad to find thread picked up here after recently seeing wccatv dropping archive.inc development

current site http://www.totalvision.tv has embed code from archive.org directly embedded in each story

look forward to future resolution, and offer whatever help I can provide?

moving this to critical. don't want to miss the opportunity for the new dcdc videos: 60 already in place, with another 60 to come soon. i'll work with anyone interested tomorrow during the sprint.

I meant this to be reply to #4

Hi. Generally thumbs and other format derivation is automatic at archive.org. This is not an Ourmedia feature (although we do have a new, cloud-based upload that allows additional transcoding, faster uploads and keeps retrying the ftp submission when there is a lot of traffic competing for archive.org bandwidth).

Back to our point here, sometimes archive.org will not create the expected derived files, depending on the upload format or possibly some sort of error. In these cases it is possible to re-submit the request to derive (rumor has it you can even tweak the parameters passed to ffmpeg).

Here are some references that may be useful:

http://internetarchive.wordpress.com/tag/ffmpeg/
http://www.archive.org/details/InternetArchiveVideoPresentation
http://www.archive.org/stream/InternetArchiveVideoPresentation/InternetA...

Hi. Just a quick intro and offer to share resources. My name is markus and I work on Ourmedia.org. We have been working with archive.org for a number of years and are familiar with their XML API (yes, they have one based on Lucene). Also, we have encountered many of the issues reported in this thread and have some code that may be useful. Much of this is simply handling the many kinds of errors that can occur during upload and transcoding process or when particular archive servers are off-line.

Anyway, this is just an introduction. I'm at DrupalCon DC 09 Media Sprint and got to talking with aaron, alex and arthur. I still have more to read here and start looking at the code, but hoping to contribute in the near term. In the mean-time, I'm happy to share any info we have learned.

This has been committed to the latest dev release. I'm adding a new full release right now...

From my brief testing tonight, the video component works well =)

One thing i noticed though -- a multi value field has have trouble playing on the same page. Only the first video will play (full screen on all videos is fine).

Please open a new issue for this...

@Alex UA: ok - made new issue #441792: multi value field only plays first video

Automatically closed -- issue fixed for 2 weeks with no activity.