Hi, I'm hosting a podcast using the FileField Podcaster module, which uses GetID3 to automatically retrieve information about each file. I recently switched hosting over to Pantheon (getpantheon.com) and started having issues with the site. The platform is solid, and we eventually narrowed it down to the fact that my podcast view was causing too much strain because GetID3 was parsing each podcast file every time the view was called. Normally, this wouldn't be an issue, but on Pantheon anything 5MB or over written to Pantheon files/ gets streamed to and from S3 instead of the local cluster. This means that everytime GetID3 parses a file it's downloaded for each piece of metadata it needs to get.

For example, if I want to get the file size, length, and title of a podcast file, it has to download the (25MB) file three times. Multiply that by 25 or 50 podcasts in a feed, and you're now downloading 150 files every time the feed is requested... you can see how this could cause a server to slow down quite a bit.

So, their support team suggested a module which stores the id3 info in the database when the file is added. If that could be cached, then the files won't need to be downloaded every time the view is requested.

I believe this would affect anyone using a service like Amazon S3 for file delivery, so I thought I'd bring it up here, and see if we could come up with a solution.

Thanks!

Here are some excerpts from the support issue at Pantheon, just in case they help:

I noticed that your ID3 tag tool does a bunch of seek()s on the file directly. It may be more efficient to copy the file to the local /tmp and then do all the processing there. Those seek() operations seem intended to read as little of the file as possible when a local copy is available, but it turns into loading the full MP3 over the network, possibly multiple times, when on our network file system.

It's not the location of the MP3s as much as how they're processed to get ID3 information. Ideally, MP3s would get uploaded to a temp directory, get their ID3 information extracted there (and pushed into the DB or cache), and then moved to either the Pantheon files/ directory or other storage.

Just so you know, anything 5MB or over written to Pantheon files/ gets streamed to and from S3 instead of the local cluster.

Yes, I think it is that the id3 mod is loading the files into memory to get the info. Josh suggested a module which stores the id3 info in the database when the file is added.

CommentFileSizeAuthor
#11 cache_id3_info_for-1864762-11.patch1.43 KBtravelertt
#6 cache_id3_info_for-1864762-6.patch1.42 KBbotris

Comments

mglaman’s picture

mglaman
Primary language English
Location WI, USA
commented

Comment invalid now that 7.x-2.x exists.

MickC’s picture

MickC commented
Version: 7.x-1.x-dev » 7.x-2.x-dev

I am using 7.x-2.x-dev and find that
a) metadata is not storing in file_metadata if I upload to S3
b) is IS storing if I upload to to Public files
c) if I go to admin/content/file and change a file stored in S3 to Public, it will let me, and then I see data in file_metadata - if I changed it back again to S3 it changed it appended _1 tothe file name, but the metadata was still in the table.

I'm thinking if I can't get metadata to store on the first read into S3, I may have to make public first, then change later.
Although I've been using AmazonS3, "S3 File System" may be better as it is required by a new module "S3 File System Migrate" which is designed to switch filed across using cron.

mglaman’s picture

mglaman
Primary language English
Location WI, USA
commented
Title: Cache id3 info? » Cache ID3 info for Amazon S3 storage

MickC, thanks for moving this over here.

I think the original issue was that the original 1.x branch didn't use a database to store ID3 information, it did a pull from the file each time. Later in 1.x this was added. 2.x ditches this table and ties into File Entity (less to maintain, use existing architecture, keep things unified.)

So I think it's appropriate to turn this into attention at the usage of S3 and the module.

To be honest this sounds like a bug with S3 integration, not the module. The module just listens on hook_presave() and analyzes the file. However, I'd be more than willing to debug and poke around - except I don't have access to any S3 storage (I'll see what I can do.)

For now, could you try to see what's happening when using the S3 file scheme versus public?

MickC’s picture

MickC commented

I switched from "AmazonS3" to the alternative "S3 File System", and also added "S3 File System Migrate".
Same issue if first uploading to S3 so now the plan is to:
- upload to Public
- add a custom module to write metadata values to Audio fields
- schedule cron to migrate to S3 using "S3 File System Migrate"

It's not a solution but a workaround but I think a perfectly acceptable one.
I doubt there'l be any performance penalty, and it may be complex to program a method of integrating with getID3.

The only other thing to possible try is implement "Amazon S3 CORS Upload" https://www.drupal.org/project/amazons3_cors which does the following: "This module works in conjunction with the amazons3 module to provide direct to S3 uploading from your browser. Bypassing the Drupal file system and storing all files in S3 only."

Potentially this method helps Drupal treat S3 as if it was the Public scheme - however this might be something to try at a later date.

botris’s picture

botris
Primary language Dutch
Location Creuse, France
commented

So what I discovered, thanks to codedumperror, in #2500317: S3 File System does not work with Getid3 is that the getid3 does not support stream wrappers. And therefore can only handle local files.
Basically there are two ways around this:

  1. Save a temporary version and analyse that one. Got this working in a code example over at #2500317: S3 File System does not work with Getid3. This works but you have to proces the result from $getID3->analyze() yourself.
  2. Make the getid3 library work with remote stream wrappers. This would mean either wait for version 2.0 and then make the Drupal implementation of that. Or hack 1.9 to get it to work and make user that the Drupal module uses file_create_url() in stead of drupal_realpath()
botris’s picture

botris
Primary language Dutch
Location Creuse, France
commented
StatusFileSize
new cache_id3_info_for-1864762-6.patch1.42 KB

A third option would be to download the data from S3 and analyze that data.
Although I think this is not the way to go, I needed to be able to analyze files that where already on the S3 filesystem so attached is a patch that:

  • Checks if the file scheme is S3
  • If yes downloads the first 10 bytes of that file, which contains the size of the ID3 tag
  • Then downloads the first X bytes of that file which contains the entire ID3 tag (due to the album art that can be a few mb)
  • Then analyzes that data

This should not be considered a patch that is aiming to make it into the module but a solution you might use for your custom code.

mglaman’s picture

mglaman
Primary language English
Location WI, USA
commented

boris sondagh, thanks for work! Currently on vacation, but will review this on Monday. It is GPL v2 compliant, so if it came to having a bundled and modified library to support remote stream wrappers, so be it - Drupal's tarball handling does this (obviously need to consider issues of bundling.)

dsbrianwebster’s picture

dsbrianwebster commented

Just sounding off. I'm using getid3 + amazons3, applied this patch and it worked great! Thanks Boris!

mglaman’s picture

mglaman
Primary language English
Location WI, USA
commented
Status: Active » Needs review

Whoops didn't realize this wasn't marked for review! If we could get one more person to give it a run and report back with (thumbs up) I'll commit.

botris’s picture

botris
Primary language Dutch
Location Creuse, France
commented

Just reviewing the patch again. I'm still using the patch but line 130 could be deleted, as it's after data is being returned.

travelertt’s picture

travelertt
Primary language English
Location Cincinnati
commented
StatusFileSize
new cache_id3_info_for-1864762-11.patch1.43 KB

Our site is using Storage API and has a different scheme prefix. So I refactored patch #6 to work with any external file with any scheme, after first checking if the file isn't local.

travelertt’s picture

travelertt
Primary language English
Location Cincinnati
commented
Title: Cache ID3 info for Amazon S3 storage » Cache ID3 info for External storage
delzhand’s picture

delzhand commented

#11 gets a thumbs up from me. Definitely works.

botris’s picture

botris
Primary language Dutch
Location Creuse, France
commented
Status: Needs review » Reviewed & tested by the community

  • mglaman committed e14a87c on 7.x-2.x authored by travelertt 
    Issue #1864762 by botris, travelertt, MickC, delzhand: Cache ID3 info...
mglaman’s picture

mglaman
Primary language English
Location WI, USA
commented
Status: Reviewed & tested by the community » Fixed

Thanks, everyone!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

sgdev’s picture

sgdev commented
Related issues: +#3017162: Temp filename approach for getid3_analyze_file() causes race condition

I'd recommend reviewing this new patch I just created: https://www.drupal.org/project/getid3/issues/3017162

The patch committed in this thread causes a race condition with temp files.