Importing/exporting discography

Possibly the discography feature might end up as a modified version of the discography module.

This is about right.

File uploading feels out of scope right now. Media module is a prime candidate though, let's keep an eye on it.

The short answer is that for a discography feature, file uploads aren't required. A track URL to an mp3 file combined with Soundmanager 2 will do. Users can upload a file via FTP or a cck file upload and link to that.

Somewhat longer answer: DIY audio file handling is complex, rapidly complicates a base install profile and UI and without good transcoding and id3 support is not that useful. Not to mention the added overhead of rendering files for playback/download, local vs remote storage, and permissions headaches if you want private/public files. Media module hints at things to come, but it is still a moving target.

There are several other things I am intentionally leaving out of Pushtape for similar reasons to minimize complexity. The design goal is for a slim & stable extendable base install profile - not a kitchen sink style install profile.

subscribe

Discography feature is available in 7.x.-1.x
For a discussion of Media module and media display see: http://drupal.org/node/1263926

Hi, this is Karl Giesing, aka Karlheinz. I'm the developer of the Discogs module. I'm working on a D7 version right now, but it seems like it would be better if we joined forces on this. Specifically, I was envisioning Discogs to be come more of a "discography framework," and less of a tool for people to import releases from discogs.com. This has significant overlap with this project.

I have some significant issues with how Pushtape handles their discography, specifically: #1409724: having nodes for each individual song. I can confirm that the workflow to create a single album, with tracks, is terrible. It may be fixable, however.

In any case, I think it would be the best thing for everyone if we combined forces. Do you think it would be better for you to incorporate my module into your distro, or for me to drop support for my module and send people here?

@Karl:

I just sent you an email but I think it'd be cool to find a way to collaborate on this. Right now Pushtape has a very track centric approach that allows for modification of track fields and makes it easy to use Views for list views, so tracks as nodes is pretty fundamental to how things work. Album nodes store track data using node reference, and it might be possible to do it in a different way or augment it. If it's just a workflow issue we could also just work on trying to solve that part of the problem.

Interested to learn more about how Discogs works and your goals for it!

@Karl:

I just sent you an email but I think it'd be cool to find a way to collaborate on this. Right now Pushtape has a very track centric approach that allows for modification of track fields and makes it easy to use Views for list views, so tracks as nodes is pretty fundamental to how things work. Album nodes store track data using node reference, and it might be possible to do it in a different way or augment it. If it's just a workflow issue we could also just work on trying to solve that part of the problem.

Interested to learn more about how Discogs works and your goals for it!

@zirafa:
I'm into this. I'll email you again with all the issues I've found and ideas I've had.

Everyone else: keep the ideas coming.

• Would it be possible to have the tracklist display numbers before each track with the possibility to play the tracks on that list and a "more info" and "buy" link next to the tracks? The "more info" link would then bring you to the individual track with videos, lyrics, and so on.
• A base page layout for albums and tracks would be better than just to have fields under each other.
• How about having different sorts of releases (albums, eps, singles)?.

Hi ebieymjunior

Would it be possible to have the tracklist display numbers before each track with the possibility to play the tracks on that list and a "more info" and "buy" link next to the tracks? The "more info" link would then bring you to the individual track with videos, lyrics, and so on.

Definitely this is possible, and is ultimately the goal. It will involve a couple moving pieces/modules to get it all looking right though.

A base page layout for albums and tracks would be better than just to have fields under each other.

There will be templates for the album node, track node, and list displays. Not sure if that answers your question.

How about having different sorts of releases (albums, eps, singles)?.

Probably the easiest way to do this is just to use tags field on the album content type if you wanted to categorize what type of album it is (ep, album, etc).

Thanks for the clear answers!

It might be profitable to look at the Spotify module as well, which integrates with the Spotify Metadata API. It uses the Amazon module to display album covers. It doesn't yet have a 7 version, but they're working on it.
I don't know in what extent it overlaps with Pushtapes or what the Discogs module will try to do, but it might be worth checking out.

ebieymjunior: The fact that there even is a separate Spotify module - which seems to do exactly the same thing as Discogs, except with Spotify - shows, again, how much Drupal needs a centralized Discography framework.

In fact, if I had actually gotten off my ass and made Discogs a discography framework like I should have, Spotify could have just been a plug-in to that, and I'm sure he wouldn't have had to do so much programming.

I think we should see if we can get Peter Törnstrand (author of the Spotify module) on board with this, too...

Edit: Created an issue in the Spotify queue:
#1463006: Integrate with Discogs/Pushtape

any update on this issue and the discogs/pushtape integration?

The Discogs 6 module provides a user interface to the API that is probably what many Drupal users would like, with the concept of a release node linked to a tracks table broadly similar to the Pushtape 7 one of an album node linked by entity reference field to tracks release node.

I agree the Discogs API is capable of being a great time saver. As an example, of what I see as its potential uses, images held in an array could be imported to a multiple value field. There's a default field view, to which can be added the option of display in a Javascript player (e.g. Field Slideshow module). This would be in line with recent discussions on media players for Pushtape.

Do you have in mind adding in artists and label as node types? I ask because the API allows for import of data for them. I have looked at this for my own website, but as a general solution, this would add complexity and I am not sure who else wants that. I mean is this a "would be nice to have" option, or would users like to have the media player issue resolved as a priority, as this (media/audio) stuff has been "often" discussed?

I'd be interested to hear other comments on what you said.

Changing issue title to better reflect this thread.

It sounds like each integration will have a different set of requirements, but all will require some way of importing and exporting discographies. What I'd propose is some way to import discographies using a standardized format. I'm not sure if there are existing format specs for discographies (*edit* Musicbrainz XML, Microformat, Discogs JSON, XSPF XML), but something like that would be ideal. Or even something like a .info file format.

By establishing a standardized file spec it would be easier to import and export discographies into Pushtape (even between Pushtape sites).

I haven't used MusicBrainz before but it looks good so I can see that others might want to use it for Pushtape. As it happens, Discogs provides a JSON object. I think they recently decided to move away from XML for whatever reason.

When I used the Discogs API for my website I was using the Feeds module (which has CTools dependency) and JSONPAth to import the discography. A Views type exportable can be created for each discogs resource (label, artist, release, search) and node type(s).

The Feeds request is the URL string which is a more painstaking way of working than the way Discogs 6 UI currently works, so that may need to be looked at .

The project page for the Features module describes a feature as "a collection of Drupal entities which taken together satisfy a certain use-case." Might a way forward be to create an (import) feature that met a defined set of criteria and then look at widening criteria

• export Drupal discography
• improved import UI
• many discography sources (Discogs, Musicbrainz)

@supertrader6: A Feature is definitely one way of going about it. Certainly Feeds module would be a good way to do generic mapping from external APIs into Drupal, and I could see bundling a pre-configured use case with Feeds and making a Discogs Feature, Spotify Feature, etc...but I'd still think it would be advantageous to come up with a standard exchange flat file format (JSON? XML? Exportable?) to make discographies more "portable".

@gusaus: Coming up with a standard exchange format would benefit other Drupal projects as well. Also, Pushtape Discography can already be installed as a standalone feature.

Lately have been thinking about using YAML to import/export a discography.

Advantages:

• A superset to JSON
• Human friendly format - you can create a YAML file by hand
• Machine friendly, parsers exist for almost every language making it very "portable"
• Drupal 8 already using YAML files for config stuff

So MusicBrainz has a discography standard for XML, Discogs uses JSON, and Pushtape could define a YAML discography format...it could store basic metadata as text strings and store URLs to point to mp3s, artwork, etc.

If anyone is interested, I've committed a proposed YAML implementation for discography over here:

https://github.com/zirafa/discography-yaml

Feedback is appreciated. I'm hoping to also try and get some feedback from others that may have uses for such an implementation outside of Drupal, as it'd be great to get some consolidation around something like this. I could see it potentially being useful as a way to generate Jekyll powered discographies for static sites.

I'm really interested in the implementation for Drupal.

Having looked at github page, I am guessing that the intention is that a user could create/use a Feeds parser and import the contents of discography.yaml to their website. Is that right?

Yes, the idea would then to have a Feeds parser that could import the discography.yaml to your website, creating all the track and album nodes automatically.

And for exporting, there could be an automatically generated file available via a URL, something like

http://www.example.com/pushtape/export/discography.yaml

that would be generated based on the track and album nodes for your site.

A new Drupal module could include a YAML parser e.g. Spyc as a library. The documentation suggest the YAML file is added like:

 $myDiscography = Spyc::YAMLLoad('discography.yaml');

For the purposes of importing to Feeds the new module could create a UI which allows mapping items in the source document to targets (fields). Looking at the example of Feeds JSONPath parser the module should have a function that

Implements hook_feeds_plugins

The module code then creates the form for the UI.

I would like to be able to add values to entity reference fields when I import with Feeds module from the YAML implementation for discography.

Feeds and Entity Reference issues can be located at

Feeds integration for Entity reference fields
Entity reference feeds

Parallel elements in discography.yaml which have the same left justification, and, which are imported to entities/nodes could benefit from having an id (Guid). |This id could be used to on import, when, the processer would lookup the entity/node id in the Feeds Item table.

Two separate passes of the Feeds Import could be used to import albums and tracks (which are hierarchically nested elements in discography.yaml). A workable, alternative method may be a single import utilising Feeds entity reference.

I know I'm a bit late to the party, but I've once again been working on the Discogs project.

I re-designed it as a framework. It should be pretty easy to integrate it with the Pushtape Discography. All you would need to do is convert from a standardized array to something that could save nodes in Pushtape. In other words, write one theme hook.

I just put up the 7.x-1.0-alpha1 version of Discogs. Give it a download and see what you think.

p.s. Still haven't heard anything from the guy who did the Spotify module.

Update from my end:

I'm at 7.x-1.0-beta2.

I just worked all day writing documentation for the Discography Framework (as I'm calling Discogs 7.x). There is online documentation that lists the hooks you need to implement an Entity Adapter (an adapter between the framework and some kind of content type):
https://drupal.org/node/2031967

If you download the Discogs project, there is also an .api.php file documenting the hooks, which includes a skeleton implementation. That''s in the discogs_mediator/docs subdirectory.

I really want this to get some steam. I think it would be very useful for Pushtape, but I don't know how you have the discography set up, so I'm not sure I could write the adapter myself.

But if I'm bugging you in this issue thread, tell me to get lost.

I decided to go ahead and code the module myself.

It's called the Pushtape Discography Adapter. It allows you to use the Discography framework provided by Discogs 7.x to import releases into Pushtape.

Right now, it only imports from Discogs.com, but hopefully some other intrepid souls will write Provider Adapters for other sites (like maybe Last.fm).

@KarlHeinz

I just tried out your Pushtape Discogs.com Adapter and it worked flawlessly. Great work! Let me know if you need help maintaining it.

@zirafa: Thanks! I think it's important to cater to as many users as possible, and I didn't want anyone from Pushtape being left out.

The help I need at the moment is to find any bugs that exist. (I haven't found any myself, but there are always bugs you can't think of.)

Also, spreading the word would be great.

Other than that, I would need help mainly when you make changes to Pushtape. If the Pushtape's general discography architecture remains the same, everything should work, but of course that's always a big "if."

By the way - how is the multi-module framework system working for you? That is, if you decide to include the module and framework in the Pushtape distro, would there be any issues doing so? Pushtape users wouldn't need a bunch of things from the Discography framework (Release Node, etc), and I don't know if Features can pick and choose modules from the same project.

My hope is that the general architecture should remain fairly stable...that is one of the main goals I have at least, but I'll let you know if something major changes.

I'm not sure what you mean by multi-module framework? Are you referring to the Pushtape Feature modules (Pushtape Discography, etc)? Not sure I understand your question.

Sorry if I wasn't clear. I was talking about the fact that the Discogs project is now broken up into different modules.

Say, for example, that Pushtape decided to include, in the distro, the ability to import releases from Discogs.com. You are using Features to do this, which is exactly what you should do. So, in order to include this in your distro, you would need to include dependencies on the Discography Mediator module, the Discogs.com Provider Adapter, and the Pushtape Discography Adapter module.

The Pushtape Discography Adapter is its own project on Drupal.org, so there's no issue there. The other two modules, however, are part of the Discogs 7.x project.

Would Features have to download all of the modules in the Discogs 7.x framework - including the ones you wouldn't want (Release Node, Track Node, and Track Field)? Or does Features have the ability to pick and choose from different modules within the same project?

The reason I ask is because there's been some talk about breaking Discogs 7.x into different projects (perhaps taking over the now-vacant Discography namespace on Drupal.org). However, the consensus among the (very few) Drupal.org people I've talked to is that I should just leave things as they are. To be honest, I happen to agree with them, as breaking it up would be a huge PITA, and make it a lot less convenient for Discogs 6.x users who just want to upgrade.

On the other hand, this would mean that you (and others like you) might be forced to download modules you'll never use.

Thoughts? If something like this is to happen with Discogs, it had better happen very soon, before Discogs 7.x gets adopted to any degree.

Would Features have to download all of the modules in the Discogs 7.x framework - including the ones you wouldn't want (Release Node, Track Node, and Track Field)? Or does Features have the ability to pick and choose from different modules within the same project?

At the moment if a Drupal project is packaged with a distro, the drupal.org packager downloads the whole project including all submodules. Features doesn't deal with the download process directly, it just says "look for module dependencies, turn on modules and set default variables". So making the distro is kind of a two step process: make sure all necessary files are downloaded and then run the install profile. Features give more control over configuration of other modules. Most of the Pushtape Features modules are simply this, configuration containers for a specific goal in mind.

I don't think it's a big deal if extra modules are included but not enabled. Some projects like to have lots of submodules and it's assumed you only turn on what you need, while others like to separate everything into individual projects so the distro maintainer has to explicitly declare multiple dependencies. I think it really depends on what would be easier for the maintainer(s) involved, the users, and the overall goals of the project.

I am currently planning on an export feature which will include the following:

• JSPF per release, to allow shareable JSON feeds
• Embed feature, using an iframe

The URL scheme would look something like:

http://example.com/track/title-of-track/pushtape_export/jspf/cassette.js......

http://example.com/track/title-of-track/pushtape_export/embed?parameters=...

And resulting embed code:

<iframe src="http://example.com/track/track-here/pushtape_export/embed?parameters=..."/>

The embed code could be a configurable content field that outputs a dummy form element containing the iframe code for cutting/pasting.

The aformentioned feature (JSPF and embed code) is now available as part of Pushtape Services (drupal.org/project/pushtape_services) which provides JSON endpoints for tracks & albums, and there is a pushtape discography adapter (https://www.drupal.org/project/pushtape_discography_adpt) which supports import of discography metadata from discogs.