There have been a lot of discussion regarding drupal.org's policy on 3rd party code in drupal's CVS repository:
http://drupal.org/node/594704

I think an ideal solution would be to include a downloading mechanism in Libraries API that other modules or install profiles can use.

For example, I'm developing Drupal for Churches and really wanting to include libraries in the distribution to make it easy for new drupal site builders to get started. Something that seems simple like including TinyMCE with the Wysiwyg module in a distribution on drupal.org is impossible given drupal.org's current policy.

If Libraries API could fetch libraries, it would be solve many of the headaches caused by this policy.

CommentFileSizeAuthor
#6 main.library.txt3.11 KBlpalgarvio
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

TwoD’s picture

There's a comment by sun which is somewhat related to this: #542940-7: [meta] Support for downloading libraries via Composer.
The basic functionality required by this issue and for the above one would no doubt be the same, so I'm inclined to mark this a duplicate. However if this issue just deals with how the mechanism would work, the other issue could focus on the drush-part. Having one issue would simplify the discussion itself tho. I'll just leave it up to sun to decide since it's his module...

One of the biggest problems I see with this would be how to get the download paths and the unpacking procedures correct as they will no doubt vary between libraries and possibly also between releases of the same library. And there's the choice of which version to download if two or more modules require different versions of the same library. (I'd say the latest, if one module can't deal with a later library version it needs an update.) There might also be variations of the same version, released as separate 'distributions' tailored for specific uses. Once again, a choice which is not easily automated.

Trying to figure out how Libraries API should choose between libraries required by various modules, and install them, will also cause a lot of headaches. But better once for a single module, than all modules trying to do the same. ;)

philbar’s picture

Would it be possible to set up a procedure to include a libraries.inc file along with the module and make Libraries API read it.

This libraries.inc would be formatted like so...

Library = TinyMCE;
Version = 3.2.7;
URL = http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.2.7/tinymce_3_2_7.zip;

And maybe a way to handle older versions for compatibility between modules:

Library = TinyMCE;
Version = 3.2.7;
URL = http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.2.7/tinymce_3_2_7.zip;
Version = 3.1.1;
URL = http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.1.1/tinymce_3_1_1.zip;

So if another module is only compatible with TinyMCE 3.1 Libraries API will use 3.1 instead of 3.2.

philbar’s picture

Note: I don't know why the second example is automatically hyperlinked all weird. It should look like this:

Library = TinyMCE;
Version = 3.2.7;
URL = http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.2.7/tinymce_3...
Version = 3.1.1;
URL = http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.1.1/tinymce_3...

TwoD’s picture

What happens when two modules say they depend on the same library, of the same version, but the URL to download it doesn't match? This could happen if module A was released first when the library archive could be downloaded from location X, but module B is released later and the location of the library has changed to Y (say a new version was just released, the old one was archived but B has not had time to adapt so they still rely on the now archived package).

Even if we manage to figure out that B was released later, so that URL Y should be used, module A will still be 'broken' when B is not installed because URL X now returns 404 (or if we're really lucky, a 301 'Permanent Redirect'). One approach would be to keep track of all the URLs and try them one at a time until something is downloaded -and then try to install it - or fall back to notifying the user that the library could not be installed automatically, along with some "these steps were attempted"-list and just a link to the library's website. But then we could be back to the same problem of conflicting URLs for the library site (be it with much less risk of happening), we're just putting the responsibility on the user.

Just trying to think worst-case-scenarios...

philbar’s picture

TwoD - You think very elaborately...haha. That's good!

My thoughts are to match the version numbers rather than the URL. Then the module should download from the first URL available, and if the first URL is 404, then it should try the second URL.

lpalgarvio’s picture

FileSize
3.11 KB

my concept idea:

- all scripts live inside /sites/all/libraries/.
- they can be inside subdirs, just like modules. example, /sites/all/libraries/jquery/jquery.ui/.
- any script can be accessed by any module (done i believe)
- scripts can be downloadable, updated and deleted through Libraries API - manually by admin request at Libraries API config page, or by module request (install, update, uninstall)

how to store the information for Libraries API:

the options:
- each module that requires a script should have a file storing the script metadata
- store the metadata under a metadata or repo directory in the Libraries module directory
- store the metadata under a metadata or repo directory in the /sites/all/libraries directory

to keep the code ALWAYS up to date and with single revisions (not multiple versions), perhaps Libraries should be responsible for keeping the repo along with it's entries... but should also be able to read custom entries from within the client module directory.

format for the metadata/repo files:

#comments
[ %library_group% ]
name=%library_name%
description=%library_description%
version=%library_version%
site=%library_web_site%
url=%library_download_link%
group=%library_group%
path=%library_path%

...
another library following (or not)...

example of a metadata/repo file:

#this is a comment
[wysiwyg]
name=TinyMCE
description=A WYSIWYG editor.
version=3.3.9
site=http://tinymce.moxiecode.com/
url=https://sourceforge.net/projects/tinymce/files/TinyMCE/3.3.9/tinymce_3_3_9.zip/download
group=wysiwyg
path=tinymce

name=CKeditor
description=Another WYSIWYG editor.
version=3.4.0
site=http://ckeditor.com/
url=http://download.cksource.com/CKEditor/CKEditor/CKEditor%203.4/ckeditor_3.4.tar.gz
group=wysiwyg
path=ckeditor

name=FCKeditor
description=An older WYSIWYG editor.
version=2.6.6
site=http://www.fckeditor.net/
url="http://sourceforge.net/project/downloading.php?group_id=75348&filename=FCKeditor_2.6.6.tar.gz"
group=wysiwyg
path=fckeditor

[jquery]
name=jQuery
description=The jQuery library.
version=1.4.2
site=http://jquery.com/
url=http://code.jquery.com/jquery-1.4.2.min.js
group=jquery
path=jquery

name=jQuery UI
description=The jQuery UI library.
version=1.8.4
site=http://jqueryui.com/
url="http://jqueryui.com/download?download=true&files[]=ui.core.js&files[]=ui.widget.js&files[]=ui.mouse.js&files[]=ui.position.js&files[]=ui.draggable.js&files[]=ui.droppable.js&files[]=ui.resizable.js&files[]=ui.selectable.js&files[]=ui.sortable.js&files[]=ui.accordion.js&files[]=ui.autocomplete.js&files[]=ui.button.js&files[]=ui.dialog.js&files[]=ui.slider.js&files[]=ui.tabs.js&files[]=ui.datepicker.js&files[]=ui.progressbar.js&files[]=effects.core.js&files[]=effects.blind.js&files[]=effects.bounce.js&files[]=effects.clip.js&files[]=effects.drop.js&files[]=effects.explode.js&files[]=effects.fold.js&files[]=effects.highlight.js&files[]=effects.pulsate.js&files[]=effects.scale.js&files[]=effects.shake.js&files[]=effects.slide.js&files[]=effects.transfer.js&theme=?ffDefault=Trebuchet+MS,+Tahoma,+Verdana,+Arial,+sans-serif&fwDefault=bold&fsDefault=1.1em&cornerRadius=4px&bgColorHeader=f6a828&bgTextureHeader=12_gloss_wave.png&bgImgOpacityHeader=35&borderColorHeader=e78f08&fcHeader=ffffff&iconColorHeader=ffffff&bgColorContent=eeeeee&bgTextureContent=03_highlight_soft.png&bgImgOpacityContent=100&borderColorContent=dddddd&fcContent=333333&iconColorContent=222222&bgColorDefault=f6f6f6&bgTextureDefault=02_glass.png&bgImgOpacityDefault=100&borderColorDefault=cccccc&fcDefault=1c94c4&iconColorDefault=ef8c08&bgColorHover=fdf5ce&bgTextureHover=02_glass.png&bgImgOpacityHover=100&borderColorHover=fbcb09&fcHover=c77405&iconColorHover=ef8c08&bgColorActive=ffffff&bgTextureActive=02_glass.png&bgImgOpacityActive=65&borderColorActive=fbd850&fcActive=eb8f00&iconColorActive=ef8c08&bgColorHighlight=ffe45c&bgTextureHighlight=03_highlight_soft.png&bgImgOpacityHighlight=75&borderColorHighlight=fed22f&fcHighlight=363636&iconColorHighlight=228ef1&bgColorError=b81900&bgTextureError=08_diagonals_thick.png&bgImgOpacityError=18&borderColorError=cd0a0a&fcError=ffffff&iconColorError=ffd27a&bgColorOverlay=666666&bgTextureOverlay=08_diagonals_thick.png&bgImgOpacityOverlay=20&opacityOverlay=50&bgColorShadow=000000&bgTextureShadow=01_flat.png&bgImgOpacityShadow=10&opacityShadow=20&thicknessShadow=5px&offsetTopShadow=-5px&offsetLeftShadow=-5px&cornerRadiusShadow=5px&scope=&t-name=ui-lightness&ui-version=1.8.4"
group=jquery
path=jquery.ui

the directory structure:

/sites/all/libraries/
.......................%library_group%/
................................................%library%/

an example of a directory structure:

/sites/all/libraries/
.......................wysiwyg/
.....................................tinymce/
.....................................ckeditor/
.....................................fckeditor/

inspiration taken from YUM repos. example of a linux YUM .repo file:

[base]
name=CentOS-$releasever - Base
baseurl=http://192.168.*.*/centos/$releasever/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
#released updates
[update]
name=CentOS-$releasever - Updates
baseurl=http://192.168.*.*/centos/$releasever/updates/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

downloading the libraries:

- files should be downloaded to a temporary directory (ie /tmp as defined in drupal config or /sites/all/libraries/tmp)
- Libraries API should auto detect if it's a single uncompressed file (.php, .js, etc) or a package, in the format of .tar.gz/.tgz/.tar/.gz or .zip
- Libraries should then extract or move to the proper directory under /sites/all/libraries/%library_group%/%library_name%/ and then delete the temporary files

considerations and final thoughs:

names and URLs
- jquery libraries should follow the convention and have their directory names as jquery.name. ex, jquery.ui, jquery.aop, etc
- other libraries can do the way they want, like shadowbox, colorbox, lightbox, myuber_script, another.script, etc
- URLs that are too complex (ex, include [], (), =, etc, should be escaped with double quotes ("").

repo/meta files:
- groups could be stored either with "group=%library_group%" variable or [ %library_group% ] tag. i'd pick [ group ] tag if it's easy to do
- the main repo/metadata file should be named just main.repo or main.meta. examples:
/sites/all/modules/libraries/repo/main.library
/sites/all/libraries/repo/main.library
/sites/all/libraries/repo/main.repo
/sites/all/libraries/metadata/main.meta
- custom repo/meta can be stored in the same directory or in the client module directory
- a custom.library/repo/meta file should be included blank in the same directory
- Libraries API should be responsible for automatically loading those files
- the client module meta files should be named %module%.library. example: jqueryui.library
- the main library should include as much Libraries as possible
- Libraries API should only download when asked by a module (ie even if main.library contains 100 libraries, it will only download 1 library ever if only 1 module asks and not everything when running cron or whatever)
- in case a module provides a library that already exists in main.library, it SHOULD OVERRIDE the main.library entry IF the version IS MORE RECENT - BUT ask the user if he wants to UPGRADE or NOT before any action

IMO
i would pick "repo" as the repo/meta directory and "library" as the file extension.
i would choose to store the "main.library" inside Libraries API module repo directory (/sites/all/modules/libraries/repo/main.library)
and have the "custom.library" inside libraries directory (/sites/all/libraries/repo/custom.library)

so, my ideal paths:
main.library - /sites/all/modules/libraries/repo/main.library
custom.library - /sites/all/libraries/repo/custom.library
example.library - /sites/all/modules/mymodule/mymodule.library
example library files - /sites/all/libraries/mygroup/mylibrary

hope this idea is of any good ;)

lpalgarvio’s picture

some libraries and their modules:
jquery - jquery_update & tabs (jquery ui tabs) & core
jquery.form - jquery_form_update & core
jquery.ui - jquery_ui
many jquery plugins - jquery_plugin
jquery.aop - jquery_aop
jquery.easing
jquery.scrollTo
jquery.contextMenu
jquery.cycle
jquery.jcarousel - jcarousel
jquery.jcarousellite - jcarousellite
jquery.colorpicker - jquery_colorpicker
jquery.bt - beautytips
jquery_ui_dialog - jquery_ui_dialog
thickbox - thickbox
colorbox - colorbox
shadowbox - shadowbox
tinymce - wysiwyg
ckeditor - wysiwyg
fckeditor - wysiwyg

it's sad but many modules repeat the same scripts, and with different versions.
that's the case of jCarousel which bundles jquery-1.2.3.pack.js and thickbox.js (v3)

tstoeckler’s picture

Wow, that's quite some text. I think there's a lot of valuable thoughts in there, but there's also a lot of things that have been discussed elsewhere, and are not strictly related to downloading libraries (which this issue should be about).

See the epic #719896: Add a hook_libraries_info(), which recently added a hook for modules to declare their library information. The declaring part is basically done. We want to also support information in .info files (similar to what you call repo files above), but there's no issue for that yet.

#719896: Add a hook_libraries_info() also introduced automatic loading of libraries (libraries_load('example_library')), but as you might imagine there are lot of things to sort out before we can consider this done. Loading of external libraries (#864376: Loading of external libraries) for instance, or library dependencies (no issue yet), the problem of duplicate names (#465908: Scan: Name clashes), caching (#466090: Scan: Cache results), etc. I don't think we will really work on any automated download mechanism until the 'groundwork' has all been done and sorted out.

Also of interest to you might be that sun recently started work on a drush command to download libraries (drush libraries-download). See http://drupal.org/cvs?commit=409402 for that.

lpalgarvio’s picture

sorry, i got carried away and typed in here all i had though up :P

maybe some links pointing here should be provided in those issues (which i haven't read yet, had no clues)

sun’s picture

I also slightly fear that it's too early to design this, given that the other features are not fixed yet.

Nevertheless, some quick feedback:

  • all scripts live inside /sites/all/libraries/.

    Not entirely true. That's the default, yes. But actually, "libraries" behaves identically to "modules", so you can have:

    /libraries
    /profiles/[profile]/libraries
    /sites/all/libraries
    /sites/default/libraries
    /sites/[confpath]/libraries
    
  • they can be inside subdirs, just like modules. example, /sites/all/libraries/jquery/jquery.ui/.

    This is what will bite us badly here and needs to be discussed in depth in #465908: Scan: Name clashes -- when even starting to think about automatic download mechanisms and no human intervention, then I fear we will need to resort to another directory level of "vendor" names, as one vendor will most likely enfore different machine-readable names for each distributed library on its own. In other words, if you download jQuery from Google, then it would have to be stored in /sites/all/libraries/google/jquery/, but if you download it from jQuery.com, then it would have to be stored in /sites/all/libraries/jquery/jquery/.

    Let's please discuss this only in #465908 though. (I'll crosspost this)

  • scripts can be downloadable, updated and deleted through Libraries API - manually by admin request at Libraries API config page, or by module request (install, update, uninstall)

    Actually, I hoped that we'd be able to defer the actual file handling to PluginManager resp. Update module in D7; i.e., just like it works for modules.

    However, I wasn't really involved in those D7 issues/efforts, so I don't know whether it's a pluggable system now that can also support downloading/extracting/updating libraries.

  • each module that requires a script should have a file storing the script metadata

    As tstoeckler already mentioned, this is what #719896: Add a hook_libraries_info() is about. Optional support for .info files may be added later on.

  • store the metadata under a metadata or repo directory in the Libraries module directory

    That sounds like #466090: Scan: Cache results

LPCA, if you really want to help with this project, then it would be ideal if you'd subscribe to all issues of Libraries API, and make sure that you've read the discussions in the existing issues.

@tstoeckler: #909296: Figure out how to document what we've learned and discussed in detail

philbar’s picture

Can we make a list of issues blocking this feature and then mark this as postponed until they are fixed?

grendzy’s picture

I think there's a huge opportunity here - as a maintainer of several contrib modules, there are libraries I'd like to use but am currently forgoing because of the burden on the user during installation. And I also know of many projects that disregard the d.o restriction on 3rd party code because it's so painful (a restriction, btw, that is not present on other platforms, for example the WordPress plugin directory).

If my modules could declare a library dependency, suggest a URL for downloading it, and have the update manager automatically install - that would be amazing!

tstoeckler’s picture

suggest a URL for downloading it

That's the part we're trying to sort out here. But it just isn't that easy. Just as an example, what's the download link for the latest jQuery? And more importantly, is there a way to find that out programatically based on the latest version?

As I've said in another issue, I'm thinking more and more that we have to maintain a list of known libraries, including the latest version (and respectively download link). We should only do that, though, if we can't figure out a way to avoid it, and if we go that route, we need to figure out a bunch of details...

grendzy’s picture

Yeah, I agree maintaining a big list like that would be epic. If that route does become necessary, one way to scale it might be creating a special d.o project for the library data, and recruiting a larger team. For comparison, I noticed the MacPorts team has well over 100 active committers.

On a side note, the Google Summer of Code proposal for a module / theme browser has been accepted: http://groups.drupal.org/node/145159 . This doesn't directly affect the libraries module of course, but I predict that soon user expectations of what's easy to install will be raised considerably.

RobLoach’s picture

Version: 6.x-1.x-dev » 7.x-2.x-dev
Status: Active » Postponed
tstoeckler’s picture

Status: Postponed » Closed (duplicate)

Marking duplicate on the issue in #15. That issue will probably spawn a couple sub-issues of its own, but I don't see any point in keeping 2 general/meta-issues.