- #320562-13: Libraries API contains reasoning for not supporting and scanning recursive directories.

- #287321-6: Add wysiwyg_external_plugins module talks about optional add-ons/plugins for a external library, which are equally hosted/distributed externally, but depend on a certain external library. Those plugins can have arbitrary names, such as 'g2image' or 'iespellchecker'. There, we are referring to plugins for a certain editor. However, it's perfectly possible that a completely unrelated external library has the same name.

A conflict crying for a solution.

Comments

sun’s picture

Title: Library scanner » Scan: Name clashes
markus_petrux’s picture

One possible way to deal with this:

Module A requires library X. But it doesn't necessarily mean that module A uses "X" for the library name. It could be a module option (requested via variable_get() API) with a default value of "X", but the site administrator would be able to change it, from the settings panel of module A, or from the site settings.php script ($conf array), where it is possible to define variables.

Module A could document this in the installation section of its README file, for example.

sun’s picture

So today the idea in IRC was to namespace libraries. I.e. instead of putting them directly into sites/all/libraries, we put them into "meta" categories, f.e.

sites/all/libraries/wysiwyg/fckeditor
sites/all/libraries/jquery/ui
sites/all/libraries/jquery/dimensions
etc.

I have to read up earlier considerations in the Libraries API issue/discussion once more to make sure this approach does not open another can of worms.

libraries_get_libraries() would thus not scan sites/all/libraries, but the second level of directories, record the paths, record their name ("ui"), and key them by ... oh my, maybe that was the issue.

naught101’s picture

sun’s picture

Version: » 7.x-1.x-dev

@markus_petrux: If you're in IRC anytime, please join #drupal-wysiwyg, so we can brainstorm and discuss all of this a bit faster.

sun’s picture

Issue tags: +Libraries

Tagging.

alex_b’s picture

subscribe

RobLoach’s picture

Hmmm, tricky.

tstoeckler’s picture

Just something that popped into my mind. It is very likely that it won't work for some reason, but hey:
If you (being a module) depend on a plugin, which in turn depends on some other library, then you know that that plugin relies on that other library. I would say it's save to assume that. Going from there, avoiding the name clash might be possible if we put those plugins in a subdirectory of the dependency library, that means the FCKEditor spellchecker would be at
libraries/fckeditor/spellchecker
and the super awesome stand-alone spellchecker (?) would be at
libraries/spellchecker
And since whichever module knows if what it means with spellchecker is either dependent on fckeditor or not, every module knows what to ask for.

Downsides:
* Are there libraries who have subfolders with names that are identical to external plugins to those libraries? That would kill this approach. Those would have to be weird libraries, though...
* We would have to differentiate somehow what libraries_get_library('spellchecker') means. But given that there *are* 2 different libraries with the same name we have to do that anyways.

Interested to hear how you are going to rip this proposal apart =)

tstoeckler’s picture

Also something this would not be compatible with:
* Libraries that are plugins to multiple other libraries. Like super-cool-js-filebrowser which can be integraded with TinyMCE and FCKEditor (hypothetical) Are there such things???

sun’s picture

@#9: Yes, your example of FCKeditor actually is one of the libraries that uses its own name as a sub-folder of the extracted library.

sites/all/libraries/fckeditor is the extracted library,
sites/all/libraries/fckeditor/fckeditor is where FCKeditor's internally used scripts and plugins reside.

So, while this wouldn't necessarily prevent what you suggested (since we do not talk about recursion), it would mean that you would have to extract the FCKeditor library into

sites/all/libraries/fckeditor/fckeditor

and in this particular case, FCKeditor's internal stuff would live in

sites/all/libraries/fckeditor/fckeditor/fckeditor

That looks really scary + prone to (user) errors ;)

@#10: Technically, something like http://drupal.org/project/imce could just be a library (and to my knowledge, the author provides it as stand-alone download on his website). Therefore, technically, yes, such libraries can exist.

Perhaps we need another, real-world scenario:

sites/all/libraries/prototype
sites/all/libraries/jquery
sites/all/libraries/lightbox (for which JS framework?)

Also note that we're still talking about JavaScript libraries only. However, libraries can actually be anything from PHP to JS to CSS, or whatever else we didn't think of yet. Therefore:

sites/all/libraries/spellcheck can be

http://martinezdelizarrondo.com/spellcheck/ (FCKeditor/CKEditor)
http://wiki.moxiecode.com/index.php/TinyMCE:Plugins/spellchecker (TinyMCE; actually part of the TinyMCE library, but there's no reason why it couldn't be a separate plugin)
http://www.broken-notebook.com/spell_checker/ (JS)
http://www.phpclasses.org/browse/package/2398.html (PHP)

Of course, the names slightly vary, but that means nothing for this issue.

tstoeckler’s picture

So this is a fact:
- The are 2 different libraries with the same name -
Then there are logically only 2 possibilites to be able to support both libraries
1. You rename one or preferably both of the libraries (ie php_spellcheck, jquery_spellcheck, tinymce_spellcheck)
2. You place them in different directories preferably grouped by something (php-libs, wysiwygs, ...)
That's about all your typical filesystem has to offer.
And both reside to basically the same thing: namespacing of libraries.
But if Libraries API wants to be as generic as possible, there is no way it can get that right. (Is jQuery a sufficient namespace or are there duplicate modules? What a jWYSIWYG which is a jQuery plugin, but also a WYSIWYG, etc.)

So either the module itself has to provide the path it expects the library to reside in, which is pretty bad and not flexible and simply shoves the problem to the module maintainers which then in turn get those weird "FCKEditor doesn't work when PHPSpellcheck enabled" issues for them to solve even though that would have been Libraries API's job.

A solution for this would be a validation callback. It would get a $filename to search in, the $line of code to search, and a $string to search for. If all libraries that are registered are validated, that's it. If Libraries are left unvalidated (which happens if 2 modules register libraries with the same name (because only one can be found, or none if the user decided to put the libraries into subfolders)) then the whole rest of the /libraries directory is scanned for files that match the validation callback. Then that information would be cached somehow and ideally be reused for the next scan.

naught101’s picture

What about library info files? Obviously they'd have to be added on the drupal end, which would be a hassle...

tstoeckler’s picture

That's not an option. You download external libraries from third-party sites and you should drop them into your libraries folder as they are, not add a Drupal-specific .info file. Also, for modules registering library info, see #719896: Add a hook_libraries_info()

Since I'm already following up:
I found a flaw in my logic and also a possible solution.
Contrary to my argumentation there CAN be multiple directories with the same name, because they can be INSIDE different directories. IE you can have a profiles/mycustomprofile/libraries/spellchecker and a sites/all/spellchecker, the former being the FCK spellchecker from a really cool distro and the latter a PHP spellchecker library your experimenting with.

What does need to be unique, though is the internal name that is known to Libraries API see #719896: Add a hook_libraries_info().
So a possible solution would be to allow something like a [alternate names] property. Then we could have the internal names be rather specific, ie 'fckeditor_spellcheck' and 'php_spellcheck', and both would define something like array('spellcheck', spellchecker') as their [alternate names]. Then people could call them (more or less) what they wanted, and in the rare case of collision we could use the 'version callback' to see which one is which.

sun’s picture

Sorry, slightly off-topic, but I have to amend the latest follow-ups here:

Let's not entirely ignore or suppress an .info file based attempt to approach handling of external libraries, please. There is actually nothing that prevents us from additionally scanning certain directories for .info files, parsing them, and registering them as if they were registered via hook_libraries_info(). In the end, that (and actually most) info hooks should only return very simple, structured data, which technically could be written in an .info or .ini style, too.

What I'm trying to say: We're not all knowing. Both ways could, can, and may work. We can build an API that simply supports both. Let's just see, which attempt wins.

The goal after finishing #719896: Add a hook_libraries_info() should be to actually also allow for .info files. Probably needs someone dedicated to work on it. :) As I could imagine that two people favoring different concepts/approaches can only be beneficial to the underlying API. The approaches are not mutually exclusive. Perhaps there's even yet another one, of which we did not think thus far.

lpalgarvio’s picture

sun’s picture

Cross-posting from #681034-10: Download Mechanism:

they can be inside subdirs, just like modules. example, /sites/all/libraries/jquery/jquery.ui/.

When starting to think about automatic download mechanisms and no human intervention, then I fear we will need to resort to another directory level of "vendor" names, as one vendor will most likely enfore different machine-readable names for each distributed library on its own. In other words,

if you download jQuery from Google, then it would have to be stored in /sites/all/libraries/google/jquery/,

but if you download it from jQuery.com, then it would have to be stored in /sites/all/libraries/jquery/jquery/.

lpalgarvio’s picture

in my opinion, it wouldn't exactly be vendor specific, but defined as a category at a main repo (main.library) level by the Libraries API maintainers or by the specific module maintainers, if not present in the main.library

the directory structure would mainly correlate to a script category, instead of a vendor

in the particular case of jquery, there's really a lot of scripts, so it makes sense to store them together as in /sites/all/libraries/jquery/jquery.scriptxyz

examples of categories for /sites/all/libraries/:
overlays/ (lightbox, colorbox, shadowbox, ...)
wysiwyg/ (fckeditor, ckeditor, tinymce, ...)
uploaders/ (swfupload, ...)
jquery/

javascript/ (generic js libs)
php/ (generic php libs)

or even better, others/ (generic js and php libs) to replace those 2 above

these directories (even if empty) could be created when installing Libraries API and further, documentation regarding how to store the scripts, to promote organization of these libraries.

another suggestion:
a library/info/repo/meta file which doesn't contains a group allocation, should default to "others", so it doesn't ever land on the /libraries/ root dir.

sun’s picture

@LPCA: We need to discuss this at macro-level first. I.e., before suggesting actual implementations, we need to analyze the all possible libraries and all actual available possibilities.

I even fear that this is so complex that we need something like a Wiki page (perhaps on g.d.o) or similar, so we can cleanly describe the actual problem space (all possible libraries), and for each available possibility, pros and cons (possible problems), edited and added over time. I.e., this is one of the cases where many different solution proposals exist and it doesn't make sense to have them in separate follow-ups scattered through an issue.

In any case, we need a proper, macro-level summary of this issue. An example of how such a summary should look like may be found here and here.

tstoeckler’s picture

Version: 7.x-1.x-dev » 7.x-2.x-dev

I wanted to post a D8 core issue to move all core libraries in a top-level libraries folder, but then remembered that this ugly thing is still lingering in our queue.
So I'll take a crack at the wanted meta summary.

Problem

Different external libraries can have the same name. Therefore having module maintainers which implement hook_libraries_info or authors of library info files simply pick a library machine name will lead to problems: What happens if two modules implement a library with the same name, even though they are completely different? (Currently: The module with the higher weight wins, obviously not a great solution.) And what if a module wants to load a certain library, but a completely different library with the same name is available (Currently: The completely different library is loaded and Libraries API pretends all is well.)

Possible solutions

Library groups

Idea
Group libraries into meta categories or groups. For instance: WYSIWYG, jQuery, PHP, etc.
So libraries are not located at:
sites/all/libraries/$name/this-is-the-actual-library
but at
sites/all/libraries/$group/$name/this-is-the-actual-library

Cons

  • How should categories be defined? Some predefined categories and a hook come to mind, but that basically leaves it up to the module maintainers to avoid duplication, and could also lead to unnecessary fragmentation of similar categories (think of 'User interface' vs. 'User Interface').
  • We're bound to get the categorization wrong sometime, how do updates to these categories happen?
  • How do we store library information internally? Do we key them by category? But then doing libraries_load($name) doesn't work, on the other hand libraries_load($group, $name) or similar isn't very nice either.

Prefixing

Idea
Instead of doing the namespacing in the file structure (see above), do it with the library machine names. That means that the machine names of libraries have to prefixed strongly enough for duplication to be impossible i.e. jquery_$name, etc.

Cons

  • Both the directory names and the function calls (libraries_load(), etc...) get terribly long.
  • How can we really communicate to module developers to be cautious enough about their naming that duplication doesn't occur?

Alternate names

Idea
Together with the idea above, to overcome the terribly long directory names (but not the function calls) introduce a registry of alternate names that libraries can live under.
That means a library whose machine name is jquery_$name can live in a sites/all/libraries/$name directory.

Cons

  • This doesn't prevent duplicate directory names, so doesn't really get us anywhere

Validation

Idea
Introduce validation callbacks to detect which of the possible libraries with the same name a given one is.

Cons

  • What happens when I call libraries_load($name), which library do I mean, and more importantly, which library is loaded?

Others???

naught101’s picture

Why not do something like android does, and require external libraries to be namespaced when installed in the sites/all/libraries directory? Since this would only really matter for the directory structure, it wouldn't be a huge hassle. You could do it based on the URL of the library's offical website, ie. for jquery: com.jquery, and for TinyMCE: com.moxiecode.tinymce. for projects at more complicated urls like http://sourceforge.net/project/library, you could just have net.sourceforge.project.library

lpalgarvio’s picture

group directories are a bit smaller, less complicated and easier to organize, because it makes less mess for example.

there are a lot of sites with weird names storing projects, and many only store 1 library.
it wouldn't help much IMO, because you might end with with 20 main directories, and each containing a single library - what's it for then...

revised generic groups:
flash/ - flash libs
fonts/ - typography fonts
javascript/ - javascript libs
other/ - any other libs (any that don't fit in other generic or specific groups)
php/ - php libs
silverlight/ - silverlight libs

revised specific groups:
editors/ - editor libs: fckeditor, ckeditor, tinymce, ...
jquery/ - jquery libs: jquery, jquery.ui, jquery.cycle, ...
players/ - media players libs: cortado, video-js, flowplayer, mediafront, dashplayer, ...
overlays/ - overlay libs: lightbox, colorbox, thickbox, shadowbox, ...
uploaders/ - uploader libs: swfupload, ...

any others can be easily added. i concur it will need a wiki page.
i also think it does not need to be enforced, but rather, recommended for now.

tstoeckler’s picture

The last line of my previous post somehow stuck in my head and I thought about it a lot recently.
And the more I do that, the more I'm convinced that we need to maintain an "official" list of libraries, including the machine names. Either as a docs page here on d.o or on a dedicated site (probably over-kill, at least short-term). Exactly for the stated reason. However we "obstruct" the libraries directory by grouping or namespacing or our API by bypassing the need to have a single dedicated name per library, when you do
libraries_load('foo')
in your code, there must be a single independent source to delegate which library is meant. I'm not saying it would be impossible (though probably difficult) for
libraries_load('spellchecker')
to mean different things depending on context, but I think it would be terribly confusing for the consumers of our API. And as stated I'm also against "obfuscation" of our API in the form of
libraries_load('spellchecker', 'fck-plugin')
or whatever.

I don't think maintaining such a list is the best thing since sliced bread, but I'm pretty much convinced by now, that there is no way around it.

tstoeckler’s picture

In light of the latest post here, crossposting from the other issue:
http://drupal.org/node/773508#comment-4630452

tstoeckler’s picture

Status: Active » Closed (duplicate)

Marking this duplicate off #773508: [meta] Provide a central repository of library information (i.e. libraries.drupal.org).

Like the latest posts here indicate, there simply is no way to solve this problem without a central repository of library names.