Problem/Motivation

The API module currently does not display JavaScript files or documentation. There is important Drupal functionality that is coded in JavaScript, so it probably should.

Proposed resolution

Adopt a parser for JavaScript documentation, and use its output in the API module.

Options:

a) JSDoc v. 3:
- Project: https://github.com/jsdoc3/jsdoc
- Reference: http://usejsdoc.org/
- This one has been extensively evaluated as of May 2012, and is both unstable, and unusable in its present form. There is a summary of findings, with sample output, in comment #63.

b) YUIDoc
- Project: http://yui.github.com/yuidoc/
- Reference: http://yui.github.com/yuidoc/syntax/index.html
- Preliminary evaluation (comment #65) is not favorable.

We also need to adopt some JavaScript documentation standards, which is a separate issue:
#1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser

Remaining tasks

  1. Choose a documentation parser.
  2. Develop standards for documentation that will work with the documentation parser for JS (separate issue #1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser).
  3. Integrate the parser output into the API module. The suggested process would be to run this generator separately to produce machine-readable output, and then read this output with the API module, save it in the database, and display it like PHP docs we display now.

User interface changes

JavaScript files and documentation would be parsed and displayed with the API module (and on api.drupal.org).

API changes

None.

Original report by Jaza

I was browsing around drupaldocs.org, and I noticed that the newly added JavaScript code is not documented on this site. Drupal CVS (HEAD) now has three JavaScript files - autocomplete.js, collapse.js, and drupal.js - and all up this means that there are 28 JS functions in Drupal core. The AJAX functions make up most of this code, but more recently the collapsible elements functions were added, and no doubt the number of JS functions is only going to get bigger from here on.

I understand that although Doxygen doesn't support JavaScript natively, there is a utility available (a perl script) that allows JavaScript to be parsed by Doxygen:

http://www.stack.nl/~dimitri/doxygen/helpers.html

How hard would it be to get the API module to support JavaScript? We really need to get those JavaScript functions documented in the drupaldocs.org API reference. 28 functions is a lot of undocumented code in Drupal core. I guess this isn't a huge worry for now, seeing that the JS is only in HEAD. But when Drupal 4.7 is eventually released - and although it seems like it's ages away, it's really not that far off - this really should be done.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

drumm’s picture

We actually maintain our own parser. What would the comment format look like?

jvandervort’s picture

"...and all up this means that there are 28 JS functions in Drupal core."

Not even counting the new jQuery functionality and the changes from 4.7 to 5.0.
It's like wading through a dark forest...

agentrickard’s picture

If nothing else, we can view source of the js files if we make parser.inc read js files as HTML. Same for css.

  if (preg_match('!\.(htm|html|js|css)$!', $file_name)) {
    print('Parsing "'. $file_path .'"...<br />');
    api_parse_html_file($file_path, $branch_name, $file_name);
    return TRUE;
  }

Then we could write a new parser element later....

drumm’s picture

I reviewed a patch which added js. The result was shredded javascript.

A blank type, which does not shred the code, could be made as a starting point.

RobLoach’s picture

Subscribing.... I'd really like the Javascript documentation to be available.

beginner’s picture

A parser would be nice, but much can/should be done beside this.

What is also missing:

1- better in-code documentation. drupal.js had good documentation, but collapse.js and most other core .js files are much more sparse. -> individual patches could be filed for each .js file to document them better.

2- a start-up guide is needed to explain how to use those functions, just like the FAPI start-up guide that was written for D4.7.

I am adding the sources I know about in the API section of the handbook:
http://drupal.org/node/205296

moshe weitzman’s picture

i think a parser and integration into api module is essential. the difference between pre api.drupal.org and post api.drupal.org is enormous. javascript in drupal will benefit enormously from doing same.

gabriel.’s picture

Has anyone taken a look at JsDoc toolkit (or, previously, JSDoc)? The documentation syntax is almost exactly the same as doxygen. I think it'd be a good choice (licence issues aside).

If anything, this definitely provides a good starting point I think.

luisrojas500’s picture

hola me interesaria saber si pueden traducir toda esta informacion al español por que estoy cansado de traducirla yo en google ok

luisrojas500’s picture

echo "Este texto me da error me pueden ayudar";
drumm’s picture

This will be made possible with http://drupal.org/node/300031.

caktux’s picture

caktux’s picture

To have a little follow-up from a discussion with quicksketch on IRC, making the api module parse js files would be an option. Would that be related to #300031: Rework PHP parser ?

Carl Johan’s picture

+1

drumm’s picture

http://code.google.com/p/jsdoc-toolkit/ looks like the parser to use.

mfer’s picture

@drumm If we move forward with jsdoc, what would the path forward be to get that up and running?

drumm’s picture

The basic idea is use JSDoc's machine-readable output, I think XML, and save it to our DB, with some unit tests. api_update_branch_files() defines what function is used to parse which files.

ericduran’s picture

Assigned: Unassigned » ericduran
Priority: Normal » Critical

I'm going to take this on. I also upped it to critical, as I think JS is a big part of the drupal code base.

Are we all still in agreement that jsdoc-toolkit is still the way forward?

jhodgdon’s picture

Do we have any JS docblock comments in the code base now, and if so, will this parser cover them? Will it let people use basically the same comment doc syntax we are using for PHP?

jhodgdon’s picture

Actually, that jsdoc-toolkit project looks like it could be defunct, and it is written in JavaScript (it says so anyway), so I'm not sure how that would integrate with the API module. I would think we need a PHP-based parser for the JavaScript code, wouldn't we?

Also, it's MIT license... I'm not sure that's compatible with GPL?

drumm’s picture

PHP parsing PHP works because it has a token_get_all() built in. For JS, I'd want to follow documentation standards from the JS community, and use the machine-readable output of a common documentation parser.

jhodgdon’s picture

Ah, I see.

Then only the license question remains... and I'm unclear on how this would be added to the API module -- or is this just an addition to the API site and not the module?

drumm’s picture

api_update_branch_files() has a list of what functions to use for which file extensions. js would be added and the new function would

  • Send that file to something that knows hows to parse JS and get the machine-readable output
  • Use that to build an array to send to api_save_documentation().

That gets JS documentation into the DB and basic pages. UI work is probably needed to make sure the languages happily co-exist.

jhodgdon’s picture

We'll also have to make a new standards page showing how to document functions and classes in JS I guess, and figure out what to document. They have a good reference to the @tags, but it doesn't really set standards for how to write good docs.

http://code.google.com/p/jsdoc-toolkit/wiki/TagReference

Anyway, Eric: THANKS! I'm sure this will be helpful to have on api.drupal.org -- you are right, there is now a LOT of javascript in the repository. It will be interesting to see what the parser does with it as it is, and how we can make good standards for it.

jhodgdon’s picture

MIT license is GPL-compatible, by the way, at least according to everything I found in a simple Google search (Wikipedia, and the Gnu foundation, etc.)

ericduran’s picture

thanks for all the feedback I now have a path forward.

ericduran’s picture

After doing some investigation I think the native jsdoc-toolkit project requires too much set up work. Mostly because is built on top of Java with rhino.

Instead I'm looking to using a node.js port ( https://github.com/p120ph37/node-jsdoc-toolkit) which works just like the original js toolkit project but requires a lot less set up.

Any comments, objections?

jhodgdon’s picture

That's too bad that the official and active jsdoc-toolkit project won't work out well.

I have to say, looking at the node-jsdoc-toolkit project web site that it doesn't look very promising. It has almost no activity, no stable release branch, only one issue in the queue, and just 4 little spurts of activity. So I would be skeptical about adopting it -- if it were a Drupal contrib module with that kind of activity, I would probably steer clear of it. On the other hand, it looks like most of its code comes from the base project, so maybe that's not an issue.

Anyway... Does it work? And would we be able to easily maintain it going forward (by pulling from the base project) if the current maintainer abandoned it (which seems like it might be possible)?

Regarding "too much setup work" for the base project... you mean someone who wants to run this would need to install a few other pieces of software, or is it more than that, once you/we have figured out the right configuration and commands?

ericduran’s picture

By "too much setup work" I mean there's a couple of pieces of software that needs to be install. Once they get set up it works smoothly, But I think Rhino is just too much work. The node project is pretty light in the sense that they're just bringing in the base project and replacing the run.js file to make it work with node.js.

I'll look into it more. I'm sure that in future releases they will probably also move to node.js being that it would make sense for the js project. But I'll find out.

As a side note the node js-toolkit project takes less than 1 min to set up (Assuming you have node.js install) and generated all the js docs for core in seconds.

I'll have more detail later, I need to get back to work :)

jhodgdon’s picture

So... You are saying that for the base toolkit, you have to install a few pieces of software -- that doesn't sound like a huge burden. Because for this alternate one, you also have to install node.js, which I for one don't have by default on Ubuntu... I'm not seeing a huge distinction here, but having not tried either, I'm still open to being convinced. :)

jhodgdon’s picture

Another note. We have not ("we" referring to "the Drupal Core development group") adopted any standards for documentation of JavaScript code. We need to do that too. So opening up discussion:
#1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser

jhodgdon’s picture

Apparently version 3 of JSDoc is now under development at:
https://github.com/micmath/jsdoc
(thank you ksenzee for pointing this out on the other issue)

EDIT/added: Note that this version of JSDoc apparently includes Rhino, which was cited as a concern of the earlier version above.

jhodgdon’s picture

Ugh: https://github.com/micmath/jsdoc/blob/master/LICENSE.md
(lots of licenses)

We need to be careful about license...

ericduran’s picture

hmm, v3 might be the way to go then. It looks like they might support both node, and rhino.

zzolo’s picture

Just a +1 on this. This would be so very awesome. Possibly even better than the Git migration. Well, up there at least. :) I happy to test things.

jhodgdon’s picture

Priority: Critical » Major

This is really just major not critical.

ericduran’s picture

:-) I guess not.

I've been moving so I've been completely of the grid the last couple of days/weeks. I should start moving on this ticket again some time next week.

jhodgdon’s picture

Version: master » 6.x-1.x-dev

fixing branch (we don't use branch master)

jhodgdon’s picture

@ericduran: Are you still planning to work on this, or should we unassign it?

jhodgdon’s picture

Title: JavaScript documentation needed » Parse/save/display JavaScript files
Version: 6.x-1.x-dev » 7.x-1.x-dev
Assigned: ericduran » Unassigned

It looks like ericduran has vanished from this issue -- too bad -- hopefully someone will pick it up again (or ericduran will come back). This is still important to do.

jhodgdon’s picture

Note that there's discussion going on about JavaScript docs standards on #1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser... and I just created an issue summary for this issue, with an updated link to the JSDoc project.

jhodgdon’s picture

Priority: Major » Normal
Status: Active » Postponed

I took a look at the JSDoc application today (the version at https://github.com/jsdoc3/jsdoc). Here are some notes:

- It was easy to install - I just downloaded the tarball from the web site, and it came with a script that worked fine on my Ubuntu Linux machine (it also comes with a Windows script).
- By default, JSDoc produces HTML output, but it can also be run with a different "template". The alternative template that was supplied with the tarball ("haruki") produces a JSON-format dump of some of the information that is in the HTML output (not all the information, however). I think it could fairly easily be adapted to our purposes (make it print out more information, and parse it in the API module). Looking slightly into the "haruki" template, I see that its input is some XML, so it might be possible just to dump the XML output directly.
- Both the default and "haruki" output seemed reasonable for the jsdoc.js file that was provided in the tarball. HOWEVER, neither produced any output at all when run on the entire catalog of misc/*.js files in the Drupal 7 core source. (The default produced an index.html file, but there was no documentation in it; the haruki template produced an empty {} JSON output.)

Why? Well, when I run JSDoc on the jsdoc.js file, I get two types of output: functions and namespaces. The namespaces in the file look like this:

/**
    Data that must be shared across the entire application.
    @namespace
*/
app = {
...

And the functions look like this:

/** Print string/s out to the console.
    @param {string} ... String/s to print out to console.
 */
function print() {

On the other hand, the Drupal JS code in misc is all stuff like this:

/**
 * Attaches the batch behavior to progress bars.
 */
Drupal.behaviors.batch = {

And even drupal.js, with:

/**
 * Attach all registered behaviors to a page element.
 *
...
*/
Drupal.attachBehaviors = function (context, settings) {

does not produce any output.

So, at least by default, it seems that JSDoc is not recognizing Drupal.behaviors.batch and Drupal.attachBehaviors as being things it needs to save as documentation. I tried a few things to get it to recognize them, but to no avail. So I'm demoting the priority of this issue, and marking it "postponed", since the proposed method for parsing the JS documentation is not going to help us, as far as I can tell. If anyone has any idea how to make good JS documentation out of our Drupal source code (with perhaps a few modifications of how we document our Drupal JS source code), I'd love to hear them, but until then, I don't think we have a way to parse it. I'll update the issue summary too.

jhodgdon’s picture

Issue summary: View changes

Add an issue summary

ksenzee’s picture

I downloaded it as well and played with a copy of drupal.js, as well as a simple test.js I made up. I found the following:

- The docblock can be our normal

/**
 * Does something great.
 */

style. It doesn't have to look like the examples in the jsdoc documentation.

- If you wrap your JS in (function ($) { ... })(jQuery) the parser doesn't recognize anything inside of it. That's a pretty major problem for us.

- Either var test = function () {}; or function test() {} works fine. However, obj.method = function () {}; doesn't get recognized. That's also a big problem for us.

It looks like jsdoc would need some upstream work before it would work for us, so I agree with the "postponed" status on this.

jhodgdon’s picture

Status: Postponed » Active

Thanks! After chatting with ksenzee in IRC, I found this won't-fix issue:
https://github.com/jsdoc3/jsdoc/issues/54
which led me to this tag that we might be able to try using:
http://code.google.com/p/jsdoc-toolkit/wiki/TagLends

I'll look into this...

attiks’s picture

FileSize
10.5 KB

It kind of works for drupal .js if you do the following, attached the changed version as well as the output generated using jsdoc -p drupal.js

/**
 * @requires jQuery
 * @namespace Drupal
 */
var Drupal = Drupal || { 'settings': {}, 'behaviors': {}, 'locale': {} };

// Allow other JavaScript libraries to use $.
jQuery.noConflict();

(/** @lends Drupal */function ($) {
nod_’s picture

FileSize
1.32 MB

There is an alternative, YUI Doc, which is a standalone node.js tool (npm -g i yuidocjs).

Why we should have a look at it:

  1. Has ways of documenting custom events (This is critical, as much as documenting hooks for PHP).
  2. it's easy to install/use/configure.
  3. Well documented itself, battle-tested.

YUIDoc
YUIDoc Syntax
YUIDoc command line options
YUIDoc source
Sample documented file

There is an option to only dump the JSON used to render the templates.

Example Method Block

/**
* My method description.  Like other pieces of your comment blocks, 
* this can span multiple lines.
*
* @method methodName
* @param {String} foo Argument 1
* @param {Object} config A config object
* @param {String} config.name The name on the config object
* @param {Function} config.callback A callback function on the config object
* @param {Boolean} [extra=false] Do extra, optional work
* @return {Boolean} Returns true on success
*/

Example Property Block

/**
* My property description.  Like other pieces of your comment blocks, 
* this can span multiple lines.
* 
* @property propertyName
* @type {Object}
* @default "foo"
*/

I've attached the JSON output of the whole YUI library (the data.json file is 14Mb) to see what kind of output we can expect from that. It comes from yuidoc -p yui3/.

A rendered example of what it can give: YUI API

The license is "YUI BSD" don't know enough to see what changed there.

Didn't have time to try it on Drupal code for now.

The more global issue with documentation is that Drupal JS is not in a state where it can be documented properly. I doubt we'll be able to end up with properly commented JS for D7.

(edit) ok, finally found jsdoc documentation, seems to support custom events too, nice. The overall keywords are kind of a mess though: "mixin", "mixins", "borrows" (?) how are the 3 different? I can't really read the doc for that, it's a [todo]. There are a lot of aliases all over the place too. It's doesn't seems mature yet and it carries some cruft already.

jhodgdon’s picture

RE #45 - thanks attiks! I tried using both drupal.js and autocomplete.js with this method. Adding the @lends Drupal just before the jquery closure opening:

/**
 * @lends Drupal
 */
(function ($) {

and the @namespace Drupal before the initial declaration of the Drupal variable in drupal.js was enough to get some documentation from both. That gave me all of the direct Drupal.(whatever) items in both files, but not the Drupal.(whatever).(whatever) items... so we can generate docs for Drupal.theme and Drupal.jsac, but not Drupal.behaviors.autocomplete or Drupal.jsAC.prototype.onkeydown. The good news is that our PHP-doc-like syntax for @param etc. seems to be picked up fine by JSDoc, so we would not have to adopt their alternative way of writing the docs.

RE #46 - thanks _nod! I did the following to install Yuidocs on my Ubuntu system:

sudo apt-get install nodejs npm
npm install yuidocjs

This put the yuidocs executable in ~/bin/yuidoc ... I ran it on the two Drupal files I've been using (drupal.js and autocomplete.js) and I did not get any useful output at all (lots of errors about tags, but nothing as far as output). It didn't appear to recognize anything in the file. So... if you have any hints for how we might need to modify the Drupal JS files so that YUIdocs would recognize them, _nod< that would be useful...

jhodgdon’s picture

Update on YUIDocs: I did get it to *sort of* work. It turns out that YUIDoc doesn't parse code, just documentation headers. So you have to tell it what each doc block is for. For instance, instead of doing:

/**
 * Does something.
 */
 function abc() {}

you have to do

/**
 * Does something.
 *
 * @function abc
 */
 function abc() {}

So it *might* be possible to add lots and lots of tags to the Drupal JavaScript and get YUIDocs to parse it. I would prefer to use JSDoc if we can get it to work though, because it has some code awareness.

ksenzee’s picture

Re #47: That is encouraging. I tried adding Drupal.behaviors and Drupal.behaviors.whatever as namespaces, and it picked up the docblock for Drupal.behaviors.whatever.attach = function (context, settings) {}. So as long as we declare each namespace it works fine.

Declaring all those namespaces might be kind of a pain but I don't actually hate the idea. In fact it might make it clearer to people what exactly is going on in the code. I have thought for a very long time that it's clearer and more documentable if we declare attach behaviors as

Drupal.behaviors.myModule = {};

Drupal.behaviors.myModule.attach = function (context, settings) {};
Drupal.behaviors.myModule.detach = function (context, settings) {};

instead of using object literal style:

Drupal.behaviors.myModule = {
  attach: function (context, settings) {},
  detach: function (context, settings) {}
};

Although I suppose we could add a namespace declaration either way.

jhodgdon’s picture

I got JSDoc to work! You just have to add a few tags here and there and it will create the output we want.

Attached: revised drupal.js, autocomplete.js, and a patch showing what I changed in those two files. Basically, you have to:
- Make sure each item you want documented has a doc block with /** */ comments (unlike the API module, it won't pick up items without docblocks).
- Add @class and @lends in various places (@class to tell JSDoc to treat a function() declaration as a class, and @lends to tell it that a whole section is part of another class).
It's not too onerous... I think it is viable.

Also I attached a few screenshots of the HTML output (which we wouldn't use directly, but it shows what the JSDoc is recognizing anyway), and the JSON output dump (which is what we would read into the API module, or else we'd read the base XML, which I think I can figure out how to generate).

I think this will work. Any thoughts?

jhodgdon’s picture

Ah. There's a -X option that dumps out JSON-formatted raw output from the parse. Here's an example of what you get from the autocomplete.js file. This is probably what we want to use -- should be able to use drupal_json_decode() to read it in.

attiks’s picture

-X outputs to much, it's used to debug what's going on

attiks’s picture

Issue summary: View changes

update issue status

jhodgdon’s picture

#52/attiks: too much is fine. I need some of the information generated by -X that is not included in the "haruki" formatter, and it will be easier for me to pick out the information I need from that in the API module than it would be to write a compatible formatter.

nod_’s picture

Glad you were able to get going with yuidoc quickly.

As far as the new drupal.js doc is concerned, Drupal.behaviors is nothing close to a @class. it's just misleading, and overall I have a problem with the sloppy keywords of jsdocs. Drupal.behaviors is an object. Also, classes don't exists in JS, You have either an object or a constructor. Anything you're calling class is going to be a arbitrary construct.

We don't have any arbitrary construct we can call "class" in Drupal JS.

I'm happy to go with jsdoc as long as what we tag is what it actually is. and we stick with strict keywords and depreciate/reject aliases that don't help or were kept around as backward compatibility reasons.

ksenzee’s picture

Any object we call "var x = new Y()" on is close enough to a class as to make no practical difference in the mind of most Drupal developers. Yes, Y is technically a constructor, but it represents a class-like object, with methods and properties.

However, Drupal.behaviors is not one of those things; it's more like a namespace. So calling it a namespace makes more sense to me.

jhodgdon’s picture

Let's discuss the specific standards for what to call things over on:
#1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser
JSDoc supports both class and namespace, so whatever we decide on should be fine.

nod_’s picture

yes I agree that it's close enough to a class, It's not though. Assimilating the two just demonstrate how people don't get the difference between classes and objects. As far as behaviors are concerned, namespace is much better indeed.

In the example you're giving, what should be documented? the x or the Y? One is an object and the other is a constructor I don't see any ambiguity as far as I'm concerned. Declaring something as a Class means we have a coherent declaration that uses some wrapper to extend the constructor behind the scene so that the "class" declaration fits in one object, much like YUI does it. It can be confusing for other devs but is it worth the confusion? I'm not convinced.

The js needs some refactoring, that's when we could introduce some class concepts that would fit well with jsdoc.

jhodgdon’s picture

Please take this discussion to the other issue. Thanks.

nod_’s picture

Yep, sorry, from the time I opened the issue and replied I didn't see the update.

jhodgdon’s picture

Status: Active » Postponed

Follow up to the discussion on #1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser (see comment #67 and 69)... It looks like we cannot reliably use the -X output from JSDoc, or the Haruki output from JSDoc, so I don't know if it is possible to use JSDoc after all. I'm indefinitely postponing this issue accordingly. Sigh.

attiks’s picture

Status: Postponed » Active
RobLoach’s picture

I like the idea of using YUIDocs. It would force us to improve our JavaScript docblocks....

jhodgdon’s picture

FileSize
582 bytes
166.79 KB
13.84 KB

Several of us have done quite a bit of testing and tweaking with JSDoc, and I have finally concluded that we cannot use it at this time as a first step for getting JS documentation into the API module. Reasons:

a) It is not stable/mature.

b) Although it has the advantage of parsing the JS code (meaning that it would pick up even undocumented JS items -- "items" being functions, methods, properties, etc.), it doesn't consistently recognize the right names for things (especially inside objects), so you have to put @name tags into doc headers to tell it what name items should really have. However, when you do that, it doesn't connect the documentation block with the item being documented, so it outputs two items: one for the docblock and one for the item. (To see that, see the attached zip file of sample Drupal JS code, and the attached output dump produced from it, and check out what happened with Drupal.settings.)

c) It doesn't currently have a reliable/stable/workable way to make an output dump. It has a -X option, but this is not guaranteed to be stable and isn't valid JSON. It has a "haruki" formatter, but this doesn't contain all the information we would need, and was called "experimental" by the maintainer of JSDoc. So, we would have to maintain our own template (see attached file published.js.txt). This isn't a huge problem in itself, but currently all templates need to be located inside the JSDoc directory (this is probably a bug -- see item A above), so installation would be problematic (and I'm concerned about the evolution of JSDoc making our template engine not work, see item A).

d) The output dump we are able to make is a mix of useful and not useful items. It's not clear to me how the API module would decide which are the useful and not useful items. (See attached output dump produced with the attached zip file of sample Drupal JS code.)

e) The output dump contains both the output we want and run-time messages ("Running Normal"), making it impossible to parse directly (this is probably a bug -- see item A above).

[EDIT - ADDED] f) If JSDoc encounters unknown tags (such as if we wanted the API module to handle additional tags like @code/@endcode), it outputs a complete output item for the tag, rather than just leaving it as unrecognized text within the docblock. This makes it impossible for the API module to extend the basic functionality.

So. Until items (a) to (f) are resolved, we are not going to be able to use JSDoc as our front end engine. I will update the issue summary.

jhodgdon’s picture

Issue summary: View changes

update status

jhodgdon’s picture

The issue summary is updated. I am going to do some work with YUIDoc today and see if we can use it, despite its big limitation (no code awareness -- it just parses doc headers and you have to tell it what each docblock is documenting).

jhodgdon’s picture

OK, I'm looking at YUIDoc (reference: http://yui.github.com/yuidoc/syntax/index.html)... Has anyone used this before? It seems to have quite a few limitations/problems... Attached is a zip of the sample JS I ran it on (excerpts from several Drupal JS files), and the JSON-formatted dump output. If someone wants to take this sample code and make it into something YUIDoc can deal with better, that would be fine, but as of this moment I don't see that this is at all usable.

Problems and limitations I encountered:

a) As has been noted before, you have to add tags to each docblock to tell it what everything is (it is not code-aware). This is extremely tedious, but I guess we could live with it. It adds several lines of redundancy to each docblock (redundancy in the sense that it's duplicating what's in the code). Example:

...
 *
 * @class ajax
 * @namespace Drupal
 * @constructor
 */
Drupal.ajax = function (base, element, element_settings) {

b) There doesn't seem to be a way to document a file (no @file tag or anything similar as far as I can tell). There is also no way to document a namespace (you can tell it something's part of a particular namespace, but there is no way to document the namespace itself).

c) There doesn't seem to be a way to document a function that is not a class method, such as Drupal.checkPlain. It was assigned to the Drupal.tableDrag class (randomly).

d) Generally, when encountering a block with a missing or unknown "primary" tag (such as an attempt at a @file block), it just decides it must be a property of the "current" class, and if there isn't a "current" class, it just picks one that is defined (probably from a different file).

e) I couldn't figure out how this situation should be documented (from the Drupal ajax.js file):

 ... (defines the ajax class) ...
 *
 * @class ajax
 * @namespace Drupal
 * @constructor
 */
Drupal.ajax = function (base, element, element_settings) {
...
}

// NOW, what to do with this??? How do I tell it that this is a property of the ajax class, and that the
// functions within are properties of it?

/**
 * Provide a series of commands that the server can request the client perform.
 */
Drupal.ajax.prototype.commands = {
  /**
   * Command to insert new content into the DOM.
   */
  insert: function (ajax, response, status) {

f) It got pretty confused about namespaces and classes. For instance, this one:

 ...
 *
 * @class ajax
 * @namespace Drupal
 * @constructor
 */
Drupal.ajax = function (base, element, element_settings) {

(which followed the instructions on the YUIDoc reference *exactly* by the way), came out as this element in the 'classes' section:

        "Drupal.ajax.commands.Drupal.ajax": {
            "name": "Drupal.ajax.commands.Drupal.ajax",
            "shortname": "Drupal.ajax",
... 
            "namespace": "Drupal.ajax.commands",
            "file": "ajax.js",
            "line": 82,
...

[EDIT - added] g) The parameters got these things all over them: "~YUIDOC_LINE~!" -- I guess it has a bug?

Unless anyone has any clue as to how to fix these problems, I don't think we can use YUIDoc at all...

jhodgdon’s picture

Issue summary: View changes

Update status

jhodgdon’s picture

Status: Active » Postponed

I'm not optimistic about either YUIDoc or JSDoc in their current states, so I am once again postponing this. If anyone has a clue on how to fix YUIDoc or JSDoc, or if one of them improves and becomes stable/usable, or if there is another parser to try, let's open this up again. Until then, I'm marking this as "postponed".

nod_’s picture

FileSize
22.91 KB
8.46 KB

Things are different with yui terminology i've attached the updated JS files and json output. This is a more or less good output from YUIDoc with not too much messing around with tags and source order. I've said it before but our code is a mess, YUI rely on strict ways of organizing code that we just don't follow. This is what the whole AMD effort is about and one way or another this will happen for D8 so we'll be able to use YUIDoc without abusing it.

What is attached is the best output we can get from yuidoc while having a messy JS code (D7). for D8 we'll move towards a stricter way of organizing JS code that will fit well with what YUIDoc is expecting.

(edit) the yuidoc_line problem is because the @param description needs to start on the same line apparently.

jhodgdon’s picture

nod_: You spent several comments above berating me and insulting my JS knowledge, because I called namespaces anything but "namespace", for calling anything but actual classes "class", and because you thought I could not tell the difference between a constructor function and a class.

Yet here (looking at the output, which is all the API module would see), you've apparently decided that Drupal and Drupal.behaviors (which you insisted above are namespaces) are now "modules", and that Drupal.ajax.commands is a class, which it definitely isn't (it can't be instantiated and has no constructor). Besides which, Drupal.ajax (in the output again) cannot even be identified as "Drupal.ajax" -- it is just "ajax" in the "drupal" module (whatever that is). So based on your own comments above, this is not acceptable. We can't use this output, and the code comments are definitely weird. Still postponed.

nod_’s picture

First I'm sorry, I was not trying to insult your knowledge and I looking back I agree that I've been too blunt. To be clear, the constructor/class was not meant for you but for the group of "most Drupal developers" that ksenzee was talking about and that typically doesn't know the difference in between object/class in PHP to begin with.

Now about YUIDoc, it's not meant for documenting data structures. There need to be an API, modules and methods to document for YUIDoc to be happy. Behaviors and ajax commands, should not be documented at all actually, turning them into a class is the only way to make them show up more or less properly and I agree that it's not acceptable. It will be in the near future. Like you've seen, the way YUIDoc uses namespace is different from what JSDoc does and doesn't fit well for behaviors and ajax commands either.

In the end "Drupal" will be a module, Drupal.behaviors won't exist, there will be proper methods to add/replace attach/detach behaviors that can be documented, same for ajax commands. I wasn't trying to reopen the issue, just show that it's not as that bad if we're looking at this with an better overall architecture for JS code.

jhodgdon’s picture

Apology accepted.

Anyway... If we are going to do this, we need a parser that will cover both our existing (D6/7) code structure with functions, namespaces, and classes, as well as any future structure we might be using (with modules etc.). IMO we should not adopt a parser that doesn't cover JS concepts well. It seems to me that JSDoc (if it ever stabilizes and the problems identified above are fixed at that point) is a closer fit to covering all possible JSDoc structures than YUIDoc (which doesn't seem to even cover the concept of "My JS code is defining an ordinary function here").

jhodgdon’s picture

I will just note that if we ever want to go the route of tokenizing JavaScript ourselves (such as adding the ability to Grammar Parser to parse JavaScript the way we currently do for PHP), this might be of help:
http://timwhitlock.info/blog/2009/11/14/jparser-and-jtokenizer-released/

attiks’s picture

Status: Postponed » Needs review
FileSize
415 bytes

Can we just add them as text files for now so they at least show up?

jhodgdon’s picture

I'll take a look at how they display as text files -- good idea.

attiks’s picture

I worked a bit more on this (jsdoc), for a sneak peak have a look at clientside_validation.js with color coded source view

PS: For the moment namespaces are mapped to classes

Off-topic: I also added color coding to css files

attiks’s picture

nod_’s picture

Would love to have a look, "Access denied" though :)

attiks’s picture

Sorry, try again

@nod_ I would love to hear your feedback on the js-samples

nod_’s picture

pff stupid internet connection made me lost my reply.

Very nice start. It's exciting to see usually hidden code. Seems to be missing events but I guess I'm getting impatient. Events would be one of the first things I'd look for in the js doc. It looks like it picks up everything else though, real nice. It's been a while I don't have jsdoc in mind anymore i'll need to look things up. I know they've been doing a few things since last time.

The biggest questions is to know if it suits jhodgdon. If that's the case I'm happy to spend a significant amount of time throwing doc patches at the issue queue or anything else to help api.d.o have js docs on it.

jhodgdon’s picture

As to whether it suits me: attiks can you give me some idea of what you did (e.g., a patch)? And if you want to discuss CSS files, please file a separate issue.

Anyway, the JS stuff looks somewhat good... It will need some work though... Things I noticed right off:

- The page for autocomplete.js
http://drupalapi.attiks.com/api/js-samples/autocomplete.js/7
doesn't seem to list everything that is in that file.

- The View Source link on a class page
http://drupalapi.attiks.com/api/js-samples/autocomplete.js/class/Drupal....
seems to be listing the entire file, not just the part concerned with the class.

- Constructor functions don't seem to be shown for classes

attiks’s picture

#78 "Seems to be missing events": events are detected, but API doesn't support events, so for the moment they are added to 'throws', see http://drupalapi.attiks.com/api/clientside_validation/clientside_validat...

#79 "autocomplete.js"
Drupal.behaviors.autocomplete and Drupal.behaviors.autocomplete.attach are missing, because the API module cannot find the parent class, the comment is also in the wrong place, it should be something like

/**
 * Autocomplete behavior.
 */
Drupal.behaviors.autocomplete = {
/**
 * Attaches the autocomplete behavior to all required fields.
 *
 * @property {function} attach
 *   Description of the attach function goes here.
 */
  attach: function (context, settings) {

#79 View Source shows the source of the whole file, I created an issue for this https://github.com/jsdoc3/jsdoc/issues/317

#63 f) see https://github.com/jsdoc3/jsdoc/blob/master/plugins/sourcetag.js so intercept unknown tags

attiks’s picture

#79 The link to the source file now include the line number, ex http://drupalapi.attiks.com/api/clientside_validation/clientside_validat..., for the moment you have to expand the 'View source'

jhodgdon’s picture

A few more thoughts:

a) I don't think I would want the JS classes, functions, etc. to be merged in with the PHP classes, functions, etc. So if you are saving the JS stuff in the api_documentation table, you should use different object types. I just don't think that if we're listing Drupal functions, we'd want the JS ones merged with the PHP ones, for instance... it would be too confusing. So, you can create whatever object types you need for JS (jsclass, jsfunction, jsevent, etc. would be good choices).

b) RE #81 - I am not sure what you're talking about. That link doesn't work for me at all (even after expanding View Source I am still looking at the top of the source file, which is why the API module is not currently doing that for its links.

c) Can you please provide a patch? I can't evaluate whether I'd add this to the API module without seeing how you are doing this. Along with a short explanation of what you're doing, where the JS parser code is coming from, if that is patched, what patches will be needed for the docs, how you got around the objections above for the JS parsers I already tried, etc. Thanks.

attiks’s picture

#82
a) Good idea
b) Is fixed and now the partial code is shown (still testing), http://drupalapi.attiks.com/api/clientside_validation/clientside_validat...
c) I need to do some cleaning first, but in short

  • parser is latest version of jsdoc, patched to support code extraction, it uses the -X option
  • lots of small changes against api_save_documentation, but maybe it's easier to split it and introduce something like api_save_js_documentation
attiks’s picture

jhodgdon’s picture

- Yes, probably a separate save function for JS docs in the API module would make sense.
- A patched jsdoc scares me. Do you think the JSdoc maintainers will accept your patch? If not, we have no way to upgrade etc. and will be stuck maintaining our own branch forever. I don't think that is a good idea... but if the patch is only the part that extracts code per-function/class instead of per-file, maybe we can maintain that as a separate function until they add it, rather than patching the JSDoc code itself?

So... attiks: Can you and nod_ confer on:
- Documentation patches that would be needed for Drupal Core to support this -- on a separate issue:
#1337022: [policy, no patch] Create/adopt JavaScript docs standards compatible with a JS documentation parser
- How best to integrate this within the API module... I don't think we want PHP and JS stuff mixed, so we'll need to think about how to make JS stuff findable on api.d.o.

At that point, if you can post your patch, I can work on some of the details of integration with the API module.

attiks’s picture

Regarding #82 a) the problem is that it means we'll have to duplicate a lot of function, almost everything from api.pages.inc and a huge part of api.module.

Since all of these functions know the file type (extension) isn't it easier to use that to differentiate between php and js?

#85 integrate on api.d.o I would alter the search so it searches everything regardless of file type, but display the overviews separately. Maybe add an extra page for 'PHP related' and 'Javascript related' to all projects/branches.

attiks’s picture

Namespace support added, works across files so for the example it shows all 'members' of Drupal.behaviors, http://drupalapi.attiks.com/api/js-samples/drupal.js/namespace/Drupal.be..., keep in mind some of the labels are still using 'class'

jhodgdon’s picture

Regarding #82a/#86 (the problem of where to store/display the JS stuff)... Let's think about this a bit.

a) What types of items will we find in JS files?
- function
- class [is there class inheritance in JS and if so, do we care about it?]
- namespace
- ... ? Can we make a full list?

b) If we're on api.drupal.org and we click on "Functions" in the sidebar nav, for instance, does it make sense to display both JavaScript and PHP functions? I don't think it does. To me, they are separate types of information, and I think they should be displayed on separate listing pages. I have recently made all of these listing pages into Views, so adding new ones is not much of a problem. This will become especially apparent with namespaces, which are totally different in PHP and JS (we'll have namespace pages in the API module shortly -- that is what I am working on currently). So in the sidebar navigation, we'll have something like "Functions", "Classes", "Files", "Globals", "Namespaces", "JavaScript Functions", "JavaScript Classes", etc.

c) On the other hand, in the global search/autocomplete box, we are already mixing up files, functions, classes, etc. and we should include the JS things too. Also, the files should be listed together on the Files pages -- we're already listing a bunch of different types of files anyway.

d) Given that, I think we need the parsing part to add the JS things to the existing api_documentation table, but with different object_type fields (so the PHP items use 'function', 'class', etc., and the JS ones could be 'jsfunction', 'jsclass', etc.). And I think we should add the JS functions to the api_function table too (that stores their param/return information) -- that will facilitate minimal code changes for the Function display pages. Classes and their members should be handled similarly to how they are in PHP too, so that the class member pages will work similarly.

e) Let's divide this effort up:
- attiks and nod and the Core JS maintainers can agree upon the docs standards (I may have some input on this too, like making it as close to the PHP standards as we can where that makes sense).
- attiks and nod can collaborate on making a canonical sample set of documented JS code that we can use for testing in the API module.
- attiks can integrate JSDoc parsing and get the JS items saved in the appropriate tables (parser.inc updates).
- I can work on getting them displayed on listing pages and item pages.
- I can also write the tests.

attiks’s picture

a) object_types

  1. 'class'
  2. 'constant'
  3. 'event'
  4. 'file'
  5. 'function'
  6. 'global'
  7. 'group'
  8. 'interface' I don't think we use it, but jsdoc supports @interface and @implements
  9. 'member'
  10. 'namespace'
  11. 'package' Probably not needed
  12. 'property'

b) I had the same idea, make a clear distinction between PHP and js

c) Agreed

d) I think it's cleaner to add an extra column to api_documentation to indicate the language, PHP, js for now, but who knows we'll add sass support in the future. It makes more sense because they both are functions, we only need to adapt the views to use this to filter.

e) Good plan, but I'm also working on the output for the moment, so most listings and item pages are already covered.

jhodgdon’s picture

I'm OK with that idea of adding an extra column to api_documentation to indicate the language. The Views stuff will need to be updated to use that column too.

And I'm OK with you handling the output. We definitely will need a set of canonical samples and some tests too.

If there really are 12 item types in JS, or 11 that we might use... Maybe we should think about the navigation block. It may not make sense to add 10 new items to the list?

And a heads up: I'm reworking the views for the listing pages now on #1507476: Support namespaces in API module -- adding namespaces and exposed filters. parser.inc is also changing, and a bunch of the tpl.php files. This should be committed sometime in the next 5 days (or maybe even today if we're lucky).

jhodgdon’s picture

Can I just add... This is unfortunate timing. I am really doing extensive modifications to almost everything in the API module due to the namespace issue mentioned in the previous comment. It is going to be difficult to merge in your changes. Sigh. :(((( It would probably be a good idea if you waited until that issue is finished before doing anything else. I hate to curb enthusiasm, but...

attiks’s picture

#91no worries, most of the things I'm doing are javascript specific, so they don't interfere with the rest.

But I have some questions:

  1. jsdoc can report the scope: private, protected, public, static but I have no idea where to store it?
  2. see: to reference another function: same branch, other branch, or external, but don't know how to store this, I stored it as a reference, but the output is strange. Drupal.tableDrag.hideColumns has a @see tag linking to Drupal.tableDrag.showColumns
  3. Can we add an api_params table to be used by function and members so we can store all details (type, name, description, optional, default) separately, meaning we can just build the signature on the fly, simplifying api_page_function and have more flexible theming. If not, can we store this information in a structured way?

Good news: I found a way to link to the documentation, as a proof-of-concept, I added link for some common keywords. See http://drupalapi.attiks.com/api/clientside_validation/clientside_validat... or http://drupalapi.attiks.com/api/js-samples/autocomplete.js/function/Drup...

jhodgdon’s picture

RE #92:

1. Hm. There are similar things in PHP of course for class methods. We aren't currently doing anything with them. Probably would be a good idea... separate issue? I am not sure if we are getting this information from Grammar Parser for PHP, so I'd need to investigate that.

2. The way we do @see currently for PHP functions is that we store it as-is basically, and then at display time we search to find whether it's referencing something that exists. We do it this way because at parse time, the referenced item might not exist yet, or it might change location over time (like moving a function to a different file, which ends up changing its documentation ID, URL, etc.).

The code for handling @see at parse time is in parser.inc/api_save_documentation -- search for "@see" in a code comment. It basically just pulls the @see lines out of the main documentation block and saves them in the "see" field in {api_documentation}.

The code for handling @see at display time is in functions like api_page_file() etc. There are lines like:

  $see = api_link_documentation($file->see, $branch, $file->did, NULL, TRUE, TRUE);

These turn the function names, class names, etc. that were saved in the 'see' field in the database into links to the documentation for those items. Then in the template it generates a "See Also" section if there were any @see lines in the docs.

3. I'm open to ideas on parameter storage, but what's there now is working fine for PHP and I'm not sure I want to parse out the pieces for PHP. I guess the questions I would have for PHP (or JS) are:
a) Where would you get all the information from? (in PHP I am not sure we have that information that you are getting from JSDoc?)

b) What would you do with it at display time that is different from now?

c) What would this extra complexity do for you? (this may be the same as (b))

What I'm getting at is that right now we are storing everything we need for the current display and I don't think I want to break it apart without a good reason.

attiks’s picture

jhodgdon’s picture

That's great that the JSDoc maintainers are receptive to your patches! So are you now able to run with JSDoc directly from Github, or are there still outstanding issues?

attiks’s picture

Now running on latest master

jhodgdon’s picture

Just checking in... what's the status of this, in regards to the JS documentation standards and the code?

attiks’s picture

#97 I was waiting for the conversion to views to be finished, but for the moment not much time to work on this.

jhodgdon’s picture

The conversion of the API module to using Views is pretty much complete. The only thing that isn't using views is the "functions that call this" stuff, which is pretty complicated and may not be done any time soon.

jhodgdon’s picture

Status: Needs review » Active

attiks: are you still working on this? If so, would you care to supply a patch? Setting status back to "active" in the meantime since I haven't yet seen a patch.

jhodgdon’s picture

Issue summary: View changes

update status

nod_’s picture

I think the only solution left is to go with JSDoc since the YUI lib is dead and YUIDoc is based on it.

jhodgdon’s picture

Status: Active » Closed (won't fix)

I had a discussion with nod_ and dawehner today in IRC about whether it would be better to:

a) Put a stand-alone JSDoc site on *.drupal.org (probably want it at api.drupal.org/js ?)

b) Continue working on this issue, which would integrate the parsed JS output into the API module.

It seems as though it's better to choose (a) [EDIT: typo! We want a separate site!]:

1. Much much much much simpler - we can use an industry-standard tool that someone else maintains.

2. Cleaner - really there's not that much cross-linking between JS and PHP so there's not a lot of benefit to having them integrated together. And people would normally be expected to know whether they are looking for JS or PHP. So as long as we have a clear link to the JS docs, why not just keep them separate?

So... This issue went 100+ comments already, and we might sometime need to revive it. I decided to file a separate issue about adding a JSDoc site to api.drupal.org in a subdirectory:
#2501135: Add JSDoc site to api.drupal.org

I'll be filling that issue in shortly with steps to complete etc.

Meanwhile I'm going to close this one as "won't fix".