| Project: | API |
| Version: | master |
| Component: | Code |
| Category: | task |
| Priority: | normal |
| Assigned: | moshe weitzman |
| Status: | active |
Issue Summary
The problem described in this issue is that our large array documentation, especially the Form API Reference and the needed but nonexistent Render API documentation are unmaintainable, and also poorly maintained and not as usable as it could be. This issue is about creating a new way to generate the Form API doc and documents of its type.
This is the summary of this issue by @jhodgdon as of #58 .
Desirable Requirements
- Easy to edit, unlike the current document.
- In the Drupal code base, maintained by patches, so it can get updated with the patch that changes the code.
- Parseable by the API module (see #13 for notes on that)
- (#15, #16) Keep the doc and the definitions as close to each other as possible in the code, so people remember to patch the doc. As a note, the code that defines form elements and properties is: hook_elements/hook_element_info; process, preprocess, and theme functions [these define which properties are used to theme each element]; and examples from existing form builder functions.
- (#28) Ability to define descriptions/doc for both elements and properties, and examples for both elements and properties. So both need to have doc headers/files/whatever.
- (#28) Cross-referencing of examples/properties should only be defined in one place (otherwise it will conflict or not be up to date), and probably on properties, with the ability to say "ALL" (e.g. everything uses the #process property). Cross-reference table generated automatically by API module.
- #34 outlines the desired end product
Suggested Implementations
- #33 - Leave it as it is, but move it into the Handbook
Advantages: No repository access needed for editing
Disadvantage: It's still impossible to maintain/edit, separate from the code, not patchable - #57 - Use Drupal, node references, and Views to build the document
Advantages: Sounds like it would be easy to edit.
Disadvantages: Separate from the definitions, changes cannot be done in patches, not part of API module - #2 - .info files
Advantages: In the code base, we have code in Drupal for parsing .info files
Disadvantages: Tons of little files would result, separate from the definitions, not similar to how we do other documentation (new thing to learn, new thing for API module to parse) - #7 - make things that look more or less like PHP class definitions
Advantages: In the code base, API module already parses classes
Disadvantages: Separate from the definitions - #35 & #36 (see also comments in #48 ) - element headers in hook_element_info, properties defined in element_properties.api.php
Advantages: Some information could be automatically parsed by API module from code (such as defaults), doc for elements is with the code, uses familiar syntax, API module shouldn't have trouble parsing it
Disadvantages: Doc for properties is in a separate file (not sure how to get around that anyway)
Conclusion
@jhodgdon remains in favor of something like #35 & #36 (probably because she proposed it), making sure that everything in #34 can be generated from the doc.
Can we move forward and make this a concrete proposal? I think the next step would be to take a subset of system_element_info() and add doc headers to it, and make a sample element_properties.api.php, and also make sample desired output, and solidify that into a proposal. Obviously, drumm would need to buy in to say "This can be done within the API module", and everyone else would need to buy in and say "it's clear and easy how to maintain this as a coder", and "the output is what we need".
OP by merlinofchaos:
We desperately need to be able to document forms api in API.module. And as we move forward, the new nodeapi stuff is going to need this sort of documentation as well.
I believe we need a way to document that's similar to our existing hook structure. But instead of using functions, we use something that isn't exactly PHP code
Something like?
<?php
/**
* Control the size of a Forms API widget.
*
* has meaning for: textfield, textarea, foo
*/
$api_key = "#size";
?>Note: this is just a napkin drawing, so to speak. I haven't thought this through, but I'd like to get some discussion going on this, because I see the difficulty in documenting forms API and other API like it to be a very serious impediment to developing Drupal. I'm constantly forgetting specifics and have to look in code to figure stuff out. That shouldn't be the answer we give people.
Comments
#1
I would say a more generic "big array structure" documentation is needed. Links, forms, menus, etc, are all big arrays.
#2
I had a inspiration on this - Lets define FAPI properties and elements in .info files. Each element gets a file where it describes self and points to an "example function" which demonstrates its use. Below are example .info files. I think this gives us all the data we need to to substantially recreate http://api.drupal.org/?q=api/file/developer/topics/forms_api_reference.html. We just need some PHP to glue it all together. The "example function" pointer isn't perfect, but it does is always up to date which is a huge plus. I think this system is extraordinarily easy to maintain.
name = radiosdescription = "Format a set of radio buttons."
example = "comment_controls"
And now a property:
name = #optionsdescription = Selectable options for a form element that allows multiple choices.
elements[] = radios
elements[] = checkboxes
elements[] = select
values = "An array in the form of array(t('Display value 1'), t('Display value 2')) or array('return_value1' => t('Display Value 1'), 'return_value2' => t('Display Value 2')) if specific return values are required."
example = "comment_form"
#3
@Moshe - hmm, perhaps instead it should move more to being like the hooks documentation where you can just write comments and sample code? The example above looks about as hard to maintain as the current HTML.
#4
I thought about that but we'd either have to invent a new doc syntax or shoehorn elements and properties into function definitions like element_textfield() and use standard doxygen above that. it gets even weirder for documenting properties.
I don't agree that maintaining the .info files is as hard as the big HTML doc. i think adding a new element or property will be far easier in the model where we just add a new small file. I hate messing with the huge HTML doc because I can break its structure. I think it is pretty clear that the community agrees with me because that HTML file is ugly stepchild who is barely looked after. Further, my proposed new page will use theme('table') for the element+property matrix and so will look more consistent with the rest of the site.
#5
Ok, I agree 100% we need a different process than the current one.
#6
Cool ... even if we invent a new syntax for documenting elements and properties inline, I have no clue how we implement the matrix and show which properties are valid for which elements. I think the array features of .info files are great match here.
#7
I haven't gotten to this level of thinking about the parser yet. The whole documentation parser is being rewritten in the next few weeks. The first step, breaking code into constants, functions, etc, is at http://drupal.org/node/300031. I will do a proper writeup of the whole process in the next few days.
These giant arrays have some similarities to objects, http://jaspan.com/dx-files-abandon-anonymous-arrays-attributes. Maybe we can document arrays as if they were objects
<?php
/**
* A form element.
*
* Apparently codefilter has a spacing bug here, this comment is spaced correctly in code.
*/
$form_element = array(
/**
* The type of the form element, can be radios, checkboxes, textfield, textarea, ....
*/
'#type' => 'radios',
/**
* The element's default value.
*/
'#default-value' => '...'
);
?>
Or maybe we need to make it easy to make tables in existing documentation comments. Doxygen does not appear to define a way to do this. There is a precedent with list building from lines starting with dashes.
#8
#9
@drumm - from your comment, I'm going to assume you are not especially interested in pursuing the .info file solution. Thats a bit sad, since this issue has API module has been waiting for a solution for this since 4.7. If I understand you wrong, let me know. I'm not qualified to implement the parser based solution. Nor do I know how that gets us a matrix based on the individual docblocks.
#10
I think .info files needs more thought and maybe examples. I don't think they area bad idea, but I'm not sure how good they are yet. The info files are a new use of that syntax to learn.
A couple questions:
What would the ideal Form API and other big array documentation look like? Is what we have on the site good or can we do better?
API documentation is maintained and relevant because it is in-code. How can we document big arrays as close to the code as possible with easy syntax?
#11
subscribing. I need more time to digest and get my head to thinking about this before I speak up. I do agree we need something better than what have though. :-)
#12
HAH! You want unintelligible and unmaintainable? Doxygen! http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/panels/pane... . The stuff in that ALIASES definition is what's used to build http://doxy.samboyer.org/panels2/panels_api_plugins_content_types.html, which I think is quite analogous to the forms api reference. At least, I intended it to be when I wrote it. Bottom line, I raise this as what I hope is an example of why really need a system for this (the automated hot interlinking with functions definitions laced throughout that page is CRUCIAL to it being remotely useful) but also as an example of what we really, really must avoid in order to get there. The source doc that generates that output, btw, is here: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/panels/docs...
Anyway, to say something more substantive. There are a few approaches to solutions to this that I can think of, but it's also not totally clear to me what the goal is here. So I'm a touch lost. Are we just trying to implement a system for reproducing the current page that's in HTML? Or do we want something richer - say, such that API docs could automagically turn instances of these big array properties in sample code into links pointing back to documentation on that form item? To be sure, such a thing would be easier if we were to switch over to explicitly using classes for this stuff...but it's far from impossible now.
#13
The parsing architecture I am working toward is:
1. Split file into documentation comment & code pairs.
2. Parse documentation comment into a data structure.
3. Save.
The form API documentation should be in a format that can cleanly go through step 1, either PHP code, specialized HTML or something else. Globals documentation set a precedent for documenting not-exactly-standard PHP, http://cvs.drupal.org/viewvc.py/drupal/contributions/docs/developer/glob.... The same convention can be built up around arrays and other PHP documentation.
Once step 2 is improved, it should be doable to add more @tags. @tags might be added for form documentation.
As always, everything should be as generally useful as possible, a specialized form API hodgepodge would not be good.
The slow option is to wait until API module handles objects (weeks) and form API is fully object-based (years).
#14
subscribing.
#15
One thing to consider: I think it's important to keep the doc and the definitions as close to each other as possible, or they will get out of synch pretty quickly. I have more to say, but will say it later... for now, subscribing...
#16
So I'm thinking about what I had to go through the last time I did a major overhaul of the FAPI Reference guide for Drupal 6, and wondering if the API module could somehow do the same thing.
Here is the process, more or less, that I followed in order to see if all of the elements and properties were in there, and if the matrix of "this property works with this element" was correct -- keeping in mind that we are just documenting the defaults for drupal core, since any theme can add to/override this stuff:
a) Collect all of the elements defined by hook_elements() implementations in any core module. [What is this hook called in Drupal 7?] Some properties are defined directly in the returned array for each element.
b) Take a look at the #process functions that are applied to that element, and see what properties they are using. Maybe these functions could have a special @element-property section to let the API module, e.g.:
/*** Expands checkboxes element into multiple checkboxes.
*
* @param $element
* Checkboxes element to be expanded.
*
* @element-property #options
*/
function expand_checkboxes($element) {
}
c) Take a look at process functions that are applied to all elements, and at drupal_render(), and see what properties they are using.
d) Take a look at the theme function for that element and see if there's anything there.
e) Compile the whole mess together into a totally unmaintainable document. Pull in an example from somewhere. Hope the HTML formatting doesn't break. etc.
#17
Could we do a brainstorming BOF about this at Drupalcon SF? This is tremendously important, and we *can* resolve it. It will just take a significant effort like webchick's original creation of the FAPI reference doc.
#18
I forgot this issue was here, and filed this one yesterday (which I'm closing as a duplicate).
#745214: Automagically generate the Forms API Reference
May have some other ideas on it...
#19
I'm still fond of my proposal in #2. I'm willing to write the code; I'd like a little more +1 before I undertake that.
#20
I don't really like the idea of introducing .info files for form elements and properties, unless we also get rid of hook_elements(). Anything that leaves the documentation of form elements separate from the places they are defined will end up not being maintained properly.
Also, the rest of the API module reads /** */ comment blocks, and I think the form API stuff should be built using /** */ comment blocks too.
My 2 cents...
#21
@jogdon - i don't love it either. but we have been waiting for a magical solution for years now and it isn't coming anytime soon even if we all wished it would. this issue has been open since 2006. time for a "good enough" approach, IMO.
#22
+1... also wishing for something better. But this approach (with some needed additional fields for example code and more description) will get us farther along.
I disagree about it not being maintained. This at least gives us a way to include an API doc change with the issue it's done on, in the same way we manage the hooks documentation. When you change a hook, the system.api.php has to be changed for the patch to go in. This will work the same, and it's straightforward and doesn't involve too much work (I don't think).
If it doesn't work out, we can do something better. I don't think we have to do it "perfect". But we have to get off the starting block. Let's do it.
#23
Can't it be done in an elements.api.php file (or something like that) using doxygen headers, rather than introducing .info files?
#24
The current doc has so very much redundancy because it's trying to be an all-in-one. I think using the api.php technique would leave us with the same unmaintainable mass we have now (at least without the HTML). I'd rather have us maintaining the actual non-redundant info.
I'm operating under the belief that the property is the proper level to document, and that everything can be brought in from that.
#25
I guess that my thought would be that instead of making tons of little .info files, we would instead make one .api.php file, and put the information in there, in some appropriate format, with a docblock in the .api.php file corresponding to each element and each property.
Or better yet, but the element docblocks into the hook_element_info() implementations where the elements are being defined in the first place.
#26
This would work, if we preprocessed it.
So would a larger file. But it deserves documentation *only* at the property level.
So we could make a Doxygen-type file, then preprocess it to grab all the elements and create a cross-referenced doc out of it.
The bottom line is that we should only be creating one level of documentation (properties, I think), and the rest of the cross-reference should be generated automatically.
#27
Subscribing. I don't have a strong opinion on format: whether it's .info files or a big api.php file or something else, as long as it's not a giant HTML file that's scary to touch and that's hard to read diffs for. I agree with #22 that the big win would be something whereby a patch on an issue that adds/removes/changes a property/element-type could also include the changes to documentation for it. We've been successfully rolling patches that add a new drupal_alter('FOO') step in one piece of code, and then in a separate file, document the resulting hook_FOO_alter(). Maybe it's a bit annoying to do that, but it hasn't been too bad. What's been making it work is that a single patch file can contain both changes and be readable.
I'm not sure we can get around having two levels of documentation. One level for element types (those need at least a description) and one level for properties. I could see documenting element types in the docblocks of hook_element_info() implementations. I could also see documenting properties in the docblock for hook_element_info() itself. But I can also see using .info files.
#28
RE #26, #27: Yes, we really do need descriptions on both elements and properties, and examples for both elements and properties. So both need to have doc headers/files/whatever to document them. I think?
But you are right (#26) that the cross-referencing of examples/properties should only be done in one place (otherwise it will conflict or not be up to date), and I think it should be on properties, with the ability to say "ALL" (e.g. everything uses the #process property).
#29
I'll also add that an advantage to .info files is that once we have them structured in a way we like, we could probably remove hook_element_info() and use the .info files for real, not just for documentation. OTOH, a disadvantage of .info files is that rolling patches that add/remove files is annoying.
#30
I don't think we want to write yet another .info file parser in Drupal core. The hook is more Drupal-ish, IMO, and isn't there a hook_element_info_alter() too?
#31
Re #30: perhaps we don't want to replace hook_element_info() with .info files, but if we did do that, it wouldn't violate Drupal practice with respect to hook_*_info_alter(). For example, we have hook_system_info_alter(), but what it alters is parsed .info files: there's no hook_system_info().
#32
Now that we have a real parser finding documentable items from PHP, there are a lot more possibilities than there were a few weeks ago. Traversing through code like #7 is doable now.
#16 reminded me I want to do more recognizing Drupalisms, like callbacks, hooks, and alter functions. Those are separate issues, like #294218: Turn hook names in theme() and module_invoke() into links and #745204: Turn strings that are callback function names into function links.
I think we should take a step back and make sure we know what the end result should be.
* Quickstart guide- that is still a lot of HTML, lets make it a handbook page.
* Reference is a summary, defaults, '#type' => 'elements', and #properties
* Is there any value in having defaults separate? I think that can be part of the latter two sections.
* The elements and #properties- we want one page for each item. What would a good API.d.o page look like for each? What would good in-code documentation look like?
* Once we know the ends, filling in the middle should be relatively straightforward.
* Let's not worry about the summary for now, assume whatever fills out the rest can be queried and rendered.
* This leaves two big new things to get right and linked to each other. Would tackling something more-flat, like hook_menu(), be a good thing to use as a test case, or do we want to go straight for the hard problem?
#33
Quick note...
Issue on moving things into the Handbook already exists, and already suggests Form API quickstart gets into the Handbook:
#642976: Move 3 documents out of contributions/developer/topics and into the handbook
#34
My thoughts on the ideal end product:
a) Each form element type should have a page with the following sections:
- Name of the element
- One line description
- Long description
- Links to all associated properties
- Default values for properties that have defaults (perhaps just the hook_element_info() definition for the element would be good enough, but some defaults are sort of assigned in preprocessing functions, I think?)
- Links to its default process and theme functions
- Links to all forms in core that use this element (like the "functions that call this function" section on function pages).
- A good usage example (could be a link to a particularly illuminating use in core, or a @code/@endcode section in the docblock)
b) Each property should have a page with the following sections:
- Name of the property
- One line description
- Long description
- Links to all elements this property is used on
- Links to the processing functions that use this property
- A good usage example.
- Maybe links to all forms in core that use this property, but that could be overkill.
c) [Note: this added July 2010] Automatically generated cross-reference page, with matrix similar to current FAPI guide, with links to each element/property page.
That's what I think would be ideal.
#35
Possible format for an element header within hook_element_info() implementation:
/*** Text field form element.
*
* Corresponds to an HTML input element, with type="text".
*
* Put some information here about autocompletes.
*
* @example
* @code ... @endcode
*/
$types['textfield'] = array(
'#input' => TRUE,
'#size' => 60,
'#maxlength' => 128,
'#autocomplete_path' => FALSE,
'#process' => array('ajax_process_form'),
'#theme' => 'textfield',
'#theme_wrappers' => array('form_element'),
);
In this case, the array there would be displayed on the api.drupal.org page, which would give you the defaults for size, maxlength, and autocomplete_path. Ideally, a link would be generated to theme('form_element'), theme('textfield'), and the ajax_process_form() functions automatically.
#36
Possible format for a property definition in an element_properties.api.php file -- I'm just going out on a limb here and defining new doxygen tags -- not sure how else to do this?
/*** @elem_property #options
* Selectable options for a form element that allows multiple choices.
*
* @element radios
* @element checkboxes
* @element select
*
* @value
* An array in one of two forms:
* @code
* // Return values are same as displayed values.
* array(t('Display value 1'), t('Display value 2'))
* // Return values are different from displayed values.
* array('return_value1' => t('Display Value 1'), 'return_value2' => t('Display Value 2'))
* @endcode
*
* @example
* comment_form()
*/
In both this and the element idea above, @example could be either a function (which would turn into a link) or a code example.
#37
Tackling smaller problem first might make sense. :) Suggest:
#745204: Turn strings that are callback function names into function links
Pieces of that would be useful here.
#38
We already have our own documentation system, so it is okay to go off-Doxygen. And these smaller issues covering Drupalisms do justify having our own parser. The official list we should use as a guide is http://www.stack.nl/~dimitri/doxygen/commands.html. If something fits decently, we should use that.
#39
OK. There is http://www.stack.nl/~dimitri/doxygen/commands.html#cmdexample -- @example indicates that the entire docblock is an example. So we shouldn't use that (I think it's a bad idea to use @commands in a different way from official doxygen?)...
I'm not seeing anything else that would apply to the above ideas.
#40
Just a note, because it came up on another issue: We need to be sure to document interal-use properties as well. They should probably be marked as mostly internal use. See #279246: FAPI #array_parents is not documented
#41
OK, so I'm worried. We've talked, but I don't see us having any clear goals or timeframes. We need to solve this problem. Earlier, Moshe agreed to take it on, but it looks like the tack we took has taken it out of his hands?
Who is working on this? How will it be solved? How can we help?
-Randy
#42
I shouldn't barge in here because I'm a Drupal newbie, but I'm hugely into documentation. My experience with Drupal's documentation is similar to Sun's Javadocs - it's very frustrating because:
I think a wiki style is far more flexible (errors don't persist forever) and allows more contributors to help the documentation do what it's supposed to do: convey accurate information clearly.
I would suggest that design elements not contain the final documentation. Could the code documentation actually be in fields that are read from and written to a database that is also the back-end for wiki-like pages? (I'm sure someone has already said this...)
#43
Drupal has both wiki-style doc (click on Documentation on drupal.org) and API doc (embedded in the code, which is really IMO the only viable way to maintian API doc, and is standard across the industry for just about any software that has a documented API). The API doc is updated continuously, with every checkin, not with every release. IMO, it needs to be reviewed, using the patch process we have in place.
If you find documentation on api.drupal.org that is badly written or erroneous, please file an issue. There is a link at the bottom of every api.drupal.org page (if you are logged in to drupal.org). I personally have made it my mission to make the programmer doc on api.drupal.org better, but I cannot do it alone.
If you find documentation in the drupal.org handbooks that has issues, you can file an issue with project "documentation". Or you can log in and (in most cases) edit it yourself.
#44
How to add new types of doc blocks:
1. api_documentation_loop() needs to pick up the doc block and associated code. This is the place to pull out what is needed from grammar parser objects. That will get the basics in the DB.
2. Add UI to make listings and pages for the new type of doc block. This is a lot of copy paste from your favorite of functions/groups/globals/constants
3. Add additional @tag parsing to api_save_documentation() as needed.
#45
Just another note that eventually, in the FAPI reference, each form element should have links to all of its default process functions.
#46
Maybe the process functions are the place for extensive documentation of how the form elements are built, examples, etc.?
My reasoning is that:
a) You can override the process functions if you want to, or add to them.
b) People should be aware of process functions and how they work.
c) It is really the process functions that define what properties of the form elements are used, and how they are used.
#47
We don't just need to handle the direct array stuff - $form_state, for example, is a large and complex data structure with hidden meanings everywhere, and there is very little about it in the FAPI reference. We need to figure out how to document $form_state as well.
#48
Ideally, those docs would not live within regular code, but in separate PHP files. Not necessarily in the existing $module.api.php files.
Documenting element types and renderable array properties via process functions would be the wrong place, as some types do not have any process functions in the first place. Some only define #theme_wrappers, some others define a #theme function only. All of those function and callback names heavily differ and documentation should not force us to use certain function names.
Initially, links to those callbacks should be done via @see, like everywhere else. Requires manual work and maintenance, but automated linking of property values is a different issue, and we need to solve the most pressing documentation issue first (removing that bloat of HTML and moving the docs into code).
#type "page" is not a Form API type.
#type "hidden" can only be used within forms.
#type "fieldset" and others should be able to be used in forms as well as any renderable array.
Could perhaps be solved via @ingroup.
Again, the already existing
@ingroup formstags would allow us to identify the scope/context of a function and selectively apply references and links to element types that are valid for forms.Of course, this is based on the assumption that the parser is not able to know of what kind a certain function is (i.e. a page callback, a form constructor, a form alter hook, a hook_menu() implementation, etc.).
#49
Subscribing.
#50
This is much better lived in .api.php files than a .info file. Then PHP IDEs could pick up on it, as well PHPDocs.... Subscribing as this would allow the API changes to go along with the actual documentation through a single patch instead of after the fact.
#51
Sounds like most of us are in agreement that we want to use .api.php files and not .info.
#52
Sure.
Problem is, there is not a patch or even a design doc for that solution. Its just wishful thinking, despite lots of people thinking about this for over four years. rfay groks this in #41 ... drumm says we have a parser now but I don't see anyone writing code or even a spec for it.
I proposed a solution thats fully designed and can be implemented in a week. But hey, you guys can keep parroting about how much better api files are. It would also be great to eat ice cream every day instead of vegetables.
#53
Moshe: see #35-36.
All: I'd much rather have help on bugs and critical release blockers. This is a feature.
To move this forward, we need a full sample file for tests/sample/….php and tests/api_update_branch_php.test. Same for any formatting, but I do prefer PHP to info files. Shouldn't take more than a few hours to get the basics parsed.
The front-end code of API is a bit of a mess and I hope to improve that, which would make the front-end of this less-painful than it is for classes. The file format does not affect this at all.
I do not expect this to be solved soon, stick with what we have for the D7 docs.
#54
For API, this may be a feature, but for Drupal development and documentation it's a must-have BUG, with much waiting on it.
I know there's a conflict between those two, and I respect your time and effort, @drumm, but suspect that you might underestimate the overall significance of this problem on the core development effort. Instead of important items being documented as we go along in the commit cycle (and being insisted upon there) they get pushed off to "needs documentation" land, which is a death-knell. And the developers don't really take responsibility for that.
I'm changing this to "bug" and to "critical" because IMO that's what it is in the land of Drupal. Even though this is not an API bug, it really is a bug, and we really have to fix it, soon.
Alternately, we can do this outside of API module, if it looks like API will be the blocker, but it would certainly be preferable to consolidate functions like this in API module.
Again, @drumm, *ALL* your wonderful effort over the years are appreciated, and your opinion on this is understood, but we disagree on the priority.
-Randy
#55
This can be reprioritized after a release of API module happens. There are too many improvements to delay further. There is always next release cycle.
#56
(The next cycle will be shorter.)
#57
I am currently editing the FAPI reference.
I will go on strike and never do this again. This is a travesty. I just had to add an element type and I messed it up. It means editing every row (and the comment indicating the correct row is in the wrong place).
Let's quit talking about this and do something sustainable to solve it, or we won't have a FAPI reference.
You know, we could just make an additional new Drupal instance just for large arrays, and use nodereferences to build the docs using views.
Please, put me out of my misery!
#58
Welcome to the club Randy. As the previous "person who edits the FAPI reference", I decided after my last edit to postpone all the "edit the FAPI" issues, as you may recall, in favor of figuring this out. Then you decided to take it on, and I am not surprised that you have come to the same conclusion.
So. I decided a summary of what has transpired above could be useful here. Comment numbers are given for reference.
Desirable Requirements
Suggested Implementations
Advantages: No repository access needed for editing
Disadvantage: It's still impossible to maintain/edit, separate from the code, not patchable
Advantages: Sounds like it would be easy to edit.
Disadvantages: Separate from the definitions, changes cannot be done in patches, not part of API module
Advantages: In the code base, we have code in Drupal for parsing .info files
Disadvantages: Tons of little files would result, separate from the definitions, not similar to how we do other documentation (new thing to learn, new thing for API module to parse)
Advantages: In the code base, API module already parses classes
Disadvantages: Separate from the definitions
Advantages: Some information could be automatically parsed by API module from code (such as defaults), doc for elements is with the code, uses familiar syntax, API module shouldn't have trouble parsing it
Disadvantages: Doc for properties is in a separate file (not sure how to get around that anyway)
Conclusion
I'm personally in favor of something like #35/#36 (probably because I proposed it), making sure that everything in #34 can be generated from the doc.
Can we move forward and make this a concrete proposal? I think the next step would be to take a subset of system_element_info() and add doc headers to it, and make a sample element_properties.api.php, and also make sample desired output, and solidify that into a proposal. Obviously, drumm would need to buy in to say "This can be done within the API module", and everyone else would need to buy in and say "it's clear and easy how to maintain this as a coder", and "the output is what we need".
#59
Subscribe ;) Thanks Randy for crosslinking to this; I'd no idea this existed!
I'm going to +1 the #35/#36 proposal. Also, I think if we start making concrete stuff we'll spot the pitfalls quicker.
* @elem_property #options* Selectable options for a form element that allows multiple choices.
*
* @element radios
* @element checkboxes
* @element select
So I get what this is saying -- 'here is the property '#options', and it applies to elements of type 'radios', 'checkboxes', and 'select'.
But isn't this making a special case of the '#type' property? Shouldn't we be more general and say something like:
@dependency #type radios@dependency #type checkboxes
@dependency #type select
Also -- and this is probably going too far too fast -- what about deep arrays like the ones found in hook_views_data? Do we have any like that in core?
#60
@jhodgdon: Thanks for the summary.
Though I'm not an expert on this by any means, I think that #57 or #2 would be the easiest to implement. #2 would indeed be a "new thing" but so would a *.api.php file, afaik (and there's no spec for that, currently). #57 on the other hand would move the problem out of the API module, but hopefully then there would be a lower barrier to entry for people who wanted to keep it up to date. All they would have to do would be to join the docs team, and get privileges to create or edit a node of the new node type.
#61
Updating title
#62
@jhodgdon, I'll bet you already put this on your list for discussion in Vancouver :-)
#63
:)
#64
Any reportback from discussion in Vancouver?
In particular, is there a working model at all for api.php-style docs and API module, or would the short-term, last-minute solution be to do major new documentation (Render API), for now, in a separate, easily-parsable way? (though i like the nodereference and views idea ;-) )
#65
We don't have a working model solution for this yet. I'm not sure what you're thinking of for the Render API -- isn't that really just the Form API plus some new renderable items that aren't really form elements?
#66
@drumm... We respect how busy you are and very much appreciate your contribution.
However, you've made this one your own and it's still hanging out. This is a 2006 issue. With a good consensus for like a year.
If you would like to do it your way, would you plan to do it before long? Or would you let someone else do it your way? Or would you agree to let someone else do it another way?
Not whining here. *So* much appreciate how much you've accomplished in a busy year. Just would like to nudge this issue.
#67
I've been editing the FAPI reference for a couple of months now, and here are my thoughts. I know I'm in the minority here, but I'd keep it the way it is.
I know that document is a huge, ugly thing, but it contains a wealth of information. Afaict, none of the options floated so far would retain all of the information in it now. The usage examples would probably be lost, maybe the descriptions of element callbacks. And again, it's ugly, but it's real easy to go from elements to properties and back, and easy to see it all in one place rather than a lot of little definitions. And with #648218: Make API changes in Drupal core be nodes resolved, there would be a way to notify when the FAPI reference needs updating because of code changes.
If the problem was finding someone willing to update the document, that problem has been solved.
#68
I greatly appreciate all of the work you have been doing to update the Form API document.
However, I wouldn't say that the problem of finding someone willing to update the document has been permanently solved. We have had people updating it in the past (rfay, myself to name two recent ones), and they've all burned out eventually. You may not, but I cannot count on it.
The other problem is that although you've been great about responding to the reported issues on the FAPI reference, that in itself doesn't mean that the document is maintainable. Also, there is a lot of information there that IMO shouldn't need to be maintained. For example: examples. We should just be able to say "look here, there's an example" and use the wealth of Drupal Core code as examples, instead of trying to maintain the examples in that doucment, which are nearly impossible to edit in their current format, and do not automatically keep up with code changes.
What we still (IMO) need is a way for the thing to be generated from the code, so that it stays up to date.
#69
It's true that nothing lasts forever, but I'm fine with maintaining the FAPI reference for the foreseeable future. Once the next couple of issues are resolved, there should be much less to do to keep it up to date. And I really don't mind editing the usage examples.
What I'm a little confused about is the idea of 'automatic' updates. If the information is pulled from core and Doxygen, it must still be put there through patches, through people working to make sure the documentation is current and accurate, just as I am trying to do in the HTML doc. And getting patches committed to core can be really slow. Most of these recommendations sound like another format for doing the same work.
#70
Those are good points...
Maybe we should just move the FAPI reference to drupal.org instead of maintaining it in a source code repository at all? It would likely be easier to edit there. We could also consider making a lighter-weight reference page for all FAPI and render elements, generated from code comments, on api.drupal.org, and a more complete reference on d.o. Thoughts?
#71
And the advantage of having the API reference generated from in-code comments is that there is at least some chance that, if the comments are near the code, that if the code changes the API doc changes. As things are now, with the FAPI reference in a separate file (and now in a completely separate project as well), it was a very large undertaking last year or the year before when I updated both the 6 and 7 guides to match what was actually in FAPI, because no one had bothered to update the doc... That will continue to be an issue. Things are not perfect with the other API doc (sometimes functions get updated and the doc doesn't go with them), but at least most of the doc tends to get updated with function changes.
#72
I like the idea of maintaining the current document and adding a lighter-weight reference page on api.drupal.org. Best of both worlds, with some documentation near code changes.
But moving the FAPI reference to drupal.org wouldn't work very well for me. It's easy to make mistakes in a doc like that, and being able to get it all ready locally before I push is a big help. I need to be able to save, check, save, check, experiment, etc, and I couldn't do that with the doc on drupal.org.
#73
So others can get an idea of the workload involved, where does the source for the current FormAPI docs live?
(I take it we mean this document: http://api.drupal.org/api/drupal/developer--topics--forms_api_reference....)
#74
joachim, it's in the documentation project. http://drupal.org/project/documentation.
@jn2, I see no reason why we should lose any part of the current content, at least theoretically. The idea is to make it so it gets maintained *with the code*, as hook documentation does now.
Thanks for your editing and care for the doc. But long term, it needs to be maintained by the community, IMO, as a result of development.
#75
So it's this: http://drupalcode.org/project/documentation.git/blob/refs/heads/7.x-1.x:...
and it makes this: http://api.drupal.org/api/drupal/developer--topics--forms_api_reference....
Let's talk practically about what parts of that document could be generated automatically, and what data we'd need in the code to make that.
1. Form Controls: a table of which attributes go on which types.
I'm not sure this huge table is that useful. Certainly I only ever use it to jump to the element I want to know about. But further down we need to know which attributes an element type supports. How is that represented in code?
2. Default values
These can be extracted from system_element_info().
3. Elements
a) Description: how could we put that into code? A docblock on each element defined in system_element_info()?
b) Properties: same problem as 1 above
c) Values: in the docblock?
d) Usage example: could this go in the docblock? Or could we have a '@see foo_function' and the documentation generator smart enough to find the half dozen lines in the function that are relevant? (I note in passing that the examples have had a CSS problem for about a year now... IIRC I filed a patch :/)
#76
Other parts:
3. e) Code recommendations. See #managed_file (http://api.drupal.org/api/drupal/developer--topics--forms_api_reference....) for an example.
4. Properties.
a) Each property is listed with the elements it supports. Associated elements and properties link to each other.
b) Callbacks with the function parameters. See #element_validate (http://api.drupal.org/api/drupal/developer--topics--forms_api_reference....) for an example. There's an issue to document these more fully #800162: Documentation problem with forms_api_reference.html: give function params for all callbacks.
c) Usage examples here as well.
#77
There's a related issue that will be affected by what's decided here, and that's Render elements, like the 'link' element, that are not technically part of the Form API. (There may only be one now, but seems there will be more in the future.) They also need to be documented somewhere and have many of the same issues.
#1190658: Where to document the link element (and future Render elements)? asks where they should be documented.
#1190720: Documentation for link element is about the 'link' element in particular.
#78
Is FormAPI now technically a subset of 'RenderAPI'? Or do they intersect?
Let's start simple though. Consider the textfield element. How can we add to the code documentation something that the API code can process?
We have the declaration here: http://api.drupal.org/api/drupal/modules--system--system.module/function...
Sticking docblock within function body would be a big style change. What we could do instead is document at the theme function, http://api.drupal.org/api/drupal/includes--form.inc/function/theme_textf....
The API parser can then start by looking at system_element_info(), deduce the theme functions from that ('#theme' => 'textfield',) and then get data on each element from its theme function.
Here's a patch that represents a very rough stab at that, for just that one element by way of demonstration and to get the ball rolling.
Some questions it throws up:
- do we need a formal marker instead of 'Properties used'?
- 'Format a single-line text field' describes the element. 'Returns HTML for a textfield form element.' describes the theme function. Where do we put the element description?
- supposing we got all this data in nicely-defined places. Where do the instructions go that tell the API parser to turn it all into a documentation page at a specific URL?
#79
Are you assuming that API module is running on the same version of Drupal that it's documenting? That's typically not true. Right now, api.drupal.org is D6 and it documents 5, 6, 7, and 8...
BTW, there's a reason this is named "And other big array documentation". We'd like a solution that works for lots of big weird Drupal arrays.
And about elements, form elements, and render elements: They're all part of one big "thing". Really there's very little difference. When fixing up the D7 element example... and then the Render example... I was really struck by how artificial and misleading it is for us to treat these three things separately.
#80
> Are you assuming that API module is running on the same version of Drupal that it's documenting? That's typically not true. Right now, api.drupal.org is D6 and it documents 5, 6, 7, and 8...
Not at all... currently I'm just throwing down a rough idea of how the documentation data could be in the code, for a hypothetical new piece of API module to slurp up and turn into docs.
> We'd like a solution that works for lots of big weird Drupal arrays.
Good point.
Which ones?
> Links, forms, menus
Links are part of the renderables. Menus are pretty much covered by hook_menu, aren't they?
#81
It would be wrong to document render elements and their properties in the theme functions they define (by default). The #theme function is a property of the render element itself, it may be dynamic, can be replaced, and may not exist at all (in which case drupal_render() returns the rendered #children).
#82
I'd just like to point out that the bulk of the doc currently in the FAPI reference is already based on the base implementation of the theme preprocess, process, and template/function. Otherwise, there is no way to say "this element uses these components", because that is where those components are used. Or properties, or whatever we call them.
#83
Below is a suggestion i was making in #1242780: Find a better way to organize the docs for FAPI vs. RenderAPI before I was aware of this issue... This doesnt really address the automation of documentation, but does propose how we might organize it.
... I was having an issue with render arrays (I was trying to use #attach instead of #attached) so I went over to [#930760] and saw that "#attach" wasn't listed. Had "#attached" been listed I clearly would've been ok, but alas....
So I propose that we reorganize in the following way:
Thoughts?
#84
I updated the issue summary with #58, which was the last solid proposal and has not been followed up.
#85
I just spent a bit of time with API module and am not happy with what it churns out, since it stores rendered HTML rather than the key information we need. I suspect that the API module approach would be inadequate for this. However, the proposals on how to organize the information could still (perhaps) be used.
We definitely need to expand/add Render API documentation, and the FAPI doc really is confused already about whether it's about forms, elements, or render arrays. So I'm sure the same approach will work for all.
#86
I would actually like to see the metadata about form elements and properties available to code like form generators. The first form generator would be one that explained how to use the FAPI! That would mean *not* hiding metadata in the comments, I think. So I'm now -1 about documenting in comments. I'd rather have it be explicitly provided in a hook or api.php type structure.
#87
I like the idea of declaring (some of) the metadata for elements in code. It would mean that there actually is a place where the abstract "element type" is defined.
Right now, I want to heartily agree with jhodgdon's #82, for really, it is up to the theme/process etc. functions which properties to use, so they should document what they use. In the CURRENT setup, sun's argument in #81 does not fly:
That is true, but the fact of the matter is that when you replace the theme function, the it is entirely up to the new one which #-properties to use. Which means, if you change the theme function (in theory), you (can) change how the element behaves. I guess the idea is that it should not, but still.
What the quote shows is a way to think about element types (and probably the INTENDED way to think about them) as like a contract between the function using an element type (and supplying values for the #-properties) and the theme/process function tasked with implementing the functionality. A contract very much like a function signature. The problem right now is that this "contract" is defined nowhere. Having a hook to declare this metadata would give us a place to define this "contract".
A bonus would be that there would be a way to ensure, for example, that required properties are present. Right now, the elements defined by core (or at least some of them) don't do any validation of this (e.g. 'html_tag' nowhere (not even in the theme doc) says that the '#tag' property (which does not have default value) is required---and the theme function just assumes it is there. Less than ideal.
#88
Right. The thing is, the current FAPI reference doc is a human-made compilation of a bunch of information from process, pre-process, and theme functions (and possibly other spots).
Problems with that fact:
a) It's not really possible to keep up with changes in these functions to maintain the correctness of the FAPI reference doc.
b) Human compilation is subject to error.
c) The FAPI reference is just a compilation of the standard, default behavior, which can be overridden by themes and modules etc.
Generating it from code via the proposal that is now in the issue summary will only possibly take care of (a) and (b). We have no way of taking care of (c) and shouldn't worry about it (IMO).
#89
> I'd rather have it be explicitly provided in a hook or api.php type structure.
That could be a good way to go. As my attempt further up showed, trying to fit everything we want to present about FormAPI in function headers isn't really doable.
If we go that way though, we should still try to put documentation for each element type in its theme function, as that keeps it close to the code.
I also think we need a 'way in' to the new documents -- in the same way that when you look at user_menu() it directs you to hook_menu(). Am I right in thinking that all the large arrays we are concerned with all have one function that consumes them -- hook_menu(), drupal_get_form(), and drupal_render()? These would logically take us into the deeper documentation.
#90
Looking at http://api.drupal.org/api/drupal/includes--form.inc/function/theme_table... as an example it seems to me that if we formalize this text:
element: An associative array containing the properties and children of the tableselect element. Properties used: #header, #options, #empty, and #js_select. The #options property is an array of selection options; each array element of #options is an array of properties. These properties can include #attributes, which is added to the table row's HTML attributes; see theme_table().into something like this:
* @param $variables* An associative array containing:
* - element: An associative array containing the properties and children of
* the tableselect element.
* @properties
* #header, #options, #empty, #js_select.
* @property #options
* An array of selection options;
* each array element of #options is an array of properties. These
* properties can include #attributes, which is added to the
* table row's HTML attributes; see theme_table().
*
then we're well on the way.
We then need:
- a place to put all the properties and their 'default' meaning, ie what '#empty' does for a form element if that form element doesn't tell us anything different about it with a @property. This is the bit that could go in an api.php type document, though I'd be just as inclined to put it on drupal_render() or drupal_get_form() as that's what consumes the array.
- a parser to turn all that into a table and monster list of stuff as our current Form API reference shows
#91
Thinking about it some more, what we are basically documenting is possible values in an array, where some have defaults values and some are optional.
That looks a lot like theme functions now that they take a $variables array.
So it's almost like sub-variables in a way...?
#92
I like the idea in #90 pretty well... It seems like an excellent idea, and documenting things in the right place (near where they are used).
We would also need to have some way of knowing "This should be part of that reference page", so we could mark theme_tableselect() etc. as being the things to collect. I guess having @properties would maybe be enough, but maybe we need to add a line like this to the doc:
* @form_element tableselectthat would tell us this is where the documentation for this particular form element lives? We could also have @render_element for render elements that aren't for forms.
Then we could make both a Render API reference and a Form API reference page. Hmmm... possibilities...
#93
A quick comment on #92: I think, moving forward, the more-or-less artificial distinction between Form and Render API should be given up, so I don't like @form_element ... really any form ends up being drupal_render()ed, so Form API arrays really are simply Render API arrays that undergo some preprocessing. Preprocessing which, I might add, could (and probably will, in due time) largely be reimplemented using Render API methods.
#94
Veto. Form API arrays are rendered as render arrays, but that's about it. Form API uses dedicated #properties on elements to build and process user input and output for forms and form elements. Form API requires a full form context and state of the form element within the form. Form handling is an entirely different topic and concept than element rendering. That "some preprocessing" refers to ~3,500 lines of code, which is responsible for making sure that your Drupal site stays secure. This won't be folded into the render system (or any other system) anytime soon.
A 'form element' shares the rendering semantics of an 'element', but a 'form element' is a lot more than a rendered 'element'.
All elements share a common set of generic properties, which apply to all elements.
All form elements also share a common set of generic properties (in addition to the generic render element properties), which apply to all form elements.
Each element, regardless of whether form element or render element, may have additional custom properties that only apply to the specific element type.
#95
@sun - you can dislike the proposal (I do too), but you can't veto it as you have no authority to make unilateral decisions (only catch and dries and angie for D8).
#96
My perspective has changed since #67.
So I also like the suggestion in #90-#92, at least as a starting place. #94 makes valid points about the difference between render and form elements, so +1 to #92. Also, keeping the list of render properties in drupal_render() and form properties in drupal_get_form() would help make that distinction. The end of the element listing would need @see to the relevant function, or that could be included with the @properties list.
These would be in drupal_render()?
These would be in drupal_get_form (and essentially cover the list currently in the FAPI reference)?
#97
If we want the new version to be as useful as the current FAPI reference, the entries will need to include more of the information in it now. So for tableselect, the entry in #90 would become:
* @param $variables* An associative array containing:
* - element: An associative array containing the properties and children of
* the tableselect element.
* @properties
* #header, #options, #empty, #js_select.
* @property #options
* An array of selection options;
* each array element of #options is an array of properties. These
* properties can include #attributes, which is added to the
* table row's HTML attributes; see theme_table().
*
* @form_element tableselect
* A table created with a far left column of radios or checkboxes. Build the
* table headings and columns with the #headers property, and the rows with
* the #options property. See @link http://drupal.org/node/945102#drupal7 Adding
* checkboxes to a table @endlink for a full explanation.
* Other settings:
* - To set accessibility tags for the radios or checkboxes, build one of
* the cells in the #options property using the 'title' => array( 'data' =>
* array(#title => 'mytitle'))) construction. Drupal will create invisible
* label tags for the left column based on this value.
* - To disable the default "check all" button for the checkboxes, set
* #js_select property to FALSE. This is FALSE by default for radios.
* - Setting #multiple to TRUE will give you radios instead of checkboxes.
This version incorporates all the main element description, leaving the properties listing to the @param section.
#98
Okay, it seems that my #93 was understood as suggesting that the Form API is trivial (or a trivial extension of Render API), or that I was suggesting that the implementation of Form API should be (re)done inside the Render API. I meant to suggest neither (though I can see how this impression could arise, in particular the latter one, since I pointed out that they COULD---and made the prediction that the will be at some time. I did not mean to say that this should be a goal).
What I meant to talk about (and which perhaps is equally disliked) is to give up the distinction between Render arrays in the documentation / the public side of the API---and all I "proposed" was speaking out against the proposal of using a special Form API tag (And @moshe: sun really only vetoed my statement of fact---i.e. said he did not agree that Form API arrays are Render API arrays that are preprocessed ;).
From the viewpoint of a USER of the API (and I understand that therefore my opinion has limited weight), i.e. someone who implements hook_element_info() to "define" Form API elements and Render API elements alike (in so far they are defined), and who writes form builders in which I can do everything I can do with Render arrays, and if I am trying to figure out how Form API arrays are turned into HTML and I find that they are by calling #theme / #theme_wrapper callbacks with just the semantics as for Render API arrays, it simply seems that Form API arrays are special Render arrays---and, yes, as special Render arrays, they have special #-properties.
This is getting off-topic, but I still don't quite understand why the fact that Form API #-properties are implemented in a special way should influence the way these are documented, and in particular, have them documented as a completely different thing than Render API #-properties.
#99
RE #96 - yes, this is a good idea to put the generic properties into the docs for drupal_get_form() and drupal_render(). Putting an @see to drupal_get_form()/drupal_render() also makes sense, though on api.drupal.org we could definitely generate that automatically if the docblock contains @form_element or @render_element.
RE #97: I agree that we need to make sure that all the information that is currently in the FAPI reference gets into the individual element theme docs. However, I think it should just be incorporated into the text of the theme function, rather than being put into a separate section. The automatically-generated FAPI reference (and render reference) could just link to the relevant theme function for docs, rather than repeating sections of it (I think?).
And I think it would be useful to have a link to a function that actually uses one of those elements as an example, rather than trying to maintain a separate copy of the code as in the current FAPI reference.
I started to rewrite the tableselect example from #97 but I got bogged down in the details (I'm not sure what's currently in the FAPI reference or the page on d.o is correct actually). Let's choose a simpler element as our base example please? :)
#100
One other thought:
I think to aid parsing and building the FAPI/Render reference, we should use something like the following @ tag system:
- drupal_render() -- will contain a list of @render_property items (each will give the default meaning of one of the properties).
- drupal_get_form() - will contain a list of @form_property items (default meanings)
- individual theme functions - will contain either @render_properties or @form_properties (a list of the properties it uses), and also @render_property_override/@override_form_property_override for any standard properties whose meanings it is overriding, and @render_property_specific/@form_property_specific for any specific properties it is defining that are not "generic".
Does this make any sense?
#101
I think we need to have one more set of tags: @render_property_all/@form_property_all -- to indicate properties like #theme and #pre_render that apply to all elements, as opposed to properties like #options that only apply to selects/checkboxes/radios. Or something like that...
I will try to make a partial patch for drupal_render(), drupal_get_form(), and a couple of these theme functions as an illustration so we have something definite to look at.
#102
So... In writing this patch, I ran into an issue: Not all renderable elements have a theme function or template. For instance, the 'markup" element does not. So where would we document this element? Also, I realized that drupal_build_form() is probably the best place to document the form-specific elements, and we also need to remember that everything is passed through drupal_render(), so all render properties are also form element properties.
Anyway... Here's a proposed [PARTIAL] patch for drupal_render(), drupal_build_form(), and sample form and render elements.
I decided that the form/render elements could just have @properties, and the API module could figure out if they were render or form properties. Maybe we don't need the other distinctions here... Thoughts? I've attached this both as a patch and as a text file to see what the result is.