[policy for now] New standard for documenting form/render elements and properties

Here is the first pass at a documentation standard (well, actually it's not anywhere near the first pass -- you can read the history on #100680: [meta] Make API module generate Form API documentation if you're interested).

Thoughts?

----

Documenting form/render elements and properties

Drupal's Form and Render APIs make use of elements (defined in modules' hook_element_info() implementations) and properties (array keys in form/render arrays starting with #).

Document an element foo by adding a few tags to the documentation header of its theme, process, or preprocess function. Example:

/**
 * Returns HTML for a select form element.
 *
 * @param $variables
 *   An associative array containing:
 *   - element: A form element array containing the properties of the element.
 *     @properties
 *        #title, #value, #options, #description, #extra, #multiple, #options.
 *     @property_specific #size
 *       The width of the select element.
 *
 * @ingroup themeable
 * @form_element select
 */
function theme_select($variables) { }

Notes:
- Use the @form_element or @render_element tag at the end of the documentation block to indicate which form/render element you are documenting.
- At an appropriate place that makes sense in the documentation (such as within $variables['element'] documentation, as in this example), use the @properties tag to list the standard properties that this element makes use of. Do not include properties that are common to all form or render elements (such as #pre-render or #disabled).
- For any properties that either have documentation that is different from the standard, or that are specific to this form/render element, use the @property_specific tag to document the meaning of that property on this element.
- Wrap all documentation to 80 characters and indent, similar to other documentation tags.

Document core Drupal properties that are common to form elements on the drupal_build_form() function (or its equivalent in earlier/later versions of Drupal). We make a distinction between "all" properties (such as #disabled) that are inherited by all form elements, and "some" properties, which have a common meaning but are only used on some form elements (as indicated by them appearing in the @properties tag in the form element documentation, see above). Do not include render properties that are also used in forms (such as #pre_render). Examples:

   ... Besides the common renderable array
 *  properties listed in drupal_render(), the following properties are common
 *  to all form elements:
 *    @form_property_all #disabled
 *      If TRUE, the form element is disabled (greyed out).
...
 *  The following properties are used by some (but not all) form elements:
 *    @form_property #description
 *      The translated description of the form element, typically displayed
 *      below the input field.
 *   @form_property #size
 *      The width of the form element.

Notes:
- "all" elements are documented with the @form_property_all tag, indicating they are inherited by all form elements.
- "some" properties are documented with the @form_property tag, which gives the default documentation for the property. If a form element uses this property, it can either include it in its @properties tag, or override its documentation by using a @property_specific tag (see example above).

Document core Drupal properties that are common to render elements on the drupal_render() function (or its equivalent in earlier/later versions of Drupal). We make a distinction between "all" properties (such as #pre-render) that are inherited by all render elements, and "some" properties, which have a common meaning but are only used on some render elements (as indicated by them appearing in the @properties tag in the form element documentation, see above). Examples:

... The following properties
 *  are common to all renderable arrays:
 *    @render_property_all #type
 *      The type of element to render, as defined in a hook_element_info()
 *      implementation. Default property values for this element type from
 *      hook_element_info() are added to the $elements array.
...
 *   The following properties are used by one or more types of renderable
 *   arrays, but not all types:
 *   @render_property #markup
 *      HTML markup for this element. If #type has not been set and #markup is
 *      present, #type will be set to 'markup'.

Notes:
- "all" elements are documented with the @render_property_all tag, indicating they are inherited by all form and render elements.
- "some" properties are documented with the @render_property tag, which gives the default documentation for the property. If a form or render element uses this property, it can either include it in its @properties tag, or override its documentation by using a @property_specific tag (see example above).

RE #2...

1. I believe we do need the difference. Processing is different for elements passed through drupal_*_form() functions vs. just drupal_render(), and form elements will not work correctly if they are just passed through drupal_render(), in my experience.

2. Not sure what you are talking about. You mean you don't want @properties within @param? If so, that makes some sense.

3. Good idea.

4. OK, maybe this can be unified... but we do (I think) want to have a way to say "Use these standard properties and then add these specific/overridden properties", and we don't want to list all the ones that apply to everything (the "all" vs. "some" standard properties). I think this proposal accomplishes this, but I think we could also do it with a tag called @property vs. @property_specific.

5. I can't either, which is why I suggest either the theme, process, or preprocess function. That is where the properties are really used, so I think that is the best place to document them (as close as possible to the code that uses them to get the best chance at maintenance).

Anyway... let's wait for a few more comments and I'll address these ideas with a new proposal.

Oh, and one more thing regarding:

That is, because most form elements can be rendered as pure render elements outside of a form, too (or at least, if that's not possible for a certain element, then that is a bug).

I don't really think this is true... For instance, I tried recently to use a D7 fieldset in a render array (not in a form), and it totally failed to work (didn't render right, didn't open/close, etc.). drupal_render() does not do the same processing as drupal_get/build_form() -- the form elements are not properly processed or preprocessed. I think this is generally true for any complex form element (I tracked down the reason at the time but I'm not remembering right now)... but I'm sure we *do* need the distinction. Form elements are not meant to be used outside of forms in any case (for instance, it is probably not valid HTML to have an INPUT element outside a FORM anyway, is it?).

Ah, I wasn't aware of that... So. Any other opinions -- should we make any distinction at all between form elements/properties and render elements/properties? It sounds like there *is* a distinction between form properties and render properties (those ones listed in #5 that the form builders add)... but maybe there is not a distinction between form/render elements (at least if that bug is fixed).

And I also remembered that in my last proposal on the other issue, I made a section for "internal" properties that the Form/Render API adds/uses (not meant to be set up in a form/render array or used in theme/preprocess/process functions). So this would be added to the render section of the proposal (and maybe the form section too):

 *   The following internal-use properties are common to all renderable arrays:
 *   @render_property_internal #printed
 *     During rendering, this function flags each element with a #printed
 *     status to indicate that the element and its children have already been
 *     rendered. If an already-rendered element is passed in, nothing is
 *     returned from this function.

Again, I think we do want to make a distinction between properties you should be using when defining form/render arrays, and ones that you shouldn't (but which you might see if you use print_r and wonder what they are for).

Taking (I hope) the comments above into consideration, here's a second pass at a standards proposal. A couple of notes pertaining to the comments above:
- We decided there was no distinction between form and render elements, so those have been combined.
- We decided there were some properties specifically used/added during form processing, so those are using a different tag.
- Added data types.
- Moved the documentation of elements/properties out of @param section.
- Changed @render_property_specific to just @render_property.
- Made three separate types of "common" properties: _all, _common, and _internal. I do think we want to have a distinction in the documentation between _all and _internal -- the _all properties are the ones that you would normally want to define in your form/render array, and the _internal ones are not generally defined when making a form/render array. So if we're building something like:
http://api.drupal.org/api/drupal/developer!topics!forms_api_reference.ht...
from this documentation, I think we want to be able to list the standard properties, internal-use properties, and specific properties.

----

Documenting form/render elements and properties

Drupal's Form and Render APIs make use of elements (defined in modules' hook_element_info() implementations) and properties (array keys in form/render arrays starting with #).

Documenting elements

Document an element "foo" by adding a few tags to the documentation header of its theme, process, or preprocess function. Example:

/**
 * Returns HTML for a select form element.
 *
 * @param $variables
 *   An associative array containing:
 *   - element: A form element array containing the properties of the element.
 *
 * @render_element select
 * @render_properties
 *   #title, #value, #options, #description, #extra, #multiple, #options.
 * @render_property int #size
 *   The width of the select element.
 *
 * @ingroup themeable
 */
function theme_select($variables) { }

Notes:
- Use the @render_element tag near the end of the documentation block to indicate which element you are documenting (there is no real distinction between Form API and Render API elements).
- After the @render_element tag, use the @render_properties tag to list which optional standard form/element properties this element makes use of (if any). Do not include render_property_all or render_property_internal properties (see below).
- For any properties that either have documentation that is different from the standard, or that are specific to this form/render element, use the @render_property tag to document the specific meaning of that property on this element. Give the data type of the property (see the @param/@array data types section for details), the name of the property, and a description.

Documenting Render API properties

Document core Drupal properties that are common to render elements on the drupal_render() function (or its equivalent in earlier/later versions of Drupal). Some of these properties are common to all render elements (either they are added to all elements by drupal_render() or another function, or all elements need to define them). Others are used by several, but not all, elements, and have a common meaning for elements where they are used. Element documentation (see above) should omit the common-to-all properties entirely, and should use the @properties tag to indicate which common-to-some properties pertain to this element. Examples of render API property documentation on drupal_render():

 * The following properties are common to all renderable arrays:
 *   @render_property_all string #type
 *     The type of element to render, as defined in a hook_element_info()
 *     implementation. Default property values for this element type from
 *     hook_element_info() are added to the $elements array.
...
 * The following properties are common to some renderable arrays:
 *   @render_property_common string #markup
 *     HTML markup for this element. If #type has not been set and #markup is
 *     present, #type will be set to 'markup'.
...
 * The following properties are used internally by the Render API:
 *   @render_property_internal bool #printed
 *      Set to TRUE to indicate that a section of the render array has already been processed.
...

Notes:
- Common-to-all properties are documented with the @render_property_all tag, indicating they are inherited by all form and render elements. Form/render elements should not list these properties.
- Common-to-some properties are documented with the @render_property_common tag, which gives the default documentation for the property. If a form or render element uses this property, it can either include it in its @render_properties tag, or override its documentation by using a @render_property tag (see example above).
- Internal-use properties are documented with the @render_property_internal tag. Form/render elements should not list these properties.
- When documenting a property, give the data type of the property (see the @param/@array data types section for details), the name of the property, and a description.

Documenting form properties

Some properties are added to form elements during form processing with drupal_build_form() and related functions, or are used during form processing, but are not added/used in the basic render API. These should be documented in a manner similar to the render properties (see section above) on the drupal_build_form() function, along with common-to-some properties that are only used by form elements. Examples:

 * Besides the common render array properties listed in drupal_render(), the following
 * properties are common to all form elements:
 *   @form_property_all bool #disabled
 *     If TRUE, the form element is disabled (greyed out).
...
 * The following properties are used by some (but not all) form elements:
 *   @form_property_common string #description
 *     The translated description of the form element, typically displayed
 *     below the input field.
 *   @form_property_common int #size
 *     The width of the form element.
 ...
 * The following properties are used internally during form processing:
 *   @form_property_internal string #id
 *     The HTML ID assigned to the form field for this element.

Notes:
- Common-to-all properties for form processing (if they aren't also used during generic render processing) are documented with the @form_property_all tag, indicating they are inherited by all render elements that undergo form processing. Form elements should not list these properties.
- Common-to-some properties that are meant for form elements are documented with the @form_property_common tag, which gives the default documentation for the property. If a form element uses this property, it can either include it in its @render_properties tag, or override its documentation by using a @render_property tag (see example above).
- Internal-use properties specific to form processing are documented with the @form_property_internal tag. Form elements should not list these properties.
- When documenting a property, give the data type of the property (see the @param/@array data types section for details), the name of the property, and a description.
---

RE #9... a few thoughts that may clarify this:

a) When we add a document header for a form/render element, the doc header is only going to list the properties that are specific to that element, or that have special meanings for that element, not any of the default properties that are common to all form elements or all form/render elements (#theme, #pre_render, etc.). We don't want to get into a situation where every element has to list these, or even encourage it, because down the road, some default form/render element property will be added/removed and we'd have to go through all of the elements and edit their documentation blocks. That is not easy to maintain, as anyone who has tried to maintain the current FAPI document knows, I'm sure!

b) From #9:

...just because a property cannot be overridden doesn't mean it might not be of use in some situations. (In which case people would want to know about it.)

I totally agree, but at the same time, see (a). Keep in mind that on api.drupal.org, we can do anything we want for listing and cross-referencing (and I do think we want to list all properties, all elements, and cross-reference them) -- but we should discuss the specifics of the api.drupal.org output on #100680: [meta] Make API module generate Form API documentation, not here.

c) I still think that the information in the doc headers in the proposed standards will be enough to do any cross-referencing and listings that we would want to do (including generating something very much like the FAPI reference). I could be wrong, though, so if you can think of some additional information that we need to include, let's make sure it gets added to the standard.

d) From #9:

I just wonder how to show that some elements can have access to the form_builder default properties.... I'm making the assumption that the elements referred to in #5 are only form elements and that simple render elements are not affected this way by form_builder() and co. even when used in a form.

My read of sun's comments above is that this is incorrect, and if you look at form_builder(), it is definitely doing the same thing with each array element in the form, independent of what type of element it is, right? I don't see anywhere in there where it says "If this is an ordinary render element, skip these steps and just render it with drupal_render()". It does do something extra with elements that have the #input property set to TRUE... and maybe we need to make another property distinction for properties that are specific to form input elements (anything used/output by _form_builder_handle_input_element()), and maybe we do need to mark some elements as being form *input* elements? Hmmm.

So... There are probably three things we need to add to this standard:

1. Some way to indicate a function or functions where the element (and property?) is used (an example usage).

2. Some way to indicate that a particular element is a form input element, since these are handled specially in form_builder().

3. Some way to indicate that a property is specific to form input elements.

Regarding the way to indicate that something is a form input element... maybe the place we should document elements is inside the hook_element_info() implementation and not in the theme/preprocess function? Then we could pull the element definition array out to display it as the source code, which would also tell us what the default values for the properties are (including the #input = TRUE on the input elements).

That would also avoid the problem I've been ruminating on, of separating out the element documentation from the theme function documentation, because it would be totally separate.

RE #11/1 => You are correct -- I think that we can automatically generate the list of "functions that use this element". If we also move the documentation into the hook_element_info() so it is separate from the theme function, then we can also use @see in those docblocks if we want to. I think this has a lot of advantages... OK, it's time for a new standard proposal:
----

Documenting form/render elements and properties

Drupal's Form and Render APIs make use of elements (defined in modules' hook_element_info() implementations) and properties (array keys in form/render arrays starting with #).

Documenting elements

Document an element "foo" by adding a documentation header to the definition of the element in a hook_element_info() implementation. Example:

// Inside system_element_info():
  /**
    * Form input element for a select list.
    *
    * Optional longer description would go here.
    *
    * @input_element select
    * @render_properties
    *   #title, #value, #options, #description, #extra, #multiple, #options.
    * @render_property int #size
    *   The width of the select element.
    *
    * @see function_that_uses_this_element()
    * @see theme_select()
    */
  $types['select'] = array(
    '#input' => TRUE,
    '#multiple' => FALSE,
    '#process' => array('form_process_select', 'ajax_process_form'),
    '#theme' => 'select',
    '#theme_wrappers' => array('form_element'),
  );

Notes:
- The first line should either start with "Form input element for" or "Render element for", and should describe the purpose of the element.
- An optional longer description can also be included, after a blank line (similar to function documentation).
- After the descriptive lines, use either a @input_element (for form input elements only) or @render_element (for all other elements) tag, giving the machine name of the element you are documenting.
- After the @input_element/@render_element tag, use the @render_properties tag to list which optional standard form/element properties this element makes use of (if any). Do not include *_all, *_input, or *_internal properties (see below).
- For any properties that either have documentation that is different from the standard, or that are specific to this form/render element, use the @render_property tag to document the specific meaning of that property on this element. Give the data type of the property (see the @param/@return data types section for details), the name of the property, and a description.
- You can also use @see tags at the end of the documentation block, to point people to functions with especially good examples of usage of elements, to theme and processing functions that might need to be overridden, etc.

Documenting Render API properties

Document core Drupal properties that are common to render elements on the drupal_render() function (or its equivalent in earlier/later versions of Drupal). Some of these properties are common to all render elements (either they are added to all elements by drupal_render() or another function, or all elements need to define them). Others are used by several, but not all, elements, and have a common meaning for elements where they are used. Element documentation (see above) should omit the common-to-all properties entirely, and should use the @properties tag to indicate which common-to-some properties pertain to this element. Examples of render API property documentation on drupal_render():

 * The following properties are common to all renderable arrays:
 *   @render_property_all string #type
 *     The type of element to render, as defined in a hook_element_info()
 *     implementation. Default property values for this element type from
 *     hook_element_info() are added to the $elements array.
...
 * The following properties are common to some renderable arrays:
 *   @render_property_common string #markup
 *     HTML markup for this element. If #type has not been set and #markup is
 *     present, #type will be set to 'markup'.
...
 * The following properties are used internally by the Render API:
 *   @render_property_internal bool #printed
 *      Set to TRUE to indicate that a section of the render array has already been processed.
...

Notes:
- Common-to-all properties are documented with the @render_property_all tag, indicating they are inherited by all form and render elements. Form/render elements should not list these properties.
- Common-to-some properties are documented with the @render_property_common tag, which gives the default documentation for the property. If a form or render element uses this property, it can either include it in its @render_properties tag, or override its documentation by using a @render_property tag (see example above).
- Internal-use properties are documented with the @render_property_internal tag. Form/render elements should not list these properties.
- When documenting a property, give the data type of the property (see the @param/@array data types section for details), the name of the property, and a description.

Documenting form properties

Some properties are added to form elements during form processing with drupal_build_form() and related functions, or are used during form processing, but are not added/used in the basic render API. These should be documented in a manner similar to the render properties (see section above) on the drupal_build_form() function, along with common-to-some properties that are only used by form elements. Examples:

 * Besides the common render array properties listed in drupal_render(), the following
 * properties are common to all form elements:
 *   @form_property_all bool #input
 *     If TRUE, this is an element for form input. If FALSE or omitted, this is a generic form/render element.
...
 * The following properties are used by all form input elements (form elements that take input):
 *   @form_property_input bool #disabled
 *     If TRUE, the form element is disabled (greyed out).
...   
 * The following properties are used by some (but not all) form elements:
 *   @form_property_common string #description
 *     The translated description of the form element, typically displayed
 *     below the input field.
 *   @form_property_common int #size
 *     The width of the form element.
 ...
 * The following properties are used internally during form processing:
 *   @form_property_internal string #id
 *     The HTML ID assigned to the form field for this element.

Notes:
- Common-to-all properties for form processing (if they aren't also used during generic render processing) are documented with the @form_property_all tag, indicating they are inherited by all render elements that undergo form processing. Form elements should not list these properties.
- Properties that are used by all form input elements are documented with the @form_property_input tag.
- Common-to-some properties that are meant for form elements are documented with the @form_property_common tag, which gives the default documentation for the property. If a form element uses this property, it can either include it in its @render_properties tag, or override its documentation by using a @render_property tag (see example above).
- Internal-use properties specific to form processing are documented with the @form_property_internal tag. Form elements should not list these properties.
- When documenting a property, give the data type of the property (see the @param/@array data types section for details), the name of the property, and a description.
---

#15 - Thanks for those corrections! Let's see if anyone has any other comments on the proposed standard ==> the proposal is now "#14 with the example corrections from #15"

One more addition to the standard: We should define a @defgroup render_elements (or something like that, to give an overview of what render elements and properties are), and make sure that every element has @ingroup (group_name) in it.

So.... Does anyone besides jn2 and myself want to comment on this standard before we officially adopt it?

RE #19 -

1. Why not? The API module can definitely parse this without any trouble, and putting the comments within the hook_element_info() will ensure that the documentation is with the code, which is always good. Regarding this being non-standard for documentation in general, standard JSDoc definitely uses this type of thing (/** comments within function bodies and {} closures).

2. How about @form_input_element then? The point is, it's only elements that have #input => TRUE when used in forms.

3. The idea is only to document properties that are specifically used in this element's default process/pre-process functions. Sure, themes and processing can be overridden, but we've always documented the default behavior in the FAPI reference... Don't you agree that is the right thing to do? We can/should put wording in the standard saying you are documenting the default behavior.

Let's see if we can get some other core developers to weigh in on the argument in #21. I doubt we would adopt it for D7 (and I would *really* like to get this documentation incorporated into D7 if possible)... and it sounds like it's kind of an interim solution for D8, right? How about adopting the interim solution of putting the docs into hook_element_info() instead? :)

*could* - yes. *would* - ? And does it really make sense to change the coding style for all hook_element_info() implementations just for a documentation standard? It seems relatively extreme, and I still don't see why having /** */ comments within a function is so horrible/wrong/evil/whatever.

As another note, this coding style change would need to be made in 6 core functions:
http://api.drupal.org/api/drupal/modules!system!system.api.php/function/...
(not just system_element_info()). And any contrib modules that wanted to comply with the new standards.

RE #25 - if we get elements onto api.d.o (which is most of the point of this issue anyway), you will be able to search for them on api.drupal.org, with or without there being a function with that in the name. So moving them to additional functions will not really help the developer experience on api.drupal.org.

RE #28 - I see your point (although I am not sure how often someone needs to comment out pieces of system_element_info() to debug). So... As I see it, we have three options for where to place the property/element documentation:

a) Put the docs for elements within function doc blocks (either theme functions as was proposed in #1 and other proposals above, hook_element_info() implementation function headers, or dedicated "generate this element" functions). The problem with this (the reason the proposal changed to putting element docs inside system_element_info() in the first place) is that if you want to use @see, @code, etc. and have multi-paragraph docs for the elements (and I think we need to), it's really not possible to tell which paragraphs or sections of doc belong to the element and which belong to the function. [We incidentally would have the same problem with the property documentation, but have (somewhat tacitly) decided that a one-paragraph description of each property is sufficient -- if it isn't, we need to rethink the idea of putting the property docs within other doc blocks too (drupal_render(), drupal_build_form() and individual element blocks).] I really don't think this is a feasible option, the more I think about it.

b) Make individual doc blocks for the elements, and locate them where the elements are defined (inside the code for hook_element_info() implementations) [the latest proposal, #14/#15, does this]. This has problems outlined nicely in #28.

c) Put the element (and possibly the property) documentation into new .api.php files. This way they could each be their own doc blocks [avoiding the problems of (a)] and we would also avoid the problems of (b). The disadvantage here is that like hook docs, having the documentation separate from where the elements/hooks are actually defined or used in code means that hooks/elements can easily be added/changed without the docs being updated. (For instance, somewhere in the D7 dev cycle, I searched for where hooks were being invoked, and found about 30 that were completely undocumented, and we have issues all the time about hooks whose documentation hasn't kept up with the code.)

I can't think of any other options... and I don't think any of the three options is ideal -- they each have advantages and disadvantages. My feeling is that (b) [the current proposal in #14/#15] is the best option. But I would be willing to go with (c) if we have to -- I'm not excited by the idea, because I already think maintaining the hook docs is problematic, but willing. :)

Thoughts?

Resistance to #21 -- see comment #29 (a). I just don't see, as a practical matter (as the API module maintainer), how to disentangle the element docs from the function docs, if we put element docs inside a function doc-block. So I don't want to put element docs in function doc-blocks, whether they're hook_element_info() or special-purpose functions.

So... I believe we have pretty much agreed on standards for how to document the properties and elements. We just need to settle the question of where to put the element documentation.

I just updated the issue summary with what I think are the latest ideas that we all agree upon for property and element documentation (http://drupal.org/node/1617948#summary-proposed-resolution). Please check those over and see if I've messed anything up, or if we need to add anything there.

I also updated the issue summary with a list of ideas for where to put the element documentation, and the advantages and disadvantages of each idea (http://drupal.org/node/1617948#open-question). What should we do on this?

My current opinion: I'm becoming more and more inclined to put the element documentation in api.php file(s), given that sun and effulgentsia are so opposed to putting it inside the system_element_info() function body (and I understand their reasons), and that as API module maintainer, I'm opposed to putting it inside of any function documentation block (because it's not clear to me how to separate out the parts of the block that are function docs vs. element docs). Having the elements in a separate api.php file is probably the only way to overcome both of these objections. And since we also probably want to have a @defgroup to give an introduction to form/render API, putting it all together in system.elements.api.php or something like that seems like it might make the most sense.

Thoughts? Can we all get behind this compromise idea of api.php files for the place to put the element docs, and go forward to (a) a patch and (b) some new API module code? :)

RE #33 - Agreed, that is precisely the problem -- I don't agree with you that the function *is* the element, even if we follow the suggestion in #21. The function needs to be documented as a function (http://drupal.org/node/1354#functions), and the element needs to be documented as an element (see issue summary above). How are they the same thing, even if the sole purpose of the function is to generate the element (even that language gives it away -- the element is definitely a concept separate from the function)?

And having an element be an alias of a function doesn't make sense to me either, in the context of the API module. We don't, for example, want the "@ingroup elements" to apply to the function -- we want the *element* to be part of that group.

Regarding deferring until some OO conversion is done... ideally, the solution we choose would work for both Drupal 7 and Drupal 8. In which case, I find the idea of putting the element documentation into an api.php file even more compelling. If elements turn out eventually to be handled by plugin classes, then it becomes even more apparent that they're different things (to me, anyway), since classes have member variables, member functions, inheritance, etc. -- none of which apply to "elements" in the abstract sense. I definitely wouldn't want their documentation to be merged.

That sounds like a good idea.

Note to webchick: the issue summary is updated as of comment #31. I suggest you read the issue summary and then comments #32-35 to see what you are being asked to mediate. Thanks!

We asked webchick to comment and we're waiting for her to do so. Otherwise, we are apparently at an impasse.

Yes it would *definitely* be good to get this resolved... but as noted above we have come to an impasse for the moment and are waiting for hopefully webchick or dries to weigh in.

You've nicely illustrated my point with this patch. Take this example:

/**
  * Implements hook_element_info().
+ *
+ * @render_element field_ui_table
  */
 function field_ui_element_info() {

field_ui_element_info() is a function. It's an implementation of hook_element_info(), and will be listed in the "implementations of hook_element_info()" page on api.drupal.org, as well as in the "Functions" section of the field_ui.module page. So yes, it *is* a function, and it needs to have that first line to comply with our standard for hook implementations.

And what you've added to this would make it also be the render element documentation for the "field_ui_table" element, which would need to have a *lot* of other information, like this example from the issue summary:

/**
    * Form input element for a select list.
    *
    * Optional longer description would go here.
    *
    * @form_input_element select
    * @render_properties
    *   #empty_option, #empty_value, #multiple, #options
    * @render_property int #size
    *   The number of lines in a listbox-style select element.
    *
    * @see function_that_uses_this_element()
    * @see theme_select()
    * @ingroup elements
    */

I just don't see how to merge what we need for the function doc with what we need for the render element doc -- for instance if you have an @see or an @ingroup, which do they apply to, the function or the element doc? So my conclusion is that they need to be separate and we should probably just put both render element and render property documentation in separate api.php files, which just contain a bunch of documentation blocks like the one here, one after the other, with no code on them (and the API module will be quite happy with that).

Regarding the patch in #42 -- it is nice and clean, at least in part, because it does not have anywhere near all the necessary documentation for form/render elements. I invite you to flesh out a couple of the items in the patch with all of the documentation for an element (see #14 for what documentation needs to be included) and demonstrate how it all fits together with the function docs, given that they need to have different summary lines, etc. I'm skeptical that it makes sense, but possibly willing to be convinced by an actual patch that includes all the necessary documentation for a couple of elements. I would like to see one example that goes into one of the newly created functions, and another example that goes into a hook_element_info() that isn't being broken up.

I would also like to see a D7 maintainer comment that they would be willing to commit a patch like #42 (but with a lot of added docs), since it does change the code and not just documentation.

Well, it's probably time to revive this issue.

Would someone who likes the idea of #42 like to expand the patch with all the necessary documentation that a render element would actually need (see comment #47)?

Or should we go with an elements.api.php file, as advocated on other comments?

I have a new idea for this issue, which would involve zero code change from how we are currently doing things, and would only add inline ( // ) comments.

Here's a patch that illustrates the type of comments we'd add to system_element_info().

The idea:
a) Any hook_element_info() implementation with this type of documentation would have @element_list added to its doc block. This would clue the API module that it needs to do some special custom parsing. I could probably detect this from the function name, but I'd prefer to just have this tag (hook names tend to change over time -- I think this was hook_elements in the past wasn't it?), and it's only one line to add.

b) Within a function with the @element_list tag, // comments starting with @element or @property would get parsed for the form/render API reference.

c) Each documented element would need one of the following in the first line of its doc header:
- @element - if it is a render element
- @element_form - if it is a form-specific element (a render element that only works properly if it's inside a form)
- @element_form_input - if it is specifically a form element that takes user input
This would be followed by the name of the element (I could probably parse it out but I think it's easier if it's just provided). And then the next lines, indented two spaces, would describe it.

d) Each property would only be documented once. In one chosen occurrence of the property, we'd use one of the following:
- @property - if it is a property that is used on one or more elements but doesn't automatically apply to all
- @property_all - if it is a property that automatically applies to all elements
- @property_internal - if it is a property that is used internally by the render system
- @property_form_input_all - if it is a property that automatically applies to all elements declared as @element_form_input
- @property_form_input_internal - if it is a property that is use internally by the form/render system, but only applies to form input elements
This would be followed by the type and name of the property. And then the next lines, indented two spaces, would describe it.

I think this would be enough information to generate something pretty much like the Form API reference. Thoughts?

I agree that we could get rid of the element and property names in:

// @element this_element_name
and
// @property bool this_property_name

It would make the parsing slightly more complicated, but yes it could be done. I'm not strongly in favor of either approach.

Regarding the properties that are not defined anywhere, I was just thinking about that too.

For all-element properties, how about if in system_element_info(), we added a "dummy" declaration like this:

// This array is just for documenting properties that do not explicitly appear
// in any element definitions.
$dummy_property_definitions  = array(
  // @property_form_input_all string[] #element_validate
  //   An optional array of functions that will be called to validate the element.
  '#element_validate' => array(),

   (continue with other properties here)
);

For properties that are not globally applicable, we could do something like this:

  // @element_form_input submit
  //   Submit button for a form.
  $types['submit'] = array(
    // @property_form_input_all bool #input
    //   TRUE if this is a form input element. Default: FALSE.
    '#input' => TRUE,
    // @property string #name
    //   The name attribute of the button.
    '#name' => 'op',
    // @property bool #is_button
    //   TRUE if this is a button.
    '#is_button' => TRUE,
    // @property bool #executes_submit_callback
    //   TRUE if when this button is clicked, the form submit callbacks should
    //   always be executed. Used for AJAX processing.
    '#executes_submit_callback' => TRUE,
    '#limit_validation_errors' => FALSE,
    // @property_all string[] #process
    //   Array of names of functions that will be called to process this
    //   element during rendering.
    '#process' => array('form_process_button', 'ajax_process_form'),
    '#pre_render' => array('form_pre_render_button'),
    '#theme_wrappers' => array('input__submit'),

    // Other properties that do not have defaults are listed below.

    // @property string #value
    //   The text to display on the button.

  );

This would be an argument for why it is useful to have the property or element name on the line with the @property/@element tag -- because then if there is no code below it, we could still parse it out.

Thoughts? Sigh. Maybe this is becoming too complicated again and we should go back to #42's idea.

Update current status/proposals for this issue

I have added this to the agenda for the next meeting of the Documentation Working Group, which is a new Governance group that is at the goal-setting stage.

No, we haven't figured this out yet.

I have completely rewritten the issue summary for this issue, outlining the specifications for what we want, and the various proposals that have been made.

We really need to resolve this. The current situation is that we have no documentation for render elements, out of date and unmaintainable documentation for form elements. Not OK.

Any thoughts on the proposals? Are any of them viable? Can we adopt one, flesh it out, and move forward? Please? :)

Silly testbot, this was a proposal that needs review, not a specific patch. :)

Added historical note to issue summary.

Add strong tags to highlight what each proposed resolution is.

RE #68 - Each property has a number of components to document (see item 6 in the Requirements section of the summary). So I don't think that putting all the property documentation into one function header (like drupal_render() docs for instance) will work very well, because you'd need to have a bunch of @ tags for each property. This is also why we don't want to put all the elements documentation into the hook_element_info() doc header, because elements also have a lot of components to document (see item 4 in the Requirements section). [This idea is proposal e in the summary by the way, and has been discussed previously on this issue and/or the other one. It's just not really viable.]

Making them into dummy functions in an api.php file... yeah, we could, but they really aren't functions or even remotely close to being functions, so it's fairly weird. We also do not want them to appear in the Functions list on api.drupal.org, I think? Too confusing. But we could do it that way I guess. I'm not totally opposed.

Anyway, I've added this as proposal (g) to the issue summary, and noted on the Element proposals that this is an alternative to YML files.

There was discussion early in the D8 coding cycle (and in this issue) about turning elements into plugins or classes or something. But that did not happen. So please do not discuss it here; way out of scope and not happening until Drupal 9. I think there's even an issue link somewhere above in one of the comments. We have completely impossible documentation now, and I really really don't want to wait several more years to resolve this problem. The 7 years that it's already been is plenty.

Regarding YML vs. PHPDoc... Yeah, I think we need to stick with PHPDoc too. (I think we should do this for service tags too; off topic for here.)

Regarding properties... Really it is not possible to document properties completely (with examples etc. like they are in the current Form API reference) inside of another doc block. This is extensively discussed in the Issue Summary and previous comments. They just have to be elsewhere, and I don't think there is any way around it, even though it will probably mean their docs are not updated as often.

So....

OK, here is a full proposal for what I think we should really do. I've noted this in the issue summary.

Elements

[Proposal (f) in the summary] For elements that are defined by modules that have only one element to define, put the element documentation into the docs header for the hook_element_info() implementation. For modules that have multiple elements, change the hook_element_info() implementation so that it calls individual functions, each of which defines one element, and put the docs for elements in those individual functions.

So it would look like this:

// in system_element_info():
$types['textfield'] = _system_element_textfield();

/**
 * Defines a text field form element.
 *
 * Corresponds to an HTML input element, with type="text".
 *
 * Further documentation goes here, like some information about autocompletes.
 *
 * Usage example:
 *    @code ... @endcode
 *
 * @form_input_element textfield
 *
 * @see ...
 */
function _system_element_textfield() {
   return array(
    '#input' => TRUE,
    '#size' => 60,
    '#maxlength' => 128,
    '#autocomplete_path' => FALSE,
    '#process' => array('ajax_process_form'),
    '#theme' => 'textfield',
    '#theme_wrappers' => array('form_element'),
  );
}

The important note here is that to indicate "this is an element", you would include one of these three doc tags, in order from most specific to least (include the most specific one that applies):
- @form_input_element element_name
- @form_element element_name
- @render_element element_name

In the function body, it would automatically show the specific properties, and on api.drupal.org, these would link to property pages.

Properties

[newly-added proposal (h)] For properties, we would make files like system_properties.api.php. Each property would be documented in a documentation block that doesn't have code associated, similar to a @defgroup doc block.

Sample:

/**
 * @property #options List of options for form elements with multiple choices.
 *
 * @property_form_input
 * @property_automatic false
 * @property_internal false
 * @property_name #options
 * @property_value
 *   A simple array of options, or an array whose keys are the internal names and whose values
 *   are the corresponded translated display values.
 * @property_default array()
 * 
 * @see radios.html.twig
 * @see comment_controls()
 */

The important thing here is that each docblock would start with a @property tag [might need to make sure this is a good tag name that isn't used for something else in phpdoc/doxygen first!], similar to @defgroup.

Additional tags would define aspects of the property, like whether it is internal, and whether it is automatically inherited, and what type of property it is (see issue summary "requirements" section for things we need to document).

We would also need a way to document properties that have different docs for different uses of the same name. TBD. But I think we can work this out.

Thoughts?

Hm. Mock classes. Maybe... The API module does not make a distinction between real and mock functions (so we have hook_*), and it wouldn't make a distinction for these fake classes either. Would that be confusing? And if we did this, the documentation for the elements would be removed from the declaration of them, which I think is not great.

Regarding splitting up hook_element_info(), this was proposed a LONG time ago because people didn't want to have /** */ comments inside of hook_element_info(). I don't think it's a huge deal to split it up, and splitting it up has the advantage that each element declaration is its own function body, so it's easy for the API module to parse, and therefore figure out which properties are used by each element declaration.

One thing I really don't like about #72 is that properties for elements are actually #name elements in arrays. Equating them to class $name properties is ... not great, IMO, since that is not at all how we define/use them, and their names are not $name but #name really.

I also don't think making up fake classes is less future-proof than splitting up hook_element_info() implementations into actual (non-fake) functions. When this was suggested more than a year ago in above comments, David even said we could backport it to 7.x. It doesn't change anything substantive in the code, just takes one function and breaks it up into a number of smaller function calls.

I realize we are trying to get rid of functions and hooks in D8, but this is one area where we didn't do that. If elements do become classes in 8.2 or 9 or whatever, I just don't see that having made fake classes in an api.php file is going to help in any way when the structure of the classes is quite likely to be quite different (they'll most likely be plugins for instance).

Regarding #80, what are the chances of this happening any time soon?

We have a very strong need to have documentation for the form/render elements for Drupal 8. Editing the existing, and really outdated at this point, Form API reference document is not happening -- every volunteer who has said they'd maintain the behemoth of a document has given up. So we have at least 20 issues where updates are needed, and no real viable way to make the updates.

So unless someone is volunteering to make elements into plugins for 8.0, and unless core maintainers are on board with committing such a patch, and unless we really think it is going to make it into 8.0, can we at least come up with a solution that we can do with minimal disruption?

Ummm "... Oh, you need to split your hook_element_info() implementation up in to helper functions, in a way that's not documented in the docs for hook_element_info()."

Why can't we then just document it in hook_element_info()? If we do this, we can surely add a one-line note to hook_element_info() saying "Note: If you want your elements to be documented, see ..." and write up some docs on how to do that at a URL on drupal.org.

Regarding having several items in the doc block, each one needs to have @tags included, blank lines, paragraphs, etc. I don't think your simple illustration does it justice, and I don't think having them all together will really be all that readable by people; not sure about readable by the API module either.

We've been promised plugins for Drupal 8. Let's see if this happens in the next couple of weeks.

Coding standards changes are now part of the TWG

So, I have a new idea, which will be much simpler, which will be OK for Drupal 8. I'm also moving this back to Core because I don't think this idea requires acceptance outside of the Core maintainers/developers. And removing the backport tag because it can't be backported.

Here's my idea.

In Drupal 8, we already have classes for all the Form/Render elements. They are annotated with either FormElement or RenderElement, and therefore listed on those class pages already, under "37 elements annotated with...":
https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Render!Annotation...
https://api.drupal.org/api/drupal/core!lib!Drupal!Core!Render!Annotation...
However neither of those lists is extremely useful, and no one knows about them. Also, these classes do not have all of the information that is currently in the Form API reference.

So what I would like to propose is a fairly straightforward fix for this:

a) The API module would make a new page called Elements. This would be similar to the Functions, Classes, Services, etc. pages that are listed in the right sidebar navigation for Drupal 8. It would collect all of the @FormElement and @RenderElement classes, and make a 2-column list giving the element name that you'd use in a render array, and a link to the class. So it might look like:

Elements

Like similar tables on api.drupal.org, this would have filters and table heading sorts (see the Functions or Class page for example). The API module would automatically gather these from the existing code/annotation and make the table. No patches to Core (yet!).

b) The API module would make a list like "38 uses of this element" (similar to "25 functions call this function") on each form/render element page. This would give you automatic links to examples of the use of the element. Again, no patches to Core (yet!).

c) Each form/render element class would get a review and documentation update (if necessary) to make sure its documentation includes the following:
- Documentation of its specific properties. We'd talked in previous proposals about having separate property pages, but no one could really agree on this, so let's just make sure each element documents the properties that it specifically uses. We can discuss a format, but I think the simplest thing would be just to do something like this in the class doc block:

 * Properties:
 * - size: The number of characters in the text box (default: 60).
 * - foo: Another property description.

- @see links to any theme templates and other functions/files relevant to rendering that element.
- @see link to the main Render/Form API pages (we'd need to figure out which ones all elements should link to)

I think we could argue that we should be doing (c) in any case... I think that most of the classes already have most of the text -- they would just need Properties documentation. We can discuss the specific format, make a meta issue, and then it would be a good Novice project to fix up all the docs, similar to efforts we did on the Annotation classes etc.

d) We would make sure that the Topics related to form/render API on api.drupal.org tell you to go to the Elements page to discover what the elements are. We would also want to make sure there is documentation somewhere on a topic of the common properties shared by all Render elements, and the extra ones that all Form elements inherit (things like #title and #required for Form elements). This would require a docs patch.

e) I'm putting in a method for each listing page (Services, Classes, Functions, etc.) to have some text at the top of it specific to the branch. We'd want to do this for the Elements page -- make it link to the relevant topics. This would also be a small docs patch.

Thoughts? I've added this as proposal (i) in the issue summary.

I am not against adding examples... At least if we just add the examples in @code...@endcode tags in the API docs though, they'll be much easier than editing the current FAPI HTML guide, which is horrendous.

The problem with putting the examples directly in the API docs, however, is that they may get out of date, whereas links to examples would be less likely to. Plus, it's duplication... So as a compromise, how about if we say that the docs should have a sentence like:

See [name of function or complete \class\with\namespace::method] for a usage example.

Then users would be one click away from a known actual usage example, rather than having to look through a list and try to find one.

Thoughts?

Yeah, @ref is just for within-one-doc-block links. So this idea is not going to work, too bad!

There is no reason why we cannot update the doc blocks for form/render elements and add more documentation and usage examples -- that sounds like a find idea.

This will not work in that patch suggestion in #97, if that is what you are asking:

+ * Properties:
+ * - @ref property_access "#access"
...

I'm actually not sure what it is supposed to do, but that is not what @ref is for. And then there are some -# lines later in the patch that ... not sure about those either but that is not complyiing with our docs standards.

So. There's no reason we cannot go ahead and update docs for any/all of the form/render element classes. And there's no reason we cannot make a Form/Render Element group/topic and make sure that all these classes have @ingroup elements or whatever in their doc blocks. But the suggested documentation in the patch in #97 is not going to work as it is.

So... At this time, the actionable part is:

Make sure that each Form/Render element class is well documented. Each one should contain basically what the existing Form API Reference contains (or something similar for elements not in the old Form API reference). So: a description, and either a usage example or an @see link to a good usage example.

We do not have a topic yet called form_elements, so we should not have an @ingroup form_elements in any documentation.

And starting from the proposed D8 patch for the Form API docs is probably a good idea. You might want to look at other issues in the Documentation project tagged "FAPI reference" to see what other suggestions people have had for the D8 form API reference too, but for now at least getting more/better docs on the classes would be great. Thanks!

The proposals in the issue summary are outdated since we don't have hook_element_info() any more in Drupal 8.

Also, metzlerd has done a great job of adding some documentation to all of the existing element classes in Drupal 8 on related issue #2486967: [meta] Move/Create Form Element Documentation. All of the classes now have Usage examples and some documentation of their special properties.

So... I still think that proposal (i) in comment #94 is the best idea so far for making a substitute for the Form API reference. It would require:

- API module - #839550: Ability to add documentation headers to global listing pages (Classes, Functions, etc.)

- API module also needs to make a new listing page called Elements (no issue yet), as described in #94, which would list the machine names and classes for each; you'd click through to the class to get the full docs (which are now in pretty good shape, or at least they will be once the remaining two issues there are committed).

- We'd also need to write some documentation for the standard properties of all form/render elements.

Really that is all we would need. It is quite simple.

The API module changes have been completed (though not deployed to api.drupal.org yet). See #100680-118: [meta] Make API module generate Form API documentation for details. I'm just going to mark this as a duplicate, since it was only spun off from that issue because we were thinking we'd need to change how hook_element_info() was documented. In 8, we don't have this hook any more, and we're documenting render array stuff on the render element plugin classes, so we don't need a new standard any more and this issue is obsolete.

Issues to work on for Drupal 8 to get the docs finished are the children of #2486967: [meta] Move/Create Form Element Documentation, so this is also partly a duplicate of that issue.

Anyway... we just don't need this issue open any more. Hooray!

The list of form/render elements is now up on api.drupal.org -- I think it is complete now, but if not, it should be complete within the next hour as api.drupal.org reparses files (there may be a few left to go):
https://api.drupal.org/api/drupal/elements/8

On #2603818: Add defgroups for listing page headers for api.drupal.org I have a patch that will add a documentation header to that page (as well as the Classes, Namespaces, and Services listing pages). A review of that patch would be greatly appreciated, and would further the cause of having documentation available. Thanks!