Last updated April 26, 2012.
Drupal currently lacks a system for writing really good technical documentation. We should have one, and in particular, the Drupal Documentation Team needs such a system for two purposes.
First, we've discussed a new system for in-line help in Drupal (i.e., what you see if you have the Help module enabled in a Drupal site, or Advanced Help, or other help-related modules, and have permission to view help) on this issue:
#1031972: Discussion: What would a better help system for D8 be?
Second, we've also discussed building a new system for "curated documentation" for the Drupal project (i.e., a way for the core Drupal project, and individual contributed modules/themes, to provide well-structured "official" documentation and manuals, with editorial control over who can create and edit the pages) on this issue:
#1291058: Discussion: Make a curated docs section/system
Since both of those involve a structured system for writing good technical documentation, we've pretty much decided that they're really the same system. This page contains a proposal and specifications for what the in-line help ("Help") and curated documentation ("Docs") system would include (features, requirements, etc.), and how to build it.
We propose to build this Help/Docs system in three phases:
- Phase 1: Build the minimal Docs system on a *.drupal.org web site and start creating/editing Docs (starting with an official Site Building with Drupal manual, and opening it up immediately for contributed module/theme projects to write their official Docs too).
- Phase 2: Add advanced features to the Docs system.
- Phase 3: Make the Docs importable into other sites as Help.
Note that Phases 2 & 3 could be simultaneous, or in reverse order. To track progress on building this system, see issue:
#1549580: Track progress of building the Help/Docs System
This page details what would be involved in building this system, with features marked Phase 1, 2, or 3 based on the list above, and considerations marked as Docs or Help depending on whether they apply to the curated documentation idea or the help system idea (or both). Contents of this page:
- General considerations and requirements
- Help topic entity
- Contextual and topic-to-topic linking
- Maps (outlines)
- Conditional text
- Workflow and permissions for editing
- Documentation updater
- User interface considerations
- Implementation and plan
Terminology for this page
The following terms have specific meaning for this page:
- The site where the Help and Docs will be edited. It could be a sub-site like help.drupal.org or docs.drupal.org, or it could be on drupal.org itself, but for purposes of this page, it's referred to as "Help.d.o" throughout.
- In-line help within a Drupal site, such as what is currently provided by the core Help module, the contributed Advanced Help module, contributed contextual help modules, etc.
- "Curated" documentation: official, well-structured manuals and documentation for Drupal as a whole and for contributed modules/themes, with editorial control over who can create and edit the pages.
- Each individual Help/Docs item/page is called a Topic (this definition is expanded on below in the Help topic entity section).
- An outline for navigating through Topics (this definition is expanded on below in the Maps (outlines) section).
General considerations and requirements
- [Phase 1] Help and Docs will be organized into modular, atomic, structured "Topics". This idea comes from DITA (Darwin Information Type Architecture), and should help people write better help, by giving structure to the Topics (the structure of Topics is expanded on below in the Help topic entity section).
- [Phase 3] All Help for Drupal Core and contributed projects should be done in a consistent manner. So, the Help system must be flexible enough to cover:
- In-line help pages for Drupal core and contributed modules/themes, including what the core Help module [hook_help()] and the contributed Advanced Help module can do now.
- Contextual pop-up help for admin screens within a Drupal site.
- Help supplied to community member-type users (not admins) on a site. For instance, if you create a Drupal site for your basketball league, you might want to display Help Topics for the team coaches, explaining how to use the features of the site, set up their team roster, etc.
- [Phase 1/3] Help and Docs Topics should be editable by technical writers (as opposed to coders), and removed from string freeze:
- Editing should be done on-line on Help.d.o, not in PHP code or with patches.
- Editing should not require a huge amount of knowledge of HTML, XML, etc.
- [Phase 1] Help and Docs editing needs editorial control - we don't want just anyone to be able edit or add to the Help/Docs at any time (free edits will of course still be allowed/encouraged in the Community Documentation pages). Details of the needed permissions and workflow are in the Permissions and Workflow section below.
- [Phase 1] Help for Drupal Core and contributed projects should be viewable outside the context of a Drupal site (i.e., anonymous users should be able to view all of the Help Topics on Help.d.o).
- [Phase 2] Help (and maybe Docs?) should be translatable by the Translation teams on localize.drupal.org. Meaning:
- The translation teams for different languages need to be able to supply translations for Help Topics.
- Translation teams need to have a way to have team leaders moderate the translations as they do now (less expert members supply proposed translations, and these are edited/selected by more expert members).
- The translation team needs to be notified when the base Topics have been changed or new Topics have been added, so they can see if a new translation needs to be made.
- Drupal at run-time needs to be able to load the correct translation of Topics from the local server, based on the language context for the given page load [i.e., a mechanism like what t() takes care of for smaller UI text chunks].
http://en.wikipedia.org/wiki/XLIFF might be the right standard to use for translation of larger chunks of information? PO files are great for UI text, but not good for larger text chunks.
- [Phase 1] Help/Docs Topics should have navigation features, such as searching, sorting, and hierarchical outlines (Maps). The Drupal documentation team should be able to create multiple Maps from the Topics (and this mapping ability might also be applied to the current Book module, or perhaps take its place? -- phase 2).
- [Phase 1] It should be possible to to put images and possibly other media into Topics.
- [Phase 3] A site builder should be able to pull Help Topics, translations, and Maps from Help.d.o into their site, add Topics, and revise/add Maps. They need to be notified when Topics or translations are updated, added, or changed. Or [NOT DECIDED if this should be a feature or not?] they should be able to use the Help.d.o site as their site Help, or as part of their site Help. [Note JH - I think this might make the Help system overly complex to mix local and remote Topics especially, but it seems that many people contributing comments to the issue thread think this is desirable to not pull the Help in locally, so it's always linking to up-to-date Help. But if the remote Help uses conditional text, we would want a way to request display of only the right conditions, since people on the local site might not know what version of things they are using.]
- [Phase 1/2] Where appropriate, we should aim to be able to create DITA XML output, so that DITA tools can be leveraged to present the help in other formats (books, PDF manuals, etc.).
- [Phase 1/3] Topics should have cross-linking ability (see contextual linking section below)
- [Phase 1] Topics need conditional text ability, so that we can write once for multiple Drupal versions, etc. See conditional text below for more details.
- [Phase 3] For some (but not all!) sites, we probably want display of Help to be by permission. For example, if you don't have 'administer blocks' permission, you probably shouldn't see the "Adding a Block" Topic, if you are looking at help within a Drupal installation. On the other hand, on Help.drupal.org, that Topic should be visible to everyone.
Help/Docs topic entity
[Phase 1] We'll define a new entity called a "Help Topic", to represent a Topic for both Help and Docs. [We don't want to use the Node entity, because the Help Topics entity serves a different purpose, might have different permissions, shouldn't be part of site-wide RSS feeds, etc.] This corresponds to the DITA "topic" concept: each Help Topic item will be a small, focused, structured, atomic documentation item.
[Phase 1] There will be several types of Help Topics (corresponding to DITA topic types, plus some additions or sub-types). Each will be an entity bundle (analogous to Node's content types) having its own field structure, which we will use to enforce the DITA topic structure for each type. Note: Kristof Van Tomme has already done some work on defining a "Poor Man's DITA" with CCK in Drupal 6. We can build on this.
Some things this entity needs to do or properties it needs to have:
- [Phase 1] Bundles (i.e., needs to have individual help topic types, and accept fields)
- [Phase 1] Revisions
- [Phase 3] Access control - on some sites, you probably want to have only people of certain roles be able to see certain topics. I don't think we need something as complex as the Node Access system -- but since we do need it to work for viewing lists (e.g. search results), we probably do need to have a hook that would let a help topic access module define that certain roles can see certain topics. This could be stored in a help_topic_access table, where a row with a given entity ID and given role ID means that the role can access that topic. And if there is nothing in the table, that would mean that there are no access restrictions. This is simplified from Node's grant/access hook system, and not as flexible, but it is likely enough for Help? There would also need to be a way to batch/queue rebuild the permissions.
These are all Phase 1, except where noted.
- Short description - this would show up as the excerpt in search results and outlines that show excerpts, as an aid to figuring out whether the page has the information you are looking for.
- UUID - some type of universal identifier, possibly the URL of the page on its site of origin? Suggested URL structure:
The idea would be to go from the project down through a project-specified hierarchy (which might or might not match its admin screen hierarchy), and have the conditional items as URL queries, so that if the conditions don't exist they could be ignored.
- [Phase 3?] The revision should also have a universal ID, so you would know whether you have revised it locally or the remote copy was revised.
- Modules - which module or modules does this topic pertain to (optional)
- Versions - which versions of these modules does this topic pertain to (optional)
- Permission - what permission level is needed to have this topic be relevant (optional)
- Editor notes - a place to record information pertinent to future writers/editors revising this topic, such as where the information was found, other topics that are depending on this one, etc.
- Keywords for indexing - to be useful for an index, each keyword should start with a specific noun or verb, and be in the order people would look it up. Examples: "content, creating", "block, deleting", "installing a module", "module, installing". Only start with a verb if it is pretty specific and unlikely to be used for many entries. For instance, "deleting" is likely to be used in many areas, but "installing" maybe not as much, so "block, deleting" would be appropriate, and "installing a contributed module" would also be appropriate.
- Body - structure depends on what type of topic it is -- see below.
- Related topics - multiple-valued field; each is a reference to another Topic or an external link.
Note that the Keywords and Short Description could also be used for the HTML meta-tags in the HTML header, if this topic is displayed on its own web page. It could be useful to have some additional site-wide keywords feeding into the meta-keywords field, such as "Drupal, help" for instance, and maybe the Module field too?
Task help topic type
(Corresponds to DITA standard Task topic.) A Task topic describes the steps needed to accomplish a real-world task. The body would contain the following sections (probably easier to edit as one body field though):
- Requirements (prerequisite steps and software requirements)
- Steps (ordered list of actions, each starting with an imperative verb, an "if..." clause plus imperative verb, or (Optional) plus an imperative verb, and optionally describing system response to the action). Example steps: "Navigate to ...", "If you are a new user, set up an account.", "(Optional) Customize the colors.") (Note that the steps can be nested -- for instance, after "set up an account" or "Customize the colors" in the example above, you could give the steps for how to do that.)
- Follow-on tasks - common tasks that you might want to do after completing this task.
Note that for a high-level task, the steps might just be links to other task topics.
Concept help topic type
(Corresponds to DITA standard Concept topic.) A Concept topic contains background information, definitions, rules, explanations, or guidelines. The body has no particular structure.
Module Overview help topic type
This is basically a sub-type of the Concept topic type, in DITA terms, but it is useful for Drupal purposes to have a specific topic type for it. It would give some background information about what a module does, and then the Related Topics field would list the tasks you can do with this module. Could also be used for Themes.
Glossary help topic type
A Glossary item topic is a simple definition of a term. The title is the term being defined (probably displayed as TERM (definition)), and the body is the definition. Additional fields:
- Variants (multi-valued text) - synonyms and variant forms such as plurals
We could use this to (a) automatically build an alphabetical glossary list and (b) have a text filter that would make automatic pop-up (or regular) links to glossary items or variant forms appearing in text.
Reference help topic type
A reference item type gives supporting facts, such as programming syntax, definitions of scenarios used for illustration of other topics, or descriptions of sample data.
Contextual help topic type
A short help topic that is specific to a particular field or section of a particular page. These topics would not appear in the index or navigation maps, and are probably mostly reference or concept information.
Contextual and topic-to-topic linking
Topic to Topic links
Besides the "related topics" field, it is helpful for Topics to have links to other topics within their Body field. Suggested syntax:
- [helplink UUID] - makes a link to the Topic with that UUID within the text, using the Topic title as the link title
- [helplink UUID title] - same as above but overrides the title
- If the UUID contains the URL of the help's origin, and if the Help Topic doesn't exist (hasn't been imported) in this site, it could make an external link.
- Turning these link indicators into actual links would be the job of a text filter.
The help system needs to have the following contextual linking abilities (between Help Topics and the site):
- Linking a Help Topic to an admin page on the site, by specifying a path, query, and anchor, in a syntax for saying "this is a link". Thoughts:
- One example: the main help page for the Blocks module might want to have a link to the main admin page for Blocks (admin/build/blocks in D6, etc.).
- Is there a standard DITA solution for this type of linking?
- Possible syntax in the help text: [pagelink text, path, query, anchor]
- Possible implementation: a text filter that turns it into a link appropriate to the site. So if we are displaying this Blocks help page within a Drupal site, we might link within the site to the admin/build/blocks page, and if we are displaying on Help.d.o, we might make a link to example.com/admin/build/blocks)
- Linking a module to a help outline or a Help Topic -- to define what the main help is at the module level (to display on the Modules admin page presumably)
- Contextual link on a site admin page to the Help (the page as a whole or an element within the page -- element-level links to be used sparingly). The idea would be to make a unique ID for the Help Topic, and then you can make a php function or some other functionality to make a link to a UUID help topic (themeable, etc.). A topic should be small enough you never need to link into an anchor for part of the topic. Hopefully. For a form element, it could be that it has a new attribute:
#help_link => 'UUID',
Note that Core help would not have the ability in the admin UI to make a link. This would be for module developers to add to pages/forms generated in code.
And as a note, we might want the links to be stored as part of the Map rather than either part of the module or part of the Help Topic entity, if possible. See Maps/Outlines section just below.
The DITA concept of a Map includes the idea of a hierarchical outline and cross-linking. Both could be useful. We need to define both a database structure for a Map, and a user interface for creating multiple Maps. We want the ability for multiple Maps to exist that use the same Topics. See also #995370: Want the ability to create multiple outlines/maps and #5901: Let a book page appear in multiple books (the latter applies to Books).
If we expand the definition of a Map to include contextual links between Topics and between Topics and admin screens or other pages on the site, then the modules and help pages become more portable, and the Map becomes a navigable, context-aware system, which could include local or remote pages. A module could include a default Help Map with links to the pages on Help.d.o. Modules to look at for this: Contextual Links, and Advanced Help Inject http://drupal.org/project/helpinject -- in order to allow the Map to specify that a particular form/render element or page should have a contextual help link.
<help page reference>
<original help path>http://help.drupal.org/drupal/configuration/system/site-information#7</original help path>
<actual help path>help/92</actual help path>
<help display places>
<contextual link pattern>admin/config</contextual link pattern>
<contextual link pattern>admin/config/system/site-information</contextual link pattern>
<contextual link pattern>admin/config/system#page-content-system_main-main</contextual link pattern>
</help display places>
<help page weight>5</help page weight>
<parent help page>http://help.drupal.org/drupal/configuration/system</parent help page>
</help page reference>
<help page reference>
<original help path>http://help.drupal.org/drupal/configuration/system</original help path>
<actual help path>http://palantir.net/help/12</actual help path>
</help page reference>
[Phase 1, except where noted below]
NOTE: This has been (at least mostly) implemented as the Conditional Text module.
The basic idea of conditional text is to evaluate an expression, and display some text if it's true, and hide the text if it's false. However, in some cases (such as displaying on Help.d.o), we would want to display all conditional text in fieldsets and let the user choose (possibly by cookie) whether to view or hide particular conditional text. See also #995362: Conditional Text ability needed for book pages
The Conditional Text feature should be a module that acts as a text filter, so that it can be applied to the text in the documentation topic entities. Additional features of this module:
- Ability to specify conditions at the topic level or within a topic.
- Ability to have Drupal-specific conditional expressions, like "module x is version y.z or more" -- we can probably borrow the parser for simpletest requirements? or drush make requirements?
- Ability to have non-Drupal-specific conditions, such as for experience level, audience (themer, module developer, site builder, etc.), database engine, web server, O/S, etc.
- Each condition has a class (which would be the plugin class, see below), and an expression (e.g., 'level=beginner')
- Possible condition syntax -- insert something like this into the text:
[condition installed views>6.x-3.x]text[/condition]
[condition generic level=beginner]text[/condition]
Note on syntax: We probably want to use square brackets  and not HTML/XML tags. The main reason is that in Drupal, HTML/XML tags will get stripped out by some text filters, and also they are a problem if they are entered into WYSIWYG editors. So let's stick with square brackets.
- We might want a WYSIWYG condition inserter, but this is probably not necessary in the base module, if the syntax is straightforward enough.
- [Phase 3] Ability to export the list of defined conditions, along with the help topics themselves, in the Documentation Updater, and import them on the other end.
- A doc writer/admin needs a UI that will let them define conditions, and define human-readable labels for them (for purposes of fieldset display and for the site builder UI; some types of conditions plugins might define the human-readable labels in code).
- A site builder needs a UI that will let them specify which conditional text to display in which output mode. Possible output modes:
- [Phase 3] Show/hide completely before sending to the browser -- for instance, on a site that has Views 2.x, completely omit the Views 3.x conditional text. Or if the user is role A, completely omit some of the conditional text for things they don't have permission to do.
- [Phase 1] Use a fieldset -- for instance, on Help.d.o, have the Views 2.x and Views 3.x text available in collapsed/open fieldsets.
- [Phase X] RDFA [lower priority] -- Output using RDFA tags
- [Phyase X] DITA [future]
The site builder should have the flexibility to define behavior for each condition type for each role.
Possible implementation notes:
- Needs to be a contrib module for 6/7 so we can use it for the d.o docs. http://drupal.org/project/collapse_text might provide at least a starting point.
- Should be plugin based, and until Butler's plugin architecture is defined, the CTools object-oriented plugins would be good to use.
- Each plugin will define a class for conditional text. Possible methods:
- evaluate(expression, permisssions) will return 'show', 'hide', or 'fieldset' to say what to do with the enclosed text given the permissions that have been set up (or that might be a value on the class?)
- help() will return a description of how to write an expression for this plugin type (such as a description of how to do views>3.2, or a list of possible type=value pairs for the generic condition).
- permissions() [needs a new name?] will return a list of view "permissions" to set by role. Each role/permission is a setting for how to display the types/values that are included in that permission by the plugin.
- Some way to let the plugin define configuration options (I assume CTools has this?)
- There are several cases for "permissions":
- [Phase 3] example: module versions. On a Drupal site, we probably want to hide any conditional text where the version doesn't match, but on Help.d.o we probably want to show it all in fieldsets and not check installed versions, and there we might also want to have the fieldsets expanded/closed based on some cookies.
- example: user-entered properties (e.g., type:level, values: beginner, intermediate, advanced). In this case we might want to say that a given role means the condition evaluates to true or false. And then we would again say whether if the condition is true/false, we would choose between show/hide of the text, or using a fieldset.... so this case means that for user-entered properties, we would let config say by role which types/values evaluate to true/false, and then we would let them configure by type they've defined, what to do for true/false [either show/hide based on true/false, or use a fieldset independent of true/false].
- example: maybe some other module has a different way to define how to evaluate the conditions, so it might have a different config for that. And then it can configure globally or on a per-type basis what to do (use true/false or use fieldset).
- The base conditional text module should define two basic plugins: an 'installed' plugin to deal with module versions, and a 'generic' plugin to let the user define types/values such as type="level", values="beginner, intermediate, advanced".
- Maybe there should also be a validate that at edit time will check to see if the conditions are entered correctly? e.g. does the type/value exist, or is the syntax correct for module version? do input formats have the ability to validate the text? I am not sure, but maybe they do. If not I guess use watchdog or drupal_set_message at render time.
Workflow and permissions for editing
[Phase 1 mostly]
There are several workflow and permissions functions that need to be implemented for this system, on the Help.drupal.org site (for editing the Topics):
- Module maintainers should be able to designate who can maintain the Help/Docs Topics and Maps for their module.
- There should be separate permissions for creating new Topics, editing existing Topics, creating new Maps, and editing existing Maps, with the ability to designate permissions by user or role (with Authenticated Users being one possible role assignment, for projects where the maintainers want to let the community manage their official documentation).
- [Phase 2] Ideally, it would be nice to let anyone in the community propose a revision, but let designated project people or roles make the decision to accept it as the "live" revision after seeing it on their dashboard.
- We need to be able to designate this for Drupal Core and Documentation projects as well. The Drupal Core project would oversee the core Help Topics, and the Documentation project would oversee the official Drupal Docs (of course, there would be overlap between the two, but for instance the official Site Building guide might use Views and other modules that would not be covered by the Drupal Core Help).
- [Phase 2/3?] The translation teams from localize.drupal.org for different languages need to be able to supply translations for Help Topics.
- [Phase 2/3?] Translation teams need to have a way to have team leaders moderate the translations as they do now (less expert members supply proposed translations, and these are edited/selected by more expert members).
To import Help Topics to local sites, there will have to be something similar to the Localization Update module. It will allow you to do the following:
- Select among the enabled modules and languages, and maybe other criteria (such as Topic conditions) to choose which Help you want to download.
- Select how often you want to check for updated Help.
- On individual Help Topics, you should be able to set an option "do not overwrite if external source is updated" (or something like that). Maybe you should also have a global option blocking overwrites of all local Help pages.
- Set a path to the server to get Help updates from (maybe?).
Note that the Server for the documentation updater does not need to be in core, but the Client would need to be in core. This is similar to what is done for localization. Given that Help.d.o would probably need to be a d7 site (since it would use entities), the Server would need to be a d7 module.
Additional thoughts on this:
- We'd need to be able to import both Topics and Maps.
- We'd probably also need to import the Conditional Text input filter settings -- at least any custom groups of values that are used in the Topics (e.g., if we've defined a group for "audience" with values coder, themer, etc., we'd need to be able to import that and set up the text format.
- Speaking of which, during import you would need to define what text format would be used for the body field of the Topics
- How about if the transfer format was XML, and in specific, how about DITA-compatible XML? That would buy us for free the ability to export our Help/Docs to DITA, which buys the ability to use DITA tools to create PDFs and other formats of output. I think it would, anyway.
User interface considerations
[Phase 1, except where noted]
This Help/Docs system will have the following user interface components for the end user (both in-site Help and Docs on Help.d.o, except where noted):
- Displaying and navigating through Maps
- Displaying and navigating through the A-Z glossary
- Concept Topics could be displayed in an alphabetical list
- Keywords could be used to build an index, if the keywords were chosen carefully
- Conditional text - expanding and hiding fieldsets by doc viewer
- Conditional text admin screens - defining conditions, setting up display options
- Searching for Topics - via official keywords, meta-data (module/version), and/or full text search
- "What links here" block could be useful, for finding related Topics?
- [Phase 3, Help only] Contextual linking (admin pages and Help Topics in both directions). Note that ideally an admin page would have one link to a Help Topic (such as a ? icon at the top), but occasionally there might be a need for a particular form element or page section to have its own contextual link. These should be kept to a minimum.
And for the site builder:
- Building Maps
- [Phase 3] Bringing in Topics/maps from Help.d.o
- [Phase 3] Defining which conditional text to display to whom
Implementation and plan
Possible list of the modules we would need to build this (see sections above for details):
- [Phase 1] Conditional text - define conditional text plugin structure and have an input filter for displaying conditional text (see the conditional text section above). This was created as a 2011 Google Summer of Code project by Yorirou, and is now the Conditional Text project.
- [Phase 1] Help - define the Help Topic entity, the Map entity, the topic types and their fields. It would also handle the contextual links (see descriptions above). It would associate the module overview topics with the module (like the existing hook_help system). And it would have options for display, such as hiding topics by the Permission field vs. showing all topics. This is being taken on as a 2012 Google Summer of Code project by temaruk.
- [Phase 1] Map/Outline Builder - user interface for building Maps. There are several options here. One is being worked on by Itangalo on #995370: Want the ability to create multiple outlines/maps, and there is also a related issue at #5901: Let a book page appear in multiple books that will hopefully also be resolved. Several other people have working prototypes of this too. We probably just need to pick one and make sure it has the features we need. Probably this is part of temaruk's 2012 Google Summer of Code project (but perhaps without the user interface).
- [Phase 2?] Glossary - would build the A-Z glossary and have a text filter for pop-up glossary links
- [Phase 3] Updater Client and Server - Server would live on Help.d.o (not a core module probably) and export the Help. Client would be a core module and would let a site builder pull in Help and be notified when new Help is available.
- [Phase 1/2] Workflow - For the Help.d.o site itself, we would need an editorial workflow module (see details above). We might be able to use the Workbench and/or Revisioning modules for this (with some custom components probably?). An issue has been started for discussing this, but we need someone to take on the project of building it: #1283496: Help/Docs system: editorial/translation workflow and permissions