Updated: Comment #105

Problem/Motivation

The top-level page on https://api.drupal.org/api/drupal/8 is one of the primary entry points for new developers into Drupal 8, but its list of topics is woefully short and doesn't contain any of the new concepts in Drupal 8. That can be found at http://drupalcode.org/project/documentation.git/blob/refs/heads/8.x-1.x:....

That links to certain select things under "Topics" (https://api.drupal.org/api/drupal/groups/8) which are effectively the @defgroups in core, but these are also missing key new concepts such as "Plugins," "Services," likely many others. These are important though, because they're directly related to the discoverability of these things.

It would also be good if all of the developer documentation was centralized on api.drupal.org, because although https://drupal.org/developing/api exists, many people don't know about it.

Anywho, this is a meta issue to sort this out.

Proposed resolution

Note: Items 1-3 are relatively straightforward and need to be done.

Items 5 on are a new idea and they need review. Update: The ideasin #5 and beyond were rejected by the community, but are left here as a record of what happened.

  1. New api.d.o D8 landing: Identify (Done - see comment #6) -- Identify an outline for the Drupal 8 api.drupal.org landing page.
  2. New api.d.o D8 landing: patch (Done, see linked issue) -- Take this outline and make a patch so that it becomes the Drupal 8 api.drupal.org landing page. Issue has been filed: #2148255: [meta] Make better D8 api.d.o landing page, linked to high-level overview topics, and put it in Core api.php files
  3. New api.d.o D8 landing targets: patch (Started - these are sub-issues on the previous linked issue) -- We can then get people going on fleshing out each of the landing topics, which will be core patches.
  4. developing/api tutorials - write (well underway) -- Everyone can continue working on the API tutorial docs on d.o for now under https://drupal.org/developing/api
  5. current developing/api tutorials - new home - proof of concept (A proof of concept was completed, but the community didn't like it, so it was abandoned.) The idea is to get the documentation currently on developing/api for Drupal 8 onto api.drupal.org, so that people will find it and know about it more easily, and so that all developer-related docs would be on the same site. We could do this in two ways:
    Method Details Pros Cons
    Nodes
    • Move the nodes from developing/api to api.drupal.org
    • Open up api.drupal.org so that people can edit nodes there
    We're used to this way of editing. Everyone can edit, so it's easy to contribute. It's fairly uncontrolled - leads to spaghetti documentation. You have to edit in HTML on the browser (HTML is not the best format for writing in). Also, right now there are no nodes on api.drupal.org or permissions for themm and not many people actually have accounts there (only if you have commented, which is 90% spammers anyway). And version (D7, D8, ...) is not obvious -- do you fork or modify docs when a new version comes out?
    Markdown/Git
    • Make a Git repo with text/Markdown syntax files containing the docs
    • Display them on api.drupal.org as pages
    • This is workable/working on a test site, see below
    Markdown is (arguably?) easier to write than HTML. You can use a text editor to edit. Developers like and know how to use Git and text editors. We can make the output into PDF and other formats as well very easily. (!!) When it's time to go to the next Drupal version, it's easy to branch the docs as a starting point. We're not used to AsciiDoc or editing docs this way (except of course that API docs are already in the code base). We would have to make the Git repo discoverable (with a block explaining where the pages are coming from?), or possibly editable on-line with a tool that automatically makes a fork/pull request or a patch. Edits do not go live immediately. Editing process (edit, make a patch, file an issue) is a bit more tedious than with Drupal nodes, even if the creation of the patch can be done from the Edit page right on the site.

    We thought maybe the Markdown/Git/Import method would be better. So, jhodgdon did the following to test out that method (see comments #11-13 for more discussion):
    - Made a preliminary set of documents taken from the API tutorial docs on d.o, and it put into AsciiDoc into a sandbox repo: https://drupal.org/sandbox/jhodgdon/2127335
    - Figured out how to compile AsciiDoc into HTML and PDF and display both on api.drupal.org. The (sandbox) module for this is: https://drupal.org/sandbox/jhodgdon/2127337 Note that we could also create other formats from the same source if we wanted to (.mobi, etc.).
    - Deploy it on a test site: [test site is no longer active, sorry.]

  6. former developing/api tutorials - convert (a few were started, but idea abandoned) -- We will need to convert the d.o files in https://drupal.org/developing/api into AsciiDoc and commit them to a repo (marking each page at the top with a note saying not to edit any more outside of the docs repo). And we will also need to make both sandboxes into real projects. And give more people commit privs to the docs repo. We will need to figure out who should be maintainers of the docs repo (unless we want to put it into Core?). The concept would probably be to think of the docs maintainers as "documentation editors", and people would most likely self-select and request maintainership if they think of themselves in an "editor" role.
  7. former developing/api tutorials - deploy (not started, and idea abandoned) -- At some point, when a critical mass of conversions has been done, we need to put the new module on api.drupal.org and set up Jenkins jobs to update the docs from git, and run the scripts. That is probably an hour of work for someone like @drumm. The scripts require that 3 Linux packages be installed on the server where api.drupal.org is living, but are otherwise straightforward to run and quick. Details are in the project README.
  8. Live git editing on the web (proof of concept was working, but idea abandoned) -- We need an "Edit" button somehow, so that people who find problems or want to add to the pages can edit them easily. This would need to leverage some existing technology to find the Git source, allow you to edit it, and post the result as a patch or pull request, and thereby allow people who aren't Git gurus or who don't want to figure out where the Git repo is to propose edits to the documentation. Technology that could be used:
    - GitHub: Github generally allows people to edit files in the web interface via a browser. If they don't have commit access to the file it says "Clicking this button will automatically fork this project so you can edit the file". Then, submitting a change to this file will write it to a new branch in your fork so you can send a pull request. [Symfony uses this]
    - GitLab: http://www.gitlab.com/ - has a web editor (and forking, pull requests, and a web UI for merging).
    - Prose.io (built by former Drupal shop Dev Seed for use on Jekyll/GitHub) is one option.
    - PHP.net itself has some tool we could look into borrowing.
    - Home-grown Drupal-based solution (this is what was used)

    There are some comments below where these possibilities were investigated and the home-grown solution was chosen.

Remaining tasks

a) Take care of items 2 and 3 in the Proposed Resolution. [Done!]
b) Decide whether we want the AsciiDoc approach presented here for the developing/api tutorials. [Community rejected the idea]
c) If we do go with the AsciiDoc approach [we decided not to!], decide whether it should go into the Drupal Core repo or a separate repo, and if it's a separate repo, decide on a framework for designating maintainers.
d) If we do want to use the AsciiDoc approach [we decided not to!], convert the existing documentation to AsciiDoc and deploy on api.drupal.org, along with the technology to display and edit the resulting docs. There are several UI issues that need to be resolved if we do this:
- Theming - the current Proof of Concept site has a toolbar that isn't like other stuff we see in Bluecheese, and the editor window is too wide.
- URL - we need to pick a good URL scheme for these docs. Maybe something like /reference/8.0/xxxx.html ?
- Navigation - make sure people know that these docs exist when on api.drupal.org.
- Make ability to edit more obvious - make a big Edit button somewhere (the current links are too subtle), make sure the procedure for what to do when you've downloaded the patch is more obvious, better text around "file an issue" links and maybe a big File an Issue link near the big Edit button?
- Make the Preview button more obvious in the toolbar.
- Make it more obvious that AsciiDoc is being used and a more prominent link to the AsciiDoc user guide
- Make the Return to Book link take you back to the page you were on
- Make sure the editor is accessible.

User interface changes

More relevant topics list on api.drupal.org [Done!], and new developer documentation book on api.drupal.org [Abandoned]. Editing developer doc book via patches, a Git repo, and AsciiDoc markup (with on-line editing capability) instead of Drupal nodes and HTML markup. [Abandoned.]

API changes

None

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jhodgdon’s picture

Topics we need:

a) Services / Dependency Injection Container. This is all over core obviously, and even after much searching I still do not know how you would go about actually trying to override one of the services that a core module defines.

b) Plugins. We have an Annotations topic, which is OK but it's not really the same as Plugins.

c) Routing. This would cover routing.yml files and controllers of the various types (regular pages, forms, entities, etc.)

d) Configuration and State.

It's possible we have some of this information in topics already, I'm not sure...

webchick’s picture

That looks like a pretty comprehensive list to me. Or at least a very good start. :)

jhodgdon’s picture

One more, assuming it gets done/into 8:

e) New migration system.

xjm’s picture

f) Entity Field API

eojthebrave’s picture

It might be nice to also have a list of "Classes that I will likely extend". I know for me at least it can be a little confusing to figure out which base class or interface I should be starting from when doing things like creating a form, plugin, basic page with DI, etc. So, perhaps some "Common base classes" link/list.

jhodgdon’s picture

Issue summary: View changes

make a summary

webchick’s picture

Ok, Jennifer and I sat down tonight, with input from chx, yesct, heyrocker, asherry, and came up with the following outline:

Essential background concepts

* Drupal's architecture (modules, info.yml files...)
* How to extend Drupal (hooks, plugins (+annotations), event listeners)
* Object-oriented conventions used in Drupal (PSR-*, namespaces, link to reference on OO, class naming conventions (base classes, etc.)?)
* Best practices (internationalization, security, automated tests, documentation)

Interfacing with the outside world

* Routing, page controllers, menu entries
* Forms
* Blocks
* Ajax
* REST / integrating third-party apps

Storing and retrieving data

* Configuration and State system (simple config; config entities)
* Entities (inc. config vs content entities, bundles...)
* Fields (widgets, formatters, etc.)
* Node system (node access, hook_node_save()...)
* Views
* Database abstraction layer (Entity field queries, PDO)

Utility APIs

* Internationalization
* Caching
* Permissions/roles/user access
* Theme system and render API

Advanced topics

* Services / Dependency Injection Container
* Typed Data
* Migration
* Automated tests (PHPUnit, Simpletest)

Further information

* All topics
* Examples modules
* API change notices
* Drupal 8 for Drupal 7 developers

webchick’s picture

@eojthebrave: I personally really like that idea, but Jennifer said that it might be better to cover these base classes by API/task instead, so that's what we attempted to do here. But we could always add another @ingroup later if we want.

eojthebrave’s picture

This list looks great. My initial reaction is that I would like to see this '* Theme system and render API' called out as it's own section. I realize that sure maybe it's just a handful of helper/utility tools but it's an extremely important concept and shouldn't get buried. I propose we add a Outputting data or similarly named section under the storing data one.

I'm totally fine with covering the base classes in the appropriate API section, my big thing is just wanting to make sure it's easy for people to figure out that there is a FormInterface and a FormBase and which one you should use. And also that there are things like ConfirmFormBase which I can and should make use of in certain scenarios.

jhodgdon’s picture

Just to clarify...

We are not planning at this time to get rid or combine of any of the topics currently listed on Topics. What we'd like to do instead is make some more introductory topics, which would give people new to Drupal and/or 8.x an introduction to the concepts, APIs, etc., and then link them to where they can find more information (which could be Wikipedia, drupal.org docs pages, the key classes/interfaces, or other topics). The list in #6 is intended to replace what is currently on the D8 landing page on api.drupal.org:
https://api.drupal.org/api/drupal/8

So... yes, there's a lot of stuff to cover under "theme and render". So, we'll make a quick landing page that gives a brief overview and links to the more detailed topics.

Sorry, it was pretty late last night when we finished collaborating on the list in #6 and the details of what we actually want to do with it didn't get posted until this morning. We just wanted to get it pasted in here before it got lost. :)

Berdir’s picture

I'm pretty sure I opened an issue to add a entity topic, can't find it right now.

I'm also trying to work on https://drupal.org/developing/api/entity and write about all the things we have in detail. It would be awesome if someone could help out there. Be that proofreading, fleshing out my notes and making it so that people can understand it, with code examples and so on. Contact me if you're interested :)

jhodgdon’s picture

Hey there! We reorganized and updated the Config API docs yesterday at the BADCamp sprint. Maybe I can help out today with Entity? :) If you're around today on IRC, ping me. I have a session to present in about 6 hours, but will probably be available most of the day today and tomorrow other than that and eating, and I'll be sprinting on Drupal 8 developer docs and/or perhaps the API module.

The outline on the landing page looks like a good start/direction to go anyway... might have some tweaks... but let's discuss via IRC.

jhodgdon’s picture

Angie/webchick and I were talking about this again today at BADCamp.

What we'd like to propose doing is to have the documentation that is currently under https://drupal.org/developing/api for Drupal 8, as well as a few more of the sections under http://drupal.org/developing -- comprising a set of esseniall docs for d8 core, contrib, and custom module developers -- be displayed on api.drupal.org. This way, api.drupal.org would become the one central place to go for API documentation (both reference docs and more expanded docs/tutorials).

This new addition to api.drupal.org, at least for now, is called the "Extended Drupal Developer Documentation" (ED3 in this comment).

So. Here's the plan we came up with:

a) We'd continue to have the Topics section (https://api.drupal.org/api/drupal/groups/8) being built from @defgroup doc blocks in Drupal Core repo files, so that they can list related classes, functions, etc. (everything that has the @ingroup reference to the group/topic). Generally, the text of each Topic should be fairly short, with a link to ED3 pages as necessary. Any reference documentation that a Topic wants to link to should probably be part of the ED3.

b) We'd revise the docs landing page for Drupal 8 (https://api.drupal.org/api/drupal/8) so that it looks like comment #6 above. The individual links there would be to new or existing Topics, or possibly to ED3 pages. If possible, it would be good if this landing page was either moved into the Drupal Core repo or could be edited as a node on api.drupal.org (we have an issue about making the landing page a node in the API queue now, and I don't think it would be at all hard to do).

c) Here is the list of what definitely needs to be included in ED3... and note that this proposal is for Drupal 8+ only. We don't want to go back and get D7 and earlier stuff at this point:
- Extended API docs/tutorials from https://drupal.org/developing/api, such as the Config API, Entity API, etc.
- Module development tutorial from the same section of d.o
- Updating Modules from 7 to 8 section from the same section of d.o

d) Infrastructure/implementation:
- The ED3 would be stored in a git repo as RST files (this is a Markdown-like syntax which Symfony is using for it's "Book" -- see http://symfony.com/doc/current/book/index.html ). This will allow core developers to make patches as the primary means for editing/writing this doc, and the .rst syntax is easy to read/write. Also it allows production of PDF and HTML output. Another possibility would be to use AsciiDoc (which I personally am familiar with -- wrote an entire book in it -- also easy to use and very similar to RST). I'll need to look into the tool chain for each and figure out which tool would be easier, with a slight bias towards RST because of Symfony. Links:
RST (ReStructuredText): http://docutils.sourceforge.net/docs/index.html
AsciiDoc: http://www.methods.co.nz/asciidoc/

- Angie thought we should use the Documentation project repo, but I'm actually thinking it would be better to make a new repo. The problem is the issue queue -- the Documentation issue queue has traditionally been for discussing d.o online docs issues, and mixing that up with patches etc. has been a problem in the past for the few files that are currently part of that git repo (Form API reference, d6 hook docs, etc.). So we should probably start a new repo.

- In any case, having it in a git repo allows us to have version control, branching, and all of those other things we want to have.

- We would need to have a Jenkins job that would periodically pull this repo and rebuild the HTML and PDF versions of our developer doc book, and put the output in a standard location.

- We would need to have a small module deployed on api.drupal.org that would display the book HTML. This module would read the HTML output of the Jenkins job, and display the pages in the main content area of api.drupal.org, with a link to the PDF output by Jenkins, and the book navigation as a sidebar block. This should hopefully be just a few lines of code. If the RST or AsciiDoc solution does not make this easy, I'd be surprised.

f) We would add a sidebar to api.drupal.org with a link to the new ED3 section, and a short description of how to edit it (like where the git repo is).

jhodgdon’s picture

I took a look at the two proposed (so far) tools for docs:
- RST (ReStructuredText): http://docutils.sourceforge.net/docs/index.html
- AsciiDoc: http://www.methods.co.nz/asciidoc/

The text you have to enter

- Internal (to a different section or page) links in RST are dependent on the human-readable names of the sections staying the same. This seems stupid. AsciiDoc uses a token system - you assign a particular section header a [[machine_name]] and then refer to it with <>, so if the human-readable section title changes, the token stays the same and the link does not break.
- Headers are kind of annoying in RST as well:

Chapter 1 Title
===============

Section 1.1 Title
-----------------

Subsection 1.1.1 Title
~~~~~~~~~~~~~~~~~~~~~~

Section 1.2 Title
-----------------

Chapter 2 Title
===============

vs. this in AsciiDoc:

= Document Title (level 0) =
== Section title (level 1) ==
=== Section title (level 2) ===
==== Section title (level 3) ====
===== Section title (level 4) =====

and you can also omit the trailing === signs. This is a lot more compact and generally easier to figure out IMO.

- The formatting for bold, italics, etc. is very very similar between the two, and similar to Markdown and other systems.
- Bullet lists are about the same between the two.
- Numbered lists seem to be a bit annoying in RST - you have to tell it what numbers to use. AsciiDoc lets you just use "." for it to choose the numbers, with the ability to start at a different number if necessary.
- Both RST (at least as it is used by Symfony -- not sure if it is standard?) and AsciiDoc (natively) have good code syntax highlighting.

Output

AsciiDoc will take its formatted text files and make all kinds of different output:
HTML/XHTML (chunked and unchunked), PDF, EPUB, DVI, PS, LaTeX, man page, HTML Help, text file, DocBook
Because it makes DocBook, which is a fairly standard format, there are also other options you can get to from there, and there are a lot of options for how to convert the DocBook to HTML, PDF, and other formats.

I do not see any documentation on how to convert RST to PDF, but Symfony is doing it, so it must be possible. It does convert to HTML and to ODT and to LaTex; probably from any of these you could generate a PDF.

We'd have to install a few command-line tools on a server Jenkins can access in order to use either of these, but it shouldn't be a problem.

Conclusions

From what I can see, I think I would go with AsciiDoc as it's slightly more usable in a few areas and I think fairly mature. O'Reilly uses it for authoring books (that is where I had my experience with it)... but I'll have to try a test. There may also be other systems that would work for us.

jhodgdon’s picture

Just spoke to drumm about this idea.

He generally thinks it's a good idea (we need to get the details); mentioned that we need to think about search.

i.e. it would be good to have a Search function on api.d.o that would search the API docs and these extended docs for a major version of Drupal (and/or contrib if we ever get that up there).

Which probably means getting both these new docs and the API docs reference into Solr. There's already an issue for the API docs pages.

drumm’s picture

It may make sense to import the docs via Feeds module or something that makes nodes, so we get normal Apachesolr indexing, and anything else we might decide we want later that comes with nodes. Or maybe a custom indexing hook can get it done cleanly.

I don't think search needs to be solved for the first round of deployment, but there should be a straightforward plan ready.

Gábor Hojtsy’s picture

I'm not entirely sure that moving from the handbooks to a git repo is very accessible. Right now our main problem is lack of overview docs on the API docs section. I don't think the problem evolves around where it is. The problem is that it is not at all on drupal.org or at least burried in some otherwise pretty excellent change notices.

I think the best way to ramp up the API docs is to keep them accessible as much as possible and therefore NOT moving it out to a git repository would help make it evolve quicker. Once we have a better built-out API docs section, it may be a good idea to move it to api.drupal.org THEN.

jhodgdon’s picture

I agree that for this go-around, ramping up the API docs on d.o is not a bad idea... but they do tend to get ramped up into a mess, because people can add child pages at will, and at some point people start making incoming links and bookmarks... For Drupal 9, we should be able to ramp up by branching the git repo, but of course we can still use d.o as a temporary place for starting new API docs if we want.

Also, the intention would be to give quite a few people commit access to the repo. That may not have been stated above, but I don't want it to be limited to the Group of 7 (core committers).

Anyway, this isn't happening for a while so please do keep editing on d.o for now! Meanwhile, I'll probably start working on the toolchain and see if this idea is even feasible. If I can just get my clients to go away for a while so I have time...

webchick’s picture

This issue title is no longer accurate for what's being discussed here, but let's keep going and see where we end up. :)

On a call earlier today, Gábor had expressed concern about where to put information to avoid duplication of effort, etc. Here's basically how I see this shaking out, but I welcome others' thoughts as well:

 API topics [in core], Extended API docs [in the handbook], and blogs, tutorials, etc. that all reference the extended API docs.

In an ideal world, those extended API docs would be in version control so that we can pull off fancy crap like Symfony has where you can easily switch between versions and download a PDF of everything:

Screen Shot 2013-10-30 at 9.59.12 AM.png

But in the meantime, I agree that getting the content at all is the first step, and the handbook has lowest barrier to entry. We can figure out what we ultimately want to do later on.

jhodgdon’s picture

#18 - Fine, but amend "link to handbook" to the new intent that we link to a docs page in the new Dev Docs home on api.drupal.org. :) Also... graphics are great but not everyone can see them. :(

webchick’s picture

True, and they provide other challenges such as translation, etc. However, for the ~99% of contributors who can see them, they are often a much faster vehicle to understanding, which is the goal here. The images can always be explained / expanded on in text and/or alt attributes for the 1%.

Any idea on the timeline of Dev Docs on api.d.o? I know it's very tricky to nail down, but if it feels like it's, say, in the next month (ish) it probably makes sense to hold off on any large-scale cat-herding efforts until that's in place. But if it's, say, in the next 3-5 months (ish), then we should probably kick-start the cat-herding efforts around API docs on Drupal.org for now, and then pull from there to api.d.o once that stuff is ready (and set up redirects). To me, it feels like this is more of a 3-5 month thing, between the infrastructure and tooling requirements, the inevitable discussion about how much to lock down edit access and to whom, etc. In other words, something we can likely get in place before Drupal 8.0, but not something we can get in place in time to save people who are feeling scared off from Drupal 8 (like right now).

jhodgdon’s picture

Would you be able to squelch all of my current clients and buy me some time? :) Barring that... I'm really swamped with work at the moment, and I have several other demands on my contrib time, like Search module and API Docs commits and updates to the API module itself... So.

Here's what we need to do (and this should go into the issue summary probably?):

a) We can get the skeleton of the D8 landing page built -- meaning the list in comment #6. This means: a @mainpage doc block which we'll put into the Core repo and take out of Docs). And its landing topics (just the titles with no body in @defgroups) - that is a 1 hour project and a Core patch. The @mainpage and @defgroup items can probably go into a .api.php file in core/modules/system (individual items may have more appropriate places; we talked on #2115011: We need a place to put documentation for core subsystem hooks, other than system.api.php about making a generic "core.api.php" file for "stuff that doesn't need to be in a separate api.php file", so that would be a good place?

b) We can then get people going on fleshing out each of the landing topics, which will be core patches.

c) Everyone can continue working on the API tutorial docs on d.o for now.

d) We need to figure out the AsciiDoc tool chain (see #12-13, and AsciiDoc could be some other markdown-like tool chain instead, but I'll call it AsciiDoc here for now). What we need:
- A preliminary set of documents taken from the API tutorial docs on d.o -- possibly starting with just the Config API docs since they are in reasonably good shape -- and put into AsciiDoc into a sandbox repo tentatively called core_dev_docs. This is a 1-hour project.
- Someone needs to try out the AsciiDoc tools and make a README describing what you need to install, and scripts that will (a) take the AsciiDoc and make it into a PDF and (b) take the AsciiDoc and make HTML that we can use for display on a Drupal site (hopefully separating out the table of contents and the body so we can make a nav block). I expect this will take several hours of playing around. This should probably go into a separate sandbox repo tenatively called asciidoc_tools or something like that.
- Someone needs to make a Drupal module that will read the output of the AsciiDoc tools scripts and display them in a Drupal site. I expect this will take a few hours of playing around as well and it should go into the same sandbox as the tools stuff.

e) Once (d) is figured out, we need to convert the d.o files in (c) into AsciiDoc and commit them to the repo. And make both sandboxes into real projects. And give more people commit privs to the docs repo.

f) Once that is done, we need to put the new module on api.drupal.org and set up Jenkins jobs to update the docs from git, and run the scripts. That is probably an hour of work for someone like @drumm, once the scripts are well-defined.

So... I think the question in #20 is about when I would expect it all to be done, and the answer is... I don't have a lot of time now but a month doesn't sound totally out of the question if I did have more time... I am expecting that the big time-sucking project I've been working on will wind down (finally) in a few more weeks. So probably 2-3 months is more realistic. At the same time, I don't think we need to hold up people editing/creating the docs that we need while the tools get created, no matter what the time frame is.

jhodgdon’s picture

Issue summary: View changes

add item to edit the landing page

jhodgdon’s picture

Assigned: Unassigned » jhodgdon
Issue summary: View changes

Added the tasks listed in comment #20 to the issue summary.

And by the way I have some free time this week so I'm going to start on the AsciiDoc scripts/infra part now.

jhodgdon’s picture

sorry, that was comment #21's tasks that I added to the summary.

jhodgdon’s picture

I have a proof of concept of this working on my local site!

I put the beginning of the Config API section from https://drupal.org/node/1667894 into a sandbox project:
https://drupal.org/sandbox/jhodgdon/2127335

And a working (!!) version of some scripts and a module to create the AsciiDoc output in another sandbox project:
https://drupal.org/sandbox/jhodgdon/2127337
This module and scripts will take AsciiDoc files from the first sandbox project, make HTML and PDF output, and display the HTML output and a link to the PDF output in a Drupal site. The table of contents is put into a block, similar to what is in the Book module. The rest of the output goes into the page body.

I think I'll request an api.d.o clone dev site to work on so I can display this for everyone to see.

Really it's close to being usable. The only things it probably needs are:
- The display module needs some CSS for the navigation block.
- It would probably be better if the nav would show/hide different levels of navigation instead of the whole thing.
- The display module has some hard-wired paths in it right now and needs a settings page where you could set up a book to display.

Anyway... it took me most of today to do it but it wasn't too hard.

jhodgdon’s picture

Actually I think the navigation could be made expand/contract-able with suitable JQuery, which wouldn't be too hard. It's just a nested DL list. hm....

jhodgdon’s picture

I'm adding issues to the sandboxes to cover what needs to be done there. Once I get the test site up, anyone else can do that as well.

webchick’s picture

Awesome!! :D Can't wait to check this out!

jhodgdon’s picture

FileSize
29.31 KB

I'm building a test site; need to attach a file... more coming shortly!

jhodgdon’s picture

OK, I have a test site up:
http://docs-api.redesign.devdrupal.org/asciidoc/_introduction.html

It has some problems, like weird characters appearing in the text (especially in the headers), which I'm not seeing on my local site. I'll have to sort that out -- probably can work it out without too much trouble.

And we can probably make the output a bit nicer with more XSL style sheets -- for instance, it should be able to colorize the PHP code if we can find the style sheet to do it.

But meanwhile, it is at least a proof of concept!

More info:

The PDF and HTML are generated from the files in this (currently sandbox) project (you might want to take a look at the files in the repo to see what the formatting is -- I just made a few pages by copying the existing drupal.org docs pages and putting them into this format, which isn't difficult):
https://drupal.org/sandbox/jhodgdon/2127335

The files are displayed as HTML and PDF using this other sandbox project:
https://drupal.org/sandbox/jhodgdon/2127337

Any thoughts on the proof of concept, besides the obvious special character issue? Should I keep working on this -- do you think it's worth pursuing?

jhodgdon’s picture

I've updated the module. It looks a lot better (no weird characters) and it is configurable.

The URL for viewing the book is now:
http://docs-api.redesign.devdrupal.org/d8book/index.html

jhodgdon’s picture

FileSize
36.42 KB

I've made some updates to the module - syntax highlighting! Will get it up on the new site shortly, but I need to upload these files first....

And, there! You can see the syntax highlighting on
http://docs-api.redesign.devdrupal.org/d8book/state_api.html
and other pages.

Enjoy!

We need to figure out soon if this is something we want to do for sure...

jhodgdon’s picture

jhodgdon’s picture

Issue summary: View changes

Time to update the issue summary.

jhodgdon’s picture

Issue summary: View changes

update to discussion of node vs. asciidoc idea

jhodgdon’s picture

Title: [meta] Expand the api.d.o "Topics" list to cover new concepts in Drupal 8 » [meta] Make Drupal 8 developer docs more discoverable
mikey_p’s picture

I see two major wins to the git repo w/ markdown approach:

1) This is pretty obvious, but having all the developer docs on one site (and searchable) would be a huge win. For a long time things like hook_menu were only thoroughly documented on d.o itself in the handbook (I think hook_schema may still be this way). Being able to quickly and easily link to those docs on the same site would be a huge benefit.

2) Versioning: For docs on api.d.o versioning is obvious, the tabs at the top always let you know *exactly* what version of the API you are looking at. This gets more confusing when important API docs are placed on d.o, for example hook_schema types.

This confusion is due to not wanting to copy/fork doc pages for multiple Drupal core versions. For example, if I'm looking for the D8 Schema API types, should I look under D8 APIs, or under Database/Schema? It seems that using a branched Git repo would help immensely with this problem since there would be a *complete* set of docs for each branch. I can't speak to the difficulty of maintaining all these docs branches, but I have to think it'd be easier to manage porting patches with the issue queue compared to trying to keep updates to nodes in sync.

Mile23’s picture

+ a million on better structured documentation with a review process.

+ a couple hundred thousand on mimicking Symfony's Book and Cookbook.

+ a lot for making this new doc stuff into a repo instead of nodes. People can submit patches instead of editing nodes; this allows for a baked-in review process from documentation maintainers.

Speaking of which, I'd like to point out that Examples is a maintained, structured, discoverable, api.d.o-browseable documentation system that already exists and that theoretically already has a process for inclusion in core. What it lacks is strong maintainership (ahem) and enthusiasm from anyone else, as evidenced by the existence of this issue.

Basically: Update Examples and your content is mostly written, along with useable, testable code.

This leads me to the obvious question: Whither Examples?

jhodgdon’s picture

The Examples project is good for what it is, and yes we should get those examples fixed up and into the Core repo. They are also already on api.d.o, at least for 7.

However, I do not think that Examples fully takes the place of the documentation that is currently on d.o/developing/api. It's also kind of a pain to write narratives and tutorials as code comments. Markdown, or even Drupal nodes, are a lot easier to deal with, and with Markdown we end up with a nice PDF document as an alternative.

Mile23’s picture

However, I do not think that Examples fully takes the place of the documentation that is currently on d.o/developing/api.

Who said it did, or should? In fact, IIRC, Examples exists specifically to fill in the knowledge gap which this current issue attempts to provide: Narrative descriptions of core dev info, browseable in api.drupal.org, with the bonus of being useable in a local site to illustrate solutions.

I love markdown. I'd rather use it for everything, because I'm a modern web developer. :-) Examples uses docblocks because that's How It Has Always Been, because that's how the API module works, and that's how Drupal coding standards work, and Examples is code.

In an ideal world, each example would have a .md file, and this file would be parsed by this new project for inclusion. This way you get the content and the installable code to go along with it.

Is that possible?

jhodgdon’s picture

RE #39 - That sounds like a totally different proposal, really. What we're proposing here is to take the narrative tutorials on developing/api and put them on api.drupal.org. The Examples serve a related but separate purpose than those narratives. Can we discuss that elsewhere?

webchick’s picture

Yeah, not sure I see Examples : /developing/api/FOO as a 1:1 correlation. Ideally, /developing/api/FOO could link to /examples/FOO where one existed, but the handbook pages are much more for a global overview of the concept, a comprehensive reference to all of its various features, whereas examples are ideally focused specifically to how to use the API, not the what and why.

Mile23’s picture

1:1 correlation not required. :-) It's an editorial decision. I think there's a misconception that I'm trying to say replace this project with examples, which I'm not.

The way I learned Drupal API stuff was to convert inline comments to docblocks in the Examples project, and I did that because learning doxygen was preferable to the disappointment of trying to read the handbook stuff that existed then for a newbie. That's why I said, right up there, + a million (and more) for this project.

developing/api looks like the current handbook. The goal here seems to be to change it to be a more structured, canonical handbook that is managed differently. Thus, there's an opportunity to include the documentation that's in examples, if it's something people want to do.

The D8 examples don't exist yet for the most part and can be written to whatever purpose we determine. If there is a structured handbook project, examples can support that editorially, rather than just being another project with links. It might be that someone submitting a patch to examples could more effectively submit a handbook patch or vice-versa.

And that last paragraph is the question I'm trying to hammer out.

jhodgdon’s picture

That still sounds like a separate issue, an enhancement for later. Please file a separate issue.

mradcliffe’s picture

I'm used to markdown from github now so that's a plus. I think I would be slowed down by the Git process. It is easy for me to click edit when I feel confident and bold enough to make a change to the documentation pages.

On the other hand the git process might make it easier to import Readme.markdown or Readme.md files from contrib projects (eventually?).

I'm going to ask around to people who don't normally contribute and see what they're opinion is. If a git process would make a greater number of people feel more comfortable editing and contributing, then +1 forever.

jhodgdon’s picture

Title: [meta] Make Drupal 8 developer docs more discoverable » [meta][policy, no patch] Make Drupal 8 developer docs more discoverable
Issue summary: View changes
Status: Active » Reviewed & tested by the community
Issue tags: +Coding standards

I know that there are some people who watch coding standards and RTBC issues. So even though I do not think this has been sufficiently reviewed to consider it adopted, I am currently marking it RTBC and tagging it so that people who watch these issues will see it.

Crell’s picture

There are tools that allow edit-through-web for Git. Prose.io (built by former Drupal shop Dev Seed for use on Jekyll/GitHub) is one option. PHP.net itself has some tool we could look into borrowing. I think Symfony just goes through normal GitHub PRs, which means it supports the quick edit-through-gui-and-spawn-PR function.

Don't assume that using Git means "only command line gurus can do things". There are plenty of existing options to simplify that.

jhodgdon’s picture

Issue summary: View changes

Add idea from Crell about live editing to issue summary

pwolanin’s picture

So, am I the only one who remembers that a lot of the developer docs (like the FAPI guide) used to be in CVS to get deployed to api.d.o?

That certainly had benefits in terms of versioning and review, but indeed cut down the number of possible contributors. I think combining with live editing and/or making initial versions as handbook pages and then migrating them into git could alleviate a lot of the problem.

jhodgdon’s picture

You are not the only one who remembers, and in fact, the FAPI guide is still there even in 8.x.

However... I think the main problem with that situation was that the hook docs were in that repository, and they are API reference docs and needed to be closer to the code, which is why they were moved into the core repo in D7.

We're not talking here about API reference, we're talking about narratives... it's not quite the same thing. But maybe we should consider putting them into the Core repo, as we did with the hook docs, because we all make patches for Core all the time. The only downside really is that it adds to the core committer backlog, and it also might contribute to a perception that we'd have to update those docs with every patch. I think that would be very undesirable as a huge hurdle, much as it would be great to have updated docs -- it's just not realistic.

webchick’s picture

Status: Reviewed & tested by the community » Needs review

Sorry, but I really don't think this is RTBC yet. It probably makes sense time timebox the discussion so it doesn't drag on forever, but it seems like there are still many open discussion threads happening atm:

- Where best to keep these files? In a separate repo? In the Examples module repo? In Core?
- How open and unfettered do we want the access? Anyone with a pulse? Anyone who asks? Only a special select elite group of editors?
- How will we address editing of the files? Stuff it, you're a developer, use vi/emacs? Couple deploying this with also deploying Prose or some other GUI-ish tool? Use a Drupal-based solution?
- When should we do this, and what do we move in? ASAP and everything under developing/api? Move over only a couple of test pieces of content from developing/api? Other?
- Probably other things I forget because I can't see the blankety blanking comment stream when I blankety blanking post a blankety blanking comment. :(

jhodgdon’s picture

I also did not think it was RTBC. See #45. I only set it to RTBC because I know there are some people who will not look at issues until they are RTBC and no one was commenting on this one. I just don't want to put energy into a proposal here, adopt it, and then have people say they weren't consulted.

jhodgdon’s picture

Regarding the questions in #50:

I am about 1000% against putting the docs in the Examples project. I think they should be either in a separate project or in Core itself. We've discussed several times here that they are not the same as Examples and I don't really want them merged to muddy the waters further.

Regarding editing, I have no strong opinion. I personally hate editing anything on the web if I don't have to, and I don't see using a text editor and git to be a large burden for your average core developer. But if others really like editing on the web, then we should look into a tool. I think that could be a phase 2.

Regarding access, I would be happy to give commit access to anyone who has been participating and asks. Otherwise, we can use patch/review by anyone, commit by a few, as we do in all of our other projects. I don't see it as a big deal either way. I think some review before commit is beneficial once the docs become more stable, but it slows things down while docs are being developed... so maybe we could open up access to anyone who wants it (within reason) and have a social policy that says "for now, you can commit at will to 8.x" and later on change that to "8.x is stable, be more restrained, but commit at will to 9.x"? I mean... really, we have revision control, so sure, commit at will ... maybe we can do the deployments to api.drupal.org more carefully than the commits?

Regarding timing, I would say ASAP and move it all. Having some in one place and some in another will just lead to more confusion. But if it's seriously going to stand in people's way who are writing the docs now, then let's delay a while. We need to get the docs written too. So maybe a phased approach would be best -- like start with some of the docs that people think are relatively stable? I don't know, what do others think?

Crell’s picture

As far as access, I think part of the intent here is to ensure that the docs are more cohesive and consistent rather than the disjoint, uncurated, and often wrong information we get in the handbooks since they were thrown wide open, no? That to me suggests we do want an "editor gatekeeper" on edits (probably a group of people). How large that group should be I defer to jhodgdon, but I'd say "larger than core, smaller than the old docs team standard of anyone with a pulse who didn't run away fast enough".

jhodgdon’s picture

@Crell/#53: +1. Yes. We do want them to be somewhat curated. So if we open it up to "people who agree to act as editors", that will probably self-select a group who is interested in ... editing. :)

heather’s picture

Re: "I don't see using a text editor and git to be a large burden for your average core developer. But if others really like editing on the web, then we should look into a tool. "

I guess maybe you knew this already, so I might misunderstand the problem... Github generally allows people to edit files in the web interface via a browser.

If they don't have commit access to the file it says "Clicking this button will automatically fork this project so you can edit the file". Then, submitting a change to this file will write it to a new branch in your fork so you can send a pull request.

jhodgdon’s picture

Yes, the Github solution has already been mentioned. Thanks!

mitchell’s picture

I don't see using a text editor and git to be a large burden for your average core developer. But if others really like editing on the web, then we should look into a tool.

Similarly to Github, GitLab has a web editor (and forking, pull requests, and a web UI for merging). That tool may satisfy the needs of documentation contributors. Also for current and future reference, there is a d.o improvement discussion to Use gitlab for all projects.

mradcliffe’s picture

I'm going to reiterate that not having web editing is a regression. It is not a large burden for average core developers (#52), but it is a burden for other contributors.

I asked several people in the following categories their opinion about the ability to web edit:

  • Current non-contributor, but has wanted to contribute to documentation: would feel less encouraged to contribute. Related this to Wikipedia.
  • Current non-contributor, no willingness to contribute at the moment: would feel less encouraged to contribute.
  • Current "average core developer": feels comfortable with contributing only with git patches via the issue queue. Thinks those without the documentation repository setup may feel less encouraged to contribute.
  • Current contributor and developer: would feel less encouraged to contribute. Related past experience of only contributing to documentation because of a web edit interface
  • Myself: I am comfortable Git, AND I would feel less encouraged to contribute if I had to go through the issue queue. I probably wouldn't contribute documentation at all as my process is "Oh, I found something I want to add/contribute. Goes to docs page. Welp, this is a lot of effort. I guess I'll go do something else like code."

I will continue to ask those I know. I can't say that this does not contain bias, but I'll stand by my hunch and say that the plan as-is is UX and contributor regression.

Aside: What does the word average or normal mean anyway other than make people feel not normal?

Edit: I want to mention that the plan is to NOT have a web edit interface initially and that is a regression. I think that not having a web edit interface via gitlab / github / drupal.org is a regression.

jhodgdon’s picture

Issue summary: View changes

I've added some notes about specific technology for on-line editing to the Proposed Resolution, since that seems to be the main thing that people commenting here would like to see and there have been several suggestions. Also added some recent points made to the summary.

So... there has been a lot of discussion about on-line editing, and mradcliffe has pointed out that this may be a large burden to some contributors... However, I'd like to point out that we're only talking about the documentation at developing/api, not all of the on-line docs. I think most of these core API docs are mostly written by core contributors who know the APIs (and who would presumably be comfortable working in an off-line text editor)... We'd lose the ability for random passers-by to fix a typo, but we'd also have some control over proliferation and "spaghetti docs"... I guess we should go look at the list of who has edited docs in developing/api, and see if they are mostly core subsystem maintainers or other contributors, before we decide on this.

But I'd like to point out that if we demand that this be part of the initial roll-out, then we will not do this any time soon, because although the technology certainly exists to do an automatic branch/pull request, we (a) don't do pull requests at this time in Drupal projects and (b) don't have this type of technology deployed. So, if we need this functionality in order for this proposal to be acceptable, we might as well postpone it until we have the technology in place.

mradcliffe has just added some notes... What do others think: Should we move forward now, or wait until we have on-line editing capability?

jhodgdon’s picture

mradcliffe: I'm sorry if my "average Drupal core contributor" statement made you or anyone else feel "non-average". That was not my intention and it was insensitive on my part.

Also, I don't think we've decided what the plan is, on whether or not we need web editing now or later. I've edited the issue summary to hopefully make that clearer. Having the docs in Git and plain/markdown text is a clear UX win for me, personally, and having them more curated should be a good DX win for the consumers of the documentation.

But I see your point that it might be a hurdle to potential docs contributors. What I don't know is whether there are many people who actually are currently contributing *to the https://drupal.org/developing/api documentation specifically* for whom this would be a hurdle or a step backwards. If it keeps people who are now or have recently been contributing to this section of the docs away, then that's bad. If the need/desire to edit on-line mostly affects other sections of the docs, and not the "Official How To Use The Drupal Core APIs" documentation, then it's not a problem for this particular initiative.

Mile23’s picture

I think the proper order of importance should be:

1) Editorial control by someone (or some people) in an editorial role, recognized as such by everyone. Call it 'curation' if that helps. :-)

2) Tools which enable #1.

I'd add #3:

3) Funding. Because it's a lot of work, which is why no one has done it yet.

eojthebrave’s picture

It's already been said a couple of times, but I do want to echo the "significantly raises the barrier to entry" sentiment, along with sharing that for my own part I know that without a simple, almost transparent, mechanism for editing docs I'm far less likely to contribute. I currently edit stuff quite regularly while doing research for training materials I'm working on, and it's generally more than a typo but less than writing a whole new page. Generally helping to try and make things clearer and better explained.

So, I want to emphasize the importance of having a GUI tool for editing these docs online. And personally I dont' think it's something that can or should wait for phase 2.

On the other hand, I love the idea of having a complete set of documentation for each version of Drupal and think that would help a lot with my concerns of discoverability expressed earlier in this thread.

Is there any concern about trying to do this right now, with D8 right around the corner, and fragmenting focus and effort on what should be priority #1, getting the documentation actually written? Perhaps it's not a problem at all, but it's something that comes up when I started thinking about what this would look like.

Finally, markdown++

jhodgdon’s picture

Issue summary: View changes

Thanks for all the thoughtful feedback!

Your consistent message that without web editing capability, or at least some easy/viable/friendly way to click "edit" while reading a doc and quickly make a patch, we will exclude people from contributing, has been received. I am updating the issue summary accordingly, and will check and see if there is a viable solution that we can deploy now/soon that will enable this.

If you have any suggestions that have not already been made for how this could be accomplished, please comment here. (The suggestions already made are: Symfony (uses Github), GitHub, GitLab, Prose.io, and whatever PHP.net is doing.)

jhodgdon’s picture

It turns out that most of the suggestions here involve using Gitlab or Github. Please, do not discuss here whether all Drupal source should be on those platforms... But at present, unless we mirror the API reference tutorials on Github or Gitlab or some other site that offers web-based editing, I don't think solutions involving GitHub or GitLab are viable for this initiative. We don't have pull requests, it wouldn't mesh with other issues and people using the Drupal Git repo directly, etc.

The only other solution that was mentioned here was php.net: if you're viewing a page you can click Edit and it takes you to http://edit.php.net, which is a DocBook editor running on top of SVN. If you make an edit, it submits a patch via an email system (or at least, an email-like system), and if you are logged in as someone who has permission, you can apply patches. The source for this is under the LGPL and is available at:
http://git.php.net/ (see web/doceditor.git).
It might be possible to patch this to use Git instead of SVN as a back end, but it definitely needs read/write access to a repository in order to work. And as we're using AsciiDoc and not DocBook, a different editor might be better.

So... I had another idea. Is it possible we could do something simple and somewhat home grown? Such as:

a) User clicks edit. The AsciiDoc source file from a local clone of the Git repo is downloaded into an editor on a web page.

b) User edits in a nice AsciiDoc web editor. I found this video post on how to do that using the JQuery MarkItUp plugin, and the AsciiDoctor.JS library, which looks cool and lets you live preview the result as you type:
http://www.lordofthejars.com/2013/07/asciidoc-editor-with-markitup.html
This looks *really* cool.

c) When done edting, the user clicks "Make patch" and a patch file is generated. This would probably involve doing a git clone into a temporary directory, replacing one file with the contents of the editor box, doing a git diff, and downloading this for the user.

d) Then maybe there would also be a "file an issue" link where they could go and attach their patch to an issue.

This is not *quite* as clean as just clicking "Edit" and saving the edits on a node, but I think it would not be too much of a barrier. Thoughts?

One other missing piece is that we would need to have a way to figure out which AsciiDoc source file(s) a given page we are looking at is coming from. This isn't necessarily one to one, because you can break your docs into files however you want. But I was thinking that I could have at the top of each AsciiDoc file something like
[source-file:whatever.txt]
and then that could be processed into a hidden div with the file name or something like that. Then when displaying each AsciiDoc page, we could figure out which file(s) made up the source for the page. Or there might be a way to put that in automatically during processing. I'll have to see what's feasible.

I'll see if I can get something working.

joachim’s picture

I'm very much in two minds about this.

My thinking for a long time has been that the current documentation node system should be more like wikipedia, not less. That, to me, means easier syntax, and an easier way to create hierarchies of content (in wikis you create hierarchies and categories much more easily than you can with book module).

Going to markdown in a git repo is going in the opposite direction. I can see the reasons for it, but I am concerned about the barriers that raises to contribution.

We do gain that documentation will perhaps be more carefully thought-out, and planned as a whole, which is something I find is quite difficult to do with book nodes.

Markdown is definitely nicer to write in than HTML. Though if we have to keep everything wrapped to 80 characters, then it's going to definitely be a worse editing experience. I *hate* wrapping documentation text to lines of 80 chars. It completely destroys any flow you might have while writing and for me it's the biggest problem about writing docs.

fgm’s picture

There are really at least two topics in this issue: one about the new topics, which is excellent, and the other about tooling, with which I think there is a fundamental flaw.

Like most here, I don't mind having doc in a repo like we used to, as @pwolanin mentioned, but the problem is with the interpretation of this choice, the subtext if you will:

  • we have been developing, for many years, what is arguably the best CMF/CMS a this point
  • originally, we used versioned text as our documentation source, then we switched to using our CMS, and there was much rejoicing
  • now, we feel limitations in the use of said CMS/CMF for managing documentation content:
    What is that content ?
    Text (markdown, RST...), images, possibly videos although this has not been mentioned
    What do we need with it ?
    ease of editing, non-HTML formats, maintenability, versioning, translations, stable linking
    Are these needs supposed to be covered by a CMS ?
    You bet: Drupal, for example, has inline editing, fields, input filters, relation modules, node versioning, node and field translations, link management, in
  • So what solution do we plan on using ? Drupal ? Oh no, let's not use a CMS. Let's go back to text files.

Do you realize what this move would be telling the world about the actual value of Drupal as a CMS ? We are not developing a MVC framework (yes, Sensio's narratives are just excellent, but that is not a consequence of their using git), but a CMS: and here we would renounce using it because, as the more knowlegeable users of the CMS, we cannot figure how to use that CMS to manage content ? What is the unavoidable lesson to draw for users considering switching to Drupal, or from Drupal ?

Technically, I do not feel like any of the - very real - problems outlined here, like the inconsistency caused by too many contributors (which we long strove to attract) cannot be solved within Drupal, because they are pretty much typical of any Drupal project out there: providing a rich and controlled user contribution platform. What the heck, even if for some reason, we decide we actually want text in git, I'm pretty sure a number of core devs could write an entity using Git as an entity/field storage and keep using Drupal.

Like someone repeatedly tells around me: "people problems, people solutions". If the main problem is that users do not contribute optimally, the solution is probably not to get shiny new technologies, but think more about people needs.

mikey_p’s picture

I would agree with most of the points that @fgm has with a few exceptions. In regards to videos, images, et al, I don't think that's really going to apply as much here. We're not talking about the documentation about *using* Drupal, but just the developer docs. I'm not saying it's not possible, but I don't think they're would be many videos about how to code for the APIs. Videos might help explaining lines of code, but not as much as something with more graphical elements, like videos of the admin theme.

On the "what do we need to do with it?" point, I would also add another item: branch the documentation for each major version of the Drupal API. This is where Drupal as a CMS falls short. Making a copy of a documentation node for each major version of Drupal, keeping the changes in hierarchy organized between the different handbooks for each version, and applying changes to multiple nodes is going to be difficult. Now imagine the difficultly when anyone on drupal.org can edit any of those nodes. It would be impossible to ensure that changes are properly synced across versions of the API where appropriate if anyone could edit an individual version.

Fortunately we have a system that is good as this sort of branching and we already have a workflow for applying change to more than one branch, when appropriate. I would also argue that the type of deeper developer documentation that is being discussed here might sometimes need to be written by the very developers that are contributing new features to Drupal core, so they are already familiar and comfortable with Git. Even when that's not the case the chance of someone wanting to contribute to *developer* docs knowing Git, is higher than someone wanting to contributing to documentation about *using Drupal* would be.

webchick’s picture

Right, I see this less as saying Drupal is a bad tool for writing documentation, and more that "there is already a well-used tool for managing content across multiple branches that might diverge in subtle and not-so-subtle ways, and it's called a version control system." A CMS/CMF != a Version Control System, nor should it aim to be one. There is nothing technically stopping us from abandoning Git and using Drupal nodes for all code changes: the hooks are all there. But we don't, because that would be completely ridiculous. :) So while we can try (and some have) to write tools on top of Drupal to try and make it have some elements of a version control system (e.g. the Diff module), at the end of the day it's going to fall short to something built to do exactly that, which also happens to be the industry standard way to go about solving this very problem.

The sweet spot IMO would be some kind of Drupal wrapper around a Git repo that exposes node-like forms but on save does a commit (or, lacking permissions to do a direct commit, creates a patch). This would have wider applicability outside of just documentation, like for example a Github-like in-browser editor for minor docs/markup patches and the like.

jhodgdon’s picture

So... I'm confused by the last few comments here. If we are able to offer the option of on-line editing, or at least on-line editing leading to a patch, what exactly are we losing in terms of barriers, access, usability, etc.? We are not launching this without some type of on-line editing -- that change has already been made to the Proposed Resolution in the issue summary. We will gain the ability to use a markdown-type syntax, as well as branching, and curation, and I am not seeing what we will lose except perhaps the immediacy of having your edits appear right after you make them (which is not necessarily desirable anyway -- I think we want the "Editor" team to review changes anyway?).

And yes, we are ****only talking about the Drupal Core API developer documentation***. I do not think videos and images are probably necessary since we're talking about developer docs, but there is no reason that we cannot insert an image or a video into an AsciiDoc file anyway.

Crell’s picture

I see no reason why the front-end needs to be Drupal-based. Basically we're talking about a non-GitHub version of Prose.io, and then a static generator akin to Jekyll or Sculpin or whatnot. Right? That's what it sounds like...

jhodgdon’s picture

RE #70 - we don't have the ability at the present time to accept pull requests easily within our usual Git workflow, so I don't see how we can adopt all this other technology. Anyway, I'm about 90% ready (hopefully later today) with a Drupal-based solution, which I'll put up on the test site.

jhodgdon’s picture

Yippeeee!!!

Just in time for (U.S.) Thanksgiving -- I have managed to get an on-line editor working on the test site!

You can see it in action by going to
http://docs-api.redesign.devdrupal.org/d8book

Features:
- On any page, you can download the PDF of the entire API doc book (right now it is just a few topics).
- A sidebar block called "Source information" tells you where to find the project that generated the text.
- If you are on a page with content (and in the PDF), you will see the specific file name each section came from displayed in italics just under the section header.
- Also if you are on a page with content, you will see a link that says "File an issue". And another that says "Edit source file ...".

So if you click the Edit source file link, it gives you a simple editor with some buttons to help you learn how to make AsciiDoc. There is also a link on the sidebar to the AsciiDoc user manual if you want to get more complicated. And a preview button (that is the checkmark button) so you can see the output as you type.

When you're done, you can download a Git patch file, which you can then attach to an issue (there's a file an issue link).

So... It may not be *quite* as convenient as editing a Drupal node page, but I think it is workable. Thoughts?

I'm of course going to be away for a few days seeing family for the holidays... but will respond next Monday. I'm excited!!!

jhodgdon’s picture

And yes, I know the editor box is too wide for that page. But. It's a proof of concept. We can work out the details later, if we decide to go forward with it.

I'm also not sure the editor is accessible. That needs to be checked on.

jhodgdon’s picture

Issue summary: View changes
webchick’s picture

Wow, this is really cool! :D :D

A few things:

- (major) I have no idea how to get to this book from http://docs-api.redesign.devdrupal.org/api/drupal/8. Assuming this is just something that hasn't been gotten to yet, but just noting it. :)
- (minor) We should think a little bit about the path, and look at what other projects do so that we get decent SEO and the URLs are hackable between D9, D10, etc. Off the cuff, it feels like it might be better as "/reference/8.0/xxxx.html" or something rather than "d8book."
- (major) I looked for an "Edit" tab in the usual place I'd find an "Edit" tab on the page and didn't find one, so I completely missed the "Edit source file" link the first 2-3 pages I went to, almost giving up thinking I didn't have permissions to see what you can see. I wonder if there's a way to make this stand out more, and/or add it as a MENU_LOCAL_TASK here (along with "File an issue"?) so it's more visible.
- (major) You already pointed out the editor looks a bit silly with the right side cut off. :)
- (minor) I also found the chrome on the editor rather jarring, personally; is there a theme that is more "flat" and less "shiny" and would go better with the Bluecheese design?
- (major) The "Download patch file" button does not tell me what to do with that patch once I have it. Maybe add some text after it like ...and <a href="drupal.org/node/add/project-issue/XXX/fill/out/a/title/automatically">post it to the Core API Documentation issue queue</a>.
- (normal) Would love a way to preview my edits, so I don't submit bunk patches due to screwed-up Markdown. (I am very good at screwing up Markdown. :P~)
- (normal) The AsciiDoc reference is a great resource when editing the page, but it is only a tiny link that appears magically in a section that I already looked at and assumed I know everything that's in there.
- (normal) Come to think of it, we probably need an introduction at the top of this edit form that explains what the heck I'm looking at. Because it *looks* a lot like Markdown, people might be inclined to just edit it like they do stuff on Github, but there's special syntax they need to be aware of. We could link such a reference there (maybe with a BIG GREEN BUTTON :P).
- (minor) It would be nice if "Return to book" returned me to the page I clicked here from, rather than the top-level. What I imagine doing is reading deep into the outline, finding a typo, quick clicking in to correct it, and then wanting to get back out again. If I'm jumped back to the main ToC that's going to be frustrating to find my way back to subsection V.I.IX or whatever again.

Overall, though... although it's certainly very clever, I definitely do have to say that the editing experience is far less awesome than that of nodes on d.o. :( I frequently jump into/out of editing mode in handbooks as I'm reading and find problems, and when those changes go live immediately it's real quick to get back into the flow of what I was reading and continue where I left off. With this, I'm reading reading reading, spot a typo, click a thing, make an edit, click a button, leave my browser, find the patch on my computer, open the patch in my text editor to verify it looks correct, go back to the browser, go *completely off site* to a *totally different context*, fill out the issue form, upload a patch, sit around and wait for the patch to post, obsess over the issue summary a few minutes, then finally try and remember what I was reading when I found the issue in the first place. :(

Would be great if we could arrange something like people with "you're not a moron" permissions could just commit directly from the UI, but I gather from above that this is an enormous can of worms. :\

jhodgdon’s picture

Thanks for taking a look! I wish you had put (a), (b), etc. in comment #75 so that I could be clearer in my responses, but anyway...

So, yeah, the URL, the navigation, look/shinyness, etc. -- none of this was meant to be the final answer, and you made some excellent points. ***IF*** we want to move forward on this, I'll put some time into making it look nice, making an edit button, some help, etc. At this point, the vast majority of the comments here have been largely (as I see it) against this whole idea, and I didn't want to take the time to make it all pretty etc. if no one was going to even be willing to make patches and file issues instead of editing Drupal nodes.

Anyway, all good points and I'll be happy to fix them all if we are going forward.

One thing: Preview -- this is available, actually. You just click the check-mark button (far right on the editor buttons). It even updates every time you click a button or hit Enter. It's really a very slick editor and I think if you had noticed the Preview you would have been more positive about the whole thing... :)

jhodgdon’s picture

Issue summary: View changes

Incorporated feedback in #75/webchick into issue summary. Also #73/jhodgdon.

jhodgdon’s picture

So.....

There's a table in the issue summary that I think accurately captures the pluses and minuses of using Drupal nodes vs. Git/AsciiDoc.

The main point is this.

If we continue to use Drupal nodes for the Developer docs, we have an editing mechanism that is obvious and that we're used to, with all edits going live immediately. But we are stuck with having no way to branch the docs, and no editorial control (at least with things how they are now using Drupal book nodes), which leads to the developer docs (like the rest of the Community Documentation) being kind of a mess. And everything has to be formatted in HTML.

If we switch to using AsciiDoc in a Git repository, we gain editorial control and branching, at the expense of having a more cumbersome editing process. Even with a nice big Edit button (use your imagination) and the ability to edit/preview on-line, you still have to download a patch file, create an issue, describe the problem, and attach your patch, and the edit will not go live until someone who has volunteered to be an editor for this set of docs has reviewed and committed it. But we also get to use a markup language instead of HTML to write the docs, and the ability to make output in PDF and other formats, besides displaying them on a web site.

So there are definitely advantages to each. At this point, I am not going to put any more time and effort into polishing up the AsciiDoc proof of concept unless the community thinks the advantages are enough that we want to go this route.

My vote is for AsciiDoc, but I am not the community. I would like to see more input from the people who have actually been creating and editing the developer docs for Drupal 7 and Drupal 8 that are currently under https://drupal.org/developing/api. If this set of people is strongly in favor of keeping the current system, or of moving to the new system, I think we should go with their opinions.

Is there some way we can make a list of who has most often created and made edits to just the docs under developing/api, so we can get their opinions? Or am I wrong to think those are the people whose opinions we need the most?

webchick’s picture

That seems sensible. I'm honestly not sure which way I land yet, because they both have pros and cons, as you say.

That sounds like a useful way to contact folks who might be interested in this issue more directly. Unfortunately, my SQL-fu is way too deteriorated to possibly come up with such a query, but if someone manages to figure it out, I'll happily run it on prod.

cweagans’s picture

select distinct uid as ouid, (select count(*) from node_revision where uid=ouid) as changes from node_revision order by changes desc;

cweagans’s picture

With usernames:

select distinct nr.uid, (select count(*) from node_revision inr where inr.uid=nr.uid) as changes, u.name from node_revision nr join users u on u.uid = nr.uid order by changes desc;

fgm’s picture

Although I continue to stand on the side of using more of Drupal to document Drupal, I want to thank jhogdon for this nice prototype and the effort put into it and the underlying thinking process.

This being said, maybe I was not clear enough the first time: the way I see it, this is more of a "marketing" (boo!) position: we are handling a content management problem, for which content management systems like ours have been devised. If we can not find a better way to manage that content than not using our CMS, this projects rather badly on the capabilities of said CMS.

Technically, although I haven't got into the specifics of AsciiDoc, I do not get why you appear to assume that using nodes means HTML input: most of the sites I know use Markdown for input instead, and AsciiDoc seems rather similar, so it seems we could very well use it instead of HTML. It seems likely that we could also produce the PDF books like in your prototype by listing the nodes in a book and using the PDF toolchain from there, much like with the VCS-based method.

Not even mentioning the idea of using Git as the storage for specific Drupal entities instead of nodes and SQL, which would probably require D8.

joachim’s picture

> But we are stuck with having no way to branch the docs, and no editorial control (at least with things how they are now using Drupal book nodes), which leads to the developer docs (like the rest of the Community Documentation) being kind of a mess.

What are the specific problems? Can any of them be dealt with?

I posted an issue proposing to make it much easier to reorder the structure of books. (Can't get link right now, on train.)

Could we imagine a way to clone a whole book to kick-start a new version of the docs for a new major version?

> And everything has to be formatted in HTML.

That's surely just a matter of input formats. We could opt to have books in markdown on d.org, surely?

webchick’s picture

I still think the idea of trying to re-invent a version control system via Drupal modules is pretty silly. Building a Drupal-based tool that leverages Version Control API to directly interface with a version control system which does the branching, merging, atomic commit tracking, etc. I could get behind, and was what I was originally envisioning here, although Jennifer's work leveraging the standard AsciiDoc tools is definitely interesting.

Thanks, cweagans for the query! AFAICS though that doesn't limit it to only those book nodes below /developing/api. In any case, the query takes longer than 30 seconds to run and so it gets killed. :(

jhodgdon’s picture

cweagans: Thanks for the queries, but we are looking for something that gives us "people who created or edited docs underneath https://drupal.org/developing/api", not "people who created or edited any node", which is I believe what your query would do.

Regarding the other comments... Point taken. So the idea would be to make a system with Drupal nodes that would have the following features, for deployment on api.drupal.org (so it could have totally different input formats and modules from what we have on drupal.org):

a) Has a hierarchical structure of pages and sections [Core Book module or something else].
b) Allows branching when it's time to start a new version [deep copy of each node under a certain spot in the hierarchy.]
c) Allows editorial control, so people can propose edits and new pages, but they wouldn't go "live" until a designated editor had reviewed them.
d) Uses some kind of markdown syntax.
e) Allows export of the book into DocBook (or some other generic format with a well-developed toolchain), which would then allow us to create PDFs, stand-alone HTML, ebooks, and other types of output.
f) Allows preview of output while editing on-line.
g) Runs in Drupal 7.

I agree that all of this is potentially possible to do in Drupal nodes; I also think that Git is a better fit to the requirements than Drupal nodes (and I don't care what fgm thinks about "marketing" -- we do not claim that Drupal does everything -- for instance, we use Git to manage our source code, not Drupal, and perhaps the programmer-oriented docs fall into the category of "things for which a revision control system is a better fit than Drupal").

That said, it would be great if someone wanted to work up a "proof of concept" prototype on the same api.d.o dev site where I put the AsciiDoc/Git prototype. (I have put quite a bit of time into making the AsciiDoc/Git prototype work, and I'm out of energy.) I don't think all the modules for doing this exist at the present time, but I would love to be surprised.

jhodgdon’s picture

Ah, sorry, cross-post. What Version Control API were you talking about webchick? How would this system work?

webchick’s picture

Sorry, I mean https://drupal.org/project/versioncontrol and https://drupal.org/project/versioncontrol_git. This is the set of modules that powers Drupal.org's Git integration for things like release packaging, commit log viewing, etc.

jhodgdon’s picture

I don't understand how that would work. The Version Control projects look like they provide ways for a Drupal site to react when things happen in Git or other VCS, and to make VCS log-type listings. They don't seem to provide ways for Drupal to *do* VCS operations like commits... or at least if they do, it's not obvious from the documentation. ???

jhodgdon’s picture

Issue summary: View changes

On another, less controversial note, I just filed #2148255: [meta] Make better D8 api.d.o landing page, linked to high-level overview topics, and put it in Core api.php files to create a better api.d.o landing page for the regular API reference docs generated from Drupal Core, which is item #2 in the Proposed Resolution list in the summary.

jhodgdon’s picture

So.

We had a discussion about the prototype in IRC with sdboyer, xjm, webchick, beejeebus, and others today.

The main stumbling block with the current prototype (besides some UI things that could use improvement, like a more prominent Edit button etc., as discussed above) seems to be this:


After you click "edit" and finish your edits, you are presented with a patch file and must file an issue (there's a link, but you have to fill in the issue and upload the patch file).

So, the question is whether we could streamline this process.

And the answer that we came up with was basically yes, but only if we hosted the repo for the AsciiDoc files on Github. The idea would be that the edit button would work like it does on Symfony and take you directly to Github's editor (where you would need to have an account). You would edit the file there, after which Github would take care of making that edit into a pull request. The GitHub editor does not have Preview or buttons specific to AsciiDoc, so it would not be as good as the one in the current prototype.

sdboyer says we cannot do a more streamlined process with the repo on drupal.org as they are not willing to have people writing to the git repos from a web editor in that repo, among other issues. So our options are (assuming we want to use Git at all): use Github, or live with patch/issue workflow. Or we could actually offer both, if we had a repo on d.o as well as the one on Github (the maintainers of the docs repo would have to keep them in sync, which isn't impossible. We probably would want the repo on d.o anyway?).

jhodgdon’s picture

I think that a GitHub mirror is quite viable.

I just did a test:

a) I made a public GitHub repo.
https://github.com/jhdrupal/docs_sandbox

b) Followed https://help.github.com/articles/importing-an-external-git-repository to make it mirror the sandbox where I have the AsciiDoc source files currently. (one-time process to import)

c) Following http://stackoverflow.com/questions/7244321/how-to-update-github-forked-r... I made my local Git workspace pointing to the d.o sandbox have a second "remote" for Github (you only do this once):

git remote add github https://github.com/jhdrupal/docs_sandbox.git

d) Asked beejeebus in IRC to edit one of the files on GitHub. This resulted in a pull request (beejeebus said it was a two step process but easy to follow). Note: The GitHub text editor is somewhat lame (all black) and the 2nd edit beejeebus did resulted in a bunch of end-of-line spaces. But it did work.

e) On GitHub, I merged in the pull requests using their web site tools.

f) Back in my local working Git directory, I merged the Github changes into my d.o repo:

git fetch github
git rebase github/8.x
git push origin 8.x

g) I made another commit to the d.o repo to reverse those edits.

h) I merged that over to the Github repo. This is as simple as after the usual commit/push, you do:

git push github 8.x

So, it does work. As a maintainer, you would need to do step f whenever someone makes an edit/pull request on Github and you merge it there, and step g whenever you make a commit on the d.o repo. It's not too horrible, just a couple of extra commands.

So what I would propose, in this case, is that we could offer both options for web editing of the source files:

1) What we have now. You get a nice editor with buttons and previewing, and you stay on api.d.o to edit. When you're done editing, you download a patch and file an issue.

2) Another edit button that takes you to GitHub, where you can (if you have/make an account) edit there and make a pull request.

Anyone who wants to edit a file can use their preferred method #1 or #2. The editor/maintainers of the docs will manually (following the steps above) maintain the connection between GitHub and the d.o git repo, and they will have privs to commit/merge on both. The d.o repo will remain the canonical place for the docs. Not too horrible a thing to do.

Thoughts?

sdboyer’s picture

lovely.

if you want to get really fancy, you can even set it up to push to both remotes simultaneously. here's one quick writeup on that:

http://stackoverflow.com/questions/4255865/git-push-to-multiple-reposito...

drumm’s picture

sdboyer says we cannot do a more streamlined process with the repo on drupal.org as they are not willing to have people writing to the git repos from a web editor in that repo, among other issues.

If we find a trustworthy, mature, and maintainable tool for this, it could happen. When was this question seriously looked into last?

So our options are (assuming we want to use Git at all): use Github, or live with patch/issue workflow.

I expect we'll have pull requests on Drupal.org someday, as the software working group explores what developers needs include, and the Drupal Association staffs up to be able to work on them.

jhodgdon’s picture

RE #93 - I think sdboyer's objection was "We don't want people being able to make commits into the d.o git repos through on-line forms", not "We don't have pull requests".

In any case... The point is that we don't have that capability now in d.o git, so for now we can use GitHub. I don't think we'd want to delay this project on the basis of "we need a brand new Git management tool on d.o that everyone agrees is sufficiently secure in order to deploy this", and I don't really want to open up that discussion on this issue.

If we did eventually have a GitHub-like "edit on-line and generate a pull request" capability on d.o, we could just make the edit links go there instead of to Github. The way this Asciidoc Display in Drupal project is set up, that would be as simple as changing a URL in an api.drupal.org admin page.

eojthebrave’s picture

Just thinking out loud a little since I like where this is headed in general but am still getting a little bit hung up wanting to have the "I'm a casual user and I just want to fix a typo" experience be as seamless as possible to help encourage contribution.

As mentioned above by others I too see the biggest hurdle here being context switching. I want to make an edit, click save, and be right back where I was. The dream here for me is that I can edit a typo or some simple issue without loosing my train of thought. 99% of the time I landed on this page because I needed some of the information that's contained within the page in order to accomplish a task that I'm currently working on. Not to do a review of the content and make changes. So the easier this is for me to make those changes without impeding whatever I'm working on the more likely I am to do it.

However, I also completely understand technical limitations, politics, etc., and as long as the solution we end up with doesn't feel "hard" it also doesn't have to be perfect. Yet. :P

I haven't tried any of the github stuff yet, but that does sound pretty straight forward.

Another thought if we don't want to use the GitHub solution, based on using the prototype, that might make this a little more seamless: When I edit a page and it gives me the option to download a patch file ... could we replace that button with another text field where I can describe my change, and a "Submit" or "Create Issue" button that when clicked would generate a new issue in the queue with the description and provided and the patch attached and then redirect me back to the page I was just reading when I decided to fix the typo?

jhodgdon’s picture

Regarding your idea in #95 about streamlining the issue creation... I don't think so. The "file an issue" link is going to be taking you to a different site. I do not think we want to reproduce the issue form on api.drupal.org (which is where this is proposed to be deployed).

And ... regarding your other concerns, I think they already in the "pros and cons" table in the issue summary. There are definitely advantages and disadvantages of the "use nodes" vs. "use git" approaches. The trick is to decide which side weighs in the best.

So please, anyone else commenting here... What I really want to know is, especially if you have taken the time to actually try out the make-a-patch editing prototype and/or the Github editing option (and sorry, I haven't put in a link from the test site to the Github editing page, but see comment #91):

a) Are there any new concerns you have that are not already reflected in the issue summary?

b) In your opinion(s), which approach, on balance, do you think is the better one, given the pros and cons listed in the issue summary?

And if I had time I would now update the issue summary to put in a link to the Github stuff too...

joachim’s picture

> The dream here for me is that I can edit a typo or some simple issue without loosing my train of thought. 99% of the time I landed on this page because I needed some of the information that's contained within the page in order to accomplish a task that I'm currently working on.

One thing that REALLY made a big change for me on api.d.org was the 'Report a problem with this page' link, which takes you straight to creating an issue against the core docs.

It's this link which means that whenever I'm reading the api docs and spot a typo or missing or inaccurate information, I open the link in a new tab, rattle off a quick sentence or two about the problem, create the issue, and get back to whatever I was doing. (I have the worrying feeling sometimes I'm among the most prolific creator of docs issues...)

So provided we have just as good a link on the new docs, I think this could be ok.

jhodgdon’s picture

Thanks joachim. I have to say I very much appreciate you using that link. :) However, I also have to say that you are nearly the only person who uses that link to file good, correct, actionable issues.

webchick’s picture

Amitaibu just told me about http://www.gizra.com/content/zariz-in-pics/ which looks somewhat close to what we're talking about here. A Drupal front-end for managing Git-based content. See http://www.gizra.com/content/zariz-means-agile/ for more info.

I haven't tried it myself but it sounds potentially promising!

izmeez’s picture

@webchick and Amitaibu thank you very much for sharing the information in comment #99. It is exciting to see ideas like this emerging and in action including the Ted Talks episode linked to in the second of those articles.

amitaibu’s picture

Thanks @webchick!

> A Drupal front-end for managing Git-based content

Just to clarify -- It's managing nodes using Git's concept of branches and merging.

> I haven't tried it myself but it sounds potentially promising!

I'm happy to say that Bojanz had a similar reaction

Anyway, I'd be happy to give a demonstration to anyone willing to get one ;)

jhodgdon’s picture

I'm out of energy on this project but if someone would like to set up a "proof of concept" system, we still have the demo site to do it on. It seems to me that a "proof of concept" would be to make 7.x and 8.x branches of part of the existing documentation at https://drupal.org/developing/api and show how editing would work.

I'd also like to see a list of what modules we would need to add to api.drupal.org to make this work. It looks like it relies on Organic Groups (ugh!!!)... what else?

jhodgdon’s picture

Oh, and the take-away message from sdboyer was that we don't want to allow random people to push to d.o Git repos from the web... I don't think this module would change anything in that regard.

sun’s picture

Since I kept on bringing up the topic in several other issues that tried to delete some of our bird-level documentation of Drupal core subsystems, @jhodgdon asked me to file a dedicated issue for my concrete proposal:

#2191525: [PP-1][policy, no patch] Extend Markdown coding standards to support API module (DOXYGEN)

jhodgdon’s picture

Issue summary: View changes
Status: Needs review » Closed (won't fix)

At this point, I'm going to go ahead and close this issue. After numerous rounds of "proof of concept", there was not enough community support for this (and considerable resistance). So I'm going to let the demo site go (I'll edit the issue summary), and I think we can just close this as "won't fix".

We did manage to accomplish part of the goals: we spun off #2148255: [meta] Make better D8 api.d.o landing page, linked to high-level overview topics, and put it in Core api.php files and made a new api.drupal.org landing page for Drupal 8. That work is continuing (to revise it and fill in the topic pages), and can be found/discussed on that issue.