[PP-1][policy, no patch] Extend Markdown coding standards to support API module (DOXYGEN)

Question: Is this proposal only for core/lib?

One advantage of @defgroup over .md files is the ability to use @ingroup or the @{ @} tags to add things to the documentation group. When this is done, you get two results on api.drupal.org:

a) On the Topic page, there is a list of all the "related" functions, classes, files, topics, etc.
b) On each of the related item pages, there is a list of all the related topics.

Both of these are quite useful, and I think neither would be easy to reproduce with .md files without a fair amount of work.
I am fairly inclined to stick with the status quo unless there are really some clear advantages of the system for Drupal people... I hear what you are saying about READMEs being more standard in the greater world, but the only parts of our code that are at all meant to be portable are core/lib/Drupal/Component -- the rest is too "Drupal-ish" to be useful to people outside Drupal (tied to the database, using hooks, etc.). Of these components, most consist of stand-alone classes whose functionality is well documented on the class (or an accompanying interface), so they may not even really need a "birds-eye" view or a README. The things in there are really meant to be utility classes, right?

Also, for the Components that are well-documented utitility classes, it wouldn't really be a horror if their main class or interface had a @defgroup block at the top of one of the class files, would it? I mean, someone who wanted to use the class would have the documentation right there in the file... sounds pretty handy? Or if that is a horror for some reason, we could make a .api.php file, and put one line of @ingroup in the class/interface... I don't see why that is so horrible -- a line saying @ingroup doesn't make the code non-portable or the documentation unreadable.

So... Maybe we should put README files in a few of the components in that directory, but I think for the rest of the code base we should stick with @defgroup and/or just class documentation.

That's my opinion anyway.

> One advantage of @defgroup over .md files is the ability to use @ingroup or the @{ @} tags to add things to the documentation group.

Very good point.

A very large amount of our documentation is in doxygen. Splitting some off into a new format is not ideal.

> Object-oriented code does not have a "central" (code) file that can be used to document a component/subsystem.

> The most natural place for developers who are seeking to learn about a component is to look into the actual component folder itself.

Could we satisfy the above by having a 'documentation.php' file in the component folder, which contains doxygen?

RE #4 - The file could even be called README.php. Aside from the first line, @defgroup documentation is pretty readable... I would certainly prefer that to having to set up the API module with a Markdown parser, make some ad-hoc way of figuring out what items are part of the "topics" that a README file automatically creates. etc.

We have lost all of our bird-level documentation that explained how someone is supposed to use a certain subsystem.

#1532612: Move Examples Project into Drupal core

http://ftp.drupal.org/files/projects/examples-8.x-1.x-dev.tar.gz

Examples has the following advantages over either README or @defgroup:

• It's runnable documentation. You can enable the modules and see what they do.
• It's testable documentation. If the documentation is wrong, tests will fail.
• It's crawled by api.drupal.org.

One downside is that Examples uses @defgroup to organize itself at api.drupal.org. This would have to either change or be adopted in some way, so decisions would have to be made.

I detest Markdown as a format, in part because it's not a single format but a family of related but not-really-the-same formats as the OP notes. That said, I can't say I like @defgroup any better. It's really hard to write and to line up properly within the docblock, to say nothing of our often overly stringent 80-character rule (and the result that making one simple change can often require reformatting 30 lines of text). So in concept I'd be +1 here overall.

It doesn't make sense to mandate a README.md file in every Component or Core subsystem, but for those cases where it makes sense we can use it. (The database has/had a massive defgroup block, which is still in a file that contains a bunch of wrapper functions and nothing else.)

It's true that code in /Core is not really separable from Drupal, but that has absolutely no bearing on the documentation question. Making the documentation of Drupal-specific code more familiar to non-Drupal people can only help make Drupal code more approachable and increase the likelihood that the documentation will be maintained. (Hence why we should plan to move to PSR-5 documentation eventually.)

Coding standards decisions are now supposed to be made by the TWG

Despite personal opinions about the format of markdown, the fact is: it's actually already very widely used in other development communities and becoming the "defacto" for verbiage/topic based documentation. I think there is also some confusion about what markdown actually is. It is HTML, but "marked down" so it's easier to read and parse by
humans. Regardless, the basic syntax is relatively stable and has not really changed.

The file could even be called README.php. Aside from the first line, @defgroup documentation is pretty readable

So that's basically what happened to the "core.api.php" file. Personally, I think that was a mistake. It has become ~2,400 lines of lengthy verbiage based "documentation" which is not necessarily about specific code (yes, I know there are some functions in there, but the majority of the file is just verbiage). This isn't very "readable" when they all look like green docblocks in an IDE. there's no point of reference without having to manually slow down and read the @defgroup topic name.

I think we're actually starting to abuse DOXYGEN at this point. We're stuffing verbiage based documentation (that has nothing really to do with actual PHP code) into PHP based files. PHP DOXYGEN is meant for documenting actual PHP code. It isn't meant to be a end all, be all documentation solution.

To illustrate this point, I went ahead and converted the first @defgroup in core.api.php. Which is easier to read?

PHP DOXYGEN docblock:

A simple markdown file (hypothetically named third-party.md):

You'll notice at the top of that markdown file <!-- @defgroup --> and <!-- @summary -->. What are those you ask? Well, I've actually spent a great deal of time on this topic and trying to figure out the best way to "inject" markdown files into an API module based documentation site.

<!-- @defgroup {group_name} {group title} -->
<!-- @addtogroup group_name -->
<!-- @ingroup group_name -->
<!-- @summary description -->
# Group Title

As I stated above, markdown is actually HTML. This means we can use things like an HTML comment to denote special "DOXYGEN" like properties that would be consumed by the API module, purely as an "opt-in" process.

<!-- @defgroup {group_name} {group title} -->
The curly brackets here denote that they are optional. The reason for this being to reduce duplicating information. The reality is we would already have this information available to us during the parsing phase. So we can parse the group name automatically from the file name and parse the group title automatically from the markdown title (#). It of course would recognize any specific group name/title provided in this tag, but the idea is that you would have consistency in both the filename and title inside the markdown document if either is to change.

One advantage of @defgroup over .md files is the ability to use @ingroup or the @{ @} tags to add things to the documentation group.

That would be true if we were to always treat markdown files as "@defgroup". I honestly think that would be a mistake and we should instead "opt-in" by using something like HTML comments. There will likely be cases where we may want to create a markdown file, but not actually use the file as a topic. We could still achieve the same results of using @{ @}by instead using <!-- @ingroup group_name --> or <!-- @addtogroup group_name -->.

---

So, to my the money where my mouth is, I've actually already created a working POC. This markdown file is being parsed by the API module and subsequently created as a sub-topic:
http://drupal-bootstrap.org/api/bootstrap/docs%21project%21releases.md/g...

The source of this file:
http://cgit.drupalcode.org/bootstrap/tree/docs/project/releases.md

For those interested in how I am currently parsing markdown files, you can view the source code here:
https://github.com/unicorn-fail/drupal-bootstrap.org/blob/master/docroot...

edit: updated links

Also meant to mention that naming every markdown file "README.md" is rather counter-intuitive when there could be multiple topics in the same folder. Sure, we could still use "README.md" in certain areas where it's the only file and use an explicit group name inside the @defgroup tag. As a concept though, we should really use descriptive filenames (edit: especially if they will be used as the group machine name).

#9 seems like a reasonable solution, which the API module could use as an alternate way to define 'groups'. We should put this into the issue summary. I don't think we'd want to get rid of the ability to use @ingroup/@defgroup, with the cross-linking and listing features the API module has for this type of thing.

One thing #9 still needs is the ability to define the topic title, as well as the topic summary line. See
https://www.drupal.org/node/1354#defgroup
But that should be easy enough to do.

Just a few other notes since this issue is pretty old:

a) Many of the *.api.php files in Drupal 8 Core have been moved closer to where the relevant code is. So that part of the issue summary may not be relevant any more.

b) Many of the *.api.php files have more than one topic in them, or topics plus hook docs. We'd need to continue to put the hook docs in *.api.php files, since they are PHP functions plus PHP docs. So, the proposal here would be to, I think, take each topic out of its current *.api.php file, and move it into a *.md file instead. Plus, add the relevant code to the API module that would parse these files and display them in the Topics list along with the other topics. At least, I think that is the proposal.

Also I am not sure why we would want to have markdown files and not list them as topics on api.drupal.org?

One thing #9 still needs is the ability to define the topic title, as well as the topic summary line.

Um... it already has this ability. Maybe I didn't make that clear enough in #9? The group (topic) title is automatically parsed from the Markdown's H1 (single # at the top). It also has the native ability to allow a specific group name and title (thus ignoring the Markdown H1 and file name).

The following example would work:
<!-- @defgroup third_party REST and Application Integration -->

Just as well if the file name was third-party.md and there was a Markdown H1 (#) at the top:

<!-- @defgroup -->
# REST and Application Integration

This "feature" certainly isn't necessary and core may just always use explicit definitions to keep parity with other core documentation, that's fine. I was just showing that this kind of information could easily be parsed automatically to keep in line with more "markdown" style format and to not duplicate information.

RE: summary line
That too is also already supported: <!-- @summary My short description of what this file is -->. I know that this is rather unique (@summary isn't used anywhere in PHP DOXYGEN), but I don't see the harm in it considering this is to be used for a Drupal API site file listing table only.

I guess we could do something like this if that makes any more sense:

<!-- @defgroup group_name Topic title
My short description of what this file is
-->

I guess my goal was to keep it clean and concise so that the HTML comments start and end on just one line. As you can see, this takes up three and looks a little odd.

---

FYI, I'm still working on that site and probably just broke the examples I linked in #9. This is due to putting some stub files in and to help further develop this POC. Regardless, I will get this back up and running again shortly. It will also, perhaps, clarify a little bit of what I am talking about (hopefully).

Ah, I missed @summary. Sorry! Agreed that the # title is better too.

<!-- @mainpage --> is now also being processed:
http://drupal-bootstrap.org/api/bootstrap
(source: http://cgit.drupalcode.org/bootstrap/tree/README.md)

I've changed how the group machine name is automatically generated so that it parses the full path of the file (excluding any top level "documentation|docs|doc" folder):
https://github.com/unicorn-fail/drupal-bootstrap.org/blob/master/docroot...

Basically, what this means is that instead of having duplicate words (e.g. like it was before docs/project/project_releases.md), it will use the folder its in as the "parent" group and prepend that parent grouping to the extracted file name.

The exception to this is are README.md files. I've modeled it after GitHub in where the README.* file is associated with the directory itself. Thus, it will use the folder it's in as the actual group name instead of the "README.md" filename.

To illustrate this automatic conversion:
(path/file -> group machine name)

docs/project/README.md -> project
docs/project/releases.md -> project_releases

docs/subtheme/README.md -> subtheme
docs/subtheme/drush.md -> subtheme_drush

Hypothetically if there were more nested directories, it would be converted as such:

docs/parent/sub/group/file.md -> parent_sub_group_file

I am quite aware that this level of automation can be confusing or perhaps too much for core and y'all may want to stick with explicit group name definitions.

This is just my personal preference for documenting the Bootstrap base theme. For this issue, I am just posting this stuff here to show a related working POC and that it can be done... effectively and logically.

Yeah, that naming convention is not going to work well for Core, for converting the existing @defgroup topics to .md files, because they do not follow this naming scheme. Is it implemented as a default if the @defgroup comment line is omitted? If it can be overridden, then it seems like a sensible default.

All of this still needs to be written up in the issue summary as a proposal, if for no other reason than so it can eventually be copy/pasted into some documentation on how to do this.

Yeah, that naming convention is not going to work well for Core, for converting the existing @defgroup topics to .md files, because they do not follow this naming scheme.

Yes, I understand that is not how core currently does it. That's fine, core wouldn't necessarily have to follow this exact pattern. However, I would argue that having some sort of a "scheme" will likely be necessary (if only for organizational purposes) since individual topics would need separate markdown files.

Is it implemented as a default if the @defgroup comment line is omitted?

No. If there is absolutely no @defgroup comment line specified then the entire file is just treated as a regular ol' file by the API module (e.g. it's not processed as a group/topic and would only show up in the views file listing). This is what I was saying above in that in that this is purely an opt-in process.

The reason for this being: there could be (in the future) a reason where some markdown files shouldn't be considered a "group/topic". In order to not limit ourselves, I would much rather this require an explicit declaration of @defgroup (like it currently is in PHP) when a group should be created.

The automatic group name from path/file and automatic group title from the # H1 line is only ever implemented if there are no explicitly set group name or title definitions.
If a file specified something like the following, it would never actually use the automation stuff to determine this information from the path/filename or # H1. It would use the manually defined information:

<!-- @defgroup explicitly_set_group_name Explicitly Set Group Title -->

All of this still needs to be written up in the issue summary as a proposal, if for no other reason than so it can eventually be copy/pasted into some documentation on how to do this.

Yes, I do plan on doing this. I have just been putting it off until I'm sure the development of this is more "solid" (which it mostly is, but ya never know.. there could still be some minor things to do).

In all reality, the ultimate goal I had in mind was to eventually make these custom modules a full d.o contrib project extension to the API module.

OK... A few more thoughts:

a) If we adopt this as a Drupal Project Coding Standard (which is what this issue is about, after all), then I think the base API module will need to support it and I would rather not see us have a separate contrib module needed to support the coding standards, especially those used by Core.

b) If we convert existing @defgroups in Core to .md files (the issue title and current summary do say that this is the purpose of this issue), then if possible we would want to have the same @defgroup machine name persist in the new format, so that we don't also have to go around changing all the @ingroup lines in other files. I am confused by #17 but I think/hope it is saying that this is possible. The code looks like it is.

c) I still do not understand why there would be .md files that shouldn't be listed as Topics on the topics page on api.drupal.org. It seems to me that if something is important to document with a Readme, then I think it is by definition a Topic... If not, then what distinguishes "things we should document with a readme file" and "topics"? We're talking Policy and Coding Standards here... so we if we are suggesting that people make .md files, and only some of them should be "topics", we would need to make a criterion/policy for how to decide if something should or shouldn't be a topic. I cannot figure out what that criterion would be.

a) Of course, but I suspect that will be a while before that happens. It's ok if it's a contrib project initially and then ultimately merged upstream into API.

b) Yes, you can keep existing defgroup definitions, e.g. <!-- @defgroup third_party REST and Application Integration --> will work and always be third_party... no matter where the file resides or what it is named.

c) Yes, in reality, I would suspect that most markdown files in core will be topic based. However this isn't just about core, it's about contrib too. Yes we're talking about Policy and Coding Standards. Most contrib projects will not be using (or parsed by) the API module.

Because Markdown also parses HTML, this means that they could (in theory) be used for any number of things.

I, myself, have already envisioned using some markdown files with <!-- @snippet --> to show examples on how to do things with the Bootstrap base theme. These wouldn't necessarily be "topics", but rather just standalone examples that could be attached to topics (via @ingroup).

To arbitrarily associate and then mandate every .md file as a "group/topic" is not the right solution. We don't automatically assume all PHP files are topics/groups. No, they declare a @defgroup (which is the existing standard) and so should Markdown files.

Also, another reason we can't always assume all .md files will be @defgroup is because of the newly added detection for @mainpage support (which isn't a group/topic).

Fair enough.

As a minor note, at the moment in the API module you can declare that a file is @ingroup to a topic/group. I mean, you could try and it would probably be saved in the database, but the Topic pages don't currently list files. At least, I'm pretty sure. It would of course not be difficult to add this feature to the API module.

Another minor note... Regarding "most contrib projects won't be parsed by the API module", that isn't really true. There is for one thing drupalcontrib.org, which parses quite a few contrib projects (though probably not the majority? I don't know), and also there is an eventual plan to have all of contrib on api.drupal.org, although there are still several blocking issues.

at the moment in the API module you can declare that a file is @ingroup to a topic/group. I mean, you could try and it would probably be saved in the database, but the Topic pages don't currently list files

Yes this works. This is how I am defining "sub-topics" (e.g. http://cgit.drupalcode.org/bootstrap/tree/docs/project/releases.md#n1) by defining both:

<!-- @defgroup -->
<!-- @ingroup -->

I'm not entirely sure what you mean about the files list. The Topic list view shouldn't show files, just topics (and subsequently sub-topics, although those should probably not show up... separate issue).

most contrib projects won't be parsed by the API module

It is true currently. Yes, I am aware of drupalcontrib.org, but it's taken nearly 2 years to get a module I submitted to show up. I suppose what I really meant by this is that while yes, we have standards, the use cases for core don't always coincide with contrib. To limit something just because core doesn't (yet) use something is never a good approach in my book, that's all.

eventual plan to have all of contrib on api.drupal.org

Awesome! I would be interested in seeing these issues.

---

All in all, I think this issue should really be focused on proposing/adopting official Markdown standards (not implementation), so changing meta information to reflect that.

Also, I will do my best to update the issue summary soon.

This related issue may possibly influence this one.

May I strongly suggest that we use an existing PHP Markdown library, and were appropriate push fixes upstream? There are a couple, but I believe the one from the PHP League is one of the most common.

http://commonmark.thephpleague.com/

I've already done that research and have chosen erusev/parsedown-extra. I've gone ahead and appended the following to the IS.

---

PHP Libraries (most common)

Based on packagist search term: markdown:

Current PHP library of choice: erusev/parsedown-extra as it adds the right balance of Markdown Extra features and is extended from erusev/parsedown for faster performance based on michelf/php-markdown (the original PHP library).

CommonMark is still relatively new (in PHP). This is what the README for https://github.com/erusev/parsedown has this to say:

Is it [Parsedown] compliant with CommonMark?
It passes most of the CommonMark tests. Most of the tests that don't pass deal with cases that are quite uncommon. Still, as CommonMark matures, compliance should improve.

I quickly parsed this thread, so bear with me if I've overlooked elements.

Reading #24 reminded me of http://commonmark.org/ (details in this blog post) which is a long awaited attempt at standardizing the markdown format. Even though it has to mature and doesn't have the same flattering number of downloads/stars as other markdown libraries, I still believe we should explore this more as this could provide better standardization in the long run.

So, the lack installs/stars is what actually prevented me from really considering CommonMark to begin with. Perhaps that was a mistake. Regardless, I've since changed my mind after actually looking over CommonMark more. It does make more sense to go this route considering they already have all the specs and we can just reference them.

I suspect my only concern was that I wouldn't be able to do the special extra attribute parsing that I need for http://drupal-bootstrap.org. That being said, there's actually an extension for it, so I'm fine with this.

In fact, I was able to easily switch out my existing implementation of Parsedown with CommonMark and have since created https://www.drupal.org/project/commonmark. This is just the library wrapper (for now) and really has nothing to do with the above mentioned DOXYGEN keywords.

---

I have rewritten the IS to address what this issue is now really about. We can create follow-up issues as needed for implementation once this has been finalized.

One minor change btw, use @file instead of the custom @summary... they're essentially the same thing. I also didn't mention any of the "automation" for groups... that has nothing to do with this.

The issue summary looks good here! A few notes/questions:

a) If there will not be support for @{ @}, then it does not make sense to have @addtogroup. @addtogroup means "add the following things inside a @{ @} to the group/topic", so without the {} it is meaningless. @ingroup would be what you want for this, which means "Add this item to the group/topic".

And I agree that having support for @{ @} does not make sense since the entire file is one "thing" (not like a PHP file that has other "things" in it, like functions, classes, methods, etc., each of which could be added to the group/topic).

b) I'm a bit confused actually about how the @ingroup/@defgroup/@mainpage would work. Normally the scope of each of these is 1 doc block within a file. But a .md file doesn't have "doc blocks", so that means the scope is the whole file... So I guess:
- If you have @defgroup, you are saying "Treat this entire .md file as a defgroup/topic".
- If you have @mainpage, you are saying "Treat this entire .md file as the main docs page for a branch in a project".
- If you have @ingroup, and @defgroup is present also, I think you'd mean "This topic is also a sub-topic of another topic" (just like having @ingroup within a @defgroup doc block in PHP files).
- If you say @ingroup and @defgroup is not also present, I think you'd mean "List this file on the topic page in the Files section" [we would need to add support for files listing on topic pages, which we should probably do anyway for twig templates and such... filed: #2563349: Support listing files on Topic/Group pages -- independent of this issue, we should do that]

Given that, we probably need to clarify that you cannot have @defgroup and/or @ingroup in a .md file that has @mainpage in it.

c) Independent of whether you have a @file line, presumably these files would be listed on the api.drupal.org page... Actually I think @file is a bit redundant here, because you would want to require a Title line in a .md file anyway, right? So probably that would duplicate the summary line. I don't see why we need @file... maybe simplify and just not support it? That is what the API module currently does for .html files (used for the Form API Reference), and these are pretty much the same thing in another format, right?

@addtogroup means "add the following things inside a @{ @} to the group/topic", so without the {} it is meaningless.

Technically speaking, the API module treats both @ingroup and @addtogroup the same (source).

The actual interpretation of the @addtogroup syntax is far different from the current DOXYGEN spec: https://www.stack.nl/~dimitri/doxygen/manual/commands.html#cmdaddtogroup

How I read that is basically that it should behave just like if you were to use both @defgroup and @ingroup together (e.g. subtopics). This would actually eliminate the need for having to use both (which we're currently doing). That really sounds like a separate issue for the API module though. For now, I'd rather just leave it as part of this issue/proposed policy since it exists in the current grouping mechanisms already in place.

edit: created #2563453: Follow DOXYGEN syntax for how @addtogroup is supposed to work to address this

But a .md file doesn't have "doc blocks", so that means the scope is the whole file...

Yes, the entire file is the scope. We're basically doing what we did for Twig files (which normally wouldn't have docblocks either): creating parseable docblocks for Markdown files.

Actually I think @file is a bit redundant here, because you would want to require a Title line in a .md file anyway, right? So probably that would duplicate the summary line.

No, it is not redundant. The group title and file summary are two, entirely separate, things. Perhaps there is some confusion in that the @file summary is also what is used for tooltips on links (the title attribute on <a> tags). Here's a visual that might help clarify:

---

I've added most of #28.b to the issue summary, to make things a little clearer.

Moving this issue to the Coding Standards queue per the new workflow defined in #2428153: Create and document a process for updating coding standards.

The more I look into this idea, the less I like it.

The rationale behind a system such as DOXYGEN is that the documentation and the code in the same file to encourage people to keep them in sync.

I know the hard way (i.e. from projects where the documentation where separated from the code) that separation leads to documentation rot. People then tend to "forget" updating the docs when they push code patches to the repo.

Having the documentation separated from the code is bad. Having the documentation in MarkDown is even worse. The MarkDown formats (every one of them) are not really precise enough for formal documentation. MarkDown is for informal uses, such as creating a bit richer text from plain text. People who write MarkDown do it because there is no such thing as "syntax error" in MarkDown. You can't expect MarkDown users to look up the formal syntax - since MarkDown works well enough for jazz without. If you're serious about breaking out core/lib documentation into separate files, at least pick a markup format that has a less forgiving syntax and more precision, such XML.

The most incredible thing here, however, is not that you're going to use MarkDown for something it is very badly suited for (formal documentation), but that you've picked a version of MarkDown: CommonMark that is very immature project and that Drupal has no stake in or control over. The so far latest version (2016-01-12) of CommonMark is 0.24. There has been 20 different versions of CommonMark posted in the last 17 months (more than one per month). Do you really want to "standardize" on a moving target?

In addition to all of the above, the people in this thread who insist we to adopt this little-used version and immature version of the MarkDown family seems oblivious to this huge installed base of MarkDown (all based on the version of MarkDown deployed by the MarkDown Filter) - such as groups.drupal.org or README.md in the repo (rendered on screen by the Advanced Help module).

I originally didn't intend to comment on this initiative. (It looked like it was going nowhere - so why bother?)

However, it has recently come to my attention that one fan of this initiative has started to push for sitewide adoption of CommonMark, thereby disrupting other projects that builds on the existing installed base of MarkDown text.

Frankly, I cannot see CommonMark being adopted sitewide ever happening. Doing so would disrupt the huge volume of MarkDown text that is already deployed on Drupal.org and affilliated sites such as groups.drupal.org.

As long has this initiative was just going to make a mess of the core/lib docs, I didn't care much since that documentation is pretty messy already. But given that this initiative is also being "used" to disrupt other projects on Drupal.org, I think now is time to realize that this idea is not going to fly.

I move to close this issue tracker ("works as designed"). We can then start afresh - rethinking how to improve the quality of the core/lib docs.

I'm quite surprised to see this issue and #1674976: Allow README.md to optionally render as the project page are heating up and quite frankly, don't get why. Nobody's trying to impose anything. We're just trying to think about next steps and a way forward for the next 10 years, if at all possible.

When I suggested CommonMark in #26 it was because I found it to be extremely promising, like many. If Github flavored markdown was created, it was because the original Markdown implementation was no longer satisfactory. Markdown lovers are just trying to bring Markdown to the next level. A standard usually is a good way to achieve this.

Why I believe in CommonMark:

• Built and maintained by The League of Extraordinary Packages. They have plenty of top notch packages, and CommonMark is one of them. They also build every single package with the same high standards of software development
• Has implementations in many languages
• Vicent Marti (Github) is working on the spec. This means GH will very likely support both their flavored syntax AND CommonMark first and will gracefully default to CommonMark after a few years.
• It's getting frequent commits and releases. This isn't scary. It just means it's getting better and better and driving us to a better spec.

Also, let's remember why CommonMark was created:

Because there is no unambiguous spec, implementations have diverged considerably over the last 10 years. As a result, users are often surprised to find that a document that renders one way on one system (say, a GitHub wiki) renders differently on another (say, converting to docbook using Pandoc). To make matters worse, because nothing in Markdown counts as a “syntax error,” the divergence often isn’t discovered right away.

I don't know that anybody can argue with this. As such:

We propose a standard, unambiguous syntax specification for Markdown, along with a suite of comprehensive tests to validate Markdown implementations against this specification. We believe this is necessary, even essential, for the future of Markdown.

I don't want to argue with anybody and am really grateful that we have Markdown filter (started supporting CommonMark) and CommonMark to choose from already for our Drupal sites. Consensus on Markdown policy/coding standards for Markdown in code or on *.d.o sites would be even better.

I'm quite surprised to see this issue and #1674976: Allow README.md to optionally render a project page are heating up and quite frankly, don't get why.

It heated up because a different project ready to be deployed was pushed back to "Postponed" - citing this issue as grounds for postponing the project.

The argument made (for postponing) was that deploying a module to support the installed base of README.md-files (which has been been built-up long before this project started) would break the site when CommonMark was made mandatory sitewide.

Nobody's trying to impose anything.

Look again. When this person revoked the project's RTBC status I tried to get our project back on track by reverting it to RTBC. But this person immediately revoked the project's RTBC status by setting it "Postponed", citing this issue and non-compatibility with CommonMark as grounds for doing so. He was clearly trying to impose a block on our project (and will presumably continue to do so until we acknowledge that CommonMark is the only permitted MarkDown dialect on Drupal.org).

To be honest I'm not shocked that #1674976: Allow README.md to optionally render as the project page would be postponed on this issue to begin with. If we go ahead and commit #1674976: Allow README.md to optionally render as the project page first, then we would enforce a policy that might get outdated should we adopt a different policy here. I'd love for @jhodgdon to chime in and provide some guidance for us to drive this to completion, with or without CommonMark.

Also, @gisle, 'this person' has a first name. His first name is Mark. While I truly respect your passion and commitment to make the best choice here, we can better achieve this by all following the Code of Conduct and work happily together towards a common goal.

As already noted, I believe that making CommonMark our official policy and coding standards for Markdown files is undesirable for the following main reasons:

1. It breaks the installed base. MarkDown has been used by Drupal for years. The de-facto standard for MarkDown in the Drupal environment has been the dialect implemented by the MarkDown Filter module, wich has also been adopted by the Advanced Help module for rendering README.md and for comments on groups.drupal.org. The same dialect of MarkDown has been adopted for two ready to be deployed projects: #1674976: Allow README.md to optionally render as the project page and #1224506: Allow markdown on drupal.org comments (with a non-default markdown-tuned input format). The fact that you're proposing to break our installed base has not been addressed at all. I am curious about how you plan to address it.
2. CommonMark is a moving target. There has (so far) been 20 versions of CommonMark in 17 months. This is not a specification, it is an ongoing exploration. It may have a bright future, but AFAIK, all the promised goodies, including the "suite of comprehensive tests" are currently just pie in the sky. Am I scared? No, but I don't relish at the prospect of writing documentation using a syntax that gets outdated once every month on average.

(I've some minor reasons as well - but I think those two are good enough to make a case for closing this.)

So yes, I would also love to hear what Jennifer Hodgdon thinks about this idea. I've also pinged Boris Doesborg, acting chair of the DocWG.

> I move to close this issue tracker ("works as designed"). We can then start afresh - rethinking how to improve the quality of the core/lib docs.

The discussion has drifted here. An early title of this issue was: 'Replace PHP based @defgroup with Markdown (*.md) equivalents' -- so just about documentation files in Drupal core. To close this and restart that discussion in a new issue would lose a lot of work that's here.

So I would suggest we drag this issue back to its original topic, and open a new one to discuss the merits of CommonMark or Markdown standards in general.

So I would suggest we drag this issue back to its original topic, and open a new one to discuss the merits of CommonMark or Markdown standards in general.

I think the above would be an excellent way to resolve this ...

... provided the title and issue summary is revised to reflect this, and the "Postponed" status slapped on an issue that had nothing whatsoever to do with replacing "PHP based @defgroup with Markdown" is also reversed.

@markcarver - I believe it is up to you to decide what to do next.

Someone asked me to comment on this issue. However, the last few comments seem to be about some proposal that is not part of the issue summary. So please, update the issue summary and then I will be happy to comment, because right now I don't know what to comment on.

@jhodgdon wrote:

Someone asked me to comment on this issue. However, the last few comments seem to be about some proposal that is not part of the issue summary.

Glad to see you taking the time!

The sentence I specifically would like to see your comment on is this one (it is already in the issue summary):

Adopt the existing CommonMark spec as our official policy and coding standards for Markdown files. (my emphasis)

So at one point this issue warped from being about using MarkDown for documenting core and libs, and instead became an issue about a new policy to impose CommonMark as the official policy for use of MarkDown on Drupal.org (thereby breaking legacy MarkDown content).

This unfortunate warping of the issue is also evident from comments #23, #32 and #36 - which are not about core and libs, and also from @markcarver postponing deployment of ready-to-roll README.md as project pages because (I quote):

It is wrong to proceed with something that does not even have a general consensus in the community with how or what should be in markdown files, let alone what kind of markdown will be supported

So this issue, unfortunately, has warped into an discussion that have very little to do with how to document core and libs. Instead it has become a discussion about what we want to have as our "official policy and coding standards for Markdown files" for all sorts of things that use MarkDown, such as README-files and comments in issue threads and on groups.drupal.org.

I think it was unfortunate that this issue did warp into this general discussion, instead of staying focused on how to document the code in core and libs, but that is what happened and I'll stay focused on that until someone 1) narrows the scope in the issue summary to the initial scope; and 2) decouples this issue from other MarkDown issues that has nothing to with documenting code.

OK. Looking at what is currently in the issue summary in Proposed Resolution, I have the following comments:

a) I am unsure about adopting CommonMark specifically, but can't really comment on this without further investigation of alternatives.

b) I am also unsure about the idea of adding @tags to markdown. That seems to me to be a weird hybrid, that is neither PHPDoc nor Markdown. If we need @tags (and I believe that we do!), I think I would be in favor of continuing to use .api.php files rather than trying to mash them into Markdown. It doesn't seem to me that if we need to use our own "flavor" of markdown to make this viable, that there is much advantage to using Markdown, because we won't really have Markdown.

c) Looking at comment #9, which shows how an existing defgroup would be converted to a Markdown file -- I am not convinced that the Markdown version is way better than the existing PHP doc version. As far as I can tell, the idea is that the Markdown version is way easier to write or read, and that would be the only reason for doing this, right?

d) There are also downsides and development costs to doing this:
- A bunch of new code needed in the API module -- and in spite of some previous comments, I think we would need to add this to the API module itself and not have it rely on some additional contrib module, because we really do want the API module to be able to parse Core.
- We would presumably want to convert the existing topics to the new format (otherwise, why bother with this as a standard at all), and this would take some work.
- In many cases, topics are located in the same .api.php file as related hooks, or the topic defgroup is at the top of a class file. This grouping would be lost.

I don't think the benefits (which I don't really see) outweigh these costs (which would be considerable).

First, let me apologize for this being a rather long comment. There was a lot said above and I feel it's important to address most of it so there's no confusion.

---

The rationale behind a system such as DOXYGEN is that the documentation and the code in the same file to encourage people to keep them in sync.

Sure, this makes sense when there's actual code associated with documentation. When it's just a lot of verbiage discussing the overview of a particular "topic as a concept", not so much. It leads to never ending globs of same colored text as I have mentioned in #9. This kind of documentation doesn't belong in PHP DOXYGEN. It's sole purpose is to convey overarching ideas of topics, explanation of concepts as a whole and does not actually relate to specific code.

People then tend to "forget" updating the docs when they push code patches to the repo.

I am imagining that "people" here refers to developers who tend to focus on just the code and less on documentation. I can see how this may have once been a plausible argument in the past, but given the development of core gates and people, like @jhodgdon and the rest of the documentation team who's sole focus is nothing but documentation, this isn't a viable argument. Also, if we treated "Topics" (lengthy verbiage based documentation) solely as markdown files, then we'd know that a reference to said topic would exist in those files. Referencing these topics in code is still possible and quite obvious.

Having the documentation separated from the code is bad.

Agreed, we're not suggesting otherwise. This was (originally) about lengthy verbiage based documentation in PHP files. Regardless, that isn't want this issue is primarily about anymore, it merely facilitated the need for introducing the idea of markdown as documentation.

Having the documentation in MarkDown is even worse.

This is purely subjective in nature and adds nothing to the discussion about how to improve lengthy verbiage based documentation, where Markdown is clearly superior and was created and intended to "solve" since it's more "human readable".

There has been 20 different versions of CommonMark posted in the last 17 months (more than one per month). Do you really want to "standardize" on a moving target?

So having multiple succession of versions means it's "unstable"? Hardly. If anything, it proves why adopting it is necessary: they're serious about solidifying a working spec.

Also, if you actually looked at the changelog, you'll see a lot of it verbiage/term based changes to more clearly communicate the spec itself, not actual syntax changes. Nevertheless, you are again, missing the point: it's a spec... something that doesn't exist in any other "flavor" of markdown. The fact that it has a changelog is what makes it more "stable" than any other existing implementation and why adopting it is extremely beneficial for our community. It is less work that we will have to do to maintain it.

In addition to all of the above, the people in this thread who insist we to adopt this little-used version and immature version of the MarkDown family seems oblivious to this huge installed base of MarkDown (all based on the version of MarkDown deployed by the MarkDown Filter) - such as groups.drupal.org or README.md in the repo (rendered on screen by the Advanced Help module).

On the contrary. I'm very familiar with (and have used) the Markdown module. However, it's outdated solely on the fact that it implements the original PHP library. Historically, this library has received little to no active development. Newer, faster and more powerful libraries have since come out since, as I have already outlined in #25.

In case it wasn't clear in the above comments, I did not choose CommonMark. I originally wanted ParseDown. Despite this, I realized the suggestion that @Crell made (#24) was, in fact, a very important one: The PHP League developed CommonMark. They know this stuff and actually care about standards, which is something that has lacked tremendously in markdown. That's what we should really want, not glorifying an existing module just because it has existing "installs" and was the "original implementation". We should want both sustainability and growth, neither of which the original PHP library or Drupal module provide very well.

And yes, the reason the Markdown module has "installs" is because it's been around for a while. That doesn't mean, however, that it's still the right choice. Projects get replaced all the time. This isn't about "knocking it down" either. It should still remain as the "original" markdown implementation Drupal module, but here are some of the reasons it wasn't chosen:

1. It uses the original PHP library which is slow and outdated
2. Adding/changing support for any other library to this module would risk breaking existing sites (which is why a new module was created and no patch filed for the existing module)
3. It doesn't allow for much feature expansion, including plugin support.
4. It's not abstract enough. It's really just filter, nothing more. Whatever we end up using will need better facing APIs in place so it can be used by things like: the API module to parse repositories and the Project module to parse project pages, neither of which use a "filter".
5. Doesn't actually provide any useful "filter tips" for markdown syntax, just that you can "use markdown".

However, it has recently come to my attention that one fan of this initiative has started to push for sitewide adoption of CommonMark, thereby disrupting other projects that builds on the existing installed base of MarkDown text.

Not specifically "CommonMark", no. That library was chosen for the many reasons stated above. Again, you're really missing the entire point of this issue: to provide a policy/standard for Markdown as a whole in the community; regardless if it's in repositories or used on sites. It would seem to make more sense if we had a centralized place to document what kind of Markdown we, as a community, officially support and had specs to back it up.

Doing so would disrupt the huge volume of MarkDown text that is already deployed on Drupal.org and affilliated sites such as groups.drupal.org.

This argument doesn't really have much basis on truth either. Standardizing on a PHP library would obviously require supporting existing implementations, which CommonMark does. CommonMark started from the original "spec", the creator of Markdown. The reason a "spec" has been created is because there are things that were never addressed (and don't plan to be) by the original author of Markdown. CommonMark is literally just a continuation/expansion of new features on top of the original syntax. Because the Markdown module implements the original PHP library, everything that already exists on g.d.o would inherently still work.

He was clearly trying to impose a block on our project (and will presumably continue to do so until we acknowledge that CommonMark is the only permitted MarkDown dialect on Drupal.org).

No, I'm not. My goal is to discuss what policy/standards we, as a community, should adopt for Markdown. Despite having the Markdown module, there has never been any official stance (one way or the other) on how to support, maintain and implement Markdown on a *.d.o site/core. You can bluster all you want, it doesn't change this fact.

As already noted, I believe that making CommonMark our official policy and coding standards for Markdown files is undesirable for the following main reasons:

Both reasons, stated in #37 are false and have been addressed above.

The discussion has drifted here. An early title of this issue was: 'Replace PHP based @defgroup with Markdown (*.md) equivalents' -- so just about documentation files in Drupal core. To close this and restart that discussion in a new issue would lose a lot of work that's here.

Yes, that is what the "original" issue title was, but in reality... this can never happen until all of this is discussed and a policy put into place. That is why the issue was expanded/repurposed as a policy and moved to the "Coding Standards" IQ, given this scenario/use case as an example. While the OP was about improving PHP/code docs using markdown, this issue really does have much broader implications than just that. The OP was simply conveying the "idea" of using markdown files for API topic documentation, it didn't/wasn't actually implementing it. Any actionable code should be created as a follow-up to this issue.

---

In summary, I'm a little taken aback to all the above. It's clear that, while there has been no purposeful intention, offense was taken. I think it would benefit us all to recognize that we're all on the same side and attempting to solve the same issue. While I understand the desire to just "throw something in now", we really should invest in a plan that not only meets todays needs, but also future ones.

Also, for the record, I'm not just postponing issues for the sake of it. I believe it is extremely important that we, as a community, have a consensus on what, how and where we implement markdown. My participation up to this point has stemmed from a real world use case: http://drupal-bootstrap.org, the second (soon to be top) most installed theme on d.o. I've certainly been around the block once or twice, I'm not just spouting out hyperbole. I have actually implemented everything stated in the IS as a working POC. Honestly, I thought this issue was more or less done and just needed review/adoption from DTWG. Regardless, that's what this issue is all about: policy, nothing more, nothing less, so no... we shouldn't change any of this issue's meta data.

It still needs to be reviewed for what it is: policy that needs adoption by people who have actually done the research on this.

So at one point this issue warped from being about using MarkDown for documenting core and libs, and instead became an issue about a new policy to impose CommonMark as the official policy for use of MarkDown on Drupal.org (thereby breaking legacy MarkDown content).

Sigh... no, this issue isn't "warped". I think y'all are getting hung up on the fact that it's talking about PHP topic code when the IS is merely using it as an example for what policy should be implemented (CommonMark + Drupal API) and why it's needed.

We will obviously have our own "flavor" that we'd need to support for the API module which would extend the CommonMark spec we'd adopt.

This isn't a foreign concept either. While it's not widely known, we have our own "flavor" or rather a few subtle interpretations of DOXYGEN itself too (I know, I've stepped through the code).

Re #42:
I'm confused by the 180 here.

Oh, and also for the record, #2578695: Use CKEditor on *.drupal.org sites will eventually replace markdown for comments on *.d.o sites anyway... or rather there has been no mention of markdown in that discussion so it's likely not been thought of (hence the reason for implementing a community policy that all *.d.o sites should follow lol)

I am also unsure about the idea of adding @tags to markdown. That seems to me to be a weird hybrid, that is neither PHPDoc nor Markdown.

We're doing the same exact thing with Twig... using PHP DOXGEN tags in the native language's comments (which is HTML comments in Markdown).

- A bunch of new code needed in the API module -- and in spite of some previous comments, I think we would need to add this to the API module itself and not have it rely on some additional contrib module, because we really do want the API module to be able to parse Core.

Actually, it's not a lot of code, no... and I already have it working on http://drupal-bootstrap.org (https://github.com/unicorn-fail/drupal-bootstrap.org/tree/master/docroot..., has other site specific code too, but it's relatively simple). And yes, I agree it should be part of the native API module... again, this is just a working POC.

- We would presumably want to convert the existing topics to the new format (otherwise, why bother with this as a standard at all), and this would take some work.

Yep. I would imagine that this should be tackled later in a major or possibly minor release. Regardless though, it's not as daunting as it sounds. It's really just a lot of copy and paste and a few formatting issues.

- In many cases, topics are located in the same .api.php file as related hooks, or the topic defgroup is at the top of a class file. This grouping would be lost.

Not necessarily. If the topic actually pertains to code, then it makes sense to leave it. Again.... for the hundredth time: this issue is about markdown and our policy towards it, which should include syntax that gives the ability to split out very lengthy verbiage based topics (that have no code associated with them) into separate markdown files that the API module could parse.

edit:
Also, one thing that core isn't using that much is the @addtogroup tag either. An example:

Verbiage based, explains overall concept doesn't actually have code attached to it:
http://cgit.drupalcode.org/bootstrap/tree/docs/registry/README.md?h=7.x-3.x

Code/functions/methods/sub-groups that should be "added to" that group:
http://cgit.drupalcode.org/bootstrap/tree/includes/registry.inc?h=7.x-3.x

joachim wrote in #38

The discussion has drifted here. An early title of this issue was: 'Replace PHP based @defgroup with Markdown (*.md) equivalents' -- so just about documentation files in Drupal core.

markcarver responded in #43

Yes, that is what the "original" issue title was, but in reality... this can never happen until all of this is discussed and a policy put into place. That is why the issue was expanded/repurposed as a policy and moved to the "Coding Standards" IQ, given this scenario/use case as an example. While the OP was about improving PHP/code docs using markdown, this issue really does have much broader implications than just that.

Before we can make any progress on this, we must get this sorted out. I.e.:

1. Is this about improving the code standard for core/libs?
2. Or is it about replacing the current (de-facto) standard for MarkDown (resulting from the widespread adoption of the MarkDown Filter) with a different flavour of MarkDown based on the CommonMark project?

As it stands, both of these discussions seems to be wrapped up in this single issue. This seems to produce entanglements and complex implications that blocks this issue from moving forward.

For the record, I am very much in favour of splitting them into two separate issues, as suggested by joachim in #38. Splitting them should make issue 1 able to proceed smoothly to a workable solution, while issue 2 probably will result in a very long discussion before being closed without consensus being reached (i.e.: "closed - works as designed").

However, markcarver seems to insist that progress on issue 1 (creating a better code standard for core/libs):

can never happen until all of this is discussed and a policy put into place

That is bad news for people interested in just improving the coding standards for core/libs. Reaching a consensus on the a site-wide policy inpacting on all things MarkDown will be really, really difficult. While there are some people favouring CommonMark as this policy, as has been pointed out by markcarver himself, adopting CommonMark site-wide will be disruptive to legacy MarkDown because MarkDown-based content may break after CommonMark is made into policy because the format isn't the same. That, IMHO, makes it very hard to agree on this policy, as both camps seem to have strong preferences that unfortunately are mutually incompatible.

And anyway, if this issue thread is really about bringing about such a wide-ranging policy change, it is in the wrong issue queue (as this issue queue is only about coding standards).

Personally, I don't even see the point of imposing such a site-wide policy change. The MarkDown format is for sloppy people that can't really be bothered with syntax. Most people are going to muddle through just using a tiny subset common to all MarkDown dialects and the emergence of yet another dialect of MarkDown (even if sponsored by such a fine outfit as " League of Extraordinary Packages") is not going to change that. I strongly disagree with markcarver when he insists that having an "official policy" on what MarkDown variant to use is required before we can deploy projects leveraging on MarkDown (but if I'm forced to have an opinion on such a policy because it is impacts on projects I care about, it is that CommonMark should not be a candidate because it is a work in progress, not a standard).

So - marcarver, I hope you're willing to abandon the idea of imposing a sitewide policy on MarkDown, and instead refocus your efforts on improving the docs for core/libs.

If we refocus this issue thread on how to improve coding standards for core/libs, my two cents worth is that MarkDown (no matter what dialect you pick) is not a good choice for this, and the disadvantages of MarkDown far outweigh the advantages (also pointed out by jhodgdon in #42). But since I am not a core developer, it is really none of my business. Please disregard this opinion if you decide to refocus this issue thread on core/libs - my only interest here is blocking the very disruptive site-wide policy change on all things MarkDown that somehow sidetracked the original proposal here somewhere along the way.

However, if markcarver really think we need a site-wide policy on MarkDown, and that we cannot make progress on anything MarkDown-related "until all of this is discussed and a policy put into place" - then this discussion is in the wrong place (it is currently in the issue queue for coding standards). Coding standards people can of course make policy decisions about coding standard. But coding standards people don't have the mandate to make policy decisions about everything else.

To conclude: You guys really need to sort out what this very confusing issue is about (MarkDown for coding standards, or MarkDown for everything). I see no point in debating this any further without even knowing what we're debating here.

adopting CommonMark site-wide will be disruptive to legacy MarkDown because MarkDown-based content may break after CommonMark is made into policy because the format isn't the same.

THIS IS NOT TRUE.... read the flipping issue and stop making crap up.

imposing a sitewide policy on MarkDown

One cannot "impose" on something that DOES NOT CURRENTLY EXIST. This issue is about creating an OFFICIAL STANCE (not an "imposition") on how we should treat markdown as a COMMUNITY... read the flipping issue and stop making crap up.

Before we can make any progress on this, we must get this sorted out. I.e.:

1. Is this about improving the code standard for core/libs?
2. Or is it about replacing the current (de-facto) standard for MarkDown (resulting from the widespread adoption of the MarkDown Filter) with a different flavour of MarkDown based on the CommonMark project?

This issue is about #2, while ALSO keeping IN MIND the GOAL for #1.

Furthermore, the Markdown module is not the "defacto" when it's has simply been the ONLY ONE in the past. Secondly it's, again, OUTDATED. It seems to me that you are the one trying to force outdated crap on everyone and not being a team player. I've already done the RESEARCH on this topic and been a "team player". You've just come into this just spouting LIES and lengthy verbiage to do nothing but UNDERMINE the RESEARCH and WORK that has ALREADY BEEN DONE on this issue.

---

I'm sorry but I refuse to continue this "debate" any longer. It's clear that @gisle has not actually READ this issue or done ANY RESEARCH on this topic, nor has any regard for the CoC.

Use CommonMark, don't... I really do not care. I was just trying to BE A PART OF THE COMMUNITY AND HELP... instead, that somehow got twisted into someone creating a personal vendetta against me. This kind of crap is why I left core and why I'm beginning to think I cannot be involved with a community that does not actually practice community any longer. Screw this.

gisle wrote (#48):

adopting CommonMark site-wide will be disruptive to legacy MarkDown because MarkDown-based content may break after CommonMark is made into policy because the format isn't the same.

markcarver responded (#49):

THIS IS NOT TRUE.... read the flipping issue and stop making crap up.

I have read the issues and I am not making things up. But maybe you did when you used this argument to justify your postponing of our project to display legacy README.md on project pages:

These are critical questions that should be answered before something like this is implemented. What happens when that policy is adopted and existing project pages break because the the format isn't the same? (my emphasis)

Here, you clearly indicate that there is a strong possibility that going ahead with our project based upon the existing MarkDown Filter module is going to break existing project pages when your pipe-dream about a global CommonMark policy becomes reality.

(For the record: You also seems to be oblivious to what our project is about. It is not about MarkDown policy. It is a community based initiative to leverage on README.md-files that already exists in repos. That legacy of README.md have been there for years because a lot of projects make use of the Advanced Help module to convert README.md-files into on-screen help. Reusing it on project pages is not going to break anything, even if CommonMark is adopted as "policy".)

However, from the most recent quote above, it appears you've re-evaluated your position, and that your position now has changed into, quote: "THIS IS NOT TRUE".

That's great news, because that is my position as well. Maybe we can agree on this?

And I hope that if you agree, you no longer think it is necessary to Postpone other MarkDown-based initiatives until a site-wide policy about MarkDown-flavour that you (and AFAIK nobody else in this community) insist we need, is in place?

If you answer the two questions above with the affirmative, we're done.

One comment on CommonMark specifically: I just looked over the spec and it apparently has no way to create definition-style lists (DL/DT/DD in HTML tags). This seems to me to be a major oversight -- at least, we use them a lot when we write help topics.

As a note, we are using AsciiDoc (which is a similar markdown-style formatting language) in the User Guide project... that supports DL-style lists as well as (at least from a first glance) everything Commonmark supports. In fact, I think if you had some Commonmark markdown text, it would probably parse fine in AsciiDoc, although I am not sure if it would be perfect. The syntax is very similar...

Now there is a possibility that we will move to Gitlab or Github for our development and issue tracking it would be good to start up this discussion again. Both of these services are using Markdown for commenting as well as the default format for readme files and other documentation. If we end up moving to one of these solutions almost everybody in our community will get exposed to Markdown on a regular basis.

I had a look at Commonmark in relation to the Markdown flavors that are currently being used for Github and Gitlab. My first impression of Commonmark is very good. I have used Markdown extensively over the last 10 years and the standard looks very good and detailed. It addresses many of the inconsistencies that have plagued the different Markdown implementations over the years. It is also being maintained and regularly updated. They accept contributions from the community, so there is a good change to have any missing parts addressed over time.

Now I think future wise we will probably settle on Git[hu|la]b Flavored Markdown for purely practical reasons. These are both based on Commonmark. Github Flavored Markdown is essentially backwards compatible with Commonmark, with a few extensions. Gitlab in turn is inspired by Github Flavored Markdown.

This issue has stalled and I think it's primarily due to combining both the adoption and its extension in the same issue.

Perhaps it should be split into two separate issues:

1. [policy, no patch] Markdown coding standards (adopt CommonMark spec)
2. [policy, no patch] Extend Markdown coding standards (DOXYGEN/API module support)

A new issue should probably just be created for the first one as it's pretty straight forward.

Then this issue could be renamed to the latter since it's primarily about the extension of said spec.

The TWG discussed this on today's call and we would support adopting the commonmark markdown spec as the official standard.

We also think that we should move toward the now more widely used phpdoc rather than doxygen but support splitting that into a new issue.

With regard to jhodgdon's point about commonmark not supporting certain tags (DL/DT/D) - markdown has passthrough for markup tags. We can just use the HTML tags or we could add some other syntax that we like better for those tags as an extension to commonmark.

I'm personally strongly inclined to stick to markdown as opposed to asciidoc due to its ubiquity on github/gitlab/and many other places.

With regard to jhodgdon's point about commonmark not supporting certain tags (DL/DT/D) - markdown has passthrough for markup tags. We can just use the HTML tags or we could add some other syntax that we like better for those tags as an extension to commonmark.

There's some discussion about this in https://talk.commonmark.org/t/description-list/289, so when the time comes, we can see if any consensus starts to emerge there.

I like the idea of #53's suggestion to do this in steps. Are any/all of the extensions currently proposed in the issue summary necessary to agree on prior to taking the first step? Is there any progress we can make with no (or very few) extensions, and then discuss each extension (or related group of extensions) in separate issues?

I have a few questions about this proposal that are not covered in the issue summary:

a) The summary doesn't really say which (if any) documentation pieces that we are currently maintaining using PHP documentation would be converted to this new format.

b) Are the Core maintainers on board with this idea, or will we need to support both this new standard and the old one at the same time on sites like api.drupal.org?

c) Is there a time frame for converting existing docs, and how much work is involved?

d) The issue summary says we would want to support Markdown + some doxygen tags (and I think we would need this support in order to convert docs we currently have in PHP doc format into this new format). Does the Commonmark module https://www.drupal.org/project/commonmark already support these doxygen tags, and does it integrate with the API module? [If so, please update the Commonmark project page to say this, thanks!]

e) If the module doesn't support it yet, when will this feature be added? [I don't think we can really adopt this standard if it means that the documentation topics that we are currently able to display on api.drupal.org will no longer be displayed, or if the links between topics and classes/methods/etc. on api.drupal.org would go away.]

Thanks!

Regarding #55, I think the goal here is to convert some of the Topics that are currently in Core from being written in PHP doc to being written in Markdown. At least, that is what I think; the issue summary needs an update to say that explicitly (I find it to be a bit vague about what docs need to be converted, but I think that is the plan?)...

Assuming that's correct... In order not to break what we have on api.drupal.org when we do that conversion, we would need some API module integration. Basically, the API module needs to be able to recognize and parse the Markdown-formatted topics so they end up the same as they are now (meaning they are like @defgroup in PHP docs). They will need to get saved in the API module's database as topics, so that links will be made between the topics and the classes/methods/functions/etc. that have @ingroup and @addtogroup references (classes/etc. listed on the Topic page, and a link on the class/etc. page to the Topic).

I find it to be a bit vague about what docs need to be converted, but I think that is the plan?

Everything that isn't specifically tied to real code (a.k.a. topics/verbiage). We've already discussed this, at great length I might add.

we would need some API module integration

They will need to get saved in the API module's database as topics

You are more than welcome to use/extract what I have, again, already developed for https://drupal-bootstrap.org into the API module. Granted it is dependent on the CommonMark module, but that seems to be the spec/library that is finally going to be officially adopted:

https://github.com/unicorn-fail/drupal-bootstrap.org/blob/master/docroot...
https://github.com/unicorn-fail/drupal-bootstrap.org/blob/master/docroot...

Great! Can you update the issue summary please then? Thanks!

I don't have time right now to reparse and summerize what I've already written.

Marking needs work because @jhodgdon's comments in #56 do need to be addressed (not necessarily by @markcarver :-)).

For what it's worth we keep discussing this on Technical Working Group calls and would love to see this implemented. Common mark seems to be the best implemented, documented, and tested markdown implementation with clear ways to extend the syntax as needed and solid PHP support.

FWIW, both @gisle and I are now maintainers of https://www.drupal.org/project/markdown and will be merging https://www.drupal.org/project/commonmark into it.

Also, to reiterate what I said in #53, I think it makes sense to create a new issue for officially adopting the CommonMark spec (step 1) and rename this one for all the API DOXYGEN keyword specific stuff (since it'd be an extension of said spec).

Ok, I've separated out the base policy (CommonMark spec) from this issue:

#2952616: Adopt CommonMark spec for Markdown files

With #3064854: Allow Twig templates to use front matter for metadata support and https://github.com/thephpleague/commonmark/issues/442 on the horizon, I think it makes more sense to now use front matter for API module + markdown integration over my previously proposed HTML comments, e.g.:

Previous #9:

<!-- @defgroup {group_name} {group title} -->
<!-- @addtogroup group_name -->
<!-- @ingroup group_name -->
<!-- @summary description -->
# Group Title

Using front matter:

---
defgroup: {group_name} {group title}
addtogroup: group_name
ingroup: group_name
summary: description
---
# Group Title