Markdown Syntax and Processing

[skipjack]

Moving the discussion from here...

Aside from using consistent formatting and layouts on all webpack-contrib repositories, we really need to decide on what should be supported in terms of markdown as all the readmes are dynamically pulled into webpack.js.org to be used for a variety of pages. The biggest issue it seems is that while GitHub supports a lot of markdown extensions, they haven't open sourced there parser (as far as I know) which makes it painful for us to support the exact same feature set over at the documentation repo.

Right now we have an extremely long utility that extends the popular, but no longer maintained, marked library. We're still working our way through the backlog of content issues there but I'm really hoping that once that's more under control we can address this markdown issue and potentially switch parsers. So far, there have been a few mentioned that seem promising:

• markdown-it
• remarkable
• marksy
• remark

As mentioned above, it would be great if someone knew of a parser that supported most or all of what GitHub's does but, barring that, I think we need to...

1. Choose a customizable parser that's actively maintained
2. Decide on exactly what things we plan to support and what we should limit

This issue is more to discuss the second of those two and potentially add a .markdownlint file to this repo to enforce some restrictions on what features the webpack-contrib repositories can and can't use.

cc @bebraw @michael-ciniawsky

[sapegin]

Remark is also very good and we can use remark-lint.

[sapegin]

Marksy is using Marked inside and has a dependency of babel-standalone 😭

[skipjack]

Remark is also very good and we can use remark-lint.

@sapegin awesome, yeah remark does look like a very good option and I'm already seeing plugins that will let us replicate most of github's extensions plus a plugin that looks like it could stand-in for marksy. In terms of the linter though, I feel like it would be better to use one that's decoupled from the actual parser, like markdownlint. I guess remark-lint may be as well but just by name it seems like they may be locked together.

Marksy is using Marked inside and has a dependency of babel-standalone 😭

Yeah, it using marked is definitely one big turnoff for marksy. However I think the babel-standalone dep is only included for special cases and not by default, though I could be wrong.

[michael-ciniawsky]

markdown-it is very good in terms of plain transpiling && basic needs for extensions (the setup is less verbose in comparison to remark), if liniting is a requirement remark is the best solution out there atm (linter + plugins) imho 😛. No peronal 'experience' with remarkable and no idea if there are plugins for your needs available already, since it seems to be the less 'popluar' amongst markdown-it, remark, remarkable. Personally I would go with remark

[alexander-akait]

We use https://github.com/wooorm/remark-lint, it is plugable, fast and stable, vote for #73 (comment)

[skipjack]

Ok so it seems like we're pretty quickly coming to a consensus of using remark. Based off what I've read so far, I think that's where my vote would go as well. If that's what we end up deciding on, the next step would be to decide what the remark-lint or markdownlint configuration would look like.

As a sidenote, another nice thing to figure out would be automated line-length limitation to keep docs consistent.

[joshwiens]

@skipjack - Given you seem to have come up with a consensus on tooling, is this something you want to roll into the 2.0 milestone

[skipjack]

@d3viant0ne yeah sure or we could just close this for now and I'll come back once I've had some time to refactor our markdown processing on webpack.js.org. I already created a remark-loader for similar documentation work at my job so once webpack/webpack.js.org#1612 is in I think I'll try to start on this.

As for this repo, I guess we would be adding a .remarkrc linting configuration and incorporate that into the testing/ci process?

[joshwiens]

@skipjack - Do we want to try and make this happen in Defaults 2.0?

[skipjack]

Yeah I should have some time for this in late December, early January finally. Even if I can’t port the site in that time I could likely get a remark-lint config in place. When were you targeting for the 2.0 release?

[joshwiens]

In the case of markdown, we could really do it anytime as it's just a new feature. The initial version of defaults 2.x will be done before the end of the weekend.

[skipjack]

Sounds good, yeah definitely won't have something ready by this weekend but I'll be spending a lot of time on the site next week (12/24 - 12/29) so I could probably submit a PR then. If someone else wanted to dig into this before then, that'd be fine too. Here's the general rules I have in mind so far:

• Max line length of 80 characters.
• Always __ for bold and _ for italics.
• Always - for unordered lists.
• Two line breaks before secondary headers (##).
• Two space indentation.
• Code blocks must have a language specified.
• More to come...

Not sure if some of that is too opinionated for all READMEs but it should be fairly easy to standardize with remark and I think we can even automate fixing the minor issues (e.g. similar to ESLint's fix option).

[sapegin]

I guess Prettier would solve at least most of these issues.

[skipjack]

Ah didn't even realize Prettier also supports Markdown now. I'd be happy to integrate Prettier for all our code formatting on the webpack.js.org site if you all are comfortable with that here as well. We can just fill in the blanks with remark-lint if that's the case.

[alexander-akait]

We can combine prettier (code style) + remark (problem not related code style - duplicate headers and etc.)

[joshwiens]

@skipjack - As @evilebottnawi suggested, it's usually a matter of Prettier + something as Prettier is focused entirely on stylistic issues.

i.e. We use Prettier + eslint-config-webpack and the applicable eslint rules are disabled preventing collisions.