feat: support markdown

[ikatyang]

Main:

• Use remark-parse with commonmark option enabled (GFM is based on commonmark)
• Commonmark Spec
• enforce
  • header: #-style
  • emphasis: _
  • strong: **
  • listItem: -(always), +(switch between two style for continuous lists)
  • listItem (ordered): 1. (always), 1)(switch between two style for continuous lists)
  • thematicBreak: - - -(always), * * * (in list)
  • indented code block: 4 whitespaces
  • fenced code block: ```(always), ~~~(in js template)
  • table: leading/trailing| and align table cell separator and respect alignment
• fill word by word (whitespace-based).
• See the formatted part of the snapshot file.

  export[xxx] = `
  
  unformatted
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  formatted
  
  `;

Side:

• extend align's n to be string available. (use for quoteblock)

Question:

• Are the styles above ok?
• Should name this parser markdown (current) or remark? Ans: markdown

NOTE:

• the content of code/html/yaml still remains the same.
• does not support break (trailing two spaces).
• remark is available on astexplorer.

TODO:

• <!-- prettier-ignore -->
• format the content of code
• escape html entities (<, >, &) html entities remain the same (<, etc.).
• pass the test case: https://raw.githubusercontent.com/adamschwartz/github-markdown-kitchen-sink/master/TEST.md
• add a special case for <!-- prettier-ignore --> not to print additional line break
• use ~~~-style code block for markdown in template string
• pass all the examples in the spec using AST_COMPARE (607/616)
• figure out a better way to escape necessary characters
• anything else?

Closes #2444

[ikatyang]

@Graham42

That's a test snapshot file, the top part is unformatted, the bottom part is formatted. 😅

Looks like:

export[xxx] = `

unformatted
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
formatted

`;

[Graham42]

O! well that explains it. That looks much prettier 💯

[azu]

I have a question.
Does this markdown formatter support multi-bye language like Japanese or emoji?

It seem that current implementaion has two problems.

First, current implementation will be broken in Chinese-Japanese-Korean(CJK) and emoji.
Because, some printing algorithm depened on string.length.

For example, "❤️".length is 2. Following algorithm jolt out of alignment.

• https://github.com/prettier/prettier/pull/2943/files#diff-385bd78c43a57ae55923bfea744a5ae3R359

This issue is known as East Asian Width or Unicode problem.

• UAX #11: East Asian Width
• JavaScript has a Unicode problem · Mathias Bynens

I know that East Asian Width problem is very difficult.( I don't know perfect solution...)
Some library get unicode length.

• timoxley/wcwidth: Port of C's wcwidth() and wcswidth()
• sindresorhus/string-width: Get the visual width of a string - the number of columns required to display it
• power-assert-runtime/packages/power-assert-util-string-width at master · twada/power-assert-runtime

Second, Following tokenizer can not tokenize non-English text.
Because, non-English text like cheniese doesn't put a space between words.

    .split(/(\s+)/g)

• https://github.com/prettier/prettier/pull/2943/files#diff-0f09e16c6ee7c5a40fef83b315027002R149

It will be resolved by using tokenizer like nlcst-parse-japanese, parse-english, and rakutenma.
But, It is not realistic and is not perfect. because toknizer is heavy weight(File size is large and Parse speed is slow)

Toknizer example is Text Tokenizer · Hivemall User Manual

Thanks.

[ikatyang]

@azu

Thanks for the suggestion, I've already thought about it and just merged #3003 to fix the printer first, the CJK support is working in progress now, but it should be in a separate PR I think, this PR is somehow too large.

I think there's no need to use tokenizer for CJK, since AFAIK they can be broke in any place, and that's how CJK books printed, e.g.

一串很長很長很長很長很長很長很長很長很長很長很長的中文字，一串很長很長很長很長很長很
長很長很長很長很長很長的中文字，一串很長很長很長很長很長很長很長很長很長很長很長的中
文字，一串很長很長很長很長很長很長很長很長很長很長很長的中文字。

For the string width thing, use string-width should be enough, but we have to disable the stripAnsi feature.

---

@azz

Do we have any concern to merge this PR? I'd like to send some followed up PRs instead of pushing to the current branch since this PR is somehow too large to review easily.

[azz]

I've been incrementally reviewing the changes so we're fine to merge!

[vjeux]

So cool!

[nhoizey]

Amazing!

Is it possible to also support Mardown dialects, such as Kramdown, used by Jekyll?

[ikatyang]

@nhoizey

It seems kramdown does not follow the commonmark spec so that it should have its own printer instead of merging into this one.

For new language support, see #3017 (comment).

[nhoizey]

@ikatyang thanks, I'll see what I can do!

[lipis]

Now we should format all *.md files in this project! :)

[lydell]

@lipis Good idea!

[lipis]

#3022 quite a few things to take care of though

[azz]

Big Data research part two: https://bigquery.cloud.google.com/savedquery/652929483875:69be015faec8444c8a2e7d055eeb32f8

* item	- item	+ item
642,106	413,627	10,954

type	regex
* item	/^\s*\*\s/gm
- item	/^\s*-\s/gm
+ item	/^\s*\+\s/gm

* item wins!

[ikatyang]

There's also the +-style list item, we should do research for it too, and take the top two results to be used in:

listItem: -(always), +(switch between two style for continuous lists)

(current)

- 123
- 123

+ 456
+ 456

<ul>
  <li>123</li>
  <li>123</li>
</ul>
<ul>
  <li>456</li>
  <li>456</li>
</ul>

[azz]

+ is not very popular. Edited my previous comment.

[revelt]

We should also limit the max length of the columns. For example, All-Contributors tables contain HTML code and table dash columns processed by Prettier get very very long.

[j-f1]

Can you open a new issue so we can track this better?

[revelt]

Done #4195