Semicolon as statement ender

[imteekay]

Changes

• Refactored some logic into separate functions: here and here
• Updated the baselines: commit
• Parsing statements until it gets to the EOF: commit
• Add more tests: commit

Approach

• Rather than parsing statements with the semicolon as the separator, we can parse all statements until it gets to the EOF.
• When parsing, if it gets to the semicolon (and it's not an empty statement**), the parser skips the semicolon and starts parsing the next statement
• I also decided to return the EOF node similar to how it works in TS: example below

source code	ast

[sandersn]

I think this function should be the one checking for semicolons, not parseStatement.
Although it would be best to pass in a predicate like separator, except named terminator or something.

[imteekay]

@sandersn Not sure if I got the exercise right this time. imteekay/mini-typescript@b74b6fc

The whole idea was

• Add a predicate function (terminator) to handle semicolons
• after a semicolon token, we can have
  • another semicolon: I considered that an empty statement (Allow var to have multiple declarations mini-typescript#2)
    • just added a peek function to see the next token and if it's indeed a semicolon, call the parse statement to get the EmptyStatement node
  • the rest of the tokens (string, number, eof): just parse a statement

Does that make sense? Maybe it can be improved for this particular case.

[sandersn]

Here's the way Typescript does it, as I recall:

const list = []
while (peek()) {
  list.push(element())
  terminator()
}

where peek returns true if the current token could be the start of a new Statement, and terminator doesn't consume a token if terminators are optional. This relies on the first token of statements to be distinguishable from from non-statements, but the only non-statement that can come after a statement list is Token.EOF, so peek is just going to be () => lexer.token !== Token.EOF

This way is a bit simpler than yours, and generalises to other kinds of lists, like parameter lists, arrays, etc.

[imteekay]

Simplified it, the implementation seems much better now! ✨ imteekay/mini-typescript@cad1fec