Skip to content

code_tokenize v0.2.0
Compare
Choose a tag to compare
@cedricrupb cedricrupb released this 28 Jun 11:07
· 3 commits to main since this release
v0.2.0
2705388
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Major API redesign

code_tokenize in v0.2.0 makes now mainly use of the visitor pattern for parsing the AST

Changes

  • tokenize parses source code now by parsing the AST and traversing the AST via a visitor
  • custom tokenizing visitors can be defined per language
  • For Python, we correct the tokenization process: the indentation is now AST based computed
  • Code is extensively tested in parsing large libraries (Python and Java)
  • more languages are closer integrated
Assets 2
Loading