Skip to content

Commit

Permalink
Update README and adapt docs
Browse files Browse the repository at this point in the history
  • Loading branch information
sogaiu committed Feb 28, 2023
1 parent 2e95127 commit 7577e43
Show file tree
Hide file tree
Showing 3 changed files with 137 additions and 134 deletions.
166 changes: 32 additions & 134 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,155 +1,53 @@
# tree-sitter-clojure

## Notice
A tree-sitter grammar for Clojure and ClojureScript

Although no major changes are anticipated at this point, there are no
guarantees. To get a heads-up before such changes occur, please
consider subscribing to the [Potential Changes Announcements
issue](https://github.com/sogaiu/tree-sitter-clojure/issues/33) to be
notified beforehand. The hope is that by commnuicating early enough
about these sorts of things, unnecessary breakage can be avoided
and/or mitigated.

## Status

tree-sitter-clojure has been:

* [Tested in various ways](doc/testing.md)
* [Used in some ways](doc/use.md)
* [Scoped for better behavior](doc/scope.md)
* [Brought about through cooperation](doc/credits.md)

## Prerequisites

Unfortunately, the short of it is that it may be a bit complicated depending on what you want to do.

* If you don't use any of the wasm-related functionality (e.g. previewing parse results in your web browser or you want to build a `.wasm` file for use in a plugin or extension), you probably just need:
* an appropriate version of node (I've tested with various versions >= 12, 14) and
* other typical development-related bits (e.g. git, appropriate c compiler, etc.)

* If you want wasm-related functionality, you get to have fun figuring out which version of [emsdk](https://emscripten.org/docs/getting_started/downloads.html#installation-instructions) currently works with tree-sitter. At the time of this writing, [this file](https://github.com/tree-sitter/tree-sitter/blob/master/cli/emscripten-version) indicates a version that might be appropriate. That may depend on precisely what the versions of other bits (e.g. tree-sitter-cli, web-tree-sitter, etc.) might be though, so if something doesn't work right away, you might consider trying [different versions that have been recorded](https://github.com/tree-sitter/tree-sitter/commits/master/emscripten-version).

Note that there may be an upside to using emsdk though -- it may figure out and arrange for an appropriate version of node, making a separate installation of node unnecessary. I don't use such a setup on a day-to-day basis, but it did work for me at least once.

## Fine Print

* The instructions below assume emsdk has been installed, but `emcc` (tool that can be used to compile to wasm) is not necessarily on one's `PATH`. If an appropriate `emcc` is on one's `PATH` (e.g. emscripten installed via homebrew), the emsdk steps (e.g. `source ~/src/emsdk/emsdk_env.sh`) below may be ignored.

* `node-gyp` (tool for compiling native addon modules for Node.js) may fail on machines upgraded to macos Catalina. [This document](https://github.com/nodejs/node-gyp/blob/master/macOS_Catalina.md) may help cope with such a situation.

## Initial Setup

Suppose typical development sources are stored under `~/src`.

### Short Version

```
# clone repository
cd ~/src
git clone https://github.com/sogaiu/tree-sitter-clojure
cd tree-sitter-clojure
# install tree-sitter-cli and dependencies, then build
npm ci
```
## What the Repository Provides

### Long Version
This repository provides some files used to create various artifacts
(e.g. dynamic libraries) used for handling Clojure and ClojureScript
source code via tree-sitter.

```
# clone repository
cd ~/src
git clone https://github.com/sogaiu/tree-sitter-clojure
cd tree-sitter-clojure
Please see the [what and why document](doc/what-and-why.md) for
detailed information.

# ensure tree-sitter-cli is avaliable as a dev dependency
npm install --save-dev --save-exact tree-sitter-cli
## Potential Changes Announcements

# create `src` and populate with tree-sitter `.c` goodness
npx tree-sitter generate
# populate `node_modules` with dependencies
npm install
# create `build` and populate appropriately
npx node-gyp configure
# create `build/Release` and build `tree_sitter_clojure_binding.node`
npx node-gyp rebuild
```

## Grammar Development

Hack on grammar.

```
# edit grammar.js using some editor
# rebuild tree-sitter stuff
npx tree-sitter generate && \
npx node-gyp rebuild
```

Parse individual files.

```
# create and populate sample code file for parsing named `sample.clj`
# parse sample file
npx tree-sitter parse sample.clj
# if output has errors, figure out what's wrong
```

Interactively test in the browser (requires emsdk).

```
# prepare emsdk (specifically emcc) for building .wasm
source ~/src/emsdk/emsdk_env.sh
# build .wasm bits and invoke web-ui for interactive testing
npx tree-sitter build-wasm && \
npx tree-sitter web-ui
# in appropriate browser window, paste code in left pane
Although no major changes are anticipated at this point, there are no
guarantees. To get a heads-up before such changes occur, please
consider subscribing to the [Potential Changes Announcements
issue](https://github.com/sogaiu/tree-sitter-clojure/issues/33) to be
notified beforehand. Also, please read the section on branches and
tags below.

# examine results in right pane -- can even click on nodes
## Branches and Tags

# if output has errors, figure out what's wrong
```
From 2023-03 we hope to proceed along the following lines.

## Measure Performance
There are two branches that are viewed as ongoing:

```
# single measurement
npx tree-sitter parse --time sample.clj
* [The release-related branch](https://github.com/sogaiu/tree-sitter-clojure/tree/master)
* [The development-related branch](https://github.com/sogaiu/tree-sitter-clojure/tree/dev)

# mutliple measurements with `multitime`
multitime -n10 -s1 npx tree-sitter parse --time --quiet sample.clj
```
Our intention with the release-related branch is for it to contain:

## Build .wasm
* what we consider tested code
* tagged commits to use for depending on

Assuming emsdk is installed appropriately under `~/src/emsdk`.
Our intention with the development-related branch is for it to be a
focus point for:

```
# prepare emsdk (specifically emcc) for use
source ~/src/emsdk/emsdk_env.sh
* things we are considering for the release-related branch (hopefully
before too long)
* folks to get an idea of where things might be going either by
inspection or by trying the code out

# create `tree-sitter-clojure.wasm`
npx tree-sitter build-wasm
```
Other branches might exist from time to time but it might be better to
view those as transient.

## Resources
## Other Documents

* [Guide to your first Tree-sitter grammar](https://gist.github.com/Aerijo/df27228d70c633e088b0591b8857eeef)
* [sublime-clojure](https://github.com/tonsky/sublime-clojure)
* [syntax-highlighter](https://github.com/EvgeniyPeshkov/syntax-highlighter)
* [tree-sitter](http://tree-sitter.github.io/tree-sitter/)
* [tree-sitter-clojure.oakmac](https://github.com/oakmac/tree-sitter-clojure)
* [tree-sitter-clojure.SergeevPavel](https://github.com/SergeevPavel/tree-sitter-clojure)
* [tree-sitter-clojure.Tavistock](https://github.com/Tavistock/tree-sitter-clojure)
* [vscode-tree-sitter](https://github.com/georgewfraser/vscode-tree-sitter)
* [web-tree-sitter API](https://github.com/tree-sitter/tree-sitter/blob/master/lib/binding_web/tree-sitter-web.d.ts)
Please see the [`doc` directory](doc/).

## Acknowledgments

Expand Down
17 changes: 17 additions & 0 deletions doc/resources.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Resources

Below is a list of resources related to tree-sitter and/or Clojure.

Some may be a bit dated at this point.

* [Guide to your first Tree-sitter
grammar](https://gist.github.com/Aerijo/df27228d70c633e088b0591b8857eeef)
* [sublime-clojure](https://github.com/tonsky/sublime-clojure)
* [syntax-highlighter](https://github.com/EvgeniyPeshkov/syntax-highlighter)
* [tree-sitter](http://tree-sitter.github.io/tree-sitter/)
* [tree-sitter-clojure.oakmac](https://github.com/oakmac/tree-sitter-clojure)
* [tree-sitter-clojure.SergeevPavel](https://github.com/SergeevPavel/tree-sitter-clojure)
* [tree-sitter-clojure.Tavistock](https://github.com/Tavistock/tree-sitter-clojure)
* [vscode-tree-sitter](https://github.com/georgewfraser/vscode-tree-sitter)
* [web-tree-sitter
API](https://github.com/tree-sitter/tree-sitter/blob/master/lib/binding_web/tree-sitter-web.d.ts)
88 changes: 88 additions & 0 deletions doc/what-and-why.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# What the Repository Provides and Why

This document describes what files and directories the repository
provides and associated reasoning. First it covers the bits that are
likely to be around in the near future. This is followed by a
description of things that may change or be removed.

One might be interested in this content out of academic curiosity but
more likely it might be because one is thinking of depending on the
repository in some way.

## What and Why

The following is a list of files and directories that are likely to
remain "provided" by the repository unless significant changes occur
elsewhere (e.g. upstream tree-sitter changes what / how they do things
or other grammar repositories decide to change certain practices).
The order is alphabetical and not meant to reflect relative
importance.

* `CHANGELOG.md` - this file contains a changelog.

* `COPYING.txt` - this file contains license information for the
repository.

* `grammar.js` - this file contains a grammar description and is used
in the process of generating parser source code that lives in `src`.
It's likely that this (or something comparable) will continue to be
provided assuming tree-sitter doesn't change the way it works.

* `package.json` - this file may be required if it's imporant to use
some of the capabilities of the `tree-sitter` cli. It's not
necessary for all subcommands though (e.g. neither the `generate`
nor `test` subcommands seem to require it). It also doesn't signify
necessary use of `npm`. Possibly contrary to what might be
indicated elsewhere, `npm` is not necessary for certain core parts
of tree-sitter grammar development. However, at the moment, an
appropriate version of `node` is required for the `generate`
subcommand to work.

* `queries` - this directory and the simple file it contains are
provided on request from `difftastic` folks. The file it contains
doesn't contain much and is not likely to be the sort of thing one
expects to be used in an editor.

* `README.md` - this file contains the repository's README content.

* `src` - this directory contains source files that are generated [1]
from `grammar.js`. The files are typically used to generate a
dynamic library / shared object that can be used by the tree-sitter
library to handle Clojure / ClojureScript source code. Although the
content of this directory is generated, the files are provided
because in practice, multiple parties have already become dependant
on them. There have been opinions voiced that this should not
remain so, but change in that direction has not been widespread
AFAICT. See
[here](https://github.com/sogaiu/ts-questions/blob/master/questions/should-parser-source-be-committed/README.md)
for more on the topic if interested.

* `test` - this directory contains test-related information. It
exists partly for tree-sitter's corpus test functionality to work,
but there is also other content that is used for other testing.

## Other Content

The rest of the content of the repository is subject to change /
removal so depending on such remaining in place is likely not a good
idea.

ATM, some of that includes:

* documentation
* development-related files and directories
* Node.js bindings
* Rust bindings

The bindings are not tested in any way and only really exist as a
side-effect of running the `tree-sitter` cli `generate` subcommand.
They may be removed at some point, but they should be straight-forward
to generate as long as one has a suitable `tree-sitter` cli and the
`grammar.js` file mentioned above.

## Footnotes

[1] If the grammar uses an external scanner, `src` may contain
non-generated files such as `scanner.c`, `scanner.cc`, etc. In the
current case, no external scanner is used and the `src` directory
content is entirely generated.

0 comments on commit 7577e43

Please sign in to comment.