Support for typescript generation

[fudgepop01]

Currently, the only technologies I have knowledge of how to use on the web is javascript and typescript.

The only one that kaitai struct is able to compile into at the moment is javascript. While it's still significantly better than nothing, utilizing the generated javascript is difficult (or at least very tedious) compared to other programming languages.

I attribute this tedium to the lack of autocompletion that javascript offers. IDEs such as VSCode are unable to provide "smart" autocompletion for javascript. However, it is able to do so for typescript. This removes the necessity to memorize every little part of the structure when writing code to work with the structures.

Here is an example of what I'm talking about with autocompletion: .

the suggestion "mainHeader" would not show up if this were created in plain javascript. However, because it's a snipped of something custom I developed in typescript, it shows up. This would significantly cut-down on the development time of web-based projects utilizing Kaitai Struct's generated code.

---

TL;DR:

• GOAL:
  to generate typescript code (or at least definitions)

• HOW THIS WOULD HELP:
  It would cut down on development time of web applications utilizing Kaitai Struct.

[GreyCat]

Cc @jchv

[GreyCat]

Take a look at http://doc.kaitai.io/new_language.html, if you haven't seen it yet — it gives the overall idea of scope.

That said, probably TS target should be easy enough, being mostly either a copy-pasted clone of JavaScript target, or even an extension of thereof (as with cpp_stl_98 vs cpp_stl_11), so either way it should be relatively quick to solve.

If we could reuse existing tests for JS to some extent, that would be a great start. If you want to contribute, I'd suggest to start with creation of that test infrastructure first — i.e. at bare minimum we'd want:

• ci-typescript in tests repo, which will run all the tests and produce a JSON result file (akin to ci-javascript).
• prepare-typescript in ci_targets repo to install all the prerequisites and prepare the environment prior to running these tests (akin to prepare-javascript).
• It we'll need dedicated set of tests, we'll have to clone spec/javascript in tests repo and convert it to spec/typescript.

Once that will be done, we'll start a typescript story in CI and it will get its own column in https://ci.kaitai.io/

[jchv]

I've been looking into it already, though I am just learning Scala now, so it may be a while before I am able to get code good enough to contribute. :)

So, here's the deal. Today, Kaitai Struct generates a UMD module containing old-school prototype-based classes. We could go two directions with this:

1. We could generate .d.ts files to accompany the UMD module. This would give you all of the editor goodies you want without having to re-haul the compiler; you could implement .d.ts as its own language or something. Seems like too much of a hack for my tastes, though.

2. We could generate TypeScript directly, of course. This is the one I'm most interested in working on. It has the added benefit that mistakes in the compiler would show up immediately and could be tested with automated testing.

Right now I think the best approach here is to make TypeScript just be a special case of the JavaScript compiler. To get there, we need to modernize the output so that it can be typed. So, we need to first make the compiler support outputing ES modules and ES classes. It really wouldn't hurt to add support for other module systems as well, like CommonJS.

Adding new module systems should not be too hard. It's basically a matter of using different templates based on which one we're in. For UMD, you want a return Thing, for ES you want export default Thing; and for CommonJS you want export.modules = Thing; or so. (Whether or not to use a default export is questionable, but it is more or less the current behavior. For ES modules, it might make more sense to just export differently.) This issue is already being tracked, sort of, at #267.

ES classes are a little bit different, but it's still roughly the same idea: mostly the same code generation, with different templates. I created an issue for this at #539.

The final thing would be to add type annotations everywhere. I don't think that'll be too difficult to do, either. There might be some weird things to handle, like enums. But most things, I think, will not need to be re-hauled.

One other thing we need in order to be able to make a TypeScript target is a TypeScript runtime component. I've created a PR that adds typings (kaitai-io/kaitai_struct_javascript_runtime#10) which would be enough to get started, though it would be ideal in the end if it was just converted to TypeScript (A process that will be quite easy, but I'd like to see more testing in place before it's done, to ensure that such a swap does not cause any compatibility issues.) I think it's easy enough to see the benefit of switching it to TS, since the TypeScript compiler was already able to catch a minor bug in the code, but it's very important that it doesn't break the API or browser compatibility (within reason.)

[fudgepop01]

I agree - I would like to generate the typescript directly. I've just had trouble wrapping my head around how to get started (hence why I tried for the past 6 months to try and recreate kaitai struct my own way to no avail lol).

Even more, I'd love to add support for re-serializing the structures back into binary once modifications have been made, but that is a whole other giant thing completely. For now, yeah - I'd love to get javascript / typescript working in a modern way to get those awesome IDE goodies.

Alternatively, we could in-theory use flow, but that, like the .d.ts files feels more like a hack than an actual solution.

[jchv]

@dar2355

Even more, I'd love to add support for re-serializing the structures back into binary once modifications have been made, but that is a whole other giant thing completely. For now, yeah - I'd love to get javascript / typescript working in a modern way to get those awesome IDE goodies.

You know, it never occurred to me that Kaitai doesn't support this. It's unfortunately a huge bag of worms, though; the ksy format would probably need to go through a major iteration to handle all of the tricky cases for the encode as it does for the decode :( It would probably be hard to get compex formats like TTF working both directions.

This is totally a different bug, but a bug that should be created I think.

Alternatively, we could in-theory use flow, but that, like the .d.ts files feels more like a hack than an actual solution.

I think that even if you wanted to support Flow it'd be better to go this way, personally; ES classes and modules are typed in a manner that is pretty similar in both TypeScript and FlowType.

[FireyFly]

As a JS and occasionally Flow user, I'd like to chime in a bit. In both these languages, type definition files is how you'd usually specify types for external dependencies.

To me, since AFAIK the parser itself isn't meant to be edited by hand after generation (but rather to be imported and interfaced with, and regenerated as the grammar changes), it makes sense to take a similar approach with definition files for TypeScript and Flow. It also means there's only one core code path to maintain (generating ES5 code, so it's compatible with both new and old JS), and depending on flags provided, a TypeScript or Flow typing file could be generated adjacent to it.

I think whilst it would certainly be possible to offer ES5, modern ES, modern ES with TypeScript typing, and modern ES with Flow typing backends, to me at least that feels a bit less clean.

[GreyCat]

@jchv

This is totally a different bug, but a bug that should be created I think.

#27?

[jchv]

@GreyCat

#27

Thanks, watching that now.

@FireyFly

I think whilst it would certainly be possible to offer ES5, modern ES, modern ES with TypeScript typing, and modern ES with Flow typing backends, to me at least that feels a bit less clean.

I'm looking for end-to-end type safety. I feel for my use cases, having everything in ES modules and classes is easier because my bundler, be it Webpack or Parcel or what have you, can then handle the transpilation and module wiring, and there's no need to worry about the type definitions lagging.

If you generate .d.ts files, there's nothing preventing the generated types from being incorrect, or the generated code from misusing the types. But, generating type safe code is clearly possible since there are plenty of other statically typed languages that Kaitai can generate code for just fine. So I think the aim would be to generate strict code that is able to pass the type checkers.

FWIW, I'm not suggesting that the current output should ever be removed, just that offering more modern output would be desirable to me as an optional feature, similar to offering variations depending on C++ version. I could be wrong but I don't believe the gap between the output now and the ideal TypeScript output is actually huge, and I can sort-of prove that by doing the work manually.

I haven't used Flow in a bit, but last I checked FlowType and TypeScript are not very different syntactically. On the other hand, their type definition format is quite different, which may actually make out of band type definitions harder if you wanted to share code.

[FireyFly]

Fair enough, that seems reasonable I suppose, yeah.

[fudgepop01]

ok so I created a file format (super smash bros brawl's RSTM (music) files) and have converted it to the equivalent typescript code. This has many "implicit any"s but this should display an idea of what the target should / could look like

here's the gist

[GreyCat]

@dar2355 Could you actually elaborate what will it take to get us TS testing in the CI? Is it just "we need nodejs + some dependencies + some launcher script"? If so, you chance you can start these?

[fudgepop01]

I'll need to look closer into the proper ways of adding languages, but for the most part it seems we can just extend and modify the javascript. I'll need to look into how you go about doing the different versions of C++

I have a super busy academic week (finals are soon), but I should be able to look into it much more within the next week or two

[HiveTechDev]

Any luck with this? TS with a more complicated packet would be a real huge productivity booster.

[Theaninova]

In case anyone is interested, I've put together some rudimentary type generation for a vite plugin that allows direct import of ksy files.

https://github.com/Theaninova/vite-plugin-kaitai/blob/master/src/kaitai-type-generator.ts

[nullableVoidPtr]

Thanks @Theaninova for the PR! I've an incomplete ground-up implementation from the original work from fudgepop at this compiler fork and runtime. I've attempted to shoe in BigInt, but it seems there would need to be a lot of Kaitai Ast transforms to account for strict type checking which doesn't allow implicit BigInt <-> number coercion.
This also attempts to take advantage of TS specific things like declaration merging, but all of it is wholly untested

[Theaninova]

Thanks @Theaninova for the PR! I've an incomplete ground-up implementation from the original work from fudgepop at this compiler fork and runtime. I've attempted to shoe in BigInt, but it seems there would need to be a lot of Kaitai Ast transforms to account for strict type checking which doesn't allow implicit BigInt <-> number coercion. This also attempts to take advantage of TS specific things like declaration merging, but all of it is wholly untested

Hmm, I've got a pretty much complete version of the compiler & runtime that I've been using in this project for a few days. I wasn't aware that there was already another attempt, though I've been working around some of the type safety issues by wrapping a few things in ( as any)

[nullableVoidPtr]

It's far more involved and I actually had this on the backburner for a little under a year now. Besides the BigInt stuff there's no type errors and it's strictly typed according to deno check. Of course, runtime verification is another thing :)

[Theaninova]

Oh interesting, I'll give it a run

[fudgepop01]

(aside: all this discussion makes it clear that I really need to get a handle on my github notifications ahahah 😅 )

It's really cool that @Theaninova made a version that compiles against all tests!! If desired, I'll see if I can plug that into the vscode i'm presently updating (now with a better hex editor that actually works well with scroll wheels...) - The work you've done @nullableVoidPtr is awesome too! The base I made, while it technically worked for some files, sadly did not cover all cases.

Here's @Theaninova's PR on the compiler repo: kaitai-io/kaitai_struct_compiler#249