Proposal: smarter TS compiled file management

[kevinkassimo]

Currently whenever we compile a file that has been modified, we simply create a new file using the hash of source file name + source code + version + config. This leaves the old compiled file of previous version still in the cache.

Instead I believe it might be better to introduce a similar design of storing a meta file for each compiled file. For example, we could create a file using hash from purely the source file name

hash = source_code_hash(filename);

and create 2 separate files: ${hash}.js for the compiled code, and ${hash}.meta for metadata.
The metadata might be just a string created by hashing source code + version + config.

(If we expect user to be using different configs quite often, we can consider hashing filename + config for compiled file name and store the hash for the rest in the meta file)

Whenever we run a TS file, we could see if the compiled file exists and check if the metadata matches up. If yes, we could proceed to use this compiled file. Otherwise, we could remove the stale compiled version plus old metadata, compile and replace with new ones. This would reduce the gen/ folder size quite effectively.

(Note that this does not solve the deletion case though. If we really want to handle the deletion case, we can introduce another separate garbage-collect command that walk through the cache and validate if the origin of each file still exists. This requires storing source file pathname in metadata)

[kitsonk]

I think in general I agree with this idea. Clearly we will want/need some sort of GC of cached module meta data.

(If we expect user to be using different configs quite often, we can consider hashing filename + config for compiled file name and store the hash for the rest in the meta file)

I don't think we expect this often, but if it does change, it would cause significant issues. For example, someone decides to not use a config file initially, realises that strict mode would be a good idea, enables it, and now there is no recompilation of any of the files even though there maybe now errors that TypeScript would have caught. The same would go for if they decide to enable checkjs. So while I don't think it will occur often, a change in the configuration needs to invalidate the cached files.

[afinch7]

I think a cleanup sub command would also be a welcome addition.

[lucacasonato]

Update: the gen cache is now keyed by the filename hash not source code hash.

[kitsonk]

I don't think there is anything to do here specifically, and we should consider #1687 in general to solve any problems we currently have related to this. Closing in favour of #1687.