Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Garbage collection for DENO_CACHE #1687

Closed
caijw opened this issue Feb 5, 2019 · 7 comments
Closed

Garbage collection for DENO_CACHE #1687

caijw opened this issue Feb 5, 2019 · 7 comments
Labels
chore something that we should get around to eventually cli related to cli/ dir feat new feature (which has been agreed to/accepted)

Comments

@caijw
Copy link
Contributor

caijw commented Feb 5, 2019

change a ts file and recompile it , the prev xxxx.js cached file in ~/.deno/gen is still there.
I think it is some kind of files leak(deno creates files but does not clean them when they are out of date).

@ry ry added this to the future milestone Feb 5, 2019
@ry
Copy link
Member

ry commented Feb 5, 2019

Yes, there's no automatic cache purge. I've been putting it off until someone asked for it : ) so thanks for asking.

@bartlomieju
Copy link
Member

Side question: you run program with Deno, then you update Deno to newer version. Should program still run under new version without problems using old gen files? I believe I hit problem with this once, unfortunately can't reproduce it now. Of course purging cache helped the issue.

@caijw
Copy link
Contributor Author

caijw commented Feb 5, 2019

Yes, there's no automatic cache purge. I've been putting it off until someone asked for it : ) so thanks for asking.

Do you hava any idea for this issue?
I am thinking about maintaining a file gen.cache to recode the origin filenames and their current hash filenames.
When code_cache a compiled file , update gen.cache for its hash filename.
When load_cache, read gen.cache to get its hash filename, and then read the cached file.
When deno's main thread existing, make a request to clear the cached file which is not recored in gen.cache .

@kevinkassimo
Copy link
Contributor

kevinkassimo commented Feb 5, 2019

@bartlomieju The problem last time was due to native ES module upgrade. The source files were not changed so the hash keeps the same, although the compiled version is no longer runnable under new deno release. A way to fix this is to include deno version (or the last compatible version) along with the source file content for hashing.

Regarding auto-purging-on-exit, we could possibly, instead of scanning the whole gen.cache to remove things not recorded in the last step, just remove ones that just got replaced in the code_cache step. However, storing a mapping from source file to compiled file, especially stored as a file, seems would not scale very well and adds penalty on each run as the list grows longer. On the other hand, there are some problems with how to clean compiled code for files that are deleted: the entry is still in the gen.cache, just the source file used as the key is deleted/moved to another location/renamed...

Alternatively, we can introduce an extra flag that cleans the out-of-date/invalid cache, like git clean. We can use both mechanism of hash comparison and last accessed date to determine which files to remove. In this case, we also don't need to maintain a dedicated gen.cache for the mapping, instead we can add a comment in the generated code that points to source location (so that we can check for the updated hash or just verify if the file still exists).

@caijw
Copy link
Contributor Author

caijw commented Feb 5, 2019

@bartlomieju The problem last time was due to native ES module upgrade. The source files were not changed so the hash keeps the same, although the compiled version is no longer runnable under new deno release. A way to fix this is to include deno version (or the last compatible version) along with the source file content for hashing.

Regarding auto-purging-on-exit, we could possibly, instead of scanning the whole gen.cache to remove things not recorded in the last step, just remove ones that just got replaced in the code_cache step. However, storing a mapping from source file to compiled file, especially stored as a file, seems would not scale very well and adds penalty on each run as the list grows longer. On the other hand, there are some problems with how to clean compiled code for files that are deleted: the entry is still in the gen.cache, just the source file used as the key is deleted/moved to another location/renamed...

Alternatively, we can introduce an extra flag that cleans the out-of-date/invalid cache, like git clean. We can use both mechanism of hash comparison and last accessed date to determine which files to remove. In this case, we also don't need to maintain a dedicated gen.cache for the mapping, instead we can add a comment in the generated code that points to source location (so that we can check for the updated hash or just verify if the file still exists).

Adding a comment in the generated code that points to source location is a great idea!
And i think cleaning should be transparent to users.deno can run cleaning job background and the users will never know about it.

@caijw caijw changed the title Automaticcally clear out of date cached files in ~/.gen. Automaticcally clear out of date cached files in ~/.deno/gen. Feb 6, 2019
@AbmSourav
Copy link

is that clear cash feature available on 1.0.0 ?
It's a pain "removing cash from .cash " :(

@kitsonk kitsonk changed the title Automaticcally clear out of date cached files in ~/.deno/gen. Garbage collection for DENO_CACHE Nov 5, 2020
@kitsonk kitsonk added chore something that we should get around to eventually cli related to cli/ dir feat new feature (which has been agreed to/accepted) labels Nov 5, 2020
@bartlomieju bartlomieju removed this from the future milestone Feb 4, 2023
@nayeemrmn
Copy link
Collaborator

We have deno clean now, and no strategy for automatic deletion has been proposed. Closing for now

@nayeemrmn nayeemrmn closed this as not planned Won't fix, can't repro, duplicate, stale Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore something that we should get around to eventually cli related to cli/ dir feat new feature (which has been agreed to/accepted)
Projects
None yet
Development

No branches or pull requests

7 participants