Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add unloading of tasks to GC #8671

Merged
merged 10 commits into from
Jul 15, 2024
Merged

add unloading of tasks to GC #8671

merged 10 commits into from
Jul 15, 2024

Conversation

sokra
Copy link
Member

@sokra sokra commented Jul 5, 2024

Description

Inactive tasks can be completely unloaded. This disconnects them from the graph and frees nearly all memory (except for the TaskId mapping).

When running GC for a task we check if it's inactive and unload it when it is.

But we also need to enqueue a task for GC again when it becomes inactive. So we maintain a queue of potentially inactive tasks and walk the children during GC.

Copy link

vercel bot commented Jul 5, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-nonmonorepo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 8, 2024 9:13am
rust-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jul 8, 2024 9:13am
8 Skipped Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-designsystem-docs ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-native-web ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-svelte-web ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview Jul 8, 2024 9:13am

@vercel vercel bot temporarily deployed to Preview – examples-nonmonorepo July 5, 2024 06:29 Inactive
@sokra sokra requested a review from a team as a code owner July 5, 2024 06:29
@sokra sokra requested review from bgw and arlyon July 5, 2024 06:30
Copy link
Contributor

github-actions bot commented Jul 5, 2024

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

Copy link
Contributor

github-actions bot commented Jul 5, 2024

🟢 Turbopack Benchmark CI successful 🟢

Thanks

Copy link
Contributor

github-actions bot commented Jul 5, 2024

⚠️ This change may fail to build next-swc.

Logs

error: failed to select a version for `either`.
    ... required by package `browserslist-rs v0.16.0`
    ... which satisfies dependency `browserslist-rs = "^0.16.0"` of package `turbopack-core v0.1.0 (https://github.com/vercel/turbo?rev=5d9a7ebf4e969938dc762008d39800f42ffe8323#d5975d43)`
    ... which satisfies git dependency `turbopack-core` of package `turbopack-binding v0.1.0 (https://github.com/vercel/turbo?rev=5d9a7ebf4e969938dc762008d39800f42ffe8323#d5975d43)`
    ... which satisfies git dependency `turbopack-binding` (locked to 0.1.0) of package `next-build-test v0.1.0 (/root/actions-runner/_work/turbo/turbo/packages/next-swc/crates/next-build-test)`
versions that meet the requirements `^1.12` are: 1.13.0, 1.12.0

all possible versions conflict with previously selected packages.

  previously selected package `either v1.10.0`
    ... which satisfies dependency `either = "^1"` (locked to 1.10.0) of package `next-custom-transforms v0.0.0 (/root/actions-runner/_work/turbo/turbo/packages/next-swc/crates/next-custom-transforms)`
    ... which satisfies path dependency `next-custom-transforms` (locked to 0.0.0) of package `next-swc-napi v0.0.0 (/root/actions-runner/_work/turbo/turbo/packages/next-swc/crates/napi)`

failed to select a version for `either` which could resolve this conflict

See job summary for details

Copy link
Contributor

@arlyon arlyon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with some comments

@sokra sokra merged commit c454e35 into main Jul 15, 2024
55 of 58 checks passed
@sokra sokra deleted the sokra/unload-task branch July 15, 2024 06:11
sokra added a commit to vercel/next.js that referenced this pull request Jul 16, 2024
* vercel/turborepo#8721 <!-- Tobias Koppers - fix
equal checks for cells and output -->
* vercel/turborepo#8671 <!-- Tobias Koppers - add
unloading of tasks to GC -->
* vercel/turborepo#8749 <!-- Tobias Koppers - fix
unloading stateful check -->
* vercel/turborepo#8750 <!-- Tobias Koppers -
upgrade deps to avoid conflict with next.js -->
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 25, 2024
### Description

Inactive tasks can be completely unloaded. This disconnects them from the graph and frees nearly all memory (except for the TaskId mapping).

When running GC for a task we check if it's inactive and unload it when it is.

But we also need to enqueue a task for GC again when it becomes inactive. So we maintain a queue of potentially inactive tasks and walk the children during GC.
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

Inactive tasks can be completely unloaded. This disconnects them from the graph and frees nearly all memory (except for the TaskId mapping).

When running GC for a task we check if it's inactive and unload it when it is.

But we also need to enqueue a task for GC again when it becomes inactive. So we maintain a queue of potentially inactive tasks and walk the children during GC.
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

Inactive tasks can be completely unloaded. This disconnects them from the graph and frees nearly all memory (except for the TaskId mapping).

When running GC for a task we check if it's inactive and unload it when it is.

But we also need to enqueue a task for GC again when it becomes inactive. So we maintain a queue of potentially inactive tasks and walk the children during GC.
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 1, 2024
### Description

Inactive tasks can be completely unloaded. This disconnects them from the graph and frees nearly all memory (except for the TaskId mapping).

When running GC for a task we check if it's inactive and unload it when it is.

But we also need to enqueue a task for GC again when it becomes inactive. So we maintain a queue of potentially inactive tasks and walk the children during GC.
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 16, 2024
* vercel/turborepo#8721 <!-- Tobias Koppers - fix
equal checks for cells and output -->
* vercel/turborepo#8671 <!-- Tobias Koppers - add
unloading of tasks to GC -->
* vercel/turborepo#8749 <!-- Tobias Koppers - fix
unloading stateful check -->
* vercel/turborepo#8750 <!-- Tobias Koppers -
upgrade deps to avoid conflict with next.js -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants