Graph aggregation refactoring #8082
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
7 Ignored Deployments
|
🟢 Turbopack Benchmark CI successful 🟢Thanks |
|
✅ This change can build |
|
40c155f to
759d075
Compare
7317f43 to
31eb100
Compare
eedf18b to
4d25ad5
Compare
arlyon
approved these changes
May 8, 2024
| let count = extra_followers + extra_uppers; | ||
| let target = ctx.node(target_id); | ||
| if is_in_progress(ctx, upper_id) { | ||
| drop(target); |
There was a problem hiding this comment.
What is the significance of borrowing before the branch and then dropping?
There was a problem hiding this comment.
We need to check in progress (which is an atomic) while either holding the target or the upper lock. In progress is only set during the target lock, so when we read it it need to be under the target lock. If not in progress we can continue working with the target in the else branch.
Otherwise we want to enqueue our work to the upper node. So we acquire a upper lock. In the meantime the in_progress flag might have changed, so we need to check that again.
Co-authored-by: Alexander Lyon <arlyon@me.com>
|
Can we maybe add a Markdown doc next to the code describing the overall approach (or just copy the PR description in there) |
I'll add this in a follow-up PR |
sokra
added a commit
to vercel/next.js
that referenced
this pull request
May 8, 2024
* vercel/turborepo#8082 <!-- Tobias Koppers - Graph aggregation refactoring -->
Neosoulink
pushed a commit
to Neosoulink/turbo
that referenced
this pull request
Jun 14, 2024
### Description * Deletes the aggregation tree * Adds a new graph aggregation algorithm which is more efficient The graph aggregation works as following: For the graph aggregation: Every task is a node in the graph. Every parent-child relationship is an edge in the graph. * Every node has an "aggregation number" N. * There are 2 kinds of nodes: Leaf nodes and aggregating nodes. * If a node has N < LEAF_NUMBER, it's a leaf node, otherwise an aggregating node. * A higher N for a node usually means that a larger subgraph is aggregated into that node. * Next to normal edges there are two extra kind of edges for the graph aggregation: Upper edges and follower edges. * A node is considered as "inner" to another node when it has an "upper" edge pointing towards it. * The inner node has a lower N than the upper node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * Aggregating nodes store an aggregated version of the state of all inner nodes and transitively inner nodes. * Changes in nodes are propagated to all upper nodes. * Every node has at least one upper node which is more aggregated than the node. Except for the root node of the graph, which doesn't have upper edges. * An aggregating node also has follower edges. They point to the nodes that are one normal edge after all inner and transitively inner nodes. * An leaf node doesn't have follower edges. For all purposes the normal edges of leaf nodes are considered as follower edges. * Follower nodes have a higher N than the origin node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * This means large and larger subgraphs are aggregated. * Graph operations will ensure that these invariants (Higher N on upper and follower edges) are not violated. * The N of a node can only increase. So graph operations need to "fix" the invariants by increasing N or changing upper/follower edges. That later one is preferred. N is usually only increased if two nodes have equal N. * When new edges between leaf nodes are added, the target node's N is increased to the origin node's N + 4 if it's smaller. This adds a small tolerance range so increasing N doesn't cause long chains of N += 1 between leaf nodes. ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes PACK-3036 --------- Co-authored-by: Alexander Lyon <arlyon@me.com>
ForsakenHarmony
pushed a commit
to vercel/next.js
that referenced
this pull request
Jul 25, 2024
### Description * Deletes the aggregation tree * Adds a new graph aggregation algorithm which is more efficient The graph aggregation works as following: For the graph aggregation: Every task is a node in the graph. Every parent-child relationship is an edge in the graph. * Every node has an "aggregation number" N. * There are 2 kinds of nodes: Leaf nodes and aggregating nodes. * If a node has N < LEAF_NUMBER, it's a leaf node, otherwise an aggregating node. * A higher N for a node usually means that a larger subgraph is aggregated into that node. * Next to normal edges there are two extra kind of edges for the graph aggregation: Upper edges and follower edges. * A node is considered as "inner" to another node when it has an "upper" edge pointing towards it. * The inner node has a lower N than the upper node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * Aggregating nodes store an aggregated version of the state of all inner nodes and transitively inner nodes. * Changes in nodes are propagated to all upper nodes. * Every node has at least one upper node which is more aggregated than the node. Except for the root node of the graph, which doesn't have upper edges. * An aggregating node also has follower edges. They point to the nodes that are one normal edge after all inner and transitively inner nodes. * An leaf node doesn't have follower edges. For all purposes the normal edges of leaf nodes are considered as follower edges. * Follower nodes have a higher N than the origin node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * This means large and larger subgraphs are aggregated. * Graph operations will ensure that these invariants (Higher N on upper and follower edges) are not violated. * The N of a node can only increase. So graph operations need to "fix" the invariants by increasing N or changing upper/follower edges. That later one is preferred. N is usually only increased if two nodes have equal N. * When new edges between leaf nodes are added, the target node's N is increased to the origin node's N + 4 if it's smaller. This adds a small tolerance range so increasing N doesn't cause long chains of N += 1 between leaf nodes. ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes PACK-3036 --------- Co-authored-by: Alexander Lyon <arlyon@me.com>
ForsakenHarmony
pushed a commit
to vercel/next.js
that referenced
this pull request
Jul 29, 2024
### Description * Deletes the aggregation tree * Adds a new graph aggregation algorithm which is more efficient The graph aggregation works as following: For the graph aggregation: Every task is a node in the graph. Every parent-child relationship is an edge in the graph. * Every node has an "aggregation number" N. * There are 2 kinds of nodes: Leaf nodes and aggregating nodes. * If a node has N < LEAF_NUMBER, it's a leaf node, otherwise an aggregating node. * A higher N for a node usually means that a larger subgraph is aggregated into that node. * Next to normal edges there are two extra kind of edges for the graph aggregation: Upper edges and follower edges. * A node is considered as "inner" to another node when it has an "upper" edge pointing towards it. * The inner node has a lower N than the upper node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * Aggregating nodes store an aggregated version of the state of all inner nodes and transitively inner nodes. * Changes in nodes are propagated to all upper nodes. * Every node has at least one upper node which is more aggregated than the node. Except for the root node of the graph, which doesn't have upper edges. * An aggregating node also has follower edges. They point to the nodes that are one normal edge after all inner and transitively inner nodes. * An leaf node doesn't have follower edges. For all purposes the normal edges of leaf nodes are considered as follower edges. * Follower nodes have a higher N than the origin node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * This means large and larger subgraphs are aggregated. * Graph operations will ensure that these invariants (Higher N on upper and follower edges) are not violated. * The N of a node can only increase. So graph operations need to "fix" the invariants by increasing N or changing upper/follower edges. That later one is preferred. N is usually only increased if two nodes have equal N. * When new edges between leaf nodes are added, the target node's N is increased to the origin node's N + 4 if it's smaller. This adds a small tolerance range so increasing N doesn't cause long chains of N += 1 between leaf nodes. ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes PACK-3036 --------- Co-authored-by: Alexander Lyon <arlyon@me.com>
ForsakenHarmony
pushed a commit
to vercel/next.js
that referenced
this pull request
Jul 29, 2024
### Description * Deletes the aggregation tree * Adds a new graph aggregation algorithm which is more efficient The graph aggregation works as following: For the graph aggregation: Every task is a node in the graph. Every parent-child relationship is an edge in the graph. * Every node has an "aggregation number" N. * There are 2 kinds of nodes: Leaf nodes and aggregating nodes. * If a node has N < LEAF_NUMBER, it's a leaf node, otherwise an aggregating node. * A higher N for a node usually means that a larger subgraph is aggregated into that node. * Next to normal edges there are two extra kind of edges for the graph aggregation: Upper edges and follower edges. * A node is considered as "inner" to another node when it has an "upper" edge pointing towards it. * The inner node has a lower N than the upper node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * Aggregating nodes store an aggregated version of the state of all inner nodes and transitively inner nodes. * Changes in nodes are propagated to all upper nodes. * Every node has at least one upper node which is more aggregated than the node. Except for the root node of the graph, which doesn't have upper edges. * An aggregating node also has follower edges. They point to the nodes that are one normal edge after all inner and transitively inner nodes. * An leaf node doesn't have follower edges. For all purposes the normal edges of leaf nodes are considered as follower edges. * Follower nodes have a higher N than the origin node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * This means large and larger subgraphs are aggregated. * Graph operations will ensure that these invariants (Higher N on upper and follower edges) are not violated. * The N of a node can only increase. So graph operations need to "fix" the invariants by increasing N or changing upper/follower edges. That later one is preferred. N is usually only increased if two nodes have equal N. * When new edges between leaf nodes are added, the target node's N is increased to the origin node's N + 4 if it's smaller. This adds a small tolerance range so increasing N doesn't cause long chains of N += 1 between leaf nodes. ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes PACK-3036 --------- Co-authored-by: Alexander Lyon <arlyon@me.com>
ForsakenHarmony
pushed a commit
to vercel/next.js
that referenced
this pull request
Aug 1, 2024
### Description * Deletes the aggregation tree * Adds a new graph aggregation algorithm which is more efficient The graph aggregation works as following: For the graph aggregation: Every task is a node in the graph. Every parent-child relationship is an edge in the graph. * Every node has an "aggregation number" N. * There are 2 kinds of nodes: Leaf nodes and aggregating nodes. * If a node has N < LEAF_NUMBER, it's a leaf node, otherwise an aggregating node. * A higher N for a node usually means that a larger subgraph is aggregated into that node. * Next to normal edges there are two extra kind of edges for the graph aggregation: Upper edges and follower edges. * A node is considered as "inner" to another node when it has an "upper" edge pointing towards it. * The inner node has a lower N than the upper node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * Aggregating nodes store an aggregated version of the state of all inner nodes and transitively inner nodes. * Changes in nodes are propagated to all upper nodes. * Every node has at least one upper node which is more aggregated than the node. Except for the root node of the graph, which doesn't have upper edges. * An aggregating node also has follower edges. They point to the nodes that are one normal edge after all inner and transitively inner nodes. * An leaf node doesn't have follower edges. For all purposes the normal edges of leaf nodes are considered as follower edges. * Follower nodes have a higher N than the origin node. (This invariant might be temporarily violated while tree balancing is scheduled but not executed yet) * This means large and larger subgraphs are aggregated. * Graph operations will ensure that these invariants (Higher N on upper and follower edges) are not violated. * The N of a node can only increase. So graph operations need to "fix" the invariants by increasing N or changing upper/follower edges. That later one is preferred. N is usually only increased if two nodes have equal N. * When new edges between leaf nodes are added, the target node's N is increased to the origin node's N + 4 if it's smaller. This adds a small tolerance range so increasing N doesn't cause long chains of N += 1 between leaf nodes. ### Testing Instructions <!-- Give a quick description of steps to test your changes. --> Closes PACK-3036 --------- Co-authored-by: Alexander Lyon <arlyon@me.com>
ForsakenHarmony
pushed a commit
to vercel/next.js
that referenced
this pull request
Aug 16, 2024
* vercel/turborepo#8082 <!-- Tobias Koppers - Graph aggregation refactoring -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The graph aggregation works as following:
For the graph aggregation: Every task is a node in the graph. Every parent-child relationship is an edge in the graph.
Testing Instructions
Closes PACK-3036