storage: splitPostApply can see tombstone for RHS #40470
Assignees
Labels
A-disaster-recovery
S-2-temp-unavailability
Temp crashes or other availability problems. Can be worked around or resolved by restarting.
Comments
|
cc @dt for triage |
|
Is this BulkIO? that stack trace like it is coming from core, no? Or does core think that IMPORT is mis-behaving? |
|
ok so you're saying this failure is not deterministic, and I was jut (un)lucky to see the panic when I passed the flag explicitly and not see the panic when I omitted the flag? If so that's a bummer because it will make the investigation a bit lengthy. |
|
@danhhz how do you think this should be triaged? |
|
Will be fixed by #39796. |
|
Actually if you haven't removed this cluster yet, could you grab the logs from all nodes? |
|
The crash here indicates that a split trigger found a tombstone for the right-hand side replica. We assert against that because the thinking is that this should never happen: we can't replicaGC a replica that is uninitialized (=has never received a snapshot), and (or so we thought) the RHS is always uninitialized before the snapshot because snapshots are blocked by the LHS until the split trigger applies. But I think there's a way in which this can be violated if the LHS comes into existence late on the crashing store. Something like this:
Some variant of this basically has to be it. Raft tombstones are only written in two places: when a range is subsumed and during replicaGC. On merges, we use math.MaxInt32 as NextReplicaID but the example shows a An alternative could work like this
I think the second scenario is more likely since @knz was running 19.2-alpha which never uses preemptive snaps. |
tbg
added a commit
to tbg/cockroach
that referenced
this issue
Sep 6, 2019
This is the next band-aid on top of cockroachdb#39658 and cockroachdb#39571. The descriptor lookup I added sometimes fails because replicas can process a split trigger in which they're not a member of the range: > F190821 15:14:28.241623 312191 storage/store.go:2172 > [n2,s2,r21/3:/{Table/54-Max}] replica descriptor of local store not > found in right hand side of split I saw this randomly in `make test PKG=./pkg/ccl/partitionccl`. There is one more bug here that is not addressed by this change, namely the fact that there might be a tombstone for the right hand side. This continues to be tracked in cockroachdb#40470. See also cockroachdb#40367 (comment) for more work we need to do to establish sanity. Release note: None
|
Also seen in import/tpcc/warehouses=1000/nodes=32 run on 9/2 (node 30's logs) in https://teamcity.cockroachdb.com/viewLog.html?buildId=1463583&buildTypeId=Cockroach_Nightlies_WorkloadNightly |
tbg
added a commit
to tbg/cockroach
that referenced
this issue
Sep 6, 2019
This is the next band-aid on top of cockroachdb#39658 and cockroachdb#39571. The descriptor lookup I added sometimes fails because replicas can process a split trigger in which they're not a member of the range: > F190821 15:14:28.241623 312191 storage/store.go:2172 > [n2,s2,r21/3:/{Table/54-Max}] replica descriptor of local store not > found in right hand side of split I saw this randomly in `make test PKG=./pkg/ccl/partitionccl`. There is one more bug here that is not addressed by this change, namely the fact that there might be a tombstone for the right hand side. This continues to be tracked in cockroachdb#40470. See also cockroachdb#40367 (comment) for more work we need to do to establish sanity. Release note: None
|
I still see this with #39796. |
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Fix crashes by preventing replica ID change.
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Fix crashes by preventing replica ID change.
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Fix crashes by preventing replica ID change.
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 23, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 24, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 24, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 24, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 24, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 24, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
ajwerner
added a commit
to ajwerner/cockroach
that referenced
this issue
Sep 24, 2019
We've seen instability recently due to invariants being violated as
replicas catch up across periods of being removed and re-added to a range.
Due to learner replicas and their rollback behavior this is now a relatively
common case. Rather than handle all of these various scenarios this PR prevents
them from occuring by actively removing replicas when we determine that they
must have been removed.
Here's a high level overview of the change:
* Once a Replica object has a non-zero Replica.mu.replicaID it will not
change.
* In this commit however, if a node crashes it may forget that it learned
about a replica ID.
* If a raft message or snapshot addressed to a higher replica ID is received
the current replica will be removed completely.
* If a replica sees a ChangeReplicasTrigger which removes it then it
completely removes itself while applying that command.
* Replica.mu.destroyStatus is used to meaningfully signify the removal state
of a Replica. Replicas about to be synchronously removed are in
destroyReasonRemovalPending.
This hopefully gives us some new invariants:
* There is only ever at most 1 Replica which IsAlive() for a range on a Store
at a time.
* Once a Replica has a non-zero ReplicaID is never changes.
* This applies only to the in-memory object, not the store itself.
* Once a Replica applies a command as a part of the range descriptor it will
never apply another command as a different Replica ID or outside of the
Range.
* Corrolary: a Replica created as a learner will only ever apply commands
while that replica is in the range.
The change also introduces some new complexity. Namely we now allow removal of
uninitialized replicas, including their hard state. This allows us to catch up
across a split even when we know the RHS must have been removed.
Fixes cockroachdb#40367.
Issue cockroachdb#38772 (comment)
manifests itself as the RHS not being found for a merge. This happens because
the Replica is processing commands to catch itself up while it is not in the
range. This is no longer possible.
Fixes cockroachdb#40257.
Issue cockroachdb#40257 is another case of a replica processing commands while it is not
in the range.
Fixes cockroachdb#40470.
Issue cockroachdb#40470 is caused by a RHS learning about its existence and removal
prior to a LHS processing a split. This case is now handled properly and is
tested.
Release justification: This commit is safe for 19.2 because it fixes release
blockers.
Release note (bug fix): Avoid internal re-use of Replica objects to fix the following crashes:
cockroachdb#38772 "found rXXX:{-} [, next=0, gen=0?] in place of the RHS"
cockroachdb#39796 "replica descriptor of local store not found in right hand side of split"
cockroachdb#40470 "split trigger found right-hand side with tombstone"
cockroachdb#40257 "snapshot widens existing replica, but no replica exists for subsumed key"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem
node fails with
To Reproduce
roachprod create knz-gce -u knz -c gce --geo -n 10roachprod start $CLUSTER:1,4,7-8roachprod run $CLUSTER:1 -- "./cockroach workload fixtures import tpcc --warehouses=5000 --db=tpcc --experimental-direct-ingestion"This fails within 1-2 minutes.
Relevant log lines:
Expected behavior
Import succeeds
Context
The text was updated successfully, but these errors were encountered: