Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recover nodes requesting ChangeView when possible #579

Merged
merged 153 commits into from
Feb 16, 2019
Merged
Show file tree
Hide file tree
Changes from 149 commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
dbb5454
Initial cut at regeneration.
Jan 30, 2019
121eebe
Clean up.
Jan 30, 2019
ed3027c
Fix missing set of PreparationWitnessInvocationScripts.
Jan 31, 2019
96a02a9
Fix missing initialize, save and restore of PreparationWitnessInvocat…
Jan 31, 2019
f760cc8
Fix serialize/deserialize.
Jan 31, 2019
fbf3f60
Fix ConsensusContext serialize/deserialize and add unit test.
Jan 31, 2019
301f780
Add support for tracking ChangeViewWitnessInvocationScripts and use i…
Jan 31, 2019
8a78768
Primary regenerates anyone trying to change view from a lower view nu…
Jan 31, 2019
7b1b02b
Restore expected view state and ChangeViewWitnessInvocationScripts wh…
Jan 31, 2019
7cc1f92
Allow any of the CN nodes that have received the prepare request to s…
Jan 31, 2019
7a4c8df
Obtain more Perparations from regeneration message if received in the…
Jan 31, 2019
f8d5804
Remove unused using statement.
Jan 31, 2019
1efc09b
Fix missing fields when maing the RegenerationMessage.
Jan 31, 2019
fd2b08a
Rename the RegenerationMessage to RecoveryMessage.
Jan 31, 2019
b3125c5
Load context from store only when height matches
erikzhang Jan 31, 2019
cc616d1
OnChangeView only triggers recovery if requesting a lower view than t…
Jan 31, 2019
0d64749
Fix bug in RegenerateSingedPayload method.
Jan 31, 2019
f62331d
Minor adjustment so unit tests pass.
Jan 31, 2019
ba05cef
Merge branch 'consensus/improved_dbft' into consensus/regenerateOnCha…
erikzhang Jan 31, 2019
b938f96
If committed, resend the commit periodically in case of a networking …
Jan 31, 2019
50d1f9f
Whenever the commit is sent reset the timer for resending in case of …
Jan 31, 2019
d442039
Recover view even if no PrepareRequest was available in the regenerat…
Jan 31, 2019
a8bcbce
Fix copy paste error.
Jan 31, 2019
b1b72d0
Clean-up.
Jan 31, 2019
dae481d
Perform missing validity checks on regenerated payloads.
Jan 31, 2019
03fc49a
Simplify.
Jan 31, 2019
44dcace
Fix comment.
Jan 31, 2019
b411e56
minor change
erikzhang Jan 31, 2019
d270967
Handle recovering prepare responses with no prepare request available…
Feb 1, 2019
b0ae9e5
Handle ChangeViewTimestamps.
Feb 1, 2019
c7ed961
Fix prepare request timestamp of processing a regeneration message in…
Feb 1, 2019
82c80bb
Fix one more NRE.
Feb 1, 2019
0cee42f
Log sending regenerations. Avoid Exception when creating regeneration…
Feb 1, 2019
7b1b570
Fix log message.
Feb 1, 2019
f15bba6
Fix serialization issue.
Feb 1, 2019
7721314
Allow regenerating from a change view request below expected view, si…
Feb 1, 2019
f38e2fa
Allow RecoveryMessage to be received from a higher view number.
Feb 1, 2019
89cd2d4
Really allow RecoveryMessage to be received from a higher view number.
Feb 2, 2019
9ee40cf
For the tempConsensusContext for handling recovery, we have to reset …
Feb 2, 2019
11baa69
Add missing serialization / deserialization for recovery message time…
Feb 2, 2019
802dbd7
Only regenerate change view messages for the validators present in th…
Feb 2, 2019
c43beff
For Regenerating ChangeView the NewViewNumber is all that matters, we…
Feb 2, 2019
2d52a4a
Improve comment.
Feb 2, 2019
eaefde2
Fix regeneration of change view message.
Feb 2, 2019
105f0d5
Fix initializing consensus upon accepting regeneration.
Feb 2, 2019
60de007
Fix merge issue.
Feb 2, 2019
28074f3
Very minor simplification (removal of unnecessary variable).
Feb 3, 2019
183647e
Issue a ChangeView with NewViewNumber of 0 to request recovery messag…
Feb 4, 2019
9725f41
Fix unit tests.
Feb 4, 2019
17ba1eb
Improve check for node at tip of the chain to decide whether to reque…
Feb 4, 2019
95f28c0
Optimize handling the recovery to only regenerate change view message…
Feb 7, 2019
4a3eb82
Only include ChangeView info in the RecoveryMessage if necessary.
Feb 7, 2019
a9dbb6b
Add tests for serializing / deserializing RecoveryMessage. Fix Recove…
Feb 7, 2019
99c94c0
Fix bug for regenerating the prepare request when coming from a diffe…
Feb 7, 2019
d3a791b
Fix additional bugs using the wrong primary index during recovery.
Feb 7, 2019
58f18f2
Fix missing ViewNumber when regenerating prepare responses.
Feb 8, 2019
e6cc6e4
Ensure view is restored if one of the verifications in the Recovery m…
Feb 8, 2019
d7efbbc
1. Adjust who responds to recovery to be nodes plus committed nodes.…
Feb 10, 2019
7724413
Fix typo.
Feb 10, 2019
4718171
Reduce code by adding an IO.Helper method for serializing and desrial…
Feb 10, 2019
fe19284
Handle processing commit signatures from the recovery message.
Feb 11, 2019
140936f
Handle recovery message received with same view as the current view.
Feb 11, 2019
b6731a0
Fix comment. Add one final thing to complete as a TODO comment.
Feb 11, 2019
edd4c6f
Never allow nodes with CommitSent to change view. Cause Primary being…
Feb 11, 2019
af7c833
Unify code for restoring commits. Allow non-committed not to change v…
Feb 11, 2019
37ba5db
Address PR Comments involving renaming and unused constructor.
Feb 11, 2019
898a314
Compile fix.
Feb 11, 2019
a681067
Save and restore original change view numbers in recovery message.
Feb 11, 2019
8626917
Change RecoveryMessage enum definition value according to PR comment.
Feb 11, 2019
a27e3b7
Use the most common hash as the PerparationHash in the RecoveryMessage
erikzhang Feb 11, 2019
49b1ea3
Fix overriding timestamp for ChangeView to do it before the payload i…
Feb 11, 2019
388aadb
Adjust algorithm for determining recovery message responders.
Feb 11, 2019
dd010e9
One more adjustment for determining recovery message responders.
Feb 11, 2019
6bfcf30
Fix order of operations problem in determining eligible responders.
Feb 12, 2019
e0a40b9
Fix verification for M perparations as criteria for accepting change …
Feb 12, 2019
e08cf73
When generating a prepare response, we must set our our own payload w…
Feb 12, 2019
e2a58d1
Fix overwriting the prepare request witness and timestamp when primar…
Feb 12, 2019
c3dbc87
Fix checking for primary context for when accepting our own prepare r…
Feb 12, 2019
e151ef7
If double the block time goes by with commit sent, resend a recovery …
Feb 12, 2019
601f5f7
Rename variable for uniformity.
Feb 12, 2019
7e35eed
Minor clean-up to reduce nesting.
Feb 12, 2019
6f68d97
Minor adjustment to add a safeguard check.
Feb 12, 2019
63ce1f9
Re-order serialization/deserialization to match object fields for Rec…
Feb 12, 2019
d61e47c
Reduce copying when defining ConsensusContext.M.
Feb 12, 2019
4bbc4a2
Comment clean-up.
Feb 12, 2019
63ed83a
Comment clean-up.
Feb 12, 2019
8d06306
Clean ConsensusContext by storing the ChangeViewPayloads instead of i…
Feb 12, 2019
bf09e81
When Primary accepts his own PrepareRequest, set State with the Prima…
Feb 12, 2019
17b0139
Switch to storing Payloads for Preparations instead of InvocationScri…
Feb 12, 2019
d15979d
Remove Preparations array since it is no longer necessary.
Feb 13, 2019
1f262da
Remove unnecessary ExpectedView array.
Feb 13, 2019
60551b1
Fix tests.
Feb 13, 2019
382bb27
Fix setting PreparationHash after it was broken when removing Prapara…
Feb 13, 2019
73f72d5
Fix method name for WriteVarBytesArray.
Feb 13, 2019
5b28ca4
Remove commented out line that should have been removed.
Feb 13, 2019
519310a
Add consensus maximum constants MaxValidatorsCount and MaxTransaction…
Feb 13, 2019
0d22968
Move Timestamp field out of ConsensusMessage.
Feb 13, 2019
7af17ff
Support returning an empty array instead of null for ReadVarBytesArray.
Feb 13, 2019
c28f115
Clean-up.
Feb 13, 2019
a97a0fe
Fix max items supported by ReadVarBytesArray.
Feb 13, 2019
47aa1d6
Remove unused using statement.
Feb 13, 2019
e30ca86
Restore access level of ConsensusMessageType.
Feb 13, 2019
47ea907
Simplify `RecoveryMessage` (#583)
erikzhang Feb 14, 2019
c24e26d
Optimize deserialization of `ConsensusMessage`
erikzhang Feb 14, 2019
bd5a504
Fix `ChangeView.Size`
erikzhang Feb 14, 2019
4f396e3
Move `PrepareRequest.Timestamp` to the top and fix `PrepareRequest.Size`
erikzhang Feb 14, 2019
fd33496
Fix `ConsensusPayload.Size`
erikzhang Feb 14, 2019
8efb049
Move `MaxTransactionsPerBlock` from `ConsensusService` to `Block`
erikzhang Feb 14, 2019
78f70ed
Fix cast
erikzhang Feb 14, 2019
86d40c6
Remove unnecessary cast
erikzhang Feb 14, 2019
a6268dd
Remove redundant checks.
Feb 14, 2019
01f1f0e
Restore retrieving the Blockchain snapshot.
Feb 14, 2019
e1aec0d
Re-use snapshot when creating a temporary snapshot during Recovery.
Feb 14, 2019
3d293c3
Fix criteria for moving back to prevent nodes jumping views back and …
Feb 15, 2019
fb46f0c
Only reset again if necessary, and no need to pass the snapshot on th…
Feb 15, 2019
829a9d2
Remove unnecessary method ConsensusContext.TransactionExists.
Feb 15, 2019
8d1ff49
Remove unnecssary method ConsensusContext.VerifyTransactions.
Feb 15, 2019
ac61dff
Don't allow the view to move backward until there is a proof concerni…
Feb 15, 2019
590e2fb
Fix not allowing view number to move backward.
Feb 15, 2019
5b67656
Remove comemnts related to code for allowing the view to move backwar…
Feb 15, 2019
b018071
Improve the processing of `RecoveryMessage` (#584)
erikzhang Feb 15, 2019
7fbb2ba
Fix NRE.
Feb 15, 2019
ac67372
Fix deserializing the RecoveryMessage when there are ChangeView messa…
Feb 15, 2019
a0bebb9
Fix NRE.
Feb 15, 2019
5512d8a
Fix setting TimeStamp in the PrepareRequest message.
Feb 15, 2019
7b739e0
Fix potential NRE.
Feb 15, 2019
41ff33f
Improve name of the preparation payload witnesses object for clarity.
Feb 16, 2019
620bcac
File name matches class name.
Feb 16, 2019
3e6a382
Revert "File name matches class name."
Feb 16, 2019
9e74e93
Revert "Improve name of the preparation payload witnesses object for …
Feb 16, 2019
63c7cf8
Allow receiving Transactions if view changing, but receiving messages…
Feb 16, 2019
469ba70
Fix regenerating prepare request payload.
Feb 16, 2019
93d7689
Revert "Fix regenerating prepare request payload."
Feb 16, 2019
4249b66
Fix regenerating `PrepareRequest`
erikzhang Feb 16, 2019
3eee66a
If the ViewNumber of the RecoveryMessage to be constructed matches th…
erikzhang Feb 16, 2019
1072e49
Revert "If the ViewNumber of the RecoveryMessage to be constructed ma…
Feb 16, 2019
6dc60dc
It is only necessary to include M ChangeView messages in the Recovery…
Feb 16, 2019
5b2466d
Clean up a few load variable names.
Feb 16, 2019
4e2bceb
Comment clean-up.
Feb 16, 2019
c947ad0
More comment clean-up.
Feb 16, 2019
f8f8504
Make sure `CommitMessages` is not `null`
erikzhang Feb 16, 2019
cee538f
Remove unused local.
Feb 16, 2019
2b80c0a
Merge branch 'consensus/regenerateOnChangeView' of https://github.com…
Feb 16, 2019
fa7989f
Minor clean-up reducing nesting.
Feb 16, 2019
56edc43
Remove unnecessary `null` checks
erikzhang Feb 16, 2019
a031c7a
Set `ChangeViewPayloads` in `MakeChangeView()`
erikzhang Feb 16, 2019
6d88860
Remove unnecessary `null` checks
erikzhang Feb 16, 2019
3c801dc
Should not set `ResponseSent` flag if we are the primary
erikzhang Feb 16, 2019
d25a682
Change the fields order of `IConsensusContext`
erikzhang Feb 16, 2019
90ff0c5
Set `PreparationPayloads` in `MakePrepareResponse()`
erikzhang Feb 16, 2019
9926b74
Relocate methods that get payloads form the recovery message to the r…
Feb 16, 2019
2b0eb19
Check ViewChanging below.
Feb 16, 2019
58150aa
Code cleanup
erikzhang Feb 16, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions neo.UnitTests/TestBlockchain.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
using System;
using Moq;
using Neo.Cryptography.ECC;
using Neo.IO.Wrappers;
using Neo.Ledger;
using Neo.Persistence;

namespace Neo.UnitTests
{
public static class TestBlockchain
{
private static NeoSystem TheNeoSystem;


public static NeoSystem InitializeMockNeoSystem()
{
if (TheNeoSystem == null)
{
var mockSnapshot = new Mock<Snapshot>();
mockSnapshot.SetupGet(p => p.Blocks).Returns(new TestDataCache<UInt256, BlockState>());
mockSnapshot.SetupGet(p => p.Transactions).Returns(new TestDataCache<UInt256, TransactionState>());
mockSnapshot.SetupGet(p => p.Accounts).Returns(new TestDataCache<UInt160, AccountState>());
mockSnapshot.SetupGet(p => p.UnspentCoins).Returns(new TestDataCache<UInt256, UnspentCoinState>());
mockSnapshot.SetupGet(p => p.SpentCoins).Returns(new TestDataCache<UInt256, SpentCoinState>());
mockSnapshot.SetupGet(p => p.Validators).Returns(new TestDataCache<ECPoint, ValidatorState>());
mockSnapshot.SetupGet(p => p.Assets).Returns(new TestDataCache<UInt256, AssetState>());
mockSnapshot.SetupGet(p => p.Contracts).Returns(new TestDataCache<UInt160, ContractState>());
mockSnapshot.SetupGet(p => p.Storages).Returns(new TestDataCache<StorageKey, StorageItem>());
mockSnapshot.SetupGet(p => p.HeaderHashList)
.Returns(new TestDataCache<UInt32Wrapper, HeaderHashList>());
mockSnapshot.SetupGet(p => p.ValidatorsCount).Returns(new TestMetaDataCache<ValidatorsCountState>());
mockSnapshot.SetupGet(p => p.BlockHashIndex).Returns(new TestMetaDataCache<HashIndexState>());
mockSnapshot.SetupGet(p => p.HeaderHashIndex).Returns(new TestMetaDataCache<HashIndexState>());

var mockStore = new Mock<Store>();

var defaultTx = TestUtils.CreateRandomHashInvocationMockTransaction().Object;
mockStore.Setup(p => p.GetBlocks()).Returns(new TestDataCache<UInt256, BlockState>());
mockStore.Setup(p => p.GetTransactions()).Returns(new TestDataCache<UInt256, TransactionState>(
new TransactionState
{
BlockIndex = 1,
Transaction = defaultTx
}));

mockStore.Setup(p => p.GetAccounts()).Returns(new TestDataCache<UInt160, AccountState>());
mockStore.Setup(p => p.GetUnspentCoins()).Returns(new TestDataCache<UInt256, UnspentCoinState>());
mockStore.Setup(p => p.GetSpentCoins()).Returns(new TestDataCache<UInt256, SpentCoinState>());
mockStore.Setup(p => p.GetValidators()).Returns(new TestDataCache<ECPoint, ValidatorState>());
mockStore.Setup(p => p.GetAssets()).Returns(new TestDataCache<UInt256, AssetState>());
mockStore.Setup(p => p.GetContracts()).Returns(new TestDataCache<UInt160, ContractState>());
mockStore.Setup(p => p.GetStorages()).Returns(new TestDataCache<StorageKey, StorageItem>());
mockStore.Setup(p => p.GetHeaderHashList()).Returns(new TestDataCache<UInt32Wrapper, HeaderHashList>());
mockStore.Setup(p => p.GetValidatorsCount()).Returns(new TestMetaDataCache<ValidatorsCountState>());
mockStore.Setup(p => p.GetBlockHashIndex()).Returns(new TestMetaDataCache<HashIndexState>());
mockStore.Setup(p => p.GetHeaderHashIndex()).Returns(new TestMetaDataCache<HashIndexState>());
mockStore.Setup(p => p.GetSnapshot()).Returns(mockSnapshot.Object);

Console.WriteLine("initialize NeoSystem");
TheNeoSystem = new NeoSystem(mockStore.Object); // new Mock<NeoSystem>(mockStore.Object);
}

return TheNeoSystem;
}
}
}
28 changes: 28 additions & 0 deletions neo.UnitTests/TestUtils.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
using Neo.VM;
using System;
using System.Collections.Generic;
using System.IO;
using Moq;
using Neo.IO;
using Neo.Persistence;

namespace Neo.UnitTests
Expand Down Expand Up @@ -119,5 +121,31 @@ public static Mock<InvocationTransaction> CreateRandomHashInvocationMockTransact

return mockTx;
}

public static Mock<MinerTransaction> CreateRandomMockMinerTransaction()
{
var mockTx = new Mock<MinerTransaction>
{
CallBase = true
};
var tx = mockTx.Object;
tx.Attributes = new TransactionAttribute[0];
tx.Inputs = new CoinReference[0];
tx.Outputs = new TransactionOutput[0];
tx.Witnesses = new Witness[0];
tx.Nonce = (uint) TestRandom.Next();
return mockTx;
}

public static T CopyMsgBySerialization<T>(T serializableObj, T newObj) where T : ISerializable
{
using (MemoryStream ms = new MemoryStream(serializableObj.ToArray(), false))
using (BinaryReader reader = new BinaryReader(ms))
{
newObj.Deserialize(reader);
}

return newObj;
}
}
}
Loading