Warp sync part I #9227

arkpar · 2021-06-29T08:25:23Z

Closes #1208

Initial support for syncing with GRANDPA warp sync protocol. It works as follows:

Download and verify finality proofs up to the latest finalized blocks.
Download and whole state snapshot for the block that is proven to be final.
GRANDPA and BABE are reset with authority set and epoch information read from the imported state.

Still left to do in the following PR:

Support for syncing child tries.
A background process to download full block history after initial sync. This will also need to reconstruct authority set changes in sc-finality-grandpa.

Marked with B0-silent as this is still WIP and will be considered complete after the following PR.

polkadot companion: paritytech/polkadot#3382

client/finality-grandpa/src/warp_proof.rs

expenses · 2021-06-30T12:51:55Z

What's the use of warp-syncing a full node? Won't (most?) full nodes need all the blocks?

arkpar · 2021-06-30T13:53:57Z

As I've mentioned in the description, full blocks will be downloaded in the background after warp sync is complete.

rphmeier · 2021-07-09T22:32:54Z

#1208 - does this close it?

arkpar · 2021-07-10T07:52:16Z

Right, closes #1208

rphmeier · 2021-07-15T02:16:10Z

Since #1208 is the primary source of documentation for warp sync, it would be good to preserve that documentation either in-line in the code somewhere or in a wiki document. Otherwise, it will disappear and the basis for the code will be less apparent.

…sync

arkpar · 2021-07-15T16:23:56Z

Most of what's described in #1208 was not really implemented. The actual thing is buried somewhere in the middle of the discussion there.
I've added basic description to client/network/README.md in e5991c1

arkpar · 2021-07-15T16:24:10Z

@andresilva Waiting for your review

andresilva

Overall changes LGTM, just some minor nits. It would be nice to have some tests for the sync logic changes.

Sorry for the delay in reviewing 🙏.

test-utils/test-runner/src/client.rs

primitives/consensus/common/src/import_queue.rs

andresilva · 2021-07-16T16:07:33Z

client/service/src/client/client.rs

@@ -817,18 +818,6 @@ impl<B, E, Block, RA> Client<B, E, Block, RA> where
 					}
 				};

-				// ensure parent block is finalized to maintain invariant that
-				// finality is called sequentially.
-				if finalized {


Why remove this? Maybe guard it under parent_exists instead?

This is a weird hack. Why only finalize a single parent? What if their parent is not finalized as well?
Sequential finalization is already insured by the DB layer. There's ensure_sequential_finalization function that is called on block commit. I believe this code was added before ensure_sequential_finalization and is no longer needed.

apply_finality_with_block_hash was taking the tree route from the last finalized block and making sure all intermediary blocks are finalized, but yeah you're right that this is guaranteed by the DB layer now. The only thing extra that apply_finality_with_block_hash is doing is sending finality notifications for all these intermediary blocks (although we cap it to 256 blocks), I am not sure if we still should keep doing that as some subsystems might be expecting these finality notifications (@rphmeier is this needed for parachains?).

That's a good point. I'll add it back with a comment.

andresilva · 2021-07-16T16:14:14Z

client/network/src/warp_request_handler.rs

@@ -16,58 +16,57 @@



Even though there's a generic backend for generating warp proofs this is still quite coupled to GRANDPA and feels a bit wrong to add it to the generic layer of networking.

I could not think of a good place to put it. Leaving it in a stand-alone crate would introduce a circular dependency because it uses types from sc-network and sc-network is not really designed for pluggable sync algorithms. Ideally there should be a separate sc-network-api crate which only contains the network interface, but not the actual implementation. This however turned out to be a substantial refactoring outside of the scope of this PR.

We can make this more abstract in the future, there's no point blocking this PR now on this since the only consensus engine we have right now that supports Warping is GRANDPA/BABE anyway.

client/consensus/babe/src/lib.rs

andresilva · 2021-07-16T16:32:21Z

client/finality-grandpa/src/import.rs

+			Err(sp_api::ApiError::Application(_)) => {
+				// The new API is not supported in this runtime. Try reading directly from storage.
+				// This code may be removed once warp sync to an old runtime is no longer needed.
+				for prefix in ["GrandpaFinality", "Grandpa"] {


andresilva · 2021-07-16T16:33:40Z

client/finality-grandpa/src/import.rs

+				// Read authority list and set id from the state.
+				self.authority_set_hard_forks.clear();
+				let block_id = BlockId::hash(hash);
+				let authorities = self.inner.runtime_api().grandpa_authorities(&block_id)


The reason taking this from state works is because we're querying at a finalized block, maybe we should add a comment here to that effect.

client/informant/src/display.rs

client/network/src/protocol/sync.rs

rphmeier · 2021-07-17T00:51:19Z

I'm skeptical of merging anything touching sync code right now. What is our maintenance plan for this code?

arkpar · 2021-07-17T07:49:55Z

@rphmeier Could you please elaborate? What's the reason for scepticism?
The "maintenance plan" is the same as for the rest of the code base. Again, do you have reasons to believe the sync code is poorly maintained?

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

arkpar · 2021-07-17T08:44:45Z

@andresilva Tests are in the following PR #9284

arkpar · 2021-07-22T06:34:09Z

@andresilva Care to take another look?

andresilva

LGTM. I think there's nothing here that would break warp sync for smoldot.

andresilva · 2021-07-22T11:57:54Z

client/consensus/babe/src/lib.rs

@@ -1331,14 +1412,21 @@ where
 		match self.client.status(BlockId::Hash(hash)) {
 			Ok(sp_blockchain::BlockStatus::InChain) => {
 				// When re-importing existing block strip away intermediates.
-				let _ = block.take_intermediate::<BabeIntermediate<Block>>(INTERMEDIATE_KEY)?;
+				let _ = block.take_intermediate::<BabeIntermediate<Block>>(INTERMEDIATE_KEY);


IIUC you removed the ? because it doesn't really make a difference for correctness whether we can take the intermediate or not, either it's not there in the first place or has the wrong type.

Right, when reimporting an existing block we assume it has been validated the first time.

client/network/src/config.rs

andresilva · 2021-07-22T12:04:16Z

client/network/src/warp_request_handler.rs

@@ -16,58 +16,57 @@



We can make this more abstract in the future, there's no point blocking this PR now on this since the only consensus engine we have right now that supports Warping is GRANDPA/BABE anyway.

andresilva · 2021-07-22T12:10:27Z

client/service/src/client/client.rs

@@ -817,18 +818,6 @@ impl<B, E, Block, RA> Client<B, E, Block, RA> where
 					}
 				};

-				// ensure parent block is finalized to maintain invariant that
-				// finality is called sequentially.
-				if finalized {


apply_finality_with_block_hash was taking the tree route from the last finalized block and making sure all intermediary blocks are finalized, but yeah you're right that this is guaranteed by the DB layer now. The only thing extra that apply_finality_with_block_hash is doing is sending finality notifications for all these intermediary blocks (although we cap it to 256 blocks), I am not sure if we still should keep doing that as some subsystems might be expecting these finality notifications (@rphmeier is this needed for parachains?).

client/network/src/protocol/sync.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

…sync

client/consensus/babe/src/lib.rs

client/service/src/builder.rs

client/network/README.md

client/finality-grandpa/src/import.rs

client/db/src/lib.rs

client/network/src/protocol/sync.rs

bkchr · 2021-07-29T07:22:10Z

client/network/src/protocol/sync.rs

+			}
+			if let Some(request) = sync.next_warp_poof_request() {
+				for (id, peer) in self.peers.iter_mut() {
+					if peer.state.is_available() && peer.best_number >= sync.target_block_number() {


target_block_number always return 0 until we hit the phase of downloading the state proof. So, we could also send such a warp proof request to some peer that itself is still syncing?

Sounds like a bug?

Yes. Unfortunately with the current GRANDPA warp sync protocol it is not possible to learn the latest finalized block number before downloading and verifying all proofs first. In my first implementation I've set a recently announced block as target, but in general that would not work because there's no way to know if it is finalized in advance. I'll add a generic check that a peer is not too far behind the majority of the peerset instead.

bkchr · 2021-07-29T07:38:51Z

client/network/src/protocol/sync.rs

@@ -620,6 +687,17 @@ impl<B: BlockT> ChainSync<B> {
 					return Ok(None)
 				}

+				if let SyncMode::Warp = &self.mode {
+					// TODO: wait for 3 peers


I've removed the check for local testing. Will put it back.

bkchr · 2021-07-29T07:42:28Z

client/network/src/protocol/sync.rs

+			}
+			if let Some(request) = sync.next_state_request() {
+				for (id, peer) in self.peers.iter_mut() {
+					if peer.state.is_available() && peer.best_number >= sync.target_block_number() {


bkchr · 2021-07-29T07:44:36Z

client/network/src/protocol/sync.rs

+						log::debug!(target: "sync", "Starting warp state sync.");
+						if let Some(provider) = &self.warp_sync_provider {
+							self.warp_sync =
+								Some(WarpSync::new(self.client.clone(), provider.clone()));


Does that mean we will start again from zero when the node is restarted?

Yes. Proofs and state are downloaded into memory anyway. Block history download won't be restarted however, but that's coming in the following PR.

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

…warp-sync

…sync

arkpar · 2021-07-30T15:45:30Z

@cheme @bkchr @andresilva Could you please take a look at the companion PR paritytech/polkadot#3382 as well?

…sync

arkpar · 2021-08-02T08:27:12Z

bot merge

ghost · 2021-08-02T08:27:17Z

Trying merge.

* Started warp sync * BABE & GRANDPA recovery * Warp sync protocol * Sync warp proofs first * Added basic documentation * Apply suggestions from code review Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Style changes * Apply suggestions from code review Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * fmt * Apply suggestions from code review Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Fixed chage trie pruning wrt missing blocks * Restore parent finalization * fmt * fmt * Revert pwasm-utils bump * Change error type & check API version * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Build fix * Fixed target block check * Formatting Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* copying rustfmt from root to node-template build. Also, running rustfmt on this. * Update .maintain/node-template-release/src/main.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add some important events to batch & staking to ensure ease of analysis (#9462) * Add some important events to batch & staking to ensure ease of analysis * Update frame/staking/src/pallet/mod.rs Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> * Update lib.rs * fix test Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Move client consensus parts out of primitives and into client/consensus/api (#9319) * moved client code out of primitives * bump ci * Fixup from merge. * Removed unused deps thanks to review feedback * Removing unneeded deps * updating lock file * note about rustfmt * fixed typo to bump ci * Move lonely CacheKeyId to parent * cargo fmt * updating import style * Update docs/STYLE_GUIDE.md Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Refactor Benchmarks for Less Wasm Memory Usage (#9373) * extract repeat out of benchmark * remove r * unused * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * use linked map to keep order * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Delete pallet_balances.rs * Delete out * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * steps and repeat to tuple (current_*, total_*) * idea for list command * fmt * use benchmark list in cli * handle steps in cli * move log update to cli * fmt * remove old todo * line width * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * benchmark metadata function * don't need this warm up * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fix warnings * fix node-template * fix * fmt * line width * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * improve docs * improve cli * fix format * fix bug? * Revert "fix bug?" This reverts commit 8051bf1. * skip frame-metadata * extract repeat out of benchmark * remove r * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * use linked map to keep order * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Delete pallet_balances.rs * Delete out * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * steps and repeat to tuple (current_*, total_*) * idea for list command * fmt * use benchmark list in cli * handle steps in cli * move log update to cli * remove old todo * line width * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * benchmark metadata function * don't need this warm up * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fix warnings * fix node-template * fix * fmt * line width * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * improve docs * improve cli * fix format * fix bug? * Revert "fix bug?" This reverts commit 8051bf1. * skip frame-metadata * Update .gitlab-ci.yml * fix import * Update .gitlab-ci.yml Co-authored-by: Parity Benchmarking Bot <admin@parity.io> * Warp sync part I (#9227) * Started warp sync * BABE & GRANDPA recovery * Warp sync protocol * Sync warp proofs first * Added basic documentation * Apply suggestions from code review Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Style changes * Apply suggestions from code review Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * fmt * Apply suggestions from code review Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Fixed chage trie pruning wrt missing blocks * Restore parent finalization * fmt * fmt * Revert pwasm-utils bump * Change error type & check API version * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Build fix * Fixed target block check * Formatting Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add rustfmt skip to default frame benchmarking template (#9473) This was missed in the introduction pr of rustfmt. There we only had updated the Substrate local template. * CI: stop publishing to crates.io until unleash is fixed (#9474) * CI: stop publishing to crates.io until unleash is fixed; allow restarting k8s runners * CI: fix CI if ci-release tag is pushed Co-authored-by: Eric Miller <emiller@lirio.co> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Gavin Wood <gavin@parity.io> Co-authored-by: Zeke Mostov <32168567+emostov@users.noreply.github.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Squirrel <gilescope@gmail.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io> Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io>

arkpar requested review from andresilva and sorpaas as code owners June 29, 2021 08:25

arkpar commented Jun 29, 2021

View reviewed changes

client/finality-grandpa/src/warp_proof.rs Show resolved Hide resolved

arkpar mentioned this pull request Jun 29, 2021

Companion PR for warp sync support. paritytech/polkadot#3382

Merged

github-actions bot added the A7-needspolkadotpr label Jun 29, 2021

arkpar removed the A7-needspolkadotpr label Jun 29, 2021

arkpar added 4 commits June 30, 2021 10:33

Started warp sync

f41b6ff

BABE & GRANDPA recovery

dd80234

Warp sync protocol

32983ea

Sync warp proofs first

c1bd69b

arkpar force-pushed the a-warp-sync branch from bef23a5 to c1bd69b Compare June 30, 2021 08:33

arkpar mentioned this pull request Jul 6, 2021

Warp sync part II #9284

Merged

arkpar added 2 commits July 15, 2021 18:17

Merge branch 'master' of github.com:paritytech/substrate into a-warp-…

6b3c686

…sync

Added basic documentation

e5991c1

andresilva suggested changes Jul 16, 2021

View reviewed changes

Apply suggestions from code review

61ecbd5

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

andresilva approved these changes Jul 22, 2021

View reviewed changes

arkpar and others added 7 commits July 22, 2021 15:49

Apply suggestions from code review

e562fd5

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

Fixed chage trie pruning wrt missing blocks

1c27a28

Restore parent finalization

8dfd07f

fmt

fc8b9fc

Merge branch 'master' of github.com:paritytech/substrate into a-warp-…

2e3ad35

…sync

fmt

f07d173

Revert pwasm-utils bump

401b98d

bkchr reviewed Jul 29, 2021

View reviewed changes

arkpar and others added 5 commits July 29, 2021 19:01

Change error type & check API version

d95ca0e

Apply suggestions from code review

f2c9dd7

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Merge branch 'a-warp-sync' of github.com:paritytech/substrate into a-…

0569ac0

…warp-sync

Build fix

bf12779

Fixed target block check

9aea448

arkpar requested a review from bkchr July 30, 2021 06:14

bkchr approved these changes Jul 30, 2021

View reviewed changes

Merge branch 'master' of github.com:paritytech/substrate into a-warp-…

f2cde3f

…sync

arkpar added 2 commits July 30, 2021 18:23

Formatting

78de046

Merge branch 'master' of github.com:paritytech/substrate into a-warp-…

b6e8bfe

…sync

ghost merged commit c093de5 into master Aug 2, 2021

ghost deleted the a-warp-sync branch August 2, 2021 08:27

nuke-web3 mentioned this pull request Oct 29, 2021

"Warp" synch #2698

Closed

FlorianFranzen mentioned this pull request Nov 5, 2021

sc-finality-grandpa-warp-sync: Remove left-over Cargo.toml #10188

Merged

aurexav mentioned this pull request Dec 7, 2021

Update Substrate darwinia-network/darwinia-common#977

Merged

This pull request was closed.

Warp sync part I #9227

Warp sync part I #9227

Conversation

arkpar commented Jun 29, 2021 • edited by rphmeier Loading

expenses commented Jun 30, 2021

arkpar commented Jun 30, 2021

rphmeier commented Jul 9, 2021

arkpar commented Jul 10, 2021

rphmeier commented Jul 15, 2021

arkpar commented Jul 15, 2021

arkpar commented Jul 15, 2021

andresilva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkpar Jul 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkpar Jul 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier commented Jul 17, 2021

arkpar commented Jul 17, 2021

arkpar commented Jul 17, 2021

arkpar commented Jul 22, 2021

andresilva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkpar Jul 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arkpar commented Jul 30, 2021

arkpar commented Aug 2, 2021

ghost commented Aug 2, 2021

arkpar commented Jun 29, 2021 •

edited by rphmeier

Loading

arkpar Jul 17, 2021 •

edited

Loading

arkpar Jul 17, 2021 •

edited

Loading

arkpar Jul 29, 2021 •

edited

Loading