v1.5.0 Release and Upgrade Timeline

[jennijuju]

Note: All the time below are in EST (UTC -05:00).

If you have any concerns or question, leave a comment here.

Tentative release and upgrade timeline

• Feb 19th, v1.5.0-rc1 release
• Feb 19th, v1.5.0-rc2 release
• 18:10:00, Feb 19th, calibration net reset w/ v1.5.0-rc2 release
• ~23:10:00, Feb 20th, calibration net v3 upgrade epoch 600
• Feb 23rd, v1.5.0 release
• Mar 3rd, network v10 upgrade epoch 550321 (~19:00:30, March 3rd, 2021)

[arajasek]

v1.5.0-rc1

Lotus 1.5.0-rc1 has been released here.

[jennijuju]

v1.5.0-rc2

lotus v1.5.0-rc2 has been released here for calib configurations.

Calibration net

Calibration net has been reset @18:10, Feb 19th with lotus v1.5.2 and the v3 actor upgrade will be triggered at epoch 600(~22:10).

Simply run ‘make calibnet’ to build Lotus and connect with calibration-net!

[jennijuju]

The calibration-net has passed the actor v3 upgrade epoch and new blocks have been received in the network. You can run ./lotus state get-actor t01000 and see if it returns (fil/3/storageminer) to check if the state migration was successful. We are still monitoring the network and we will send another update later to confirm the upgrade status.

[jennijuju]

How to run a disputer?

FIP-0010 off-chain Window PoSt Verification was introduced in v3 actor to reduce the gas consumption of SubmitedWindowedPost messages by optimistically accepting Window PoSt proofs on-chain without verification, allowing them to be disputed later by off-chain verifiers. Any third party that has a lotus node may dispute onchain storage proofs that were submitted in the past 1800 epochs (~15h)  by invoking DisputeWindowedPoSt.

When a dispute successfully refutes an optimistically accepted Window PoSt, the miner is fined one IPF(invalid proof fee, 5.51BR at the moment) per active sector in the partition (at the moment when the proof was submitted) plus a flat fee of 20FIL, all incorrectly proved sectors are marked faulty, and the disputer (address that submitted the dispute) is awarded a fixed DipsuteReward (currently 4FIL).

Run a Disputer

After your node is fully synced, you can run lotus chain disputer with the following options:

• --max-fee is the max amount in FIL that you are willing to spend for a DisputeWindowedPoSt message. Keep the amount DisputeReward in mind when you set this fee.
• --from is the account you want to send the messages from.

Note:  A DisputeWindowedPoSt costs ~300M in gas.
Note: You ONLY need a lotus node to run the dispute against to, miner is not required.

Start a Window PoSt Disputer

Simply run lotus chain disputer start to start a disputer. It will start dispute proof since the epoch that your disputer is started unless you specify a --start-epoch.

Manual Dispute

You can also send a specific DisputeWindowedPoSt message by running .lotus chain disputer dispute [minerAddress] [index] [postIndex], where:

• minerAddress is the miner id that submitted the proof you want to dispute. The same address is also the recipient of the message
• index is the deadline index of the proof you want to dispute for the miner, it should be in [0, 47].
• postIndex is the post snapshot index, see the definition here.

Note: In both cases, a DisputeWindowedPoST message is only sent on chain  when the simulated dispute PoSt calls are successful locally.

[jennijuju]

v1.5.0

lotus v1.5.0 is now released. This is a consensus-breaking update, so it is NOT OPTIONAL. You must upgrade before the network v10 upgrade epoch which is set to 550321 (~19:00:30, March 3rd, 2021) or you will lose sync with the chain🚨.

Despite 1.5.0 being a consensus-breaking release, you do NOT need to delete your Lotus repo. Simply shut down your node (both miner & daemon), get the latest code, and build afresh! (See Upgrade in Place for instructions on how to upgrade safely.)

[arajasek]

🚨 TL:DR - if you are running a lotus “full-node” (with large chain history datastores) you should prepare for 4+ hrs of service disruption around tomorrow’s state upgrade!

Since we put out lotus 1.5.0, the lotus team has been doing additional benchmarking on the state migration coming in Network Upgrade 10 (happening on March 3rd, 4pm PT / March 4th 12am UTC). Through this testing, it’s come to light that the time required for the migration depends on the size of your datastore and your hardware specs - meaning that nodes with larger data stores (aka, “full-nodes” with all chain history back to mainnet ignition) are actually significantly impacted by the state migration compared to an average node.

Here are our benchmarked performance characteristics:

• Snapshot-node: Running on a 24-core AMD EPYC 7402P with 64GiB of ram, an SSD, and a 150GiB datastore: pre-migration took 8m and migration took 1m10s.
• Full-node: Running on a r5.4xlarge using a gp2 (250MB/s max io 12,000 iops), datastore 2.7TB: pre-migration took 2h10m and migration took 2h, with sustained 250MB/s reads, averaged 4000 iops (reads).
  NOTE: In the future, lotus migrations will be much lighter due to the new splitstore being introduced in hot/cold blockstore segregation (aka. splitstore) #4992 🎉

While node operators, miners and clients running nodes based on recent snapshots should be able to stay in sync throughout this migration, nodes with larger data stores should expect disruption. If you fall in this category, you have two options:

• If you are running a full-node but you have no time-sensitive actions, plan for a degraded window of 4-6 hours for your node to complete the migration and catch back up. Expect your node to fall out of sync temporarily while the migration is happening.
• If you do have time-sensitive actions (ex you’re a miner with upcoming WindowPoSts or you simply want to avoid a disruption window), please proactively spin up a new temporary snapshot-node to serve your operations during this upgrade window (until your full-node catches up). See these docs for instructions: https://docs.filecoin.io/get-started/lotus/chain/#lightweight-snapshot (note, if you’d like to switch to a snapshot instead of a full-node, please follow these upgrade instructions).

Feel free to subscribe to notifications on statuspage for any additional updates!

[jennijuju]

Monitor the migration

Simply subscribe to key words STARTING pre-migration, COMPLETED pre-migration, STARTING migrationand COMPLETED migration in your lotus daemon warning log to get migration status for your node. To confirm your node has successfully finished the migration, run lotus state get-actor f00 and Code=(fil/3/system) indicates your node has finished actor v2→v3 migration successfully!

[jennijuju]

🎉Upgrade epoch has passed!

The upgrade epoch has passed and we are observing new blocks in the network! 👍🏻 this message if your node has finished migration as well! You can check that by running lotus state get-actor f00 and Code=(fil/3/system) indicates your node has finished actor v2→v3 migration successfully!

[arajasek]

🎉 PoSts are significantly cheaper!

Now that the upgrade epoch has successfully passed, we have confirmation of the results of FIP-0010 which introduced Optimistic WindowPoSt acceptance!

The GasUsed for an arbitrary SubmitWindowedPoSt message has decreased from ~130M to 8.5M, a reduction of over 90%! This is an immense saving to the daily costs of miners, and also frees up large amounts of chain bandwidth, which allows for a lower base fee and greater chain utilization.