Lotus Task Benchmarks

[Reiers]

Lotus Community Benchmarks and Hardware

❗️ You don't have to fill in everything to take part of the Discussions any feedback is appreciated, thank you.

Reply card:

Sector size

32GiB	64GiB
X	X

Sealing

Sealing Pipeline	CPU	GPU	RAM	Sealing Disk	Time
AP
PC1
PC2
WaitSeed
C1
C2
RU
PRU2
WindowPost
WinningPost

Environments

export <insert here>
export <insert here>
export <insert here>
export <insert here>
export <insert here>
export <insert here>

Long Term Storage Solution

ZFS, mdadm, RAID etc.

Local Network Bandwidth

10 Gig // Fiber // Copper // Ethernet etc.

[jamerduhgamer]

Sector size

32GiB	64GiB
X

Sealing

Sealing Pipeline	CPU	GPU	RAM	Sealing Disk	Time
AP	16C	N/A	64 GB	1 TB NVMe	3 min
PC1	64C	N/A	1 TB	7 TB NVMe	3.5 hour per PC1
PC2	16C	3 x A5000	512 GB	N/A	15 min per PC2
WaitSeed	N/A	N/A	N/A	N/A	1.5 hours
C1	Done on PC2's CPU	N/A	N/A	N/A	<1 second
C2	24C	3 x A5000	1 TB	N/A	20 min per C2
RU
PRU2
WindowPost	24C	A5000	256 GB	N/A	10 - 15 min
WinningPost	24C	A5000	256 GB	N/A	1-2 seconds

Environments

export FFI_BUILD_FROM_SOURCE=1
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1
export FIL_PROOFS_USE_MULTICORE_SDR=1

Long Term Storage Solution

Some ZFS2, some ZFS3.

Local Network Bandwidth

10 Gig // 25 Gb // 40Gb // 100Gb // Fiber // Copper // Ethernet

[lanpin]

Sector size

32GiB	64GiB
X

Sealing

Sealing Pipeline	CPU	GPU	RAM	Sealing Disk	Time
AP	N/A	N/A	N/A	N/A	0
PC1	64C	N/A	2 TB	24 TB NVMe	3 hour per PC1
PC2	16C	2 x 3090	2 TB	N/A	7 min per PC2
WaitSeed	N/A	N/A	N/A	N/A	1.5 hours
C1	Done on PC2's CPU	N/A	N/A	N/A	<1 second
C2	16C	2 x 3090	2 TB	N/A	11 min per C2
RU
PRU2
WindowPost	32C	3090	512 GB	N/A	260 seconds
WinningPost	32C	3090	512 GB	N/A	1-2 seconds

AP/PC1/PC2/C1/C2 in one.

Environments

export FFI_BUILD_FROM_SOURCE=1
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1
export FIL_PROOFS_USE_MULTICORE_SDR=1

Long Term Storage Solution

RAID6

Local Network Bandwidth

10 Gig // Fiber

[donkabat]

Thx, very good results!
How many threads do you assign by AP/PC1/PC2/C2?

[GuiPulp]

Is it convenient to communicate?

[lanpin]

OK, what do you need?

[GuiPulp]

OK, what do you need?

Learn how to optimize，Tks

[lanpin]

@GuiPulp OK, how can I contact you?

[cryptowhizzard]

Sector size

32GiB

Sealing

Sealing Pipeline	CPU	GPU	RAM	Sealing Disk	Time
AP	N/A	N/A	N/A	N/A	0
PC1	64C	N/A	2 TB	24 TB NVMe	3 hour per PC1 -> 25 parallel
PC2	10C	1 x 3090	256GB	N/A	9 min per PC2
WaitSeed	N/A	N/A	N/A	N/A	1.5 hours
C1	Default	N/A	N/A	N/A	<1 second
C2	10C	1 x 3090	256 GB	N/A	10 min per C2
RU
PRU2
WindowPost	10C	3090	256 GB	N/A	480 seconds ( For a full window )
WinningPost	10C	3090	256 GB	N/A	1-2 seconds

Environments

export FFI_BUILD_FROM_SOURCE=1
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1
export FIL_PROOFS_USE_MULTICORE_SDR=1

Long Term Storage Solution

ZFS 2 + ZFS 2

Local Network Bandwidth

25 Gbit/s

Notes

We have fluctuating times on PC2/C2 as those are not bottleneck and we do powersaving on them in expensive electricity hours.

[SBudo]

Sector size

32GiB

Sealing

Sealing Pipeline	CPU	GPU	RAM	Sealing Disk	Time
AP	2x EPYC 7262	N/A	1TB	3x 3.2TB NVMe	3 mins
PC1	2x EPYC 7262	N/A	1TB	3x 3.2TB NVMe	3h46 -> 15 parallel
PC2	2x EPYC 7302	2x QUADRO RTX 8000	386GB	2x 3.2TB NVMe	12m
WaitSeed	N/A	N/A	N/A	N/A	1h15
C1	2x EPYC 7302	2x QUADRO RTX 8000	386GB	2x 3.2TB NVMe	1s
C2	2x EPYC 7302	2x QUADRO RTX 8000	386GB	2x 3.2TB NVMe	18m
RU	2x EPYC 7302	2x QUADRO RTX 8000	386GB	2x 3.2TB NVMe	TBD
PRU2	2x EPYC 7302	2x QUADRO RTX 8000	386GB	2x 3.2TB NVMe	TBD
WindowPost	1x EPYC 7302	Tesla T4	256GB	N/A	372s
WinningPost	1x EPYC 7302	Tesla T4	256GB	N/A	3s

Environments

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1
export FFI_USE_CUDA=1
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
export FIL_PROOFS_USE_GPU_TREE_BUILDER=1
export FIL_PROOFS_USE_MULTICORE_SDR=1
export FIL_PROOFS_MULTICORE_SDR_PRODUCERS=1
export BELLMAN_CUSTOM_GPU="Quadro RTX 8000:4608"

Long Term Storage Solution

Seagate Exos 5u84 iSCSI (10Gbps)

Local Network Bandwidth

10Gbit/s

[Reiers]

Sector size

32GiB	64GiB
X

Sealing

Sealing Pipeline	CPU	GPU	RAM	Sealing Disk	Time
AP	TR Pro 3995wx 64cores	-	1TB	6x 3.2TB NVMe	2m52s
PC1	5x AMD EPYC 7F32	-	1TB	6x 3.2TB NVMe	4h -> 8 Parallel
PC2	TR Pro 3995wx 64cores	GeForce RTX 3090	4TB NVMe	8m54s
WaitSeed	-
C1	TR Pro 3995wx 64cores	GeForce RTX 3090	512GB	4TB NVMe	1 s
C2	TR Pro 3995wx 64cores	GeForce RTX 3090	512GB	4TB NVMe	14m10s
RU	5950X 16cores	GeForce RTX 3090	128GB+64GB swap	4TB NVMe	6m32s
PRU2	TR Pro 3995wx 64cores	GeForce RTX 3090	512GB	4TB NVMe	16m43s
WindowPost	5950X 16cores	A5000	64GB+64GB swap	4TB NVMe	350s
WinningPost	5950X 16cores	A5000	64GB+64GB swap	4TB NVMe	1s

Environments

PATH="<->:/usr/local/cuda-11.8/bin"
LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64
BELLMAN_CUSTOM_GPU="NVIDIA GeForce RTX 3090:10496"
RUST_GPU_TOOLS_CUSTOM_GPU="NVIDIA GeForce RTX 3090:10496"
FIL_PROOFS_MAXIMIZE_CACHING=1
FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
FIL_PROOFS_USE_GPU_TREE_BUILDER=1
BELLMAN_CPU_UTILIZATION=0
FIL_PROOFS_PARAMETER_CACHE=/media/reiers/Params/32GiB
RUST_LOG=debug
FIL_PROOFS_USE_MULTICORE_SDR=1
FIL_PROOFS_MULTICORE_SDR_PRODUCERS=3
FIL_PROOFS_SDR_PARENTS_CACHE_SIZE=1073741824
LOTUS_MAX_HEAP=27002396672

Long Term Storage Solution

ZFS Raidz2 15 x 18TB for each vdev.
Storinator (45drives)

Local Network Bandwidth

10 Gig // Fiber

[Reiers]

Bench 4090

Benchmark	CPU	GPU	RAM	Sealing Disk	Time
Specs	TR Pro 3995wx 64cores	GeForce RTX 4090	256GB	2TB NVMe	below

results (v28) SectorSize:(34359738368), SectorNumber:(1)
seal: addPiece: 2m12.257025951s (247.8 MiB/s)
seal: preCommit phase 1: 2h16m44.662727985s (3.994 MiB/s)
seal: preCommit phase 2: 9m45.622324826s (55.95 MiB/s)
seal: commit phase 1: 553.269989ms (57.84 GiB/s)
seal: commit phase 2: 8m44.82447005s (62.44 MiB/s)
seal: verify: 5.870605ms
unseal: 2h16m54.08029831s  (3.989 MiB/s)

generate candidates: 5.01216ms (6.235 TiB/s)
compute winning post proof (cold): 1.240285092s
compute winning post proof (hot): 815.784814ms
verify winning post proof (cold): 33.668728ms
verify winning post proof (hot): 1.858874ms

compute window post proof (cold): 3m39.078233652s
compute window post proof (hot): 1m17.748797873s
verify window post proof (cold): 1.682232527s
verify window post proof (hot): 12.100966ms

Environments

FIL_PROOFS_MAXIMIZE_CACHING=1
FIL_PROOFS_USE_GPU_COLUMN_BUILDER=1
FIL_PROOFS_USE_GPU_TREE_BUILDER=1
FIL_PROOFS_USE_MULTICORE_SDR=1
FIL_PROOFS_MULTICORE_SDR_PRODUCERS=3
FIL_PROOFS_SDR_PARENTS_CACHE_SIZE=1073741824
LOTUS_MAX_HEAP=27002396672
BELLMAN_CUSTOM_GPU=NVIDIA GeForce RTX 4090:16384
RUST_GPU_TOOLS_CUSTOM_GPU="NVIDIA GeForce RTX 4090:16384"

Lotus version:

Daemon: 1.18.0-rc1+mainnet+git.9fde074ff+api1.5.0
Local: lotus version 1.18.0-rc1+mainnet+git.9fde074ff