sealing: DataCid on workers

[magik6k]

Depends on filecoin-project/specs-storage#26

This PR adds a ComputeDataCid which will use add-piece workers to compute piece CIDs. This is mostly useful for scaling boost.

Followups:

• lotus-miner command to compute CommP of a file/url
• Integrate into fil-markets (maybe)

[codecov]

Codecov Report

Merging #8557 (43436f7) into master (48ffb15) will decrease coverage by 7.55%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##           master    #8557      +/-   ##
==========================================
- Coverage   40.86%   33.30%   -7.56%     
==========================================
  Files         686      670      -16     
  Lines       75582    74780     -802     
==========================================
- Hits        30883    24903    -5980     
- Misses      39374    45021    +5647     
+ Partials     5325     4856     -469     

Impacted Files	Coverage Δ
api/api_storage.go	0.00% <ø> (ø)
cmd/lotus-worker/main.go	0.00% <0.00%> (ø)
cmd/lotus-worker/tasks.go	27.27% <ø> (ø)
extern/sector-storage/mock/mock.go	45.13% <0.00%> (-15.07%)	⬇️
extern/sector-storage/sealtasks/task.go	47.36% <ø> (-21.06%)	⬇️
extern/sector-storage/storiface/worker.go	56.00% <ø> (-20.00%)	⬇️
extern/sector-storage/worker_local.go	53.10% <71.42%> (-7.42%)	⬇️
extern/sector-storage/ffiwrapper/sealer_cgo.go	44.96% <73.97%> (-13.83%)	⬇️
extern/sector-storage/manager.go	54.48% <87.50%> (-6.98%)	⬇️
extern/sector-storage/storiface/resources.go	58.20% <100.00%> (-17.55%)	⬇️
... and 223 more

[dirkmc]

Can you get the buffer from the throttle channel at the top of this for loop? That way you could read directly into the buffer instead of reading to an intermediate buffer and copying

[magik6k]

We could, but reading before throttling should be (very slightly) faster / more efficient, as you're not waiting with some threads for the read to complete - the data already sits in a buffer

[dirkmc]

That makes sense assuming that the io buffer is smaller than chunk.Unpadded(), which I think is correct 👍

[magik6k]

So the itest with the bigger buffer is really flaky, with something weird in rpcenc happening:

2022-04-27T20:46:10.614+0200	WARN	rpc	go-jsonrpc@v0.1.5/handler.go:279	error in RPC call to 'Filecoin.ComputeDataCid': storage call error 0: %!w(pr read error: http: invalid Read on closed Body [Hostname: ukaszs-MacBook-Pro.local])
    worker_test.go:60: 
        	Error Trace:	worker_test.go:60
        	Error:      	Received unexpected error:
        	            	storage call error 0: %!w(pr read error: http: invalid Read on closed Body [Hostname: ukaszs-MacBook-Pro.local])
        	Test:       	TestWorkerDataCid
    blockminer.go:326: shutting down mining

Which is weird, because I don't think we've ever saw this in AddPiece (but we've also never tested it in itests with a worker + bigger bit of data); Something about how we handle responses in rpcenc is probably racy.

[dirkmc]

It seems like the rpcenc error is unrelated to this change, so I'm approving 👍

Seems like it's worth looking into the rpcenc error as a follow-up