Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support standalone miner-market process #6356

Merged
merged 106 commits into from
Jul 13, 2021
Merged
Show file tree
Hide file tree
Changes from 98 commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
a989f60
add SectorAddPieceToAny and SectorUnsealPiece to StorageMiner iface; …
nonsense May 19, 2021
87ed228
extract pushUrl and fix return type for worker RPC v0
nonsense May 19, 2021
e275b54
export PartialFile and OpenPartialFile
nonsense May 19, 2021
5540111
add UnpaddedByteIndex.Valid()
nonsense May 19, 2021
83e55dc
move handleProvingSector to correct file - states_proving.go
nonsense May 19, 2021
1295c92
confirm that Miner struct satisfies sectorblocks.SectorBuilder
nonsense May 19, 2021
2aad7b6
update sectorstorage.New (Manager) interface
nonsense May 20, 2021
2562f2e
add ReaderParamDecoder to json rpc server to lotus-miner run
nonsense May 20, 2021
c12d802
update reader to use ReadAtLeast
nonsense May 20, 2021
cb603c6
update retrievaladapter ; add piece_provider
nonsense May 20, 2021
76bb424
update StorageMinerAPI
nonsense May 20, 2021
4693c61
re-arrange NodeBuilder and add storageminer_svc
nonsense May 20, 2021
9092899
remove DealInfo and DealSchedule from storage-sealing gen/main.go
nonsense May 20, 2021
1a9b576
add remoteGetAllocated http handler
nonsense May 20, 2021
19bd5be
update remoteGetSector
nonsense May 20, 2021
9a7b0b6
comment
nonsense May 20, 2021
8d3d3c8
introduce init service, refactor restore service
nonsense May 20, 2021
9979de2
logs and defaults
nonsense May 26, 2021
c311520
add missing messagepool override
nonsense May 27, 2021
46a368d
bump api ver
nonsense May 27, 2021
d195a12
Merge branch 'master' into nonsense/split-market-miner-processes
nonsense May 31, 2021
e4136b0
add modules.StorageAuthWithURL to set correct token
nonsense Jun 1, 2021
0a79f49
fix tests
nonsense Jun 1, 2021
88756f3
fix TestAPIDeal tests
nonsense Jun 4, 2021
7f23480
make linter happy
nonsense Jun 4, 2021
c877166
add messagepool to test node constructor
nonsense Jun 4, 2021
8bd09e3
resolve merge conflicts
nonsense Jun 4, 2021
d9a7348
use masters filecoin-ffi
nonsense Jun 4, 2021
6f125cc
add prover to fullnode and miner builders
nonsense Jun 4, 2021
c4d10bc
apply messagepool.NewProviderLite to lite nodes
nonsense Jun 4, 2021
ed634bc
rebuild docs
nonsense Jun 4, 2021
ded5341
fixup test node builder
nonsense Jun 4, 2021
042ac82
testplans: lotus-soup: new images with filecoin-ffi ; use default WPo…
nonsense Jun 7, 2021
b0cb0c1
do not depend on filecoin-ffi in api package
nonsense Jun 7, 2021
de646a5
fix node builder
nonsense Jun 7, 2021
9f3ec82
initial resolution of conflicts
nonsense Jun 8, 2021
c0607a0
Merge branch 'master' into nonsense/split-market-miner-processes
nonsense Jun 8, 2021
4dd093b
fix piece provider test
nonsense Jun 8, 2021
3f5f96d
go mod tidy
nonsense Jun 8, 2021
30ba9a2
remove double http handler decl
nonsense Jun 8, 2021
9ab84bd
upgrade docsgen
nonsense Jun 8, 2021
beb5bff
fix log.Error; rename RemoteStore
nonsense Jun 9, 2021
31e6fb1
resolve conflicts
nonsense Jun 14, 2021
c0feb36
add missing debug
nonsense Jun 14, 2021
090a163
bump miner version apiv0
nonsense Jun 15, 2021
323c78e
comment for checkV1ApiSupport
nonsense Jun 15, 2021
d45bb14
Merge branch 'master' into nonsense/split-market-miner-processes
nonsense Jun 16, 2021
338bd04
Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-m…
nonsense Jun 17, 2021
89df3cc
Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-m…
nonsense Jun 18, 2021
577b730
wip
nonsense Jun 18, 2021
6720463
resolve merge conflicts
nonsense Jun 22, 2021
5e10d53
wip
nonsense Jun 22, 2021
0061867
policy.SetPreCommitChallengeDelay to reduce WaitSeed stage time
nonsense Jun 22, 2021
b0d1668
comment out preseal sectors
nonsense Jun 22, 2021
e656aad
regenerate docs
nonsense Jun 22, 2021
9567807
Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-m…
nonsense Jun 22, 2021
a828b15
revert MockSectorMgr
nonsense Jun 22, 2021
4f2d8b0
add all subsystems to deadlines and wdpost_dispute tests
nonsense Jun 22, 2021
218c919
fix testNonGenesisMiner
nonsense Jun 22, 2021
aa58447
fix paych and sdr tests
nonsense Jun 22, 2021
fc7feff
Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-m…
nonsense Jun 22, 2021
d98eca7
Merge branch 'raulk/itests-refactor-kit' into nonsense/split-market-m…
nonsense Jun 23, 2021
51a4657
fix SDR test
nonsense Jun 23, 2021
7fb93d4
refactor kit.NewDealHarness to include main miner and market node
nonsense Jun 23, 2021
99646d2
fix TestFirstDealEnablesMining
nonsense Jun 23, 2021
21b5132
adding TestDealWithMarketAndMinerNode
nonsense Jun 23, 2021
4217ec5
make auth on remote rpc server optional
nonsense Jun 23, 2021
a8d5ae5
enable more tests
nonsense Jun 23, 2021
9ec7cd7
enable libp2p options
nonsense Jun 24, 2021
4f0a96c
resolved conflicts
nonsense Jun 30, 2021
3aa6d03
connect miner to full node , only when running markets
nonsense Jun 30, 2021
1869de7
unwire host from storage.Miner.
raulk Jul 1, 2021
4194c7f
revert back optional for Miner; remove ActorAddress call in node.Mine…
nonsense Jul 2, 2021
96245c3
lint
nonsense Jul 2, 2021
9e85492
fix test
nonsense Jul 2, 2021
287e069
missing flag, that will need to be refactored
nonsense Jul 2, 2021
8dbae00
fix lotus-soup
nonsense Jul 2, 2021
724f728
enable more tests
nonsense Jul 2, 2021
ab6e01e
remove --enable-markets flag, read from repo
nonsense Jul 2, 2021
e9d0a25
remove redundant flag from docs
nonsense Jul 2, 2021
adba1a7
add missing err check
nonsense Jul 2, 2021
4be0a7a
resolve conflicts with master
nonsense Jul 5, 2021
5a87f04
fix finalize sectors early test
nonsense Jul 5, 2021
9839d57
fix SectorsStatus to query full node for on-chain data
nonsense Jul 6, 2021
ff2772a
resolved conflicts
nonsense Jul 6, 2021
604be5f
Update extern/sector-storage/stores/remote.go
nonsense Jul 6, 2021
8cf3acc
Update itests/kit/ensemble.go
nonsense Jul 6, 2021
898687e
Update node/modules/storageminer.go
nonsense Jul 6, 2021
10dc90f
addressing comments
nonsense Jul 6, 2021
b572c2b
fix IsUnsealed
nonsense Jul 6, 2021
7a78527
remove not used sa; cleanup
nonsense Jul 6, 2021
2a7af38
fix testServ to Start with correct listener
nonsense Jul 6, 2021
566a9d5
fix signature
nonsense Jul 6, 2021
42ae876
remove ApplyIfEnableLibP2P
nonsense Jul 7, 2021
caa366b
variadic WithSubsystems
nonsense Jul 7, 2021
6b014f5
pass Subsystems to StorageMiner option; add enableLibp2p bool in Sett…
nonsense Jul 7, 2021
1fe08f5
refactor testkit in lotus-soup to include Subsystems when instantiati…
nonsense Jul 7, 2021
bb466ed
fix cast on role_miner in testground lotus-soup plan
nonsense Jul 7, 2021
82efe01
fixup
nonsense Jul 7, 2021
16784aa
remove pieceProvider from DI; small refactors
nonsense Jul 12, 2021
d89ddb9
resolve conflicts
nonsense Jul 12, 2021
a423c46
rename mock.go to net.go
nonsense Jul 12, 2021
056136e
rename argument from url to apiInfo
nonsense Jul 12, 2021
4bc9fa0
rename cfg.Subsystems.EnableStorageMarket to EnableMarkets
nonsense Jul 12, 2021
7728d6b
remove --enable-markets flag; add --name string slice flag
nonsense Jul 12, 2021
2af02af
rename --name to --type; provide empty stores.StorageConfig when init…
nonsense Jul 13, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion api/api_storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,13 @@ type StorageMiner interface {
// Get the status of a given sector by ID
SectorsStatus(ctx context.Context, sid abi.SectorNumber, showOnChainInfo bool) (SectorInfo, error) //perm:read

// Add piece to an open sector. If no sectors with enough space are open,
// either a new sector will be created, or this call will block until more
// sectors can be created.
SectorAddPieceToAny(ctx context.Context, size abi.UnpaddedPieceSize, r storage.Data, d PieceDealInfo) (SectorOffset, error) //perm:admin

SectorsUnsealPiece(ctx context.Context, sector storage.SectorRef, offset storiface.UnpaddedByteIndex, size abi.UnpaddedPieceSize, randomness abi.SealRandomness, commd *cid.Cid) error //perm:admin

// List all staged sectors
SectorsList(context.Context) ([]abi.SectorNumber, error) //perm:read

Expand Down Expand Up @@ -135,8 +142,8 @@ type StorageMiner interface {
StorageBestAlloc(ctx context.Context, allocate storiface.SectorFileType, ssize abi.SectorSize, pathType storiface.PathType) ([]stores.StorageInfo, error) //perm:admin
StorageLock(ctx context.Context, sector abi.SectorID, read storiface.SectorFileType, write storiface.SectorFileType) error //perm:admin
StorageTryLock(ctx context.Context, sector abi.SectorID, read storiface.SectorFileType, write storiface.SectorFileType) (bool, error) //perm:admin
StorageList(ctx context.Context) (map[stores.ID][]stores.Decl, error) //perm:admin
nonsense marked this conversation as resolved.
Show resolved Hide resolved

StorageList(ctx context.Context) (map[stores.ID][]stores.Decl, error) //perm:admin
StorageLocal(ctx context.Context) (map[stores.ID]string, error) //perm:admin
StorageStat(ctx context.Context, id stores.ID) (fsutil.FsStat, error) //perm:admin

Expand Down Expand Up @@ -300,3 +307,25 @@ type PendingDealInfo struct {
PublishPeriodStart time.Time
PublishPeriod time.Duration
}

type SectorOffset struct {
Sector abi.SectorNumber
Offset abi.PaddedPieceSize
}

// DealInfo is a tuple of deal identity and its schedule
type PieceDealInfo struct {
PublishCid *cid.Cid
DealID abi.DealID
DealProposal *market.DealProposal
DealSchedule DealSchedule
KeepUnsealed bool
}

// DealSchedule communicates the time interval of a storage deal. The deal must
// appear in a sealed (proven) sector no later than StartEpoch, otherwise it
// is invalid.
type DealSchedule struct {
StartEpoch abi.ChainEpoch
EndEpoch abi.ChainEpoch
}
Loading