Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: sealing: Partially sealed sector import #9210

Merged
merged 28 commits into from
Sep 19, 2022
Merged

feat: sealing: Partially sealed sector import #9210

merged 28 commits into from
Sep 19, 2022

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Aug 24, 2022 •
edited
Loading

Related Issues

Implements #9012
With input from #9079

On top of #9183

Some non-api-breaking type improvements in #9300; Not having those shouldn't be blocking for this PR

Proposed Changes

Add a new SectorReceive API method for importing sectors in a bunch of different states, optionally fetching remote sector data. Adds optional 'hooks' to sector metadata, making lotus call external APIs for Commit1 / on state transitions / after sealing is done. Details TBD as #9079 and this implementation progresses.

  1. Add new miner API for importing remote sector metadata
    1.1. Remote sector metadata can include urls to sector files
  2. Add new worker task type for downloading sector data on workers
  3. Test all that a lot

Additional Info

This is very work in progress. Will take a while to finish, test, and review this, but I'm opening this PR early for visibility.

Code in this PR isn't usable in its current form, and likely won't be until the implementation is mostly done a few days / weeks from now.

Checklist

Before you mark the PR ready for review, please make sure that:

  • All commits have a clear commit message.
  • The PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, INTERFACE BREAKING CHANGE, CONSENSUS BREAKING, build, chore, ci, docs,perf, refactor, revert, style, test
    • area: api, chain, state, vm, data transfer, market, mempool, message, block production, multisig, networking, paychan, proving, sealing, wallet, deps
  • This PR has tests for new functionality or change in behaviour
  • If new user-facing features are introduced, clear usage guidelines and / or documentation updates should be included in https://lotus.filecoin.io or Discussion Tutorials.
  • CI is green

@magik6k magik6k force-pushed the feat/recievesector branch from 44cbb16 to dc99efe Compare August 24, 2022 19:40
Base automatically changed from feat/sectornum-mgmt to master August 26, 2022 14:59
@magik6k magik6k force-pushed the feat/recievesector branch 3 times, most recently from 376614e to 07340f7 Compare September 2, 2022 11:57
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
api/api_storage.go
// Required in PreCommitting and later
TicketValue abi.SealRandomness
TicketEpoch abi.ChainEpoch
PreCommit1Out storiface.PreCommit1Out // todo specify better
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#9300

magik6k
magik6k commented Sep 2, 2022
View reviewed changes
api/api_storage.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
api/api_storage.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
storage/pipeline/receive.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
storage/pipeline/receive.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
storage/pipeline/receive.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
storage/pipeline/receive.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 2, 2022
View reviewed changes
storage/pipeline/receive.go Outdated Show resolved Hide resolved
magik6k
magik6k commented Sep 9, 2022
View reviewed changes
api/api_storage.go
Comment on lines +587 to +603
// todo spec better
Commit1Out storiface.Commit1Out
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#9300

@magik6k magik6k force-pushed the feat/recievesector branch 2 times, most recently from e70ba2c to d4c2d14 Compare September 9, 2022 13:11
@magik6k magik6k changed the title [WIP] feat: sealing: Partially sealed sector import feat: sealing: Partially sealed sector import Sep 9, 2022
@magik6k magik6k mentioned this pull request Sep 13, 2022
Closed
5 tasks
@magik6k magik6k marked this pull request as ready for review September 14, 2022 07:49
@magik6k magik6k requested a review from a team as a code owner September 14, 2022 07:49
@magik6k magik6k force-pushed the feat/recievesector branch 4 times, most recently from efc1332 to 6b663d3 Compare September 14, 2022 09:32
geoff-vball
geoff-vball reviewed Sep 15, 2022
View reviewed changes
Copy link
Contributor

@geoff-vball geoff-vball left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few small comments. Excited for this to land!

itests/sector_import_simple_test.go Outdated Show resolved Hide resolved
storage/paths/fetch.go Show resolved Hide resolved
storage/pipeline/receive.go Show resolved Hide resolved
storage/sealer/storiface/storage.go Outdated Show resolved Hide resolved
@magik6k
sector import: Check pieces in Sealing.Receive
332d83b
@magik6k
sealing: Validate piece alignment in checkPieces
ca790b4
@magik6k
sector import: Fix missing perm tag on SectorReceive
2318419
@magik6k
sector import: More check progress
2b64452
@magik6k
sector import: Put the imported sector into the sealing pipeline
39e4845
@magik6k
sector import: Plumbing for DownloadSectorData in the sealing system
fbb487a
@magik6k
sector import: Sector data download
ea99bd9
@magik6k
sector import: Fix tests
cde23be
@magik6k
sector import: Implement handleReceiveSector
cdadf5e
@magik6k
sector import: Update type-gen
3caa010
@magik6k
sector import: Initial SectorReceive itest
fef7232
@magik6k
sector import: Add missing initial fsm state transition
12a8ab5
@magik6k
sector import: Plumb sector download task canfig
9c6d531
@magik6k
sector import: Seal with the correct proof type in saas itest
3086e8a
@magik6k
sector import: FinalizeInto, send finalized cacde in itest
830c2ff
@magik6k
sector import: Remote Commit1
9f03569
@magik6k
sector import: Test remote commit1 retry
b2dfaae
@magik6k
sector import: Implement seed/ticket/commr/commd checks
1428948
@magik6k
sector import: Remote Commit2
c17ab76
@magik6k
tarutil: Strict header checks
474c602
@magik6k
sector import: Fix test build
ef834b9
@magik6k
sector import: RemoteSealingDoneEndpoint
061a990
@magik6k
sector import: Fix lint
fec9c0f
@magik6k
systar: Create file after header checks
a05593d
@magik6k
sealing: Address review
859c260
@magik6k magik6k merged commit 4cdeb6c into master Sep 19, 2022
@magik6k magik6k deleted the feat/recievesector branch September 19, 2022 11:04
@magik6k magik6k mentioned this pull request Dec 5, 2022
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shrenujbansal shrenujbansal shrenujbansal left review comments

@geoff-vball geoff-vball geoff-vball approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@magik6k @geoff-vball @shrenujbansal