Sealing-as-a-Service API Design

[dleonx]

This is a discussion to define and align on the API for the sealing-as-a-service interface. The interface should reference the example architecture of this issue.

Sealing-as-a-service aims to accelerate the ability to onboard data to the network and facilitate the onboarding of new SPs. New SPs can use a sealing service to onboard to the network without purchasing sealing hardware. Existing SPs can either use sealing-as-a-service to accelerate sealing sectors, or provide sealing-as-a-service to generate new lines of business.

Below is an example sealing architecture in Lotus. Sealing components of this sealing architecture can be extracted into a sealing service. The goal of this discussion is to align on the best way for Lotus to expose a generic interface to enable this service. If you have any design suggestions or feedback, please leave them in this discussion.

Example architecture:

[ineffectualproperty]

Hey @dleonx ! Thanks for starting this discussion. We've taken a first pass at demonstrating a potential communication flow between Sealing/SNARK providers and Lotus based on the architecture presented. We've tried to capture what we think are the important general JSON key/values, but the exact fields and data structures should ultimately be defined based on Lotus objects. Please feel free to ask if you have any questions and feedback is greatly appreciated.

---

BatchAssignSectorIDs (Assign SectorID in Diagram)

Sealing-as-a-Service > Lotus

This API asks Lotus to reserve a number of sectors (1-n) and returns the metadata required to create the replica_ID.

Inputs:
[
  num_sectors
]

Response:
[
  "ProverId": prover_id,
  "PoRepSeed": porep_seed,
  "Sectors" : [
    {
      "SectorID": sector_id,
      "Ticket": ticket,
      "OutPath": out_path, <- Used if running sidecar on same system as Lotus node
      “TrustSector”: bool <- This needs to be logged internally to Lotus but not necessarily sent to the Sealing-as-a-Service provider. This flag determines if Lotus should trust that the data should be sealed correctly or needs to download and verify it all first.
    },
    ...
  ]
]

BatchNotifyPrecommitComplete (SendSector in Diagram)

Sealing-as-a-Service > Lotus

This API call lets Lotus know that PC1 and PC2 are complete. The information required for Lotus to do the Precommit is in the "SealedCID" field. If TrustSector was set to 1 then Lotus can proceed to post the Precommit message without downloading and verifying the rest of the sector data. The replica and trees can be found in the "OutPath".

Inputs:
[
  "Sectors" : [
    {
      "SectorID": sector_id,
      "SealedCID": sealed_cid,
      "OutPath": out_path"
    },
    ...
  ]
]

Response:
[
  success
]

BatchDoCommits (RemoteCommit1/Commit2 in Diagram)

Lotus > Sealing-as-a-Service

This API for the Sealing-as-a-Service provider tells them to begin performing C1 and optionally C2 and provides them with the required WaitSeed information.

Inputs:
[
  "Sectors" : [
    {
      "SectorID": sector_id,
      "WaitSeed": wait_seed,
      “DoC2”: bool, <- If true, OutPath will be the C2 SNARK proof, if false, OutPath will be the Merkle proofs.
      "OutPath": out_path
    },
    ...
  ]
]
  
Response:
[
  success
]

BatchDoSNARK (Commit2 in Diagram)

Lotus > SNARK-as-a-Service

This API asks a Snark-as-a-Service provider to compute a SNARK.

Inputs:
[
  "Sectors" : [
    {
      "SnarkInputs": snark_inputs, (exact data structure for JSON fields to be defined, e.g. circuit_ID, public_inputs, witness)
      "OutPath": out_path <- This is where C2 SNARK proof will be placed if Lotus is running in same environment as sidecar.
    },
    ...
  ]
]

Response:
[
  success
]

BatchNotifyCommitComplete

Sealing/SNARK-as-a-Service > Lotus

This API lets Lotus know that C1 and/or C2 has been completed and is ready to be used.

Inputs:
[
  "Sectors" : [
    {
      "SectorID": sector_id,
      "OutPath": out_path
    },
    ...
  ]
]

Response:
[
  success
]

BatchFinalizePoRepSectors (Proving in Diagram)

Lotus > Sealing-as-a-Service

This API to the Sealing-as-a-Service provider lets them know that the Commit message has been successfully posted on chain and the sector data can be deleted from the service.

Inputs:
[
  "Sectors" : [
    {
      "SectorID": sector_id,
    },
    ...
  ]
]

Response:
[
  success
]

[magik6k]

BatchAssignSectorIDs

This really can just be an api to reserve a block of sector numbers, making sure that lotus won't use any numbers in the reserved block. Something like SectorReserveNumbers(count int, name string) Bitfield (name being optional metadata for miners to know where allocations came from). The returned bitfield, encoded as json-rle. Seed/Ticket/ProverID don't need to be returned by this API, they can be obtained easily by calling any chain node, no key access required.

This way lotus also doesn't need to track the state of those sectors until they are actually imported into lotus

BatchNotifyPrecommitComplete

As mentioned in #9012, I think it's better to have one API method which allows you to import sector metadata into the sealing pipeline for sectors in any state, which should be much more flexible.

I also don't think it's necessary to specify sector paths here:

• The saas sidecar would put sector files in one of miner-configured sector paths
• It would then call StorageRedeclareLocal(path) on lotus-miner / lotus-worker with access to the path - this api will then detect those sector files.
  • The sidecar can also just participate in the lotus-miner storage system
    • If you're using Go, adding this capability is just a few lines of code - e.g. https://github.com/filecoin-project/lotus/blob/master/cmd/lotus-worker/main.go#L513-L536
    • This gives you most control over how you handle sector data, and means that lotus doesn't have to make any technical choices which could limit how well this works
    • For more details see the implementation of https://github.com/filecoin-project/lotus/blob/master/storage/paths/local.go / https://github.com/filecoin-project/lotus/blob/master/storage/paths/remote.go
• After sector files are visible to lotus-miner, the sidecar would import sector metadata, and lotus-miner will pick up the sealing process at whichever stage the sector was imported.

BatchDoCommits

This api can be simpler by being synchronous, and only doing C1, which is very cheap/fast. I imagine this would be passed as a url to a jsonrpc/rest endpoint which lotus would call if it is set in sector metadata.

BatchDoSNARK / BatchNotifyCommitComplete

This doesn't need to exist. You can already do this by registering a C2-only worker with lotus-miner.

BatchFinalizePoRepSectors

Also technically doesn't need to exist as you can just observe the blockchain, but it can be another url to jsonrpc/rest endpoint in sector metadata that lotus would call when the sector is successfully proven / removed (or at any state transition)

[magik6k]

You can already do this by registering a C2-only worker with lotus-miner

To expand on this a little, here's an example template of worker interfaces which can be easily adapted to pass work to SaaS - https://gist.github.com/magik6k/0fc17efc07f5ffc7960911d3c3e5fd05.

After you get a worker serving its rpc, you just need to call WorkerConnect on lotus-miner API to have lotus-miner start passing C2 tasks to it - here's how lotus-worker does that - https://github.com/filecoin-project/lotus/blob/master/cmd/lotus-worker/main.go#L660

BatchFinalizePoRepSectors.. technically doesn't need to exist

You can use miner state diffing or state inspection helpers to monitor when sectors get proven on-chain.

[jennijuju]

@dleonx could you please ask the related parties to review Magik's feedback soon, i.e: confirm whether registering workers are something thats feasible/reasonable for them to do? If not, please let us know and we will propose design accordingly!

[hexcola]

First, we must admit that our design is not perfect. And here is our design diagram.

Control-key is the trick.
The miner must add a wallet address that sealing-svc provided to its control-key list, and this wallet address will only use for publishing deals, precommit, and submitcommit.

Pandarua is one of our products. It can help users to buy sectors, track the sealing process, etc.

Pros:

• miner's ownership is secure;
• less interactive with the client's lotus-miner(imagine m:n relations, which m represents multiple sealing-svc providers, n means multiple miners) ;
• less code modification;
• sealing-service side has more control over allocating sealing resources.

cons

• for now, we can't limit the control-key access permission. It may cause trust issues.

[ineffectualproperty]

Thanks for the feedback @magik6k! Below are our thoughts/feedback on the various APIs. One additional piece of context I’d like to provide is that as we drafted the API the goal was to eliminate the requirement to run a ‘sidecar’ alongside the Lotus node. We hope with the proper API we can have direct interaction between Lotus and Sealing/SNARK providers without any duplication of logic from Lotus such as chain interaction or storage management. In an ideal world (from our perspective) the service is purely computational and is designed to offload the heaviest computations and reduce the hardware requirements and cost of sealing for storage providers.

BatchAssignSectorIDs
Your approach looks good if there are already existing APIs to get the other required data to seal a sector. One thing we wanted to understand is why RLE is needed here? If compression is required, could that be done in the network protocol? Most languages have libraries for handling JSON and we are not sure how portable the RLE option is without writing custom logic.

BatchNotifyPrecommitComplete
As mentioned the goal of our design is to eliminate the need for a ‘sidecar’ application. The ‘outpath’ field is needed to provide a link for Lotus to download the relevant data. We chose this approach because we think it is better for security purposes if Sealing-aaS providers do not have access to a Lotus nodes’ storage. As for generalizing the API, we agree that your suggested approach makes sense. Our only comment is that you would probably always have a Sealing-aaS provider do PC1/PC2/C1 and/or C2. It wouldn’t make sense to only do PC1 and PC2 as you would then have to transfer hundreds of gigabytes to allow the Lotus node to perform C1. Below is an example API that would generalize this function:

BatchNotifySealingOperationComplete

Inputs:
[
  "Sectors" : [
    {
      "SectorID": sector_id,
      "SealedCID": sealed_cid,
      "OutPath": out_path,
      “PC2Complete”: Bool,
      “C1Complete” Bool,
      “C2Complete” Bool
    },
    ...
  ]
]

Response:
[
  success
]

BatchDoCommits
We’re not very familiar with the existing remote workers but this approach seems like it would have Lotus managing the Sealing-aaS provider infrastructure, rather than just making a request for work to be done. This may add complexity for the Sealing-aaS provider when managing jobs as there will be a many (Lotus nodes) to one (Sealing-aaS worker) relationship. Regarding only doing C1, while this is possible, it does mean that you will be required to send the Merkle paths back to the storage provider only to be returned back to a SNARK producer who may be the same entity. Do we want to require this vs. have the option to have a single entity perform them both? Regarding making this synchronous, we have heard reports that sometimes providers can’t read the Merkle paths even to do WindowPost due to heavy utilization of the underlying storage. It seems better to make this asynchronous so the connection doesn’t have to stay open since Sealing-aaS providers storage will be under heavy utilization.

BatchNotifyCommitComplete
If we follow the asynchronous support above we will need this capability, however it can be consolidated into the BatchNotifySealingOperationComplete API.

BatchFinalizePoRepSectors
The point of this API is to allow the Sealing-aaS provider to be notified when they can delete the replica data. The replica may be downloaded up to 30 days before the provecommit hits the chain so it would be great to have this API to improve the efficiency and economics of Sealing-aaS providers. Ideally this API is called after Lotus has downloaded the ‘outpath’ data from the BatchNotifySealingOperationComplete.

[magik6k]

One thing we wanted to understand is why RLE is needed here?

We already have a lot of other APIs dealing with data in this format, and I'd think that it's really easy to deal with it in pretty much any language. In Go you can use our bitfield library; I'd imagine other languages have libraries for it too, but fundamentally it should be really easy to just encode/decode the json rle array with 1/2 lines of map/reduce into any other form you need.

https://github.com/filecoin-project/lotus/pull/9183/files#diff-11f17642f37c68a4e8373aa8612f2ab0c0160e516b7d071e8d9f3922764f010eR127-R131 Implements a call which makes it possible to reserve ranges of sector numbers (SectorNumReserve(ctx context.Context, name string, field bitfield.BitField, force bool) error), but it doesn't handle assigning - I can add that to the PR, the function would basically take a count of sector numbers you want to get a reservation for, and lotus would give you a valid range of numbers to use.

---

The ‘outpath’ field is needed to provide a link for Lotus to download the relevant data

in many lotus clusters today the main lotus-miner process will have no local access to storage, and it will only be accessible from workers, and in some cases not all workers will have access to all the storage. This is done because at certain scales you simply can't push all the bytes through a single computer, even if it has multiple 100G NICs. Given that SaaS is meant to work/exceed those scales, we should start with something that will work at those scales.

One solution that could work is:

• Instead of having the sidecar as I mentioned above do the fetching, have BatchNotifyPrecommitComplete (or whatever other name, say RecieveSector) on the lotus-miner api accept sector CommD/R/etc + a set of URLs to the sealed/unsealed/cache files
• lotus-miner would import this sector metadata, and have one of its workers start fetching sector data from SaaS-provided URLs
• This way technically no sidecar is needed on the miner side, and sectors can be fetched by multiple workers on multiple machines; lotus-miner can stay firewalled from the internet for the most part (with a small proxy to pass the lighter SaaS api calls)
• note that those URLs initially may be http/https, but in future they may support other things like nfs/rdma
• cache directories have multiple files in them, and ideally would be referred to with one URL; Lotus currently just uses tar, which isn't ideal for a number of reasons (hard/impossible to use accelerate with sendfile) - cache files could still be sent as tar to be consistent with what we do today, but other things like multipart uploads are also good approaches (just require more code/time on our side)

(tldr after sector is sealed, saas would give sector metadata+sector urls to lotus; lotus(-workers) would later fetch those files)

---

As for generalizing the API, we agree that your suggested approach makes sense. Our only comment is that you would probably always have a Sealing-aaS provider do PC1/PC2/C1 and/or C2. It wouldn’t make sense to only do PC1 and PC2 as you would then have to transfer hundreds of gigabytes to allow the Lotus node to perform C1.

There are some other sector states in which it makes sense to import sectors:

• After sectors are packed, before precommit1 - this may be useful for deal aggregator services, letting them handle deal packing
• After sectors are fully sealed/committed - SaaS needs some access to miner keys for that, but in some cases this may be what miners want - and this seems to be what @hexcola described above
• In some cases you may even want to get sectors with full non-finalized cache after PC1 - this can be the case with multiple miners sharing resources a single DC, where getting TB/s-level bandwidth between racks/rooms is reasonably cheap.

---

BatchDoCommits, We’re not very familiar with the existing remote workers but this approach seems like it would have Lotus managing the Sealing-aaS provider infrastructure

That's not really the case; If your worker declares itself as having no resource constraints, lotus will just forward all C2 calls to your service with no scheduling logic getting in the way. It's also worth noting that all lotus<->worker calls are async http (jsonrpc) calls, so there's no need to worry about keeping connections alive;

Abstractly this is roughly the http requests which happen between lotus / c2 workers:

worker(W) lotus(L)

W -> L RegisterWorker(workerApiUrl)

L -> W WorkerInfo() -> TaskTypes/Resources/etc.

(at some interval) L -> W Session() -> SessUUID - lotus pings workers to check if they are alive/shutting down/restarted; The session ID is generated when worker processes start, so when lotus sees that they've changes, it will assume that all tasks which were running on that process have been aborted; Much like `/proc/sys/kernel/random/boot_id` on linux.

(lotus-miner wants to do c2 on a sector)
L -> W Commit2(commit2 params) -> CallUUID - worker starts C2, returns job UUID to lotus immediately. If SaaS would be the worker, it can do it's own scheduling now, and run the C2 whenever it sees fit.

(worker computes C2)

W -> L ReturnCommit2(c2callid CallUUID, c2out Commit2Out, error Error)

---

BatchFinalizePoRepSectors

Agree that this is useful

---

Batch*

Not sure if we really want or need all methods to support batching - it really depends what you want to achieve with it.

• If it's performance I'd think that any gains will be negligible.
  • JsonRPC supports multiple async calls (with multiple parallel calls, or over a single websocket with different request ids) - lotus should be able to accept sectors one-by-one quite quickly
  • Even if the perf was so bad that you'd be importing 5 sectors/second, you'd become the biggest miner in the network in ~5 days
• If it's for the guarantee that all sectors in a batch are imported without errors or otherwise nothing is imported, I may be able to be convinced that this is useful, but I'd like to hear more on why that property is desired - it seems to me that it isn't really needed for the service to work well, and on the lotus side it's a whole bunch of complexity to support correctly.

[Fatman13]

Hello, team,

Venus team believe that SaaS initiative is highly compatible with the design philosophy of venus-cluster. The team has brought the following proposal for review. Comments and questions are very welcomed!

[ineffectualproperty]

Hey @magik6k thanks for all of the feedback! I think we are mostly in agreement on the API approach. Below is a quick summary of where I think we're at.

BatchAssignSectorIDs
Seems like we are mostly aligned here. It would be great to see the final proposed API, and if possible, links to the other APIs that we should use to get the data that is required to begin sealing a sector (e.g. to retrieve seed, etc.). We're still not 100% sure on the need for RLE, but if that is a requirement for Lotus then we understand it will need to be used here.

BatchNotifyPrecommitComplete / BatchNotifySealingOperationComplete / RecieveSector
Seems like we are in agreement here as well in that we will pass the CommD/R/SealedCID through the API along with a set of URLs to the larger files.

BatchDoCommits
It sounds like using the existing worker structure/API will work fine here based on your feedback. It would be great to see the API that should be implemented here on Sealing-aaS providers side to accept the existing Lotus APIs for workers.

BatchFinalizePoRepSectors
Looks like we are in agreement here as well.

As for batching, our assumption was that the standard working model would be that many sectors would be requested to be sealed at once, and our sealing software seals many sectors in parallel as well. However, if it adds complexity to have batch APIs then it is no problem to make repeated API calls.

Have a great weekend!

[jennijuju]

standard working model would be that many sectors would be requested to be sealed at once

@magik6k I think this is a fair assumption and will provide cleanliness for the pipeline (compare to have single sectors being passed around all the time)

Could you please provide some insight on how much refactoring in lotus would be needed for supporting this tho, with a effort estimation?

[magik6k]

As for batching, our assumption was that the standard working model would be that many sectors would be requested to be sealed at once, and our sealing software seals many sectors in parallel as well

If batches is how you represent sectors in your internal data structures, I don't think this should influence lotus APIs which are going to be used by different SaaS impls as well.

Just calling the sector import RPC for each sector (potentially in parallel) should be more than fast enough, and makes error handling much easier.

[benjaminh83]

Hi, Out of the graphics, I find it a bit hard to understand the actual data movement steps for incoming deal data. @dleonx ?

Is this because it is assumed that a SaaS solution with have their own boost/market implementation that will do the Graphsync/HTTP transfers directly to their datacenter, so no deal data is never received or processed on the lotus-miner (SP) side?

[benjaminh83]

@jennijuju Thanks! I think the key here is to make sure the implementation of SaaS support is flexible enough to support the different scenarios a SP typically wants to operate under. I like @dirkmc comment around the unpredictable behaviour, as this is very much what we see with Estuary, were we might get blasted with 100s of deals, but maybe only 10% makes it through due to geo location (and who knows what else). In this scenario SPs might want to use this more clean scenario here:

• SP does all deal downloads (handles the issues) and packing of deals into sectors and generate the unsealed copy
• SP sends off the unsealed copy to the SaaS (but does not delete it locally)
• SP gets a sealed copy in return from the SaaS
• SP takes the locally generated unsealed copy and moves it to LTS doing Finalize
• SP takes the sealed copy downloaded from the SaaS and moves it to LTS doing Finalize

This is basically what is described in Scenario 3
@jennijuju unfortunately, issue #9159 does not have much to do with this ask. That issue is about missing unsealed copy on retrieval deals, and the ability to generate unsealed copies on the lotus-workers instead of on the lotus-miner.
The change I'm proposing is about changing the behaviour so that lotus does not delete the locally generated unsealed copy once it has been fetched by the SaaS for PC1. Instead we would like our AP worker to hold on to this unsealed copy, and once the sector needs to finalize, then move this local copy to LTS, instead of fetching this from the SaaS. Reason for this ask is to reduce the upload bandwith requirement for the SaaS by 50%, and keeping the upload/download stream symmetric between SP and SaaS, which is cost efficient. Basically we would just be exchanging unsealed copies for sealed copies...

[jennijuju]

The change I'm proposing is about changing the behaviour so that lotus does not delete the locally generated unsealed copy once it has been fetched by the SaaS for PC1

could you please create an issue?

on deals - the issue you are describing sounds more like a SaaS design on how they wanna handle jobs with deal-making. @dleonx I think it's essential for SaaS providers to take this feedback in when they design their service. Do you have a place where broader place SaaS design should live in (again, id like keeping this discussion for the SaaS <> lotus-miner API?

[jennijuju]

@benjaminh83 chatted with @magik6k , we agree that the AddPiece is a separate process either that happens on SaaS or lotus miner directly.

When it happens in lotus-miner, we should have better handling and Magik has some ideas, he will propose a design in a separate issue ticket later, and I'm looking forward to seeing you engaged there!

@dleonx can confirm when it happens in SaaS - there is no impact on the lotus APIs that we have designed here, which are things happens post AP

[dleonx]

@benjaminh83 thank you for the feedback and designs! I also think it'll be essential to properly outline and incorporate SaaS deal-making. For this first iteration, our approach has been to keep it simple for SaaS providers and SPs who might not want to handle deals right away through this new service. Do we think it makes sense to start another discussion for deal-making SaaS while we let these initial V1 SaaS designs materialize? This way we can start testing a first iteration of a SaaS while we work on the next iteration as well.

@jennijuju thank you for the confirmation!

[magik6k]

So what is described here can be done independent of the basic SaaS support, and the implementation should be relatively simple.

Here's what I'm thinking:

• Keep lotus-miner accept deal data from markets/boost, and build sectors with those (AddPiece)
• After the sector is full and goes into the Packing state, after we get the unsealed sector file read, instead of going into GetTicket->PreCommit1, have the sector go into a new state, Paused/External
• When in that state, lotus wouldn't do anything with that sector
• SaaS would pick that sector up using some (tbd) mechanism - possibly lotus calling some configurable hook
• SaaS would fetch the unsealed sector data, and seal it
• It would then just import the sealed sector data using the SectorReceive API described below

[dleonx]

@Fatman13 @hexcola thank you for providing designs! We want to make sure that all services will be compatible with the generic interface that gets implemented. Do you have any specific feedback related to the architecture and conversations that have followed in this thread? We'd love to get your input.

[dleonx]

Sounds good @Fatman13. Correct there's no concrete conclusion yet.

[jennijuju]

Note that the generic interface we are trying to design here, is for SaaS <> lotus miner, not SaaS <> all miner impl. i can imagine the venus cluster handles things way differently.

[strahe]

Is it possible to refine the permissions of the control address? If possible, I think the design of @hexcola is more in line with the actual situation, with less interaction and data transmission.

[jennijuju]

@strahe

permissions of the control address

could you please elaborate more on this?

[strahe]

@jennijuju Node owners can set a key to only do specific things, for example, a control address can only send precommit messages, another can send commit messages.

[jennijuju]

@magik6k shall provide more technical details later, tho after prototyping and careful consideration, we can see a value supporting remote C2 in lotus, and that will be included in the initial rollout.

[flyworker]

we would like to try the remote c2 support in lotus

[claydrone]

Nebula Block is interested to participate in testing

[dleonx]

@flyworker @claydrone great! Please sign up through this form: bit.ly/sealingsurvey

[magik6k]

Update on current implementation status

Right now there are 3 PRs in lotus related to this work

• feat: sealing: Use bitfields to manage sector numbers #9183, merged, introducing more advanced sector number management APIs/CLI
• feat: sealing: Partially sealed sector import #9210, in review, partially sealed sector import, main piece of the puzzle, implements all SaaS<->Lotus interactions needed for the service to work
  • Those 2 PRs will be in the next release
• [wip] sector import c1/pc1 out types #9300, in progress, giving more concrete types to rust-fil-proofs C1/PC1 outputs
  • Non-api-breaking, also no real externally visible features
  • Probably won't make it to the next release

Sector number management APIs

SectorNumFree

SectorNumFree drops a sector reservation

Perms: admin

Inputs:

[
  "reservation-name"
]

Response: {}

SectorNumReservations

SectorNumReservations returns a list of sector number reservations

Perms: read

Inputs: null

Response:

{
  "reservation-name": [5, 3],
  "other-reservation-name": [12, 5]
}

SectorNumReserve

SectorNumReserve creates a new sector number reservation. Will fail if any other reservation has colliding
numbers or name. Set force to true to override safety checks.
Valid characters for name: a-z, A-Z, 0-9, _, -

Perms: admin

Inputs:

[
  "reservation-name", # name
  [5, 3], # to reserve (json-rle)
  false # force - set this to false in basically all cases
]

Response: {}

SectorNumReserveCount

SectorNumReserveCount creates a new sector number reservation for count sector numbers.
by default lotus will allocate lowest-available sector numbers to the reservation.
For restrictions on name see SectorNumReserve

Perms: admin

Inputs:

[
  "reservation-name", # name
  42 # number of SectorNumbers to reserve
]

Response - json-rle

[5, 42]

On json-rle

Sector number ranges are handled as bitfields encoded as an array of run-lengths, always starting with zeroes (absent values)

E.g.: The set {0, 1, 2, 8, 9} is the bitfield 1110000011, and would be marshalled as [0, 3, 5, 2]

In Go those lists can be unmarshalled with go-bitfield, in other languages it should be easy to transform this run-length list into any other format with a few map/reduce calls.

Sector Import

#9210 adds "just" one new miner RPC method - SectorReceive

Example json input (values are not all valid):

[
  {
    # State specifies the first state the sector will enter after being imported
    # Must be one of the following states:
    # * Packing
    # * GetTicket
    # * PreCommitting
    # * SubmitCommit
    # * Proving/Available
    "State": "PreCommitting",
    "Sector": {
      "Miner": 1000,
      "Number": 9
    },
    "Type": 8, # RegisteredSealProof

    # Required in Packing and later
    "Pieces": [
      {
        "Piece": {
          "Size": 1032,
          "PieceCID":  {"/": "baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}, # CommP
        },
        "DealInfo": {
          "PublishCid": null,
          "DealID": 5432,
          "DealProposal": {
            "PieceCID": {"/": "baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}, # CommP
            "PieceSize": 1032,
            "VerifiedDeal": true,
            "Client": "f01234",
            "Provider": "f01000",
            "Label": "",
            "StartEpoch": 10101,
            "EndEpoch": 1010100,
            "StoragePricePerEpoch": "0",
            "ProviderCollateral": "0",
            "ClientCollateral": "0"
          },
          "DealSchedule": {
            "StartEpoch": 10101,
            "EndEpoch": 1010100
          },
          "KeepUnsealed": true
        }
      }
    ],

    # Required in PreCommitting and later
    "TicketValue": "Bw==", # base64, 32 bytes of ticket randomness
    "TicketEpoch": 10000,
    "PreCommit1Out": "Bw==", # see (1) below
    "CommD": {"/": "baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"},
    "CommR": {"/": "bagboea4b5abcbwhjwgl3vo54uefgs6gq5qkjmar3xg5st56ippg7vzy4nwkxubs3"},

    # Required in SubmitCommit and later
    "PreCommitInfo": {
      "SealProof": 8,
      "SectorNumber": 9,
      "SealedCID": {"/": "bagboea4b5abcbwhjwgl3vo54uefgs6gq5qkjmar3xg5st56ippg7vzy4nwkxubs3"},
      "SealRandEpoch": 10101,
      "DealIDs": [
        5432
      ],
      "Expiration": 10101,
      "UnsealedCid": null
    },
    "PreCommitDeposit": "0",
    "PreCommitMessage": null,
    "PreCommitTipSet": [{"/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"}, {"/":"bafy2bzacebp3shtrn43k7g3unredz7fxn4gj533d3o43tqn2p2ipxxhrvchve"}],
    "SeedValue": "Bw==", # base64, 32 bytes of ticket randomness
    "SeedEpoch": 10101,
    "CommitProof": "Ynl0ZSBhcnJheQ==", # see (2) below

    # Required in Proving/Available
    "CommitMessage": {"/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"},

    # Optional sector metadata to import
    "Log": [
      {
        "Kind": "string value",
        "Timestamp": 42,
        "Trace": "string value",
        "Message": "string value"
      }
    ],

    # Sector data references - lotus will use those for fetching files into local storage
    # * When Local is false, lotus will fetch sector data from provided URLs
    # * When Local is true, lotus will assume that sector data is available in the lotus storage system already (e.g. through a SaaS sidecar fetching and declaring sector data to the sector index before calling SectorReceive). This is useful for SaaS implementations which want to use more advanced file transport methods.
    "DataUnsealed": {
      "Local": false,
      "URL": "http://example.com/something/something/miner-567/sector-unsealed-123", # http GET, will be executed by lotus-miner or one of lotus-workers when the DownloadSector task type is enabled; Expects octet-stream with the "unsealed" sector file
      "Headers": [
        {
          "Key": "X-my-header",
          "Value": "my-header-value"
        }
      ]
    },
    "DataSealed": {
      "Local": false,
      "URL": "http://example.com/something/something/miner-567/sector-sealed-123", # http GET, will be executed by lotus-miner or one of lotus-workers when the DownloadSector task type is enabled; Expects octet-stream with the "sealed" sector file; File names and sizes must match constraints defined in https://github.com/filecoin-project/lotus/blob/6b663d33854a8797435a27de9fa278560d9a56ca/storage/sealer/tarutil/systar.go#L17-L72
      "Headers": []
    },
    "DataCache": {
      "Local": false,
      "URL": "http://example.com/something/something/miner-567/sector-cache-123", # http GET, will be executed by lotus-miner or one of lotus-workers when the DownloadSector task type is enabled; returned data must be a TAR of the "cache" sector directory
      "Headers": []
    },

    # Sealing service hooks (all are optional)
    "RemoteCommit1Endpoint": "http://example.com/something/something/miner-567/c1", # http POST, request body is json-ified https://github.com/filecoin-project/lotus/blob/6b663d33854a8797435a27de9fa278560d9a56ca/api/api_storage.go#L584-L591, return value is json of https://github.com/filecoin-project/rust-filecoin-proofs-api/blob/11126c5b91dca942712cf632917a0027634dfddf/src/seal.rs#L154-L162 (same as in (2) below)
    "RemoteCommit2Endpoint": "http://example.com/something/something/miner-567/c2", # http POST, request body is json https://github.com/filecoin-project/lotus/blob/6b663d33854a8797435a27de9fa278560d9a56ca/api/api_storage.go#L593-L599, Commit1Out is described in (2) below
    "RemoteSealingDoneEndpoint": "http://example.com/something/something/miner-567/done" # http POST, request body is https://github.com/filecoin-project/lotus/blob/6b663d33854a8797435a27de9fa278560d9a56ca/api/api_storage.go#L601-L611
  }
]

1. PreCommit1Out is base64 json encoding of https://github.com/filecoin-project/rust-filecoin-proofs-api/blob/11126c5b91dca942712cf632917a0027634dfddf/src/seal.rs#L29-L34
2. CommitProof is base64 json encoding of https://github.com/filecoin-project/rust-filecoin-proofs-api/blob/11126c5b91dca942712cf632917a0027634dfddf/src/seal.rs#L154-L162
3. For a valid example for a 2K sector after PC2 see https://gist.github.com/magik6k/72029ea06ee666be88a39e051003e910
4. Golang version of the RemoteSectorMeta from above is here

There is also a simplified integration test which does SaaS style sealing, then imports the sector into lotus-miner with all the http hooks which might be helpful to look at when integrating with this API:

lotus/itests/sector_import_simple_test.go

Lines 47 to 345 in 6b663d3

	////////
	// Reserve some sector numbers on the miner node; We'll use one of those when creating the sector "remotely"
	snums, err := miner.SectorNumReserveCount(ctx, "test-reservation-0001", 16)
	require.NoError(t, err)
	sectorDir := t.TempDir()
	maddr, err := miner.ActorAddress(ctx)
	require.NoError(t, err)
	mid, err := address.IDFromAddress(maddr)
	require.NoError(t, err)
	mi, err := client.StateMinerInfo(ctx, maddr, types.EmptyTSK)
	require.NoError(t, err)
	ver, err := client.StateNetworkVersion(ctx, types.EmptyTSK)
	require.NoError(t, err)
	spt, err := lminer.PreferredSealProofTypeFromWindowPoStType(ver, mi.WindowPoStProofType)
	require.NoError(t, err)
	ssize, err := spt.SectorSize()
	require.NoError(t, err)
	pieceSize := abi.PaddedPieceSize(ssize)
	////////
	// Create/Seal a sector up to pc2 outside of the pipeline
	// get one sector number from the reservation done on the miner above
	sn, err := snums.First()
	require.NoError(t, err)
	// create all the sector identifiers
	snum := abi.SectorNumber(sn)
	sid := abi.SectorID{Miner: abi.ActorID(mid), Number: snum}
	sref := storiface.SectorRef{ID: sid, ProofType: spt}
	// create a low-level sealer instance
	sealer, err := ffiwrapper.New(&basicfs.Provider{
	Root: sectorDir,
	})
	require.NoError(t, err)
	// CRETE THE UNSEALED FILE
	// create a reader for all-zero (CC) data
	dataReader := bytes.NewReader(bytes.Repeat([]byte{0}, int(pieceSize.Unpadded())))
	// create the unsealed CC sector file
	pieceInfo, err := sealer.AddPiece(ctx, sref, nil, pieceSize.Unpadded(), dataReader)
	require.NoError(t, err)
	// GENERATE THE TICKET
	// get most recent valid ticket epoch
	ts, err := client.ChainHead(ctx)
	require.NoError(t, err)
	ticketEpoch := ts.Height() - policy.SealRandomnessLookback
	// ticket entropy is cbor-seriasized miner addr
	buf := new(bytes.Buffer)
	require.NoError(t, maddr.MarshalCBOR(buf))
	// generate ticket randomness
	rand, err := client.StateGetRandomnessFromTickets(ctx, crypto.DomainSeparationTag_SealRandomness, ticketEpoch, buf.Bytes(), ts.Key())
	require.NoError(t, err)
	// EXECUTE PRECOMMIT 1 / 2
	// run PC1
	pc1out, err := sealer.SealPreCommit1(ctx, sref, abi.SealRandomness(rand), []abi.PieceInfo{pieceInfo})
	require.NoError(t, err)
	// run pc2
	scids, err := sealer.SealPreCommit2(ctx, sref, pc1out)
	require.NoError(t, err)
	// make finalized cache, put it in [sectorDir]/fin-cache while keeping the large cache for remote C1
	finDst := filepath.Join(sectorDir, "fin-cache", fmt.Sprintf("s-t01000-%d", snum))
	require.NoError(t, os.MkdirAll(finDst, 0777))
	require.NoError(t, sealer.FinalizeSectorInto(ctx, sref, finDst))
	////////
	// start http server serving sector data
	doneResp := new(*api.RemoteSealingDoneParams)
	m := mux.NewRouter()
	m.HandleFunc("/sectors/{type}/{id}", remoteGetSector(sectorDir)).Methods("GET")
	m.HandleFunc("/sectors/{id}/commit1", remoteCommit1(sealer)).Methods("POST")
	m.HandleFunc("/sectors/{id}/sealed", remoteDone(doneResp)).Methods("POST")
	m.HandleFunc("/commit2", remoteCommit2(sealer)).Methods("POST")
	srv := httptest.NewServer(m)
	unsealedURL := fmt.Sprintf("%s/sectors/unsealed/s-t0%d-%d", srv.URL, mid, snum)
	sealedURL := fmt.Sprintf("%s/sectors/sealed/s-t0%d-%d", srv.URL, mid, snum)
	cacheURL := fmt.Sprintf("%s/sectors/cache/s-t0%d-%d", srv.URL, mid, snum)
	remoteC1URL := fmt.Sprintf("%s/sectors/s-t0%d-%d/commit1", srv.URL, mid, snum)
	remoteC2URL := fmt.Sprintf("%s/commit2", srv.URL)
	doneURL := fmt.Sprintf("%s/sectors/s-t0%d-%d/sealed", srv.URL, mid, snum)
	////////
	// import the sector and continue sealing
	err = miner.SectorReceive(ctx, api.RemoteSectorMeta{
	State: "PreCommitting",
	Sector: sid,
	Type: spt,
	Pieces: []api.SectorPiece{
	{
	Piece: pieceInfo,
	DealInfo: nil,
	},
	},
	TicketValue: abi.SealRandomness(rand),
	TicketEpoch: ticketEpoch,
	PreCommit1Out: pc1out,
	CommD: &scids.Unsealed,
	CommR: &scids.Sealed,
	DataUnsealed: &storiface.SectorData{
	Local: false,
	URL: unsealedURL,
	},
	DataSealed: &storiface.SectorData{
	Local: false,
	URL: sealedURL,
	},
	DataCache: &storiface.SectorData{
	Local: false,
	URL: cacheURL,
	},
	RemoteCommit1Endpoint: remoteC1URL,
	RemoteCommit2Endpoint: remoteC2URL,
	RemoteSealingDoneEndpoint: doneURL,
	})
	require.NoError(t, err)
	// check that we see the imported sector
	ng, err := miner.SectorsListNonGenesis(ctx)
	require.NoError(t, err)
	require.Len(t, ng, 1)
	require.Equal(t, snum, ng[0])
	miner.WaitSectorsProving(ctx, map[abi.SectorNumber]struct{}{snum: {}})
	require.NotNil(t, *doneResp)
	require.True(t, (*doneResp).Successful)
	require.Equal(t, "Proving", (*doneResp).State)
	require.NotNil(t, (*doneResp).CommitMessage)
	}
	func remoteCommit1(s *ffiwrapper.Sealer) func(w http.ResponseWriter, r *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
	vars := mux.Vars(r)
	// validate sector id
	id, err := storiface.ParseSectorID(vars["id"])
	if err != nil {
	w.WriteHeader(500)
	return
	}
	var params api.RemoteCommit1Params
	if err := json.NewDecoder(r.Body).Decode(&params); err != nil {
	w.WriteHeader(500)
	return
	}
	sref := storiface.SectorRef{
	ID: id,
	ProofType: params.ProofType,
	}
	ssize, err := params.ProofType.SectorSize()
	if err != nil {
	w.WriteHeader(500)
	return
	}
	p, err := s.SealCommit1(r.Context(), sref, params.Ticket, params.Seed, []abi.PieceInfo{
	{
	Size: abi.PaddedPieceSize(ssize),
	PieceCID: params.Unsealed,
	},
	}, storiface.SectorCids{
	Unsealed: params.Unsealed,
	Sealed: params.Sealed,
	})
	if err != nil {
	w.WriteHeader(500)
	return
	}
	if _, err := w.Write(p); err != nil {
	fmt.Println("c1 write error")
	}
	}
	}
	func remoteDone(rs **api.RemoteSealingDoneParams) func(w http.ResponseWriter, r *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
	*rs = new(api.RemoteSealingDoneParams)
	if err := json.NewDecoder(r.Body).Decode(*rs); err != nil {
	w.WriteHeader(500)
	return
	}
	w.WriteHeader(200)
	}
	}
	func remoteCommit2(s *ffiwrapper.Sealer) func(w http.ResponseWriter, r *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
	var params api.RemoteCommit2Params
	if err := json.NewDecoder(r.Body).Decode(&params); err != nil {
	w.WriteHeader(500)
	return
	}
	sref := storiface.SectorRef{
	ID: params.Sector,
	ProofType: params.ProofType,
	}
	p, err := s.SealCommit2(r.Context(), sref, params.Commit1Out)
	if err != nil {
	fmt.Println("c2 error: ", err)
	w.WriteHeader(500)
	return
	}
	if _, err := w.Write(p); err != nil {
	fmt.Println("c2 write error")
	}
	}
	}
	func remoteGetSector(sectorRoot string) func(w http.ResponseWriter, r *http.Request) {
	return func(w http.ResponseWriter, r *http.Request) {
	vars := mux.Vars(r)
	// validate sector id
	id, err := storiface.ParseSectorID(vars["id"])
	if err != nil {
	w.WriteHeader(500)
	return
	}
	// validate type
	_, err = spaths.FileTypeFromString(vars["type"])
	if err != nil {
	w.WriteHeader(500)
	return
	}
	typ := vars["type"]
	if typ == "cache" {
	// if cache is requested, send the finalized cache we've created above
	typ = "fin-cache"
	}
	path := filepath.Join(sectorRoot, typ, vars["id"])
	stat, err := os.Stat(path)
	if err != nil {
	w.WriteHeader(500)
	return
	}
	if stat.IsDir() {
	if _, has := r.Header["Range"]; has {
	w.WriteHeader(500)
	return
	}
	w.Header().Set("Content-Type", "application/x-tar")
	w.WriteHeader(200)
	err := tarutil.TarDirectory(path, w, make([]byte, 1<<20))
	if err != nil {
	return
	}
	} else {
	w.Header().Set("Content-Type", "application/octet-stream")
	// will do a ranged read over the file at the given path if the caller has asked for a ranged read in the request headers.
	http.ServeFile(w, r, path)
	}
	fmt.Printf("served sector file/dir, sectorID=%+v, fileType=%s, path=%s\n", id, vars["type"], path)
	}
	}