Skip to content
This repository has been archived by the owner on Jul 18, 2024. It is now read-only.

[DataCap Application] <Fogmeta Labs> - <Chainsnap> #1999

Open
1 of 2 tasks
hengdingy opened this issue May 18, 2023 · 84 comments
Open
1 of 2 tasks

[DataCap Application] <Fogmeta Labs> - <Chainsnap> #1999

hengdingy opened this issue May 18, 2023 · 84 comments

Comments

@hengdingy
Copy link

Data Owner Name

Fogmeta Labs

What is your role related to the dataset

Data Preparer

Data Owner Country/Region

China

Data Owner Industry

Web3 / Crypto

Website

https://fogmetalabs.com/chainsnap

Social Media

Twitter: https://twitter.com/FogMeta
GitHub: https://github.com/FogMeta

Total amount of DataCap being requested

15PiB

Weekly allocation of DataCap requested

1PiB

On-chain address for first allocation

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

  • Use Custom Multisig

Share a brief history of your project and organization

Chainsnap is a project developed by Fogmeta Labs for maintaining blockchain snapshots. This project enables the storing and retrieval of snapshots for blockchain projects including Aptos, Avalanche, Binance Smart Chain, Ethereum, Filecoin,,Near, Pocket, and Polygon. Please refer to: https://github.com/FogMeta/chainsnap.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

These are public blockchain snapshots that are maintained by the network communities.
Please refer to the links presented below:
1. Aptos: https://bwarelabs.com/snapshots
2. Avalanche: http://186.233.187.26/
3. BSC: https://github.com/bnb-chain/bsc-snapshots
4. Ethereum: https://github.com/allada/eth-archive-snapshot
5. Filecoin: https://lotus.filecoin.io/lotus/install/start-lotus/
6. NEAR: https://near-nodes.io/intro/node-data-snapshots
7. Pocket: https://github.com/pokt-foundation/pocket-snapshotter
8. Polygon: https://snapshots.polygon.technology/

Where was the data currently stored in this dataset sourced from

Other

If you answered "Other" in the previous question, enter the details here

These snapshots are maintained by 1)the network teams, 2) community volunteers, 3) ourselves and are from several sources, such as AWS Cloud and our own storage infra.

How do you plan to prepare the dataset

IPFS, lotus, graphsplit, others/custom tool

If you answered "other/custom tool" in the previous question, enter the details here

We're going to use the `go-mc-sdk` to prepare the datasets. 
https://github.com/FogMeta/go-mc-sdk

Please share a sample of the data

1. Aptos
- https://snapshots.bwarelabs.com/aptos/mainnet/aptos20230517.tar.lz4 (133G)
- https://snapshots.bwarelabs.com/aptos/testnet/aptos20230517.tar.lz4 (179G)

2. Avalanche
- https://youravaxsnapshot.senseinode.com/avalanche_mainnet_20230507.tar (828G)

3. BSC
- https://pub-c0627345c16f47ab858c9469133073a8.r2.dev/geth-20230516.tar.lz4(1.9T)

4. Ethereum
- s3://public-blockchain-snapshots/eth

5. Filecoin
- https://snapshots.mainnet.filops.net/minimal/latest.zst(45.7G)

6. NEAR
- s3://near-protocol-public/backups/testnet/rpc/latest (469.2G)
- s3://near-protocol-public/backups/mainnet/rpc/latest (473.7G)
- s3://near-protocol-public/backups/testnet/archive/latest (5.2T)
- s3://near-protocol-public/backups/mainnet/archive/latest (9.0T)

7. Pocket
- https://snapshot.nodes.pokt.network/latest.tar (575G)

8. Polygon:
- https://snapshot-download.polygon.technology/bor-mainnet-incremental-compiled-files.txt (1467.43 GB)
- https://snapshot-download.polygon.technology/bor-mumbai-incremental-compiled-files.txt (250.05 GB)
- https://snapshot-download.polygon.technology/heimdall-mainnet-incremental-compiled-files.txt (218.07 GB)
- https://snapshot-download.polygon.technology/heimdall-mumbai-incremental-compiled-files.txt (34.40 GB)
- https://snapshot-download.polygon.technology/erigon-mainnet-archive-2023-04-21.tar.gz (2210.24 GB)
- https://snapshot-download.polygon.technology/erigon-mumbai-archive-2023-04-21.tar.gz (228.66 GB)

Confirm that this is a public dataset that can be retrieved by anyone on the Network

  • I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Monthly

For how long do you plan to keep this dataset stored on Filecoin

2 to 3 years

In which geographies do you plan on making storage deals

Greater China, Asia other than Greater China, Africa, North America, South America, Europe, Australia (continent), Antarctica

How will you be distributing your data to storage providers

HTTP or FTP server, IPFS, Shipping hard drives, Others

How do you plan to choose storage providers

Slack, Partners, Others

If you answered "Others" in the previous question, what is the tool or platform you plan to use

FilSwan platform (https://filswan.com/) is another good choice for us to choose storage providers who meet our requirements.

If you already have a list of storage providers to work with, fill out their names and provider IDs below

The storage providers we'd like to work with are presented below. Some of them are from the FilSwan platform.
f03624
f010088
f02301
f08399
f02401
f0187709
f01163272
f01402814
f01072221
f0240185
f0143858
f01390330
f01225882
f0717969
f03223
f01395673
f01786736
f0836160
f032824
f01443744
f01871352
f01907556
f01946551
f02012951
f01970630

How do you plan to make deals to your storage providers

Boost client, Lotus client, Others/custom tool

If you answered "Others/custom tool" in the previous question, enter the details here

https://github.com/FogMeta/go-mc-sdk

Can you confirm that you will follow the Fil+ guideline

Yes

@large-datacap-requests large-datacap-requests bot added the very large application For LDN applications over 5+ PiB label May 18, 2023
@large-datacap-requests
Copy link

Thanks for your request!
Everything looks good. 👌

A Governance Team member will review the information provided and contact you back pretty soon.

@large-datacap-requests large-datacap-requests bot added very large application For LDN applications over 5+ PiB validated and removed very large application For LDN applications over 5+ PiB labels May 18, 2023
@Sunnyiscoming
Copy link
Collaborator

How much data does each of the following datasets have, and how many backups do you plan to store?

  1. Aptos: https://bwarelabs.com/snapshots
  2. Avalanche: http://186.233.187.26/
  3. BSC: https://github.com/bnb-chain/bsc-snapshots
  4. Ethereum: https://github.com/allada/eth-archive-snapshot
  5. Filecoin: https://lotus.filecoin.io/lotus/install/start-lotus/
  6. NEAR: https://near-nodes.io/intro/node-data-snapshots
  7. Pocket: https://github.com/pokt-foundation/pocket-snapshotter
  8. Polygon: https://snapshots.polygon.technology/

Do these storage providers offer retrieval services?

@hengdingy
Copy link
Author

hengdingy commented May 21, 2023

@Sunnyiscoming the snapshot size is as follows, Chainsnap will store a series of public chain snapshots (Including but not limited to the following chains), and every snapshot will be sent to 15 storage providers.

Our backup plan and deal metadata can be seen here. These storage providers are from filswan platform, which is required to offer retrieval services.

Chainsnap product introduction can be found https://fogmetalabs.com/chainsnap

  1. Aptos
  1. Avalanche
  1. BSC
  1. Ethereum
  • s3://public-blockchain-snapshots/eth
  1. Filecoin
  1. NEAR
  • s3://near-protocol-public/backups/testnet/rpc/latest (469.2G)
  • s3://near-protocol-public/backups/mainnet/rpc/latest (473.7G)
  • s3://near-protocol-public/backups/testnet/archive/latest (5.2T)
  • s3://near-protocol-public/backups/mainnet/archive/latest (9.0T)
  1. Pocket
  1. Polygon:
  1. SUI:
  • s3://mysten-mainnet-snapshots/ 294.0 GiB
  • s3://mysten-testnet-snapshots/ 1.1T

@Sunnyiscoming
Copy link
Collaborator

The data samples you provided is no more than 30 TB. And you want to apply for 15 PB datacap. How many copies will you store?
Can you provide all the types of public chain and how much data does each of them have?

@hengdingy
Copy link
Author

hengdingy commented May 22, 2023

@Sunnyiscoming
Chainsnap plans to store the public chains snapshot, including but not limited to the following chains:

  • APTOS chain snapshot, testnet and mainnet (about 312GiB)
  • Avalanche mainnet snapshot (about 828G)
  • BSC snapshot (about 1.9TiB)
  • Ethereum snapshot (about 2T)
  • Filecoin snapshot (45.7GiB latest + datastore 2TiB)
  • Near snapshot (about 15TiB)
  • Pocket snapshot(about 575GiB)
  • Polygon snapshot (about 4.4TiB)
  • SUI snapshot (testnet and mainnet, about 1.3T)
  • ...

the total size is about 30TiB, Chainsnap will backup these snapshots every week and will store 10 copies for every snapshot. And every piece will be 16GiB(requires 32GiB datacap), so 30TiB*1024/16GiB*32GiB*10/1024= 600TiB/week. Chainsnap needs 2.4PiB datacap per month, 15PiB datacap is just can be used about 6 months

@Sunnyiscoming
Copy link
Collaborator

Large datasets are open to existing datasets. May I ask what data sources do you have for existing datasets? What is the corresponding data amount?
According to the information you provided, you only have 30 TB data at present, right? If so, it is recommended that you apply directly to notaries for Datacap.

@Sunnyiscoming
Copy link
Collaborator

Any update here?

@hengdingy
Copy link
Author

@Sunnyiscoming Thanks for your review, Chainsnap will download the snapshots from the above link, which is maintained by the public chain team or ecosystem project team. We are downloading every chain snapshot by the two 10Gbps bandwidths(the 30TiB dataset will be downloaded every week). And we have maintained some chain snapshots, such as Filecoin, Polygon, BSC, Pocket and so on, we will export and store these snapshot every weeks.
you can see the website: https://fogmetalabs.com/demo/All

More and more snapshot datasets will be downloaded, and hope you can support the Chainsnap project.

@cryptowhizzard
Copy link

This one looks ok. Waiting for the datacap trigger

@Sunnyiscoming
Copy link
Collaborator

Weekly allocation of DataCap requested
1PiB

the 30TiB dataset will be downloaded every week

There is a big difference between the two Numbers. Can you explain it?

@hengdingy
Copy link
Author

@Sunnyiscoming the 30TiB dataset will be downloaded every week, we will store the dataset, Chainsnap will backup these snapshots every week and will store 10 copies for every snapshot. And every piece will be 16GiB(requires 32GiB datacap), so 30TiB*1024/16GiB*32GiB*10/1024= 600TiB/week. But in the first round, there is just 256TiB datacap will be allocated.

Chainsnap needs 2.4PiB datacap per month, 15PiB datacap is just can be used about 6 months.

@Sunnyiscoming
Copy link
Collaborator

Datacap Request Trigger

Total DataCap requested

15PiB

Expected weekly DataCap usage rate

1PiB

Client address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

@large-datacap-requests
Copy link

DataCap Allocation requested

Multisig Notary address

f02049625

Client address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

DataCap allocation requested

512TiB

Id

ba088081-eca2-49b8-8fe1-f4c500f4db54

@large-datacap-requests large-datacap-requests bot added very large application For LDN applications over 5+ PiB ready to sign and removed validated very large application For LDN applications over 5+ PiB labels May 31, 2023
Copy link

nj-steve commented Jun 9, 2023

Request Proposed

Your Datacap Allocation Request has been proposed by the Notary

Message sent to Filecoin Network

bafy2bzacedcqflbjuvoonlqud2k5kskaqqw4hd6u5al5pijvqfbm7g7m4n6qw

Address

f172azgjopbvo4zmhc3ph7epx3lmknwf3h4o6wvtq

Datacap Allocated

512.00TiB

Signer Address

f1xx6555qijma7igpnjspyvdunc4vfxkawnpqy5ii

Id

ba088081-eca2-49b8-8fe1-f4c500f4db54

You can check the status of the message here: https://filfox.info/en/message/bafy2bzacedcqflbjuvoonlqud2k5kskaqqw4hd6u5al5pijvqfbm7g7m4n6qw

@large-datacap-requests large-datacap-requests bot added very large application For LDN applications over 5+ PiB ready to sign and removed ready to sign very large application For LDN applications over 5+ PiB labels Jun 9, 2023
@hengdingy
Copy link
Author

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

@Sunnyiscoming
Copy link
Collaborator

I saw the form. But it do not cover the information of organization and city of sps. Can you add these information in the form you provided?

@hengdingy
Copy link
Author

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

@Sunnyiscoming
Copy link
Collaborator

SP List provided:
[{"providerID": "f02401","City": "XYZ", "Country": "US", "SPOrg","XYZ"},
{"providerID": "f0447183","City": "XYZ", "Country": "CA", "SPOrg","XYZ"},
{"providerID": "f01163272","City": "XYZ", "Country": "RU", "SPOrg","XYZ"},
{"providerID": "f01398391","City": "XYZ", "Country": "CN", "SPOrg","XYZ"},
{"providerID": "f010088","City": "XYZ", "Country": "US", "SPOrg","XYZ"},
{"providerID": "f01390330","City": "XYZ", "Country": "CN", "SPOrg","XYZ"},
{"providerID": "f0466405","City": "XYZ", "Country": "Europe", "SPOrg","XYZ"},
{"providerID": "f0717969","City": "NorthAmerica", "Country": "US", "SPOrg","XYZ"},
{"providerID": "f01428149","City": "XYZ", "Country": "KR", "SPOrg","XYZ"},
{"providerID": "f0240456","City": "XYZ", "Country": "CN", "SPOrg","XYZ"},
{"providerID": "f01225882","City": "XYZ", "Country": "CA", "SPOrg","XYZ"},
{"providerID": "f097618","City": "XYZ", "Country": "CN", "SPOrg","XYZ"},
{"providerID": "f01861488","City": "XYZ", "Country": "Europe", "SPOrg","XYZ"},
{"providerID": "f0137489","City": "XYZ", "Country": "Europe", "SPOrg","XYZ"},
{"providerID": "f02186655","City": "XYZ", "Country": "CN", "SPOrg","XYZ"},
{"providerID": "f01927163","City": "XYZ", "Country": "KR", "SPOrg","XYZ"},
{"providerID": "f01900525","City": "XYZ", "Country": "KR", "SPOrg","XYZ"},
{"providerID": "f01924824","City": "XYZ", "Country": "KR", "SPOrg","XYZ"},
{"providerID": "f01924891","City": "XYZ", "Country": "KR", "SPOrg","XYZ"},]

@hengdingy
Copy link
Author

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Copy link

github-actions bot commented Jan 2, 2024

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

--
Commented by Stale Bot.

@hengdingy
Copy link
Author

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Copy link

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

--
Commented by Stale Bot.

@hengdingy
Copy link
Author

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Copy link

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

--
Commented by Stale Bot.

@hengdingy
Copy link
Author

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Copy link

github-actions bot commented Feb 6, 2024

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

--
Commented by Stale Bot.

@Kirameta
Copy link

Kirameta commented Feb 6, 2024

manualTrigger

Copy link

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

--
Commented by Stale Bot.

@hengdingy
Copy link
Author

manualTrigger

Copy link

This application has not seen any responses in the last 10 days. This issue will be marked with Stale label and will be closed in 4 days. Comment if you want to keep this application open.

--
Commented by Stale Bot.

@hengdingy
Copy link
Author

manualTrigger

@Normalnoise
Copy link

checker:manualTrigger

Copy link

DataCap and CID Checker Report Summary1

Storage Provider Distribution

✔️ Storage provider distribution looks healthy.

Deal Data Replication

✔️ Data replication looks healthy.

Deal Data Shared with other Clients2

⚠️ CID sharing has been observed. (Top 3)

Full report

Click here to view the CID Checker report.
Click here to view the Retrieval Dashboard.

Footnotes

  1. To manually trigger this report, add a comment with text checker:manualTrigger

  2. To manually trigger this report with deals from other related addresses, add a comment with text checker:manualTrigger <other_address_1> <other_address_2> ...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests