Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataCap Refresh] <3rd> Review of <ZCFIL+> #260

Open
zcfil opened this issue Dec 17, 2024 · 9 comments
Open

[DataCap Refresh] <3rd> Review of <ZCFIL+> #260

zcfil opened this issue Dec 17, 2024 · 9 comments
Assignees
Labels
Awaiting RKH Refresh request has been verified by Public, Watchdog, and Governance - now awaiting release of DC DataCap - Refreshed Refresh Applications received from existing Allocators for a refresh of DataCap allowance

Comments

@zcfil
Copy link

zcfil commented Dec 17, 2024

Basic info

  1. Type of allocator: manual
  1. Paste your JSON number: (v5 Notary Allocator Application:ZCFIL notary-governance#1009)

  2. Allocator verification: yes

  1. Allocator Application
  2. Compliance Report
  1. Previous reviews

Current allocation distribution

Client name DC granted
FengwoExtraordinary 2 PiB

I. FengwoExtraordinary

  • DC requested: 15 PiB
  • DC granted so far: 6.19 PiB

II. Dataset Completion

https://drive.google.com/drive/folders/1cTLYO-9vumVrpT0aY0QQAGl5E6N_boT9?usp=sharing

III. Does the list of SPs provided and updated in the issue match the list of SPs used for deals?

yes
IV. How many replicas has the client declared vs how many been made so far:

10 vs 12
Clients have been explained accordingly and reports are generated on an ongoing basis for observation with no significant subsequent increases
image

V. Please provide a list of SPs used for deals and their retrieval rates

SP ID % retrieval Meet the >75% retrieval?
f02201190 1.06% no
f01969779 0.03% no
f03175168 65.89% no
f03144777 0.00% no
f02866666 0.00% no
f02808877 60.86% no
f02808899 67.38% no
f02820000 0.00% no
f02639655 0.00% no
f03166688 0.00% no
f03226666 71.82% no
f03226668 65.13% no
f03175111 69.39% no
f02588263 0.00% no
f03253497 47.20% no
f03253580 84.03% yes

Most sp's are close to 75% with 0 retrieval rate in the ongoing generation of reports have not found that the client continues to send deal to that sp Have been following up on the retrieval rate issue on an ongoing basis

Allocation summary

  1. Notes from the Allocator

Issues identified in the previous round have been addressed accordingly

Accurate regional distribution, with VPNs accounted for and explained

Have received most of the sp's geo-location references such as server room bills, house bills, etc., all confirmed with the

image

Increase retrieval scores: current average is ~30%

Currently upgraded
Average retrievability success rate: 41.14%

  1. Did the allocator report up to date any issues or discrepancies that occurred during the application processing?

Yes, it's well documented, and we'll be the first to raise issues as they arise in our observations.

  1. What steps have been taken to minimize unfair or risky practices in the allocation process?

Regularly generate reports, retrieve sector data, etc.

  1. How did these distributions add value to the Filecoin ecosystem?

All are data types related to sustainable human development

  1. Please confirm that you have maintained the standards set forward in your application for each disbursement issued to clients and that you understand the Fil+ guidelines set forward in your application

Yes

  1. Please confirm that you understand that by submitting this Github request, you will receive a diligence review that will require you to return to this issue to provide updates.

Yes

@Kevin-FF-USA Kevin-FF-USA added Refresh Applications received from existing Allocators for a refresh of DataCap allowance Awaiting Community/Watchdog Comment DataCap Refresh requests awaiting a public verification of the metrics outlined in Allocator App. labels Dec 17, 2024
@Kevin-FF-USA
Copy link
Collaborator

HI @zcfil,
Thanks for using this template to submit a refresh request. Wanted to check in regarding timelines, with the holidays this application wont receive a Watchdog and Governance close out until January. Hoped to keep you updated so you could expect when a refresh would be coming. Warmly,
-Kevin

@filecoin-watchdog
Copy link
Collaborator

@zcfil
FengwoExtraordinary

  1. Has the client informed you in any way that they are actively receiving DataCap (DC) for this dataset from two other allocators? All these applications are active, and the client is receiving DC from all of them. The government team should follow up on this matter.
    Relevant issues:
    Genesis
    STCloud

  2. In your first comment under the application form, you mentioned:

Firstly, before I issue the datacap, I need to provide a deposit of 100 fil to deduct any subsequent fees and ensure that your data is consistent with the applied data, as well as any possible cheating situations such as data distribution. If any discrepancies are found, they will be deducted in full

Could you clarify what is ensuring data consistency in this case? Is it something you perform somehow?

  1. This "too many replicas" issue has occurred before, as highlighted in the previous review. This is now the second instance of the same problem.
    Also, why did the allocator agree to change the requested DataCap in the form? Did the client specifically request this? (I couldn’t find such a request in the GitHub issue.)
    Initially, the parameters for this application were set at 10 PiB and 7 replicas. After the client had sealed too many copies, the allocator agreed to change the parameters to 15 PiB and 10 replicas. Yet again, the replicas are incorrect.
  2. The client frequently updates about adding new storage providers (SPs), and the allocator keeps editing the application form to match these changes. However, one SP (F03166688) was never announced but has still been used for deals.
  3. 5 out of 16 SPs currently have a retrieval rate of 0%.

One final thought is that you should carefully read through the second review of yours.

@filecoin-watchdog filecoin-watchdog added Awaiting Response from Allocator If there is a question that was raised in the issue that requires comment before moving forward. and removed Awaiting Community/Watchdog Comment DataCap Refresh requests awaiting a public verification of the metrics outlined in Allocator App. labels Jan 10, 2025
@zcfil
Copy link
Author

zcfil commented Jan 13, 2025

@zcfil FengwoExtraordinary

  1. Has the client informed you in any way that they are actively receiving DataCap (DC) for this dataset from two other allocators? All these applications are active, and the client is receiving DC from all of them. The government team should follow up on this matter.

The client didn't tell me.
I've browsed in detail and the client is applying for DC at other allocators after applying at me and at the time of my DataCap Refresh
My first review is on September 8th.

Relevant issues:
Genesis
STCloud

From here you can see that he hasn't gotten a DC in that path since September 8
image

  1. In your first comment under the application form, you mentioned:

Firstly, before I issue the datacap, I need to provide a deposit of 100 fil to deduct any subsequent fees and ensure that your data is consistent with the applied data, as well as any possible cheating situations such as data distribution. If any discrepancies are found, they will be deducted in full

Could you clarify what is ensuring data consistency in this case? Is it something you perform somehow?

Here I will periodically fetch the data to ensure data consistency
I aim to urge my clients to move in the right direction and will not confiscate their FILs as long as they have done what they promised, such as the following "too many replicas" issue

  1. This "too many replicas" issue has occurred before, as highlighted in the previous review. This is now the second instance of the same problem.

I compared it in detail and it matches what the client explained, so it's not the second time
The second review was also on December 4th
Client's explanation of Dec. 4
Report for December 4th
Current report

Also, why did the allocator agree to change the requested DataCap in the form? Did the client specifically request this? (I couldn’t find such a request in the GitHub issue.)
Initially, the parameters for this application were set at 10 PiB and 7 replicas. After the client had sealed too many copies, the allocator agreed to change the parameters to 15 PiB and 10 replicas. Yet again, the replicas are incorrect.

Client discloses new co sp in advance as well as copy modification to 10
Client proposes to amend to 10 copies ago report
You can see that this report is still 7 copies
So this is a modification proposed before sealing 10 copies. Due to the long time period and the addition of new SP partners, there will be new copies and more DC requests. I don't think there is any problem with this
image

  1. The client frequently updates about adding new storage providers (SPs), and the allocator keeps editing the application form to match these changes.

My understanding is that due to the long time period for DC acquisition, the client may be working with different SPs to store copies during that time, so that's the reason for the frequent sp updates

However, one SP (F03166688) was never announced but has still been used for deals.

In my ongoing generation of reports it looks like the sp has not gotten too many deals, it has been consistently at 64GiB and has an ok retrieval rate, maybe for testing?So I'm ignoring it.I'll keep an eye on this in the follow-up
image
The question has been asked, thank you for your understanding
image

  1. 5 out of 16 SPs currently have a retrieval rate of 0%.

Client have explained that they will not resend orders to SPs with 0 retrieval rate
image

And based on the reports generated on an ongoing basis, the client did this

One final thought is that you should carefully read through the second review of yours.

Galen asked the following questions and some of the work I've done.

image

@zcfil
Copy link
Author

zcfil commented Jan 13, 2025

@filecoin-watchdog Thank you for your detailed review, if there is anything you don't understand or have questions please feel free to point them out, thank you!

@filecoin-watchdog
Copy link
Collaborator

The client didn't tell me.
I've browsed in detail and the client is applying for DC at other allocators after applying at me and at the time of my DataCap Refresh
My first review is on September 8th.

That's true, but raising this issue with the client might be good. It's not very transparent of them to perform this kind of behaviour.

The client frequently updates about adding new storage providers (SPs), and the allocator keeps editing the application form to match these changes.

My understanding is that due to the long time period for DC acquisition, the client may be working with different SPs to store copies during that time, so that's the reason for the frequent sp updates

That was a positive comment. Keeping applications up to date is very good behaviour.

I don't have other notes or questions. Thank you.

@filecoin-watchdog filecoin-watchdog added Diligence Audit in Process Governance team is reviewing the DataCap distributions and verifying the deals were within standards and removed Awaiting Response from Allocator If there is a question that was raised in the issue that requires comment before moving forward. labels Jan 13, 2025
@zcfil
Copy link
Author

zcfil commented Jan 14, 2025

@filecoin-watchdog Some of the translations may not be accurate, I made the changes, thank you again!

@Kevin-FF-USA
Copy link
Collaborator

Hi @zcfil,

Thanks for submitting this application for refresh.
Wanted to send you a friendly update - as this works its way through the system you should see a comment from Galen on behalf of the Governance this week. If you have any questions or need support until then, please let us know.

Warmly,
-Kevin

@galen-mcandrew
Copy link
Collaborator

Overall good compliance here, and summarizing areas to continue focusing on:

  • Accurate dataset size calculations, with minimal overhead or padding
  • Continue to reduce excessive duplicates across the network
  • Continue to increase geographic distribution to ensure data availability
  • Updating bookkeeping and records to maintain accuracy, such as updating SP lists
  • Increasing retrieval rates

To help support this increased compliance and data onboarding, we are requesting 5PiB of DataCap from RKH.

@Kevin-FF-USA Kevin-FF-USA added Awaiting RKH Refresh request has been verified by Public, Watchdog, and Governance - now awaiting release of DC and removed Diligence Audit in Process Governance team is reviewing the DataCap distributions and verifying the deals were within standards labels Feb 10, 2025
@Kevin-FF-USA
Copy link
Collaborator

Hi @zcfil

Friendly update on this refresh.

We are currently in the process of moving to a Metaallocator. In order for the tooling to work correctly an allocator can only use the DataCap balance they received through direct allocation from Root Key Holders, >>> or the DataCap received through Metaallocator. As a result, some of the metrics pages like Datacapstats, Pulse and other graphs might be a little confused during this update.

You will not lose any of the DataCap, but you will see that your refresh is amount of DC from refresh + remaining DC an allocator has left.

No action needed on your part, just a friendly note to thank you for your contributions and patience, and you may notice changes in your DataCap balance while the back end is updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Awaiting RKH Refresh request has been verified by Public, Watchdog, and Governance - now awaiting release of DC DataCap - Refreshed Refresh Applications received from existing Allocators for a refresh of DataCap allowance
Projects
None yet
Development

No branches or pull requests

4 participants