Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected behavior with scheduled snapshots #1152

Closed
aaroncox opened this issue May 10, 2023 · 2 comments · Fixed by #1220
Closed

Unexpected behavior with scheduled snapshots #1152

aaroncox opened this issue May 10, 2023 · 2 comments · Fixed by #1220
Assignees
Labels
actionable bug Something isn't working 👍 lgtm

Comments

@aaroncox
Copy link
Contributor

The way scheduled snapshots behaves isn't what I'd anticipate, especially from an automation standpoint.

What I want to accomplish with automated snapshots is to be able to set an interval (without a start/end) and have it take snapshots at the heights which match the block_spacing. Currently I'm trying to achieve this by manually defining the snapshot-schedule.json as:

{
    "snapshot_requests": [
        {
            "snapshot_request_id": "0",
            "snapshot_description": "Hourly snapshots starting from genesis",
            "block_spacing": "7200",
            "start_block_num": "0",
            "end_block_num": "0"
        }
    ]
}

This approach creates an easy, default way for us to automate snapshots at a given height - without the need to make API calls after the node is up and running. With Docker I can simply drop in the above configuration file and know the node is going to take snapshots at a specific block height interval.

What I have discovered though using this configuration is that when nodeos starts from a snapshot of a random height, it almost immediately takes a new snapshot, despite the height not matching the block_spacing value. For example using a recent Jungle4 snapshot, the above configuration produces a file named:

snapshot-04865af1c6ab73887325b198445aad8d9e7d6037db2c307c5fbd7cf0b437975d.bin

With that block ID hash converting to a block number of 75913969.

This also automatically modified my snapshot-schedule.json file to have a start_block_num value of 75913969.

This block height wasn't when I expected a snapshot to occur, since I specified every 7200 blocks with a start height of 0. I would expect the first snapshot to wait and occur at block number 75909600 (divisible by 7200, no remainder) - and I wouldn't expect the JSON schedule to automatically change based on this startup behavior.

One of the goals with automated backups like this is so that we can have multiple providers creating snapshots at the exact same height, which provides some authenticity to the snapshots since everyone's snapshots at a given height will have the same file hash. My hope was that a simple configuration like the above would allow us to do that.

@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog May 10, 2023
@heifner heifner added bug Something isn't working actionable and removed triage labels May 10, 2023
@vladtr
Copy link
Contributor

vladtr commented May 10, 2023

Thank you for detailed report! start_block_num = 0 is a "special" case which is treated as missing/not specified start_block_num in API call. If it is not specified, it is assumed as "ASAP"/now. That will make a call to api without parameters similar to former create_snapshot call that does it "now" (but async here)
If you set start_block_num to any other value, like start_block_num = 1 you will get behavior closer to what you are expecting.
I believe the right solution would be to treat cases with "0" specified in a config file and "not specified" api request parameter separately

@vladtr vladtr self-assigned this May 10, 2023
@BenjaminGormanPMP
Copy link

New issue added to the sprint. FYI to @bhazzard.

@BenjaminGormanPMP BenjaminGormanPMP added this to the Leap v5.0.0-rc1 milestone May 10, 2023
@vladtr vladtr moved this from Todo to In Progress in Team Backlog May 25, 2023
@vladtr vladtr moved this from In Progress to Awaiting Review in Team Backlog May 31, 2023
vladtr added a commit that referenced this issue Jun 5, 2023
Extended snapshot scheduler to better handle missing and zero params
@github-project-automation github-project-automation bot moved this from Awaiting Review to Done in Team Backlog Jun 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
actionable bug Something isn't working 👍 lgtm
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

5 participants