Testnet state migration preview.22 health check after migration error

[j1010001]

flow-go build: v0.34.0-crescendo-preview.15-atree-inlining
migration doc: https://www.notion.so/flowfoundation/TN-state-migration-test-Atree-inlining-C1-0-Apr-24-preview-15-71afa024d73042da9c2bc020cb734850?pvs=4

error:
migration-inlined-payloads-apr24-preview15-check2-output.txt

[fxamacker]

@j1010001 status of this is "in-progress" (since yesterday afternoon).

• 🧵 https://discord.com/channels/613813861610684416/1234652799380488262/1234654088730378270

[fxamacker]

@turbolent I looked into this today and it looks like the problem is in Cadence 1.0 migration's MigrateNestedValue function.

cadence/migrations/migration.go

Line 165 in e29e0f3

func (m *StorageMigration) MigrateNestedValue(

It seems when dictionary element is migrated, MigrateNestedValue() returns nil as migrated value. So parent container is unchanged even though nested element is modified.

cadence/migrations/migration.go

Lines 363 to 365 in e29e0f3

	if newKey == nil && newValue == nil {
	continue
	}

With testnet data:

• dictionary (0xa47a2d3a3b7e9133.14553) is inlined and removed from storage as result of migration
• MigrateNestedValue() returns nil (as migrated value) for this dictionary even though it is migrated
• parent dictionary (0xa47a2d3a3b7e9133.16828) thinks child element isn't migrated (nil) so it still retain the old data (slab ID storable 0xa47a2d3a3b7e9133.14553) even though the child element should be reset.
• given this, storage health check returns slab not found error for 0xa47a2d3a3b7e9133.14553.

I am going to look into this more tomorrow. Please let me know if you have some questions.

[turbolent]

Thank you for the investigation @fxamacker!

I was able to reproduce this in flow-go, and started working on reproducer in the Cadence repo here: feature/atree-register-inlining-v1.0...bastian/debug-lost-slab-during-migration. So far I'm not able to reproduce it

[fxamacker]

I was able to reproduce this in flow-go, and started working on reproducer in the Cadence repo here: feature/atree-register-inlining-v1.0...bastian/debug-lost-slab-during-migration. So far I'm not able to reproduce it

@turbolent Thanks for working on this!

I created a reproducer just now (~9pm) in Cadence repo (without flow-go migration program).

I'll let you know in the morning after I double-check and clean it up before we chat. Not being familiar with Cadence 1.0 migration code tripped me up a few times while creating the reproducer.

[turbolent]

Awesome work @fxamacker! 👏 I'm going to look into it

[fxamacker]

Awesome work @fxamacker! 👏 I'm going to look into it

@turbolent Sounds great! I opened a PR with a test and a reproducer for you to use.

• Add reproducer as test for Cadence 1.0 migration issue #3288 #3314

The testnet data for this bug happened at 5 nested levels, so I simplified the reproducer to 3 nested levels.

Hope this reproducer saves time while you resolve this issue. 🙏

@j1010001 The reproducer doesn't include a fix.

[turbolent]

Fixed by #3318 and #3314