Arbitrary Empty Chunks ("Failed to load chunk")

[louis-e]

Hi there!
I've been playing around with this project for a few days, and I've encountered a persistent issue that I can't seem to resolve. The problem manifests as certain chunks randomly becoming empty, with the following error appearing in the latest.log logfile:

[01:31:28] [Server thread/ERROR]: Failed to load chunk 8,2
dwi$a: Failed to read PalettedContainer: Invalid length given for storage, got: 256 but expected: 342
	at com.mojang.serialization.DataResult$Error.getOrThrow(DataResult.java:287) ~[datafixerupper-8.0.16.jar:?]
	at dwi.a(SourceFile:115) ~[1.21.1.jar:?]
	at aqb.a(SourceFile:553) ~[1.21.1.jar:?]
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
	at bph.d(SourceFile:162) ~[1.21.1.jar:?]
	at aqs$b.d(SourceFile:552) ~[1.21.1.jar:?]
	at bph.B(SourceFile:136) ~[1.21.1.jar:?]
	at aqs$b.B(SourceFile:561) ~[1.21.1.jar:?]
	at aqs.d(SourceFile:263) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.bv(SourceFile:852) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.B(SourceFile:840) ~[1.21.1.jar:?]
	at bph.b(SourceFile:145) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.b(SourceFile:810) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.v_(SourceFile:815) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.b(SourceFile:508) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.t_(SourceFile:345) ~[1.21.1.jar:?]
	at guo.e(SourceFile:73) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.y(SourceFile:664) ~[1.21.1.jar:?]
	at net.minecraft.server.MinecraftServer.a(SourceFile:281) ~[1.21.1.jar:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Screenshot: House structure being cut off, the entire chunk does not contain any set blocks

This error occurs sporadically and is always associated with specific chunks, which are left entirely empty apart from the ground layer (grass, dirt, etc.) from the "template" input chunk. The issue appears random but consistently affects the same chunks on repeated runs with the same data set. With "data set" I'm referring to a defined input data, which gets parsed and "converted" into a Minecraft world structure.

Observations:
The problem seems to occur more frequently in chunks where multiple block types are being set. The block types are chosen randomly from a pre defined list, but the coordinates where they are being set, always remain the same.
If I force all blocks to be of a single type (e.g., sponges), the issue does not occur.

Due to the project size I can't share everything in this issue, but this is the shortened code in question which when left in, seems to cause the problem:

let variation_index = rng.gen_range(0..building_floor_variations().len());
let floor_block = &building_floor_variations()[variation_index]; // pub fn building_floor_variations() -> Vec<&'static Lazy<Block>>

let polygon_coords: Vec<(i32, i32)> = element.nodes.iter().copied().collect(); // element.nodes contains an erray of Minecraft X, Z coordinates
let floor_area = flood_fill_area(&polygon_coords, 2); // The flood_fill_area function always has the same output, no matter if the problem does occur or not

for (x, z) in floor_area {
	if processed_points.insert((x, z)) {
		editor.set_block(floor_block, x, ground_level, z); // Set floor
		
		// Set level ceilings if height > 4
		if building_height > 4 {
			for h in (ground_level + 4..ground_level + building_height).step_by(4) {                        
				if x % 6 == 0 && z % 6 == 0 {
					editor.set_block(&GLOWSTONE, x, h, z); // Light fixtures
				} else {
					editor.set_block(floor_block, x, h, z);
				}
			}
		} else if x % 6 == 0 && z % 6 == 0 {
			editor.set_block(&GLOWSTONE, x, ground_level + building_height, z); // Light fixtures
		}

		// Set the house ceiling
		editor.set_block(floor_block, x, ground_level + building_height + 1, z);
	}
}

I wrote a world_editor.rs handler for this which takes care of setting the blocks and saving the regions:
https://gist.github.com/louis-e/522c7656960b51854f7cb5d63b859175

Additional Notes:
The error suggests that the chunk data might be corrupted due to an incorrect PalettedContainer length, which seems to be linked to the blocks being set within these problematic chunks.
I've considered that the issue might be related to overwriting other blocks during generation, but I’ve implemented checks to prevent overwriting blocks that are already set, which has not resolved the problem.

Request:
I'm seeking advice or potential solutions to address this issue. Any insights into why certain chunks might be failing to load due to the PalettedContainer mismatch would be greatly appreciated. I'm particularly interested in understanding if this could be a bug in the chunk serialization or something that could be related to block palette handling.

I've been struggling with this problem for already several days and long nights, so I appreciate any kind of help. If needed, let me know and I can share the entire project in a GitHub repository. Thanks in advance for the help!

[louis-e]

Hi there, I was able to pinpoint my problem a bit more specific. The problem I have is with resizing the data field in block_states to support more than 16 block entries in the palette. Does anyone by any chance have a working example on how to do that?