Initial MDRAID support, F2FS documentation

[Harvie]

This allows to create level 1 MDRAID with 1 device. Can be empty or prepopulated with data image.
Includes unit test using mdadm to check generated image.

Eg.:

image mdraid-md.img {
  mdraid {
    level = 1
  }
  partition data {
    image = "mdraid-ext4.img"
  }
}

It might sound stupid to create single device raid, but it actualy fits my reallife usecase where i pre-generate such raid when making image and user then can very easily add more devices during runtime when needed later. For example like this:

mdadm --grow /dev/md23 --raid-disks=2 --force
maddm /dev/md23 --add /dev/sdb1

Although it should be simple to pre-generate images for all raid members in genimage. It shouldn't be much harder than generating multiple images with the same RAID UUID and few metadata changes. But i haven't researched that so far.

Also see #191

[Harvie]

I have no idea why the test fails here. It says file mdraid.config is missing, but it's obviously there.
Also mdraid tests are passing at my PC when i do make check-TESTS:

[michaelolbrich]

https://github.com/pengutronix/genimage/actions/runs/12428468239/job/34700133730?pr=277#step:6:1000

The test run make distcheck which runs the test from the generated tarball. You forgot to add the test config to EXTRA_DIST so it's missing.

[Harvie]

Seems to be fixed now.

[Harvie]

Happy holidays.
@michaelolbrich I think i've resolved most of your objections, but i think i could really use your guidance about the multi-image thing. While i am not gonna implement generating multi device raid now in the first mdraid PR, i think it's bound to happen in the future and i would like to prepare the codebase for it, so we don't have to change config syntax in the future. What do you think would be the correct way of doing that?

It would kinda make sense for the image-mdraid.c to actualy generate two (or more) images, all being part of that RAID1 mirror. But on the other hand i don't think this would allow genimage to resolve dependencies correctly. Maybe i should manualy create other image structs and add them to list in mdraid_setup() ? would that be enough for image-hd to find them and trigger the build of all of them when needed? what would you suggest?

[Harvie]

One approach might be specifing all the output images of that array in single mdraid image node.

But maybe i should just do something like this instead:

image raid1-a.img {
  mdraid {
    level = 1
    devices = 2
  }
  partition data {
    image = "mdraid-ext4.img"
  }
}

image raid1-b.img {
  mdraid {
    master = raid1-a.img    #most of the config is gonna be inherited from master
    position = 2      #this is 2nd device in the array described by master image
  }
}

and then B image can lookup all the configuration from A image config node... but still this does not guarantee that B will be only generated after A. (UUIDs and other details of the A image need to be decided before generating B).

Is there way to enforce dependency?

[Harvie]

OK, i did binary diff of superblocks of two disks belonging to same RAID1 and highlighted the important parts:

Some things i am confused about:

• Dev numbers 0 and 2 (why skipped 1? maybe the first disk should be 1, not 0?)
• Why is there 3rd role with 0x0100 ???? Maybe the system is really confused about device ids starting at 0

Update: oh, the active role is simply number of the disk in the array (not sure why, because we already have DEV_NUMBER)

[harvie@anemophobia mdadm]$ grep DISK_ROLE md_p.h 
#define MD_DISK_ROLE_SPARE	0xffff
#define MD_DISK_ROLE_FAULTY	0xfffe
#define MD_DISK_ROLE_JOURNAL	0xfffd
#define MD_DISK_ROLE_MAX	0xff00 /* max value of regular disk role */

UPDATE: i did some research on this and it seems there are no rules on how the system should number the devices and roles down the road and that 0xFFFF roles are just OK to be ignored when there are no such disks. Therefore i no longer worry about this as long as the image we generate makes sense.

[Harvie]

I am now able to create two partitions belonging to single raid like this:

image mdraid-hd.img {

  hdimage {
    partition-table-type = "gpt"
  }

  partition mdraid-a {
    image = "mdraid-a.img"
    partition-type-uuid = R
  }

  partition mdraid-b {
    image = "mdraid-b.img"
    partition-type-uuid = R
  }

}

image mdraid-a.img {
	mdraid {
		devices = 2
		role = 0
		timestamp = 638022222
		raid-uuid = "de9980f1-0449-4e83-84bd-98e4b1ca3fe3"
		image = "mdraid-ext4.img"
	}
}

image mdraid-b.img {
	mdraid {
		devices = 2
		role = 1
		timestamp = 638022222
		raid-uuid = "de9980f1-0449-4e83-84bd-98e4b1ca3fe3"
		image = "mdraid-ext4.img"
	}
}

image mdraid-ext4.img {
  ext4 {
    label = "TEST_FS"
  }
  size = 5M
}

When i do losetup -fP mdraid-hd.img the kernel automaticaly recognizes both partitions as members of the array and assembles it automaticaly, while reporting the filesystem in the raid to be clean:

# cat /proc/mdstat 
Personalities : [raid1] 
md127 : active raid1 loop2p2[1] loop2p1[0]
      5120 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk
      
unused devices: <none>

# LANG=C fsck.ext4 /dev/md127 
e2fsck 1.47.1 (20-May-2024)
TEST_FS: clean, 12/1280 files, 1434/5120 blocks

[Harvie]

Great news. With last version the required metadata fields are automaticaly exchanged and synced between components. Therefore there is no longer need to manualy set indentical array UUID, timestamp and data to all images.

image mdraid-hd.img {

  hdimage {
    partition-table-type = "gpt"
  }

  partition mdraid-a {
    image = "mdraid-a.img"
    partition-type-uuid = R
  }

  partition mdraid-b {
    image = "mdraid-b.img"
    partition-type-uuid = R
  }

}

image mdraid-a.img {
	mdraid {
		level = 1
		devices = 2
		image = "mdraid-ext4.img"
	}
}

image mdraid-b.img {
	mdraid {
		parent = "mdraid-a.img"
	}
}

image mdraid-ext4.img {
  ext4 {
    label = "TEST_FS"
  }
  size = 5M
}

You only need to specify array format for one of the images and other images can refer to it by setting parent = "file.img" config option. Roles numbers can be assigned automaticaly for all inheritant images, when omitted.

[Harvie]

I think i am ready here. I've implemented all the features i can wish for. Added tests, documented everything. And also added documentation for f2fs, which is something i kinda owed since my f2fs PR was merged 👼

[michaelolbrich]

Mostly cosmetic stuff left.

And please squash all the mdraid commits.

[Harvie]

And please squash all the mdraid commits.

Is it OK if you just click squash and merge instead of merge?

[michaelolbrich]

And please squash all the mdraid commits.

Is it OK if you just click squash and merge instead of merge?

Hmm, I thought the F2FS stuff was already a separate commit but it's not. That should really be separate. If you create a separate PR for that and remove it here, then I can do that (hopefully I won't forget that... :-))

[Harvie]

Hmm, I thought the F2FS stuff was already a separate commit but it's not. That should really be separate.

OK, squashed this PR into two commits. MDRAID and F2FS.