fs integrity check on boot

[mudler]

Is your feature request related to a problem? Please describe.
In a deployment of mine it happened that COS_PERSISTENT was corrupted and prevented the system to boot. As I could not access to the machine except ssh from rescue mode (from host provider) and control panel it makes it really hard to debug

Describe the solution you'd like
A default cloud-config setting that checks fs in early stages and attempt to fix errors. ideally should output logs to state partition so they can be viewed afterwards, so it can just enable ideally systemd-fsck (although might not fit our partitioning logics https://github.com/systemd/systemd/blob/main/src/fsck/fsck.c#L298) . We need a mechanism to propagate back this to derivatives from a common place (like, we do have already meta/cos-light for an opinionated default)

Describe alternatives you've considered
Let derivatives handle some sane default setting by themselves, and just adding appropriate docs for it

Additional context

[mudler]

Testing locally by trying to re-use systemd-fsck

[mudler]

I've triggered an ext corruption of COS_PERSISTENT with (which should be repairable by fsck):

dd if=/dev/zero count=1 bs=4096 seek=0 of=/dev/XXXX

The system stalls during initramfs booting:

[mudler]

I couldn't work on this card today, so I'm leaving my notes from yesterday:

I'm trying to change cos-loop-img.sh and cos-mount-layout.sh by calling systemd-fsck appropriately, but currently my generated initramfs fail booting even LiveCD when customizing even comments on those files 🙄 , trying still to debug the reason why.