Metadata rework discussion

[mulkieran]

For tracking work items for RAID support.

The first is work on making our metadata more relocatable.

[jbaublitz]

Pool metadata changes for RAID (and other layers)

Motivation

While discussing our metadata layout in one of our first meetings for RAID support, @mulkieran raised the concern of device discovery if we put encryption on top of RAID. We currently encrypt the Stratis superblock so there would be virtually no way to distinguish a Stratis RAID device from a RAID device managed by another software stack. The way we worked around this with encryption with by using a LUKS2 token to identify the device as a Stratis device, but there is no user supplied field in RAID that maps to this concept.

Proposed solution: metadata rework

We have discussed metadata reworks that could be put in place to solve this problem. One key feature that has showed up in all of the options is leaving the superblock unencrypted and putting it always at the beginning of the device. This would allow us to universally determine whether a device belongs to Stratis or not without having to worry about whether the lowest layer provides a user provided field or not. We also received some feedback from @DemiMarie that it's desirable to encrypt the pool level metadata, something we expected to be a requirement for most users using encryption.

This has led us to the current design options that we see.

We would like to avoid using a second crypt devicemapper device just for the pool level metadata due to the overhead of managing devicemapper devices. For metadata that can be simply run through AES to generate a buffer from which JSON can be deserialized, AES encryption on metadata updates and decryption when a pool is set up seems to be the best balance of architectural simplicity and security.

Option 1: Encrypt pool level metadata using cryptsetup and openssl

Suggested by @oniko.

The general idea here would be to have a minimally-sized cryptsetup header to store an encrypted key in a keyslot that would then be used for AES encryption of the pool level metadata instead of unlocking a crypt device. The crypt header would allow us to reuse certain parts of cryptsetup like their PBKDF2 management without having to store things like the salt ourselves in our metadata. We would still need to manage AES encryption ourselves, but it would definitely be less work than implementing everything at the openssl level.

Option 2: Encrypt pool level metadata using openssl alone

This would require using the openssl for two separate operations: PBKDF2 and AES. My understanding is that instead of the LUKS2 header, we would need some space for PBKDF2 (such as a salt and also the digest we're using if we don't hardcode it) and AES (such as an IV and other information if we intend to use authenticated encryption).

RAID considerations

Stacking

Currently, we are tending towards putting encryption on top of RAID and putting integrity beneath RAID. This would make RAID relatively low in the stack with only integrity beneath it based on the current roadmap.

Pros and cons

Pros

• If we put it on top of encryption, this would result in encryption amplification and potential negative performance impacts because each leg of RAID would be encrypted separately. If we put it below encryption, encryption will only need to be managed a single time.

Cons

• There doesn't seem to be a straightforward way of converting from no RAID to RAID.

Conversion from pools without redundancy to pools with redundancy

We have decided not to pursue allowing conversions from no redundancy to RAID for now.

[jbaublitz]

Adding some updates after discussions we've had with @mauelsha.

We will likely attempt to provide the ability to convert from linear devices to RAID 1E and back. This depends on whether we can reasonably move the encryption layer from underneath Stratis metadata to on top of th cap device.

[mulkieran]

We decided we would not pre-allocate metadata space for RAID. The principle thing is that we decided to place the encryption layer above the RAID layer for performance reasons.