Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support full disk luks. Fixes #550 #1716

Merged
merged 76 commits into from
May 25, 2017
Merged

support full disk luks. Fixes #550 #1716

merged 76 commits into from
May 25, 2017

Conversation

phillxnet
Copy link
Member

Abstract:
Adds full disk LUKS UI components to perform initial LUKS format and set boot up configuration for the resulting container; with integral keyfile generation and registration where required. The consequent LUKS volumes also gain an ‘information only’ UI component to display their last recorded status which includes their backing container device. Further enhancements were required to the disk role improvements established in pr #1622 in order to meet the requirements of the UI components added. And to meet the expected pairing of LUKS data disk capability with encrypted system disk installs (via the “Encrypt my data” install option) the disk scan subsystem was improved to correctly propagate the otherwise incompatible LUKS in partition arrangement. Disk table view iconography was extended to indicate the open/closed status of LUKS containers.

Changes made:

  • Add capability to appropriately indicate system disk use on encrypted installs.
  • Improve iconic disk identification / tool tip info on disks page re LUKS.
  • Add LUKS format tick to role/wipe page with relevant text and title changes.
  • Add dual mode LUKS config/info page: Used for both containers and volumes.
  • Report/configure /etc/crypttab config (boot up config) for LUKS containers/volumes.
  • Partially account for existing udev sane LUKS config arrangements.
  • Report last known status of LUKS volumes, including their container device.
  • Enhance role and disk scan subsystems to accommodate the above.
  • Add generic ‘mapped to pool’ icon to further clarify device state re btrfs.
  • Add iconic indicator of pool member’s LUKS status on all main pool tables.
  • Add in context explanatory text re LUKS passphrase, boot up config, and container / volume relationship.

Existing Pathologies:

  • Removing a luks volume from an existing pool results in a detached device for up to 30mins after the end of a balance. Cause is as per commit: phillxnet@085e387 , however as a balance is often a long running process this artifact is less crucial and given the additional anticipated complexities of the same approach as per commit only applied directly after the end of a balance it was deemed worthy of it’s own issue / pr.
  • When using a non keyfile boot up configuration the system has no way to authenticate reattachment of an auto detached LUKS volume (such as when a full volume fs is wiped). And so for the time being systemd-tty-ask-password-agent is a required at the console to enable reattachment prior to redeployment. Or a reboot as the name and user text presented against the relevant options suggests. To address this inelegance only the “Auto unlock via keyfile” option is recommended via user text. This could be addressed in the future when adding manual immediate container unlock UI components.
  • Due to the extended use of the role field and it’s combined use to identify partitions and their filesystem, the existing 256 character db field limit can be exceeded upon encountering partitioned Open LUKS Volumes. However this is not a supported configuration. The existing issue overflow of disk.role db field #1709 will address this lack of robustness by way of quadrupling the disk.role maximum field length.

Configuration caveats:

  • The only recognized way to identify which device is referenced in /etc/crypttab is via the containers uuid ie 2nd column must be of the form: UUID=3efb3830-fee1-4a9e-a5c6-ea456bfc269e however this is a commonly used and robust option.
  • Keyfile location is non configurable via UI: all auto generated keyfiles will be placed in /root.
  • No account is made for keyslot availability or configuration. However appropriate error messages should be surfaced.
  • Only locked containers can be wiped and only if they also have no current boot up configuration (ie the “No auto unlock” selection). This is a safety measure and it is enforced by backend validation and front end function (ie the “Wipe locked LUKS container” link is otherwise hidden).

Noteworthy anticipated exclamations:

  • It is assumed that existing LUKS configurations, identified as custom, will experience difficulties where the chosen mapped name for the resulting LUKS volume (first column in /etc/crypttab) does not begin with “luks-”. This is not recognized by udev in the same way and so does not receive a LUKS class of by-id naming: ie will not receive a “dm-name-” prefix in /dev/disk/by-id. In this case it is advised to proceed such names with “luks-” to attain on next reboot, a udev assigned dm-name-luks- by-id name. Or have Rockstor assert the luks-{uuid} type name by way of “Boot up configuration” reconfiguration.

Testing methods employed are to follow in additional comments to this pr.

Fixes #550
Please see #550 for development history.

Ready for review.

phillxnet added 30 commits April 26, 2017 14:24
@phillxnet
back propagate crypto_LUKS partition info to the base device rockstor…
481d05f 
…#550

In order to appropriately identify a disk as hosting a LUKS container
within one of it's partitions we label the base device with this canonical
indicator. This is the same treatment used to identify mdraid disk
members when any one of their partitions is an mdraid member.
Predominantly used to accommodate default anaconda system disk
encrypted (LUKS) installs (ie swap and btrfs system pool partitions
as LUKS containers).
@phillxnet
enhance root_disk() to identify LUKS system disk rockstor#550
28526b5 
When installing via anakonda default "Encrypt my data" option
we end up with our system pool rockstor_rockstor in a LUKS
container (along with swap in it's own). Previous mdadmin root
drive enhancements added non partition ''/' capability but for
correct identification of the root drive in an open LUKS container
and subsequent auto import of system subvols (/root /home)
root_disk() required an additional clause.
@phillxnet
enhance disk table logic to better flag system disk LUKS and container …
ee24a3c 
…rockstor#550

For default anaconda ''Encrypt my data" installs ie swap and /
btrfs pool in their own LUKS we need to recognise and
appropriately flag via icons and tooltips the LUKS in partition
exception made for system disk and include LUKS container
flagging in main base flag if-else. Also reduced number of
checks by combining mutually exclusive additional flags.
@phillxnet
add LUKS and bcache devices to hasUserRole UI helper rockstor#550
686fa82 
Extend the "hasUserRole" handlebars helper to identify future
user configurable disk roles of full disk LUKS container and
full disk backing and caching bcache devices which we already
label internally via the recent roles system.
@phillxnet
refactor and enhance role flags UI code on Disks page rockstor#550
f11e28f 
Reduces the number of redundant disk status checks re role
and state of each disk and improve readability and logic
of the associated checks. The intention is to fashion a user
'funnel' to accommodate early LUKS / bcache device role
assignment while attempting to reduce the number of
redundant or unwise configuration options offered. I.e. on
freshly wiped devices a LUKS / bcache tooltip suggestion
introduces the UI entry point to apply such whole disk
configurations.  Note also that a new flag is introduced, that
of a drive mapping (essentially if the device is found to be part
of a pool) this is intended to aid in identifying fresh or already
pool committed drives from the flags alone and works in
concert with LUKS / bcache flags, ie one will replace another.
@phillxnet
initial commit of rudimentary 'LUKS format' UI template and logic roc…
0308ad4 
…kstor#550

Includes minor cosmetic and code formatting improvements to
existing disk role UI.
@phillxnet
only show LUKS format options when no existing whole disk btrfs rocks…
6e39ecd 
…tor#550

We already disable and hide LUKS options if any partitions
are found so to cater for whole disk btrfs we also check
Disk.btrfs_uuid; this also covers un-imported btrfs. Note this
doesn't cover non btrfs full disk formats but they are unusual.
@phillxnet
initial commit of LUKS backend utilities file rockstor#550
5473dc2 
Intended as a hold-all for LUKS related utilities.
@phillxnet
add backend validation for no LUKS format with role change rockstor#550
74e0997 
The front end already hides LUKS options if there are partitions
but we should block all role changes combined with LUKS format as
we do with the equivalent of disk wipe.
@phillxnet
add backend validation for no LUKS format with wipe rockstor#550
adcde86 
This way we separate these functions so that the user is
encouraged to do one at a time and we only have to deal
with each in turn and consequently monitor the result of
each in turn.
@phillxnet
add backend validation for LUKS format passwords match rockstor#550
2b8af10 
Plus minor text changes.
@phillxnet
use passphrase not password for user facing LUKS text rockstor#550
8b6113e 
This is the convention in all official LUKS
text and encourages greater length.
@phillxnet
Add LUKS intro text on Roles config page rockstor#550
0081423 
Also reduce existing word count and condense
formatting.
@phillxnet
initial front end LUKS passphrase validation rockstor#550
d868314
@phillxnet
improve roles page text brevity rockstor#550
b9f6875 
Also add links to online docs for roles and
import details.
@phillxnet
password field type (LUKS passphrase) + role page brevity rockstor#550
6a10b68
@phillxnet
Reuse Wipe warning for LUKS Format + wipe xor luks rockstor#550
6c334c9 
Emphasise exclusive nature of wipe or LUKS format by
toggling options in xor which works well with shared
wipe warning message.
@phillxnet
minor brevity improvements on wipe and LUKS tick text rockstor#550
28f9637
@phillxnet
add report of Whole Disk LUKS fstype on roles page rockstor#550
0401a59 
Plus minor rename refactoring to more accurately reflect
whole disk fstype variable.
@phillxnet
add luks-format backend command filter / wrapper rockstor#550
08ada55
@phillxnet
add backend LUKS passphrase < 14 char check rockstor#550
8ea2451 
We already have this in front end (UI) validation.
@phillxnet
add ASCII(32-126) char limit to LUKS format passphrase rockstor#550
dc78ffd 
LUKS documentating suggests limiting char use to first
127 ASCII but removing control chars and DEL makes for
easier communication. Includes front end user text and
validation filter as well as backend validation. The
reason this is advised is down to the standardized
nature of these characters across string encoding
formats.
@phillxnet
minor variable rename refactor + comments clean rockstor#550
3728f38
@phillxnet
add cryptsetup luksFormat command wrapper rockstor#550
2297c69
@phillxnet
minor debug of get_open_luks_container_dev() rockstor#550
8235349
@phillxnet
system to identify if a LUKS container is unlocked rockstor#550
77473ac 
Once unlocked a LUKS container gains a mapped open counterpart.
We query the open LUKS volumes to identify all open
LUKS containers by identifying which mapped counterpart relates
to which container via uuid. This can then be used to inform the
role system of the unlocked status of all LUKS containers/devices.
@phillxnet
surface LUKS containers unlocked status via role value rockstor#550
74e71a7 
We extend the previously unused LUKS role value to host
a dictionary which in turn has an unlocked boolean.
Existing mechanism converts dict to json for front end
javascript handlebars helper use.
Contains minor name refactoring of helper function.
@phillxnet
initial commit of LUKS UI page rockstor#550
1f91e19 
We already have a LUKS format extension on the roles
page but that is akin to the existing wipe / delete
function found there already. This page is intended
as the home for existing LUKS containers and open
LUKS volumes where we might put such utilities as
luksHeaderBackup, passphrase facilities etc. Also
the ability to open / close a given container /
volume respectively.
@phillxnet
inform the LUKS role of the device crypttab status rockstor#550
452b4e7 
A LUKS container, denoted by the LUKS role, can have
/etc/crypttab entries that can define the boot unlock
behaviour. Collect the current status and stash as
crypttab index with value of 3rd column in /etc/crypttab.
@phillxnet
display per dev crypttab config via dropdown rockstor#550
175e232 
Also adds dual mode to luks page: config for
container mode, info for open volume mode.
phillxnet and others added 18 commits May 7, 2017 18:00
@phillxnet
make fstype flag for bcache cdev consistent with role name rockstor#550
41ea0aa 
Minor flag name change - mostly aesthetic - but should help
with code readability by being consistent with consequent role
and role constants lists.
@phillxnet
remove remaining debug log for issue rockstor#550
6f97b37
@phillxnet
add user text on LUKS boot up config options rockstor#550
1752590 
Mirror formatting found on sister 'roles' config page
@phillxnet
hide user crypttab text for open LUKS volumes rockstor#550
2573b86 
Boot up config pertains to LUKS containers only and
as we have a dual personality LUKS config page we
ony show this text for LUKS container.
@phillxnet
update LUKS role on LUKS boot up config changes rockstor#550
67cd677 
Also ensure we avoid making changes to custom
(non native) crypttab configurations, even during
a LUKS config page submit when viewing these settings.
Require a config change prior to defaulting to
native keyfile creation.
@phillxnet
include LUKS Format in role page title rockstor#550
c232bcb 
Since we have partnered LUKS format with our
existing wipe on this page we should add this
key function to the page title.
@phillxnet
populate openLUKS role value with status info rockstor#550
f449f4c 
Previously we had an unused placeholder of
dm-name-uuid. Replace with output from:
'cryptsetup status dev-name' as a dict/json.
The 'openLUKS' role signifies open LUKS volumes.
@phillxnet
add Open LUKS Volume 'status table' to LUKS info page rockstor#550
781750f 
When our dual mode LUKS page is viewing an Open LUKS
Volume, display the role value which equates to a
slightly tailored cyptsetup status dev-name output where
the backing device listed is converted to a by-id name.
@phillxnet
improve LUKS page user text re custom configs rockstor#550
c0f1411 
Plus minor text enhancements for clarity.
@phillxnet
add cryptsetup to rpm-deps in base-buildout.cfg rockstor#550
1a17d9d 
Actively used by LUKS additions moving forward.
@phillxnet
improve pool mount log reporting re missing devices rockstor#550
3045722 
Add info logging when failing over to 'mount by dev'
from 'mount by label' and specify the dev names as we
go. Add error logging on missing devices.
@phillxnet
rerun systemd generators after /etc/crypttab changes rockstor#550
1159906 
As our 'source of truth' is /etc/crypttab but systemd
generators now abstract the contents to
/var/run/systemd/generators we should ensure that systemd
re-generates the abstraction in a timely manner.
@phillxnet
initial commit of systemd-escape wrapper rockstor#550
235d6df
@phillxnet
workaround upstream bug in systemd-escape rockstor#550
fa918f5 
Less flexible than original but serves our current
use. Required as upstream --template option is
broken for all input cases.
@phillxnet
after wipe of Open LUKS Volume fs, reopen volume rockstor#550
085e387 
When a full dev fs exists on an Open LUKS Volume and
that fs is wiped via wipefs -a, systemd tears down the
entire mapped block device. Re-assert there after by
starting the associated generator service if it exists.
@phillxnet
call 'udevadm trigger' after new pool creation rockstor#550
e135897 
Intermittent failure of udev to update fstype, label,
and uuid of dev after pool creation (1-2 times in 10)
was observed for a luks mapped device. After this
udev call no similar failures were observed. Also
prior to patch many mount by label fails were observed
with consequent fail-over mounts by dev, but with no
consequent dev info updates the pool was not recognized
by our UI. Where as after this patch no failures to mount
by label were observed in the test scenarios. This same
treatment has already been applied to adding disks to an
existing pool in pr rockstor#1607 commit:
0358560
@phillxnet
improve user text on LUKS page re crypttab config options rockstor#550
ff1561a 
Given our NAS remit we favour headless operation and given
LUKS volumes are closed upon wipe we must favour and recommend
auto authentication via keyfile. Also add systemd command
info for those favouring a non keyfile / command line
management approach.
@phillxnet
improve user text re LUKS boot up config on roles page rockstor#550
d6f7600 
Best we point out "Auto unlock via keyfile" and location of
this config with icon repeats to improve affordance and ease
first time configuration.
@phillxnet
Copy link
Member Author

phillxnet commented May 22, 2017 •
edited
Loading

All validation mechanisms and UI contextual behaviour was tested to behave as expected and as indicated in the pr description.

Pre-pr encrypted install system drive iconography:
3 8 16-16-luks-root

Post-pr encrypted install system drive display (here also showing a btrfs in partition virtio dev):
encrypted-install-plus-btrfs-in-partition-icon

In addition a real life scenario involving purposefully slow and constrained hardware (2G RAM single core dual thread P4) with 35 GB of data on an existing 3 disk raid 1 pool totalling 225 GB was taken through the following semi typical scenario for moving an existing pool from non LUKS formatted devices to LUKS formatted devices and their consequent mapped counterpart Open LUKS Volumes.

  • Click the ? for role / wipe / LUKS format page on spare Samsung blank disk.
    pre-luks-3drive-pool-blank-spare
    luks-format-on-roles-wipe-page

  • Disk page now shows a lock icon to indicate the container.
    post-lus-format-spare

  • Click on the lock to set “Boot up Configuration” to “Auto unlock via keyfile”
    set-container-boot-up-config

  • Revisiting the above page now indicates the keyfile’s existence and the new ‘active’ setting.
    confirm-keyfile-config-and-no-wipe-link

  • In this example we remove one of our pool disks via the resize function.

  • N.B. pending existing code issue to open here re no UI feedback during this removal.

  • While this is happening background systemd processes enact our crypttab configuration and our first Open LUKS Volume appears. N.B. as this is also a boot up config, a reboot is required for test it’s effect, this is done later in this test scenario.
    crypttab-config-enacted-via-reboot-or-wait

  • LUKS format the freshly removed from pool device via ? icon as with our first LUKS format.

  • And similarly set to “auto unlock via keyfile” with associated and required passphrase entry.

  • Reboot to prove auto unlock configuration and the consequent opening of the second luks container.

  • Now we have a 2 disk pool which we can then add our 2 Open LUKS Container to.
    two-open-luks-containers-and-volumes

  • Resize pool - add disks - select our 2 Open LUKS Containers.
    add-2-luks-volumes-to-pool

  • Let balance finish.

  • Half luks half plain pool so remove the remaining non LUKS pool member via resize.
    pool-disk-table-two-of-each-type
    pools-table-two-of-each-type
    half-luks-pool-members

  • Pool - resize - remove non luks disks one at a time in this case due to size constraints leaving each process to finish.

  • N.B. re above N.B. on pending existing code issue on UI feedback failure during this removal.

  • A disk page refresh and we have our expected all luks pool:
    pool-now-on-luks-formatted-disks

@phillxnet phillxnet mentioned this pull request May 24, 2017
Merged
@schakrava
Copy link
Member

Amazing job @phillxnet. Nice to see a PR from you again. Thoroughly done as usual :)

@schakrava schakrava merged commit a957346 into rockstor:master May 25, 2017
@phillxnet
Copy link
Member Author

@schakrava Thanks and Cheers.

@phillxnet phillxnet mentioned this pull request May 26, 2017
Closed
@schakrava schakrava mentioned this pull request Jun 1, 2017
Closed
@phillxnet phillxnet mentioned this pull request Jun 1, 2017
Closed
@phillxnet phillxnet deleted the 550_support_full_disk_LUKS branch June 6, 2017 17:55
@phillxnet phillxnet mentioned this pull request Jan 20, 2022
Closed
@phillxnet phillxnet mentioned this pull request Apr 30, 2023
Closed
@phillxnet phillxnet mentioned this pull request Mar 29, 2024
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support full disk LUKS
2 participants
@phillxnet @schakrava