pool resize disk removal unknown internal error and no UI counterpart

[phillxnet]

Thanks to forum member Noggin for highlighting this behaviour. Occasionally when removing a disk from a pool there can be a UI time out directly after the last dialog entitled "Resize Pool / Change RAID level for ..." which acts as last confirmation of the configured operation:

There is then no UI 'balance' indicated while the removal is in progress, yet the UI indicates that a balance is in progress when a balance is attempted (only attempted by Noggin as I did not attempt to execute a balance whilst the removal was in progress).

btrfs balance status /mnt2/time_machine_pool/
No balance found on '/mnt2/time_machine_pool/'

The pool resize is however indicated by the requested disk's having their size 'demoted' to zero and showing a reduced usage with subsequent executions of btrfs fi show:

Label: 'time_machine_pool'  uuid: 8f363c7d-2546-4655-b81b-744e06336b07
	Total devices 4 FS bytes used 31.57GiB
	devid    3 size 149.05GiB used 17.03GiB path /dev/sdd
	devid    4 size 0.00B used 5.00GiB path /dev/sda
	devid    5 size 149.05GiB used 23.03GiB path /dev/mapper/luks-d36d39ea-c0b3-4355-b0c5-bd3248e6bbfe
	devid    6 size 149.05GiB used 23.00GiB path /dev/mapper/luks-d7524e90-4d9e-4772-932f-d1407b6b5fe7

and then later on:

Label: 'time_machine_pool'  uuid: 8f363c7d-2546-4655-b81b-744e06336b07
	Total devices 4 FS bytes used 32.57GiB
	devid    3 size 149.05GiB used 18.03GiB path /dev/sdd
	devid    4 size 0.00B used 2.00GiB path /dev/sda
	devid    5 size 149.05GiB used 24.03GiB path /dev/mapper/luks-d36d39ea-c0b3-4355-b0c5-bd3248e6bbfe
	devid    6 size 149.05GiB used 24.00GiB path /dev/mapper/luks-d7524e90-4d9e-4772-932f-d1407b6b5fe7

As can be seen devid 4 is having it's pool usage reduced (from 5 to 3 GB) between runs. In the above example the disk removal completed successfully however there was never an UI indication of it's 'in progress' nature or any record of a balance having taken place at that time.

Reference to Noggins's forum thread suspected as indicating the same as my observations in final testing of pr #1716 which lead also to this issue creation (details of the precedence steps available in that pr):
https://forum.rockstor.com/t/cant-remove-failed-drive-from-pool-rebalance-in-progress/3319
where a 3.8.16-16 (3.9.0 iso install) version exhibited the same behaviour (pre #1716 merge).

[phillxnet]

Please update the above referenced forum thread with this issues resolution.

[phillxnet]

In working on the following related issue:
"Implement a delete missing disk in pool UI" #1700
I have been able to assess the cause of this behaviour. Essentially when running a 'btrfs device delete (dev-name and/or missing) an internal balance is initiated. This, in almost all 'real hardware' cases, causes the indicated error. Essentially a 'time out' as a balance will typically take hours and we currently fail to run the associated code resize_pool() as an async taks like we do with regular balance operations and so our db commit code is 'frozen' until such time as the 'btrfs device delete' completes it's internal balance. Note also that we previous to the as yet uncommitted changes to be associated with #1700 also then attempted to run a redundant balance after the remove (btrfs dev delete) operation.

Once the changes associated with the above referenced issue #1700 have been committed we can move to treating our resize_pool() as we currently do our balance, ie via async task with monitoring wrapper. Some progress has been made towards the design of this arrangement and is likely to mirror what we have for balance / scrub quite closely, bar the inability to query the btrfs subsystem for progress / status of this 'btrfs dev delete' initiated internal balance operation. So more work/research is required in that direction for this issue to be addressed fully.

[phillxnet]

A counter example of this issue is where the disk to be removed presents little in the way of a balance requirement prior to it's removal (via btrfs device delete mount-point) ie:

Label: 'rock-pool'  uuid: 8adf7f0b-65ec-4e00-83cc-7f5855201185
	Total devices 3 FS bytes used 39.22GiB
	devid    1 size 465.76GiB used 42.03GiB path /dev/sda
	devid    2 size 465.76GiB used 42.03GiB path /dev/sdb
	devid    3 size 465.76GiB used 0.00B path /dev/sdc

Where we are to remove /dev/sdc which has "used 0.00B".

In situations such as this our existing 'treatment' (although the tested code had the consequent Rockstor initiated balance removed) works as intended. Due to the internal 'btrfs dev delete' initiated balance completing in a sufficiently short time: hence no time outs.

Label: 'rock-pool'  uuid: 8adf7f0b-65ec-4e00-83cc-7f5855201185
	Total devices 2 FS bytes used 39.22GiB
	devid    1 size 465.76GiB used 42.03GiB path /dev/sda
	devid    2 size 465.76GiB used 42.03GiB path /dev/sdb

And our db representation is also updated accordingly.

[phillxnet]

Linking to another recent forum thread where juchong also reported this issue:
https://forum.rockstor.com/t/cant-remove-disk-even-though-i-have-enough-space/5287/6

[phillxnet]

I am currently working on this issue.

[phillxnet]

Please also update following user cferra in the following thread with this issues resolution:
https://forum.rockstor.com/t/problems-replacing-upgrading-a-disk/5515

[phillxnet]

Some visible elements added by the pending code that addresses this issue (awaiting current pr merges so that I can rebase on pending db migrations as this code also requires some db additions:

Track and surface Pool UUID:

detached disk devid and allocated message:

Enhanced final message re balance initialisation:

Freshly added disk pre allocation devid scan required notice:

last stage of disk removal attached:

running finished failed disk removal internal balance report:

Notice in the above we also surface devid and per dev allocated (%).

[phillxnet]

Code/pr for this issue's fix is in final testing/preparations.