Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve quotas not enabled behaviour #1869

Closed
12 tasks done
phillxnet opened this issue Dec 8, 2017 · 13 comments · Fixed by #1874
Closed
12 tasks done

improve quotas not enabled behaviour #1869

phillxnet opened this issue Dec 8, 2017 · 13 comments · Fixed by #1874
Assignees
@phillxnet

Comments

@phillxnet
Copy link
Member

phillxnet commented Dec 8, 2017 •
edited
Loading

Under certain as yet undetermined circumstances quotas can become disabled. One consequence of this was a failure to mount the related shares upon the following boot / reboot (addressed in issue #1867 and related pr #1868). However there are additional consequences for running with a pool / subvol that has quotas not enabled. These include:

  • Inability to create new shares in that pool.
  • Inability to resize existing shares.
  • Apparent Inability to create a snapshot (error on trying but it appears)
  • Apparent Inability to delete a snapshot (error on trying but it get deleted)
    [ ] Size reporting of the affected subvols is reported as 0 bytes.
  • Deal with quotas being re-enabled once shares etc have been created with default pqgroup -1/-1
  • Test pool import for ability to recreate/reset pqgroups
  • Test bootstrap command on quota enabled pool after having been used with quotas disabled ie -1/-1 entries in db.
  • Ensure all pqgroup's are unique while being robust to quote enable disable re-enabled cycle.
  • Add user feedback on Pool Quotas Enabled / Disabled
  • Share created when quotas disabled get fresh pqgroup during re-size re-submission if done when quotas are enabled (no change needed).
  • Share with non existent pqgroup (due to quota disable removing all pqgroups from pool) will also be re-assigned a fresh pqgroup once quotas are re-enabled and a resize is performed.
  • remove all non existent pqgroups and re-assign fresh ones during boot if quotas enabled, already done for prior -1/-1 pqgroup shares.
@phillxnet
Copy link
Member Author

phillxnet commented Dec 8, 2017 •
edited
Loading

After enabling debug on the example system that displays this issue as detailed in #1867 via:

/opt/rockstor/bin/debug-mode on

we have the following:

[08/Dec/2017 19:21:43] DEBUG [storageadmin.views.share_helpers:87] Share name = rock-ons-root.
[08/Dec/2017 19:21:43] DEBUG [storageadmin.views.share_helpers:89] Updating pre-existing same pool db share entry.
[08/Dec/2017 19:21:43] DEBUG [system.osi:104] Running command: /sbin/btrfs subvolume list /mnt2/rock-pool
[08/Dec/2017 19:21:43] DEBUG [system.osi:104] Running command: /sbin/btrfs qgroup show /mnt2/rock-pool/rock-ons-root
[08/Dec/2017 19:21:43] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rock-pool/rock-ons-root']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[08/Dec/2017 19:21:43] DEBUG [storageadmin.views.share_helpers:87] Share name = test-share-2.
[08/Dec/2017 19:21:43] DEBUG [storageadmin.views.share_helpers:89] Updating pre-existing same pool db share entry.
[08/Dec/2017 19:21:43] DEBUG [system.osi:104] Running command: /sbin/btrfs subvolume list /mnt2/rock-pool
[08/Dec/2017 19:21:43] DEBUG [system.osi:104] Running command: /sbin/btrfs qgroup show /mnt2/rock-pool/test-share-2
[08/Dec/2017 19:21:43] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rock-pool/test-share-2']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
[08/Dec/2017 19:21:43] DEBUG [storageadmin.views.share_helpers:87] Share name = test-share.
[08/Dec/2017 19:21:43] DEBUG [storageadmin.views.share_helpers:89] Updating pre-existing same pool db share entry.
[08/Dec/2017 19:21:43] DEBUG [system.osi:104] Running command: /sbin/btrfs subvolume list /mnt2/rock-pool
[08/Dec/2017 19:21:43] DEBUG [system.osi:104] Running command: /sbin/btrfs qgroup show /mnt2/rock-pool/test-share
[08/Dec/2017 19:21:43] ERROR [system.osi:119] non-zero code(1) returned by command: ['/sbin/btrfs', 'qgroup', 'show', '/mnt2/rock-pool/test-share']. output: [''] error: ["ERROR: can't list qgroups: quotas not enabled", '']
...

@phillxnet
Copy link
Member Author

post the following command:

btrfs quota enable /mnt2/rock-pool/

we have:

[08/Dec/2017 19:34:00] DEBUG [storageadmin.views.share_helpers:87] Share name = rock-ons-root.
[08/Dec/2017 19:34:00] DEBUG [storageadmin.views.share_helpers:89] Updating pre-existing same pool db share entry.
[08/Dec/2017 19:34:00] DEBUG [system.osi:104] Running command: /sbin/btrfs subvolume list /mnt2/rock-pool
[08/Dec/2017 19:34:00] DEBUG [system.osi:104] Running command: /sbin/btrfs qgroup show /mnt2/rock-pool/rock-ons-root
[08/Dec/2017 19:34:00] DEBUG [storageadmin.views.share_helpers:87] Share name = test-share-2.
[08/Dec/2017 19:34:00] DEBUG [storageadmin.views.share_helpers:89] Updating pre-existing same pool db share entry.
[08/Dec/2017 19:34:00] DEBUG [system.osi:104] Running command: /sbin/btrfs subvolume list /mnt2/rock-pool
[08/Dec/2017 19:34:00] DEBUG [system.osi:104] Running command: /sbin/btrfs qgroup show /mnt2/rock-pool/test-share-2
[08/Dec/2017 19:34:00] DEBUG [storageadmin.views.share_helpers:87] Share name = test-share.
[08/Dec/2017 19:34:00] DEBUG [storageadmin.views.share_helpers:89] Updating pre-existing same pool db share entry.
[08/Dec/2017 19:34:00] DEBUG [system.osi:104] Running command: /sbin/btrfs subvolume list /mnt2/rock-pool
[08/Dec/2017 19:34:00] DEBUG [system.osi:104] Running command: /sbin/btrfs qgroup show /mnt2/rock-pool/test-share

@phillxnet phillxnet mentioned this issue Dec 9, 2017
Closed
@phillxnet
Copy link
Member Author

enable_quota(pool) called:

  • when pool is created: add_pool() (fs/btrfs)
    Not likely candidate as when pool was created in reproducer example install shares could be created and so quotas must have been enabled as otherwise we hit a current trigger when quota not enabled on pool.
  • during a web-ui initiated import: _btrfs_disk_import() (storageadmin/views/disk.py)
    Example reproducer had pool created from scratch so no import.
  • during rescan but only for role.root (system) device: _update_disk_state() (storageadmin/views/disk.py)

The current suspicion is that we have a quota anomaly on certain pool instances where quota enabled status, such as is enacted upon pool creation, does not stick. Ie. after a reboot the pool has quotas disabled which is not currently supported in Rockstor: with the consequent limitations as noted in this issue. As such this issue may contain relevant info for issue #1592.

@phillxnet
Copy link
Member Author

My current thinking on this issue is that we have a quota status not sticking issue on certain pool instances as my understanding is that Rockstor relies on the pool / subvols to maintain their quota status (enabled / disabled) over a reboot where as in the instances noted in this issue that is not happening: hence my recent links to plans to support disabling quotas or functioning with quotas disabled which we currently don't.

@phillxnet
Copy link
Member Author

@schakrava I have found the following that may tie our disabled quotas to the recent docker update:

docker/for-linux#78

Looks like a likely candidate to me.

@phillxnet phillxnet changed the title quotas not enabled on boot. improve quotas not enabled behaviour Dec 11, 2017
@phillxnet
Copy link
Member Author

I have transitioned this issue to improving our 'quota disabled' behaviour given we now have the culprit:
docker-ce disables quotas
So we might as well deal with it as best we can at least until this is fixed upstream.

I intend to open a more focused issue on "rock-ons-root pool quota disabled by docker-ce" in the near future.

@phillxnet
Copy link
Member Author

User feedback elements re Pool Quotas status:

Column in Pools table:
quota-status-on-pools-table

In Pool column - Shares table:
quota-status-on-shares-table

On Pool details page:
quota-status-on-pool-details-page

On Shares details page re pool:
quota-status-on-share-details-page

Text in Usage / Btrfs Usage columns in Shares table changed to show quotas dependency:
"Share content - uses Quotas"
"Share content inc snapshots - uses Quotas"

@phillxnet
Copy link
Member Author

phillxnet commented Dec 13, 2017 •
edited
Loading

Enhancement of quota groups management in progress. Pull request to follow.

@phillxnet
Copy link
Member Author

Final testing and code tidy in progress.

@phillxnet
Copy link
Member Author

Please update the following forum thread with this issues resolution:
https://forum.rockstor.com/t/cant-create-new-share/4178

This was referenced Dec 17, 2017
Closed
Merged
@schakrava
Copy link
Member

Thanks for putting down all the details @phillxnet , very helpful!

We must(I really need to prioritize to do this myself) address the hard dependency on quotas in Rockstor. We need to investigate the current state of btrfs-quotas feature-set and perhaps let the user disable/enable quotas entirely or on per pool basis and also, as you indicated above, reflect the current quota enforcement to the user where it's appropriate.

@phillxnet
Copy link
Member Author

phillxnet commented Dec 18, 2017 •
edited
Loading

@schakrava Yes, bar the size enforcement and usage reporting the code associated with this issue (pull request soon) seems to allow for all features I've tested so far to function with quotas disabled. Trying to work towards #1592 that we have had for a while.

I'll get this code in soon for review as it's so far successfully managed to re-make pqgroups with multiple successive quota disable / enable cycles. All pool quota state UI elements have been fairly aggressively tested it as I've been using it during the dev of the pqgroup management improvements.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Dec 18, 2017
@phillxnet
improve quotas not enabled behaviour rockstor#1869
5dcc2fa 
Uses the existing -1/-1 default for pqgoups to represent
an unset state for the share.pqgroup and adds an active
return to this value whenever quotas are deemed to be
disabled. Pqgroup setup is moved from the bootstrap
process into the import_shares / shares refresh section.
This allows for live setting / resetting of pqgroups via
the regular share re-fresh process. Pool.quotas_enabled
and share.pqgroup_exist properties are added and all used
low level quota actions are adjusted to catch, log, and
ignore a quota disabled state.
Additionally active pqgroup assignment is added to share
resize and share refresh procedures which aid in returning
to existence the expected native 2015/n pqgroups and their
relationship to the auto generated 0/n qgroups as whenever
quotas are disabled all this info (var the 0/n) is lost.
UI elements are added in pool and share focused pages to
indicate a live status for the associated pool quotas
enabled status.
@phillxnet phillxnet mentioned this issue Dec 18, 2017
Merged
schakrava added a commit that referenced this issue Dec 21, 2017
@schakrava
Merge pull request #1874 from phillxnet/1869_improve_quotas_not_enabl…
40adc06 
…ed_behaviour

improve quotas not enabled behaviour. Fixes #1869
This was referenced Feb 4, 2018
Closed
Closed
@phillxnet
Copy link
Member Author

Linking this now closed issue to:
"Feature: Quota rebuild script?" #1785
as our associated pr address (at least initially) the recreation of required rockstor native pqgroups (2015/*) as referenced in #1785 "... and re-create any qgroups it needs ..." and so represents a partial fix for #1785

This was referenced Mar 7, 2018
Merged
Open
@phillxnet phillxnet mentioned this issue Mar 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phillxnet phillxnet

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

improve quotas not enabled behaviour. Fixes #1869 phillxnet/rockstor-core
2 participants
@schakrava @phillxnet