Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quota file exists error on Share resize #2854

Closed
phillxnet opened this issue Jun 20, 2024 · 6 comments
Closed

Quota file exists error on Share resize #2854

phillxnet opened this issue Jun 20, 2024 · 6 comments
Assignees
@phillxnet
Milestone
5.1.X-X Stable re...

Comments

@phillxnet
Copy link
Member

Thanks to forum member Stevek for highlighting this problem. We currently don't enforce Share size limiting via quotas: with the following Web-UI proviso in Shares overview & Details views:

Share size enforcement is temporarily disabled due to incomplete support in BTRFS. Until this status changes, the effective size of a Share is equal to the size of the Pool it belongs to.

But we currently, under some circumstances, error out when a Share resize is attempted:

Traceback (most recent call last):
File “/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py”, line 41, in _handle_exception
yield
File “/opt/rockstor/src/rockstor/storageadmin/views/share.py”, line 279, in put
share_pqgroup_assign(share.pqgroup, share)
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 1563, in share_pqgroup_assign
return qgroup_assign(share.qgroup, pqgroup, share.pool.mnt_pt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 1632, in qgroup_assign
raise e
File “/opt/rockstor/src/rockstor/fs/btrfs.py”, line 1580, in qgroup_assign
run_command([BTRFS, “qgroup”, “assign”, qid, pqid, mnt_pt], log=False)
File “/opt/rockstor/src/rockstor/system/osi.py”, line 289, in run_command
raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/sbin/btrfs qgroup assign 0/263 2015/7 /mnt2/Data-2. rc = 1. stdout = [‘’]. stderr = [‘ERROR: unable to assign quota group: File exists’, ‘’]

As part of our planned enforcement of quota limits, and to provide a better user experience; even on this currently unsupported limitation, we should initially fix this bad user experience.

Forum reference: https://forum.rockstor.com/t/created-5gb-share-and-put-175gb-of-data-into-it-cant-resize/9539
@phillxnet phillxnet added this to the 5.1.X-X Stable release milestone Jul 15, 2024
@phillxnet phillxnet self-assigned this Jul 16, 2024
@phillxnet
Copy link
Member Author

Updated output using current testing branch:

            Traceback (most recent call last):
  File "/opt/rockstor/src/rockstor/rest_framework_custom/generic_view.py", line 41, in _handle_exception
    yield
  File "/opt/rockstor/src/rockstor/storageadmin/views/share.py", line 279, in put
    share_pqgroup_assign(share.pqgroup, share)
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1564, in share_pqgroup_assign
    return qgroup_assign(share.qgroup, pqgroup, share.pool.mnt_pt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1633, in qgroup_assign
    raise e
  File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1581, in qgroup_assign
    run_command([BTRFS, "qgroup", "assign", qid, pqid, mnt_pt], log=False)
  File "/opt/rockstor/src/rockstor/system/osi.py", line 290, in run_command
    raise CommandException(cmd, out, err, rc)
system.exceptions.CommandException: Error running a command. cmd = /usr/sbin/btrfs qgroup assign 0/268 2015/82 /mnt2/rock-pool. rc = 1. stdout = ['']. stderr = ['ERROR: unable to assign quota group: File exists', '']

@phillxnet
Copy link
Member Author

Initially this looks like we try to assign a qgroup when it is already assigned: i.e. we have:

rockstor-core/src/rockstor/fs/btrfs.py

Lines 1567 to 1575 in 185a8d0

def qgroup_assign(qid, pqid, mnt_pt):
"""
Wrapper for 'BTRFS, qgroup, assign, qid, pqid, mnt_pt'
:param qid: qgroup to assign as child of pqgroup
:param pqid: pqgroup to use as parent
:param mnt_pt: btrfs filesystem mountpoint (usually the associated pool)
"""
if qgroup_is_assigned(qid, pqid, mnt_pt):
return True

to avoid the assignment when it already exists. Where we establish if the qgroup is assigned already via:

rockstor-core/src/rockstor/fs/btrfs.py

Lines 1500 to 1503 in 185a8d0

def qgroup_is_assigned(qid, pqid, mnt_pt):
# Returns true if the given qgroup qid is already assigned to pqid for the
# path(mnt_pt)
cmd = [BTRFS, "qgroup", "show", "-pc", mnt_pt]

@phillxnet
Copy link
Member Author

phillxnet commented Jul 16, 2024 •
edited
Loading

For the development 15.6 reproducer system we have the following, note this Pool has a little history:

btrfs qgroup show -pc /mnt2/rock-pool
Qgroupid    Referenced    Exclusive Parent                                                          Child   Path 
--------    ----------    --------- ------                                                          -----   ---- 
0/5           16.00KiB     16.00KiB -                                                               -       <toplevel>
0/268         16.00KiB     16.00KiB 2015/75,2015/76,2015/77,2015/78,2015/79,2015/80,2015/81,2015/82 -       sftp-share
2015/1           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/2           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/3           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/4           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/5           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/6           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/7           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/8           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/9           0.00B        0.00B -                                                               -       <0 member qgroups>
2015/10          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/11          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/12          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/13          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/14          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/15          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/16          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/17          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/18          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/19          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/20          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/21          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/22          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/23          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/24          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/25          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/26          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/27          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/28          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/29          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/30          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/31          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/32          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/33          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/34          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/35          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/36          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/37          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/38          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/39          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/40          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/41          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/42          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/43          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/44          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/45          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/46          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/47          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/49          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/50          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/51          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/52          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/53          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/54          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/55          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/56          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/57          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/58          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/59          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/60          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/61          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/62          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/63          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/64          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/65          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/66          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/67          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/68          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/69          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/70          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/71          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/72          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/73          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/74          0.00B        0.00B -                                                               -       <0 member qgroups>
2015/75       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/76       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/77       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/78       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/79       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/80       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/81       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/82       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>

The history may be part of the reproducer here.

N.B. as our original call is logged as:

cmd = /usr/sbin/btrfs qgroup assign 0/268 2015/82 /mnt2/rock-pool

and these same values are passed to our pre-check re assignment of qgroup_is_assigned(qid, pqid, mnt_pt) we can see that:

  • quota id 0/268 is indeed
  • assigned to parent quota id 2015/82
  • for the sftp-share (Path "sftp-share")

[edit to correct diagnostic mistake]

Qgroupid    Referenced    Exclusive Parent                                                          Child   Path 
--------    ----------    --------- ------                                                          -----   ---- 
0/5           16.00KiB     16.00KiB -                                                               -       <toplevel>
0/268         16.00KiB     16.00KiB 2015/75,2015/76,2015/77,2015/78,2015/79,2015/80,2015/81,2015/82 -       sftp-share
2015/1           0.00B        0.00B -                                                               -       <0 member qgroups>
...
2015/75       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/76       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/77       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/78       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/79       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/80       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/81       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>
2015/82       16.00KiB     16.00KiB -                                                               0/268   <0 member qgroups>

@phillxnet
Copy link
Member Author

phillxnet commented Jul 16, 2024 •
edited
Loading

Our qgroup selection in the call to qgroup_assign(qid, pqid, mnt_pt) is from:

File "/opt/rockstor/src/rockstor/fs/btrfs.py", line 1564, in share_pqgroup_assign

rockstor-core/src/rockstor/fs/btrfs.py

Lines 1555 to 1564 in 185a8d0

def share_pqgroup_assign(pqgroup, share):
"""
Convenience wrapper to qgroup_assign() for use with a share object where
we wish to assign / reassign it's current db held qgroup to a passed
pqgroup.
:param pqgroup: pqgroup to use as parent.
:param share: share object
:return: qgroup_assign() result.
"""
return qgroup_assign(share.qgroup, pqgroup, share.pool.mnt_pt)

indicating the source as from share.qgroup.

@phillxnet
Copy link
Member Author

phillxnet commented Jul 16, 2024 •
edited
Loading

It appears that this failure is around qgroup_is_assigned() and it's inability to inform us that we already have this assignment in place. There is also no test coverage for qgroup_is_assigned(), but the procedure itself has some example data and this issue can provide more.

I.e. see in qgroup_is_assigned()

rockstor-core/src/rockstor/fs/btrfs.py

Lines 1540 to 1548 in 185a8d0

# TODO: enhance to accommodate for multiple listings via:
# pqid in fields[3].split(',') or the like.
# [0] [1] [2] [3] [4]
# qgroupid rfer excl parent child
# 0/258 16.00KiB 16.00KiB 2015/1,2015/5 ---
# 0/311 16.00KiB 16.00KiB 2015/1 ---
# 0/313 16.00KiB 16.00KiB 2015/1 ---
# 2015/1 48.00KiB 48.00KiB --- 0/258,0/311,0/313
# ie 2015/1 has 3 children of 0/258,0/311,0/313 

btrfs qgroup show -pc /mnt2/rock-pool
Qgroupid    Referenced    Exclusive Parent                                                          Child   Path 
--------    ----------    --------- ------                                                          -----   ---- 
0/5           16.00KiB     16.00KiB -                                                               -       <toplevel>
0/268         16.00KiB     16.00KiB 2015/75,2015/76,2015/77,2015/78,2015/79,2015/80,2015/81,2015/82 -       sftp-share
...

We also now look to have an additional path column: i.e.

Qgroupid Referenced Exclusive Parent Child Path

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 16, 2024
@phillxnet
Quota file exists error on Share resize rockstor#2854
caebd00 
Fix insensitivity re Share quota id membership of Pool
associated quote group, when the Share's quota id has
multiple parent quota groups. Avoiding consequent failure
during redundant quota group assignment.

Includes:
- Previously missing test for qgroup assignment probe.
- Removes prior limitation re single quota parent.
- Incidental removal of unused import in test_btrfs.py
and re-black formatting of the same file.
@phillxnet phillxnet mentioned this issue Jul 16, 2024
Merged
phillxnet added a commit that referenced this issue Jul 17, 2024
@phillxnet
Merge pull request #2874 from phillxnet/2854-Quota-file-exists-error-…
887efc1 
…on-Share-resize

Quota file exists error on Share resize #2854
@phillxnet
Copy link
Member Author

Closing as:
Fixed by #2874

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phillxnet phillxnet

Labels
None yet
Projects
None yet
Milestone
5.1.X-X Stable release
Development

No branches or pull requests
1 participant
@phillxnet