Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NFS exports restored as read-only #2912

Closed
Hooverdan96 opened this issue Oct 3, 2024 · 9 comments
Closed

NFS exports restored as read-only #2912

Hooverdan96 opened this issue Oct 3, 2024 · 9 comments
Assignees

Comments

@Hooverdan96
Copy link
Member

Thanks to user simon-77 on the community forum who observed that their nfs exports were restored, however without the read-write flag set (moving from 4.6.1-0 and restoring onto a 5.0.14-0 environment). They are recreated as read-only (which might be the nfs default).

https://forum.rockstor.com/t/missing-repository-rockstor-stable-on-15-6/9643/18

The problem might occur in this area (but might also be before when preparing the restore data elsewhere):

def restore_nfs_exports(ml: list):
logger.info("Started restoring NFS exports.")
exports = []
export_groups = {}
adv_exports = {"entries": []}
for m in ml:
if m["model"] == "storageadmin.nfsexport":
exports.append(m["fields"])
elif m["model"] == "storageadmin.nfsexportgroup":
m["fields"]["pk"] = m["pk"]
export_groups[m["pk"]] = m["fields"]
elif m["model"] == "storageadmin.advancednfsexport":
adv_exports["entries"].append(m["fields"]["export_str"])
for e in exports:
if len(e["mount"].split("/")) != 3:

Here's an example of the nfs json entry taken from here in the discussion thread:

{"model": "storageadmin.nfsexportgroup", "pk": 2, "fields": {"host_str": "10.71.128.0/24", "editable": "rw", "syncable": "async", "mount_security": "insecure", "nohide": false, "enabled": true, "admin_host": null}},

where one can see that rw for the read-write setting is available.

@Hooverdan96 Hooverdan96 changed the title NFS restored as read-only from configuration backup, despite read-write flag present in configuration NFS exports restored as read-only from configuration backup, despite read-write flag present in configuration Oct 3, 2024
@phillxnet phillxnet added this to the 5.1.X-X Stable release milestone Oct 3, 2024
@phillxnet phillxnet changed the title NFS exports restored as read-only from configuration backup, despite read-write flag present in configuration NFS exports restored as read-only Nov 1, 2024
@phillxnet phillxnet self-assigned this Nov 1, 2024
@phillxnet
Copy link
Member

phillxnet commented Nov 1, 2024

With what is essentially a default NFS export created showing as:

Shares Host String Read only Sync / Async Actions
NFS-share * no async  

we have the following config-save data based on the DB dumpdata format we use, see @Hooverdan96 #2354 (comment) referencing how our config backup is created.

From the first (in our config_backup file format) DB dump of storageadmin, our default DB, we have the following relevant entries:

{"model": "storageadmin.nfsexport", "pk": 1, "fields": {"export_group": 1, "share": 1, "mount": "/export/NFS-share"}},

{"model": "storageadmin.nfsexportgroup", "pk": 1, "fields": {"host_str": "*", "editable": "rw", "syncable": "async", "mount_security": "insecure", "nohide": false, "enabled": true, "admin_host": null}},

{"model": "storageadmin.share", "pk": 1, "fields": {"pool": 1, "qgroup": "0/445", "pqgroup": "2015/3", "name": "NFS-share", "uuid": null, "size": 10485760, "owner": "root", "group": "root", "perms": "755", "toc": "2024-11-01T12:09:41.121Z", "subvol_name": "NFS-share", "replica": false, "compression_algo": null, "rusage": 16, "eusage": 16, "pqgroup_rusage": 16, "pqgroup_eusage": 16}

Where the top level model "nfsexport" references, in-turn:

  1. The "export_group" model instance id/pk=1
  2. The "share" model instance id/pk=1

@phillxnet
Copy link
Member

phillxnet commented Nov 1, 2024

With some additional debug logging we have the following config-save file replay on the original system:

[01/Nov/2024 15:27:57] INFO [storageadmin.views.config_backup:125] Started restoring NFS exports.
[01/Nov/2024 15:27:57] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.user, exports=[], export_groups={}
[01/Nov/2024 15:27:57] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.group, exports=[], export_groups={}
[01/Nov/2024 15:27:57] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.nfsexport, exports=[{'export_group': 1, 'share': 1, 'mount': '/export/NFS-share'}], export_groups={}
[01/Nov/2024 15:27:57] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.nfsexportgroup, exports=[{'export_group': 1, 'share': 1, 'mount': '/export/NFS-share'}], export_groups={1: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1}}
[01/Nov/2024 15:27:57] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.share, exports=[{'export_group': 1, 'share': 1, 'mount': '/export/NFS-share'}], export_groups={1: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1}}
[01/Nov/2024 15:27:57] DEBUG [storageadmin.views.config_backup:143] e["shares"]=['NFS-share']
[01/Nov/2024 15:27:57] ERROR [storageadmin.util:44] Exception: An export already exists for the host string: (*).
NoneType: None
[01/Nov/2024 15:27:57] DEBUG [storageadmin.util:45] Current Rockstor version: 5.0.14-2921
[01/Nov/2024 15:27:57] ERROR [storageadmin.views.config_backup:57] Exception occurred while creating resource: https://localhost/api/nfs-exports. Payload: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1, 'export_group': 1, 'share': 1, 'mount': '/export/NFS-share', 'shares': ['NFS-share']}. Exception: 500 Server Error: Internal Server Error for url: https://localhost/api/nfs-exports. Moving on.
[01/Nov/2024 15:27:57] INFO [storageadmin.views.config_backup:55] Successfully created resource: https://localhost/api/adv-nfs-exports. Payload: {'entries': []}
[01/Nov/2024 15:27:57] INFO [storageadmin.views.config_backup:147] Finished restoring NFS exports.

I.e. as expected we have no action and our exports file is unchanged:

cat /etc/exports
/export/NFS-share *(rw,async,insecure)

@phillxnet
Copy link
Member

NFS export restore

On the original system the single NFS export was deleted (Web-UI) and the NFS service left enabled. The OS was confirmed to have been updated via:

rleap15-6:~ # cat /etc/exports

returning no entries.

The same config-save as per the last comment was replayed again, with the same additional debug log entries in-play:

[01/Nov/2024 15:51:24] INFO [storageadmin.views.config_backup:125] Started restoring NFS exports.
[01/Nov/2024 15:51:24] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.user, exports=[], export_groups={}
[01/Nov/2024 15:51:24] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.group, exports=[], export_groups={}
[01/Nov/2024 15:51:24] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.nfsexport, exports=[{'export_group': 1, 'share': 1, 'mount': '/export/NFS-share'}], export_groups={}
[01/Nov/2024 15:51:24] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.nfsexportgroup, exports=[{'export_group': 1, 'share': 1, 'mount': '/export/NFS-share'}], export_groups={1: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1}}
[01/Nov/2024 15:51:24] DEBUG [storageadmin.views.config_backup:137] model=storageadmin.share, exports=[{'export_group': 1, 'share': 1, 'mount': '/export/NFS-share'}], export_groups={1: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1}}
[01/Nov/2024 15:51:24] DEBUG [storageadmin.views.config_backup:143] e["shares"]=['NFS-share']
[01/Nov/2024 15:51:25] INFO [storageadmin.views.config_backup:55] Successfully created resource: https://localhost/api/nfs-exports. Payload: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1, 'export_group': 1, 'share': 1, 'mount': '/export/NFS-share', 'shares': ['NFS-share']}
[01/Nov/2024 15:51:25] INFO [storageadmin.views.config_backup:55] Successfully created resource: https://localhost/api/adv-nfs-exports. Payload: {'entries': []}
[01/Nov/2024 15:51:25] INFO [storageadmin.views.config_backup:147] Finished restoring NFS exports.

With our issue subject 'flip' to a non-default read-only status for the restored NFS export:

Shares Host String Read only Sync / Async Actions
NFS-share * yes async  

And as expected we have the expected underlying OS config as per our Web-UI report:

cat /etc/exports
/export/NFS-share *(ro,async,insecure) 

@phillxnet
Copy link
Member

phillxnet commented Nov 1, 2024

It is currently assumed that we are failing re matching the raw DB value for 'editable' in the DB dump based config-backup file contents with what is accepted as input for NFS export creation via the API /api/nfs-exports. E.g.:

https://hostname/api/nfs-exports

From the above and our nfs_export tests:

https://github.com/rockstor/rockstor-core/blob/testing/src/rockstor/storageadmin/tests/test_nfs_export.py

there are disparities between our re-play (from config-backup file) Payload as logged in the above restore and what is used in our existing tests. With the observed behaviour of a default to read-only (ro).

@phillxnet
Copy link
Member

phillxnet commented Nov 1, 2024

Next focus is our API parsing to DB model translation: src/rockstor/storageadmin/views/nfs_exports.py

def parse_options(request):
options = {
"host_str": "*",
"editable": "ro",
"syncable": "async",
"mount_security": "insecure",
"admin_host": None,
}
options["host_str"] = request.data.get("host_str", options["host_str"])
options["editable"] = request.data.get("mod_choice", options["editable"])
options["syncable"] = request.data.get("sync_choice", options["syncable"])
options["admin_host"] = request.data.get("admin_host", options["admin_host"])

With our model defined here:

https://github.com/rockstor/rockstor-core/blob/testing/src/rockstor/storageadmin/models/nfs_export_group.py

with the referenced validators:

  • validate_nfs_host_str
  • validate_nfs_modify_str
  • validate_nfs_sync_choice

defined here: https://github.com/rockstor/rockstor-core/blob/testing/src/rockstor/storageadmin/models/validators.py

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 6, 2024
# Includes
- Instructions on fixture creation.
- Instructions on running nfs export tests.
- Modernise test_nfs_export.py by re-instating test_nfs.json fixture
and removing prior hard-coded object mock stand-ins for DB models
during these test runs.
@phillxnet
Copy link
Member

The associated tests for NFS export are in need of update/maintenance, containing many TODOs. I am including at least part of this maintenance against this issue as it should help to clarify expected function of our existing API - used in the NFS restore process. Which is where the fault lies with this issue: i.e. we restore via DB naming, where-as our current NFS export API has some differences re field names, and then some more re Web-UI naming. Improving our tests in this area can only help with current (this issue) and future development re NFS capability.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 7, 2024
- Add default ordering to NFSExportGroup models.
Addresses re-enabled test provoked warnings re:
"UnorderedObjectListWarning: Pagination may yield inconsistent
results with an unordered object_list"
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 7, 2024
Resolve remaining TODOs in test_nfs_export.py.
Includes:
- Removing redundant tests.
- Modifying previously remarked out sanity check tests
to instead cover our umbrella low-level exception. Once
more specific sanity checks are in place, we can add
tests to prove their function.
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 7, 2024
- Black reformatting test_nfs_export.py.
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 8, 2024
- Enable previously dormant field validation via addition of:
NFSExportGroup.clean_fields() prior to model.save(). Affects:
host_str, modify_str, & sync_choice during post & put.
- Additional tests to prove field validation function within
current validation capability (need refinement).
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 8, 2024
- Additional test_post() with API / DB / Web-UI mapping info.
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 8, 2024
- nfs_export_group.py comment correction & black formatting.
phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 8, 2024
- Additional NFS debug logging & comments: config_backup.py.
- Some string.format to fstring updates.
@phillxnet
Copy link
Member

Linking back to spin-off issue that is now deemed a dependency of this issue:

@phillxnet
Copy link
Member

phillxnet commented Nov 14, 2024

DB to API translation

As we simply dump our DB contents, and our API and DB have differing fields for our current NFS export create, the problem here is that we currently fail to translate NFS export DB field names back to API filed names.

I.e. from our prior example of a reproducer NFS export restore from saved config: with rw:

[01/Nov/2024 15:51:25] INFO [storageadmin.views.config_backup:55] Successfully created resource: https://localhost/api/nfs-exports. Payload: {'host_str': '*', 'editable': 'rw', 'syncable': 'async', 'mount_security': 'insecure', 'nohide': False, 'enabled': True, 'admin_host': None, 'pk': 1, 'export_group': 1, 'share': 1, 'mount': '/export/NFS-share', 'shares': ['NFS-share']}

it is evident that we are currently generating a restore payload with 'editable': 'rw', however there is no editable API field that we currently respond to; as per #2912 (comment)

 options["editable"] = request.data.get("mod_choice", options["editable"]) 
 options["syncable"] = request.data.get("sync_choice", options["syncable"]) 

Given the above, we either:

  1. Modify our API (and consequent front end code) to use only "editable", not the current "mod_choice".
  2. Introduce an translation mechanism into restore_nfs_exports().

It is proposed that option 2. is strongly preferred here as:

  • we fix only what is broken,
  • an API change is far-reaching in nature - and not appropriate at this time.

As such I will pursue the minimal fix of option (2.) give we are late RC, with comments to indicate the nature of this change.

Future testing channel phases could consider modifying our front-end to make this translation redundant. There-after, if we maintain the original editable' and syncable` payload resulting from the current direct DB dump to API filed translation, our interim fix of duplicating 'editable' & 'syncable' to 'mod_choice' & 'sync_choice' should serve to enable a smoother transition; robust to a failure to remove the translation layer: given redundancy would switch from our current "editable" & "syncable" filed use to the "mod_choice" & sync_choice" options we currently action on via our current request.data.get() use on this API.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 14, 2024
Fix translation of DB dump derived config-save field names to
NFS export create API field names.

Affects Web-UI surfaced NFS export config of:
- Access type ('Writable'/'Read only')
- Response type ('async'/'sync')
for NFS exports created via a config restore process.

Note that we maintain DB dump derived config-save field names
in the generated API calls (ignored currently), to enable
future API field name alignment to DB model field names.
At which point the additional fields added here to the API
payload would themselves become redundant and up for
deprecation, and in the change-over period ignored.
phillxnet added a commit that referenced this issue Nov 15, 2024
…read-only

NFS exports restored as read-only #2912
@phillxnet
Copy link
Member

Closing as:
Fixed by #2927

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants