diff --git a/PendingReleaseNotes b/PendingReleaseNotes index ad8ac0ed81a99..59692fe83a23d 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -279,6 +279,10 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config * RGW: in bucket notifications, the `principalId` inside `ownerIdentity` now contains complete user id, prefixed with tenant id +* NFS: The export create/apply of CephFS based exports will now have a additional parameter `cmount_path` under the FSAL block, + which specifies the path within the CephFS to mount this export on. If this and the other + `EXPORT { FSAL {} }` options are the same between multiple exports, those exports will share a single CephFS client. If not specified, the default is `/`. + >=18.0.0 * The RGW policy parser now rejects unknown principals by default. If you are diff --git a/README.md b/README.md index 3f599f9ac9c49..e51621ca8b8cf 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ To build Ceph, follow this procedure: contains `do_cmake.sh` and `CONTRIBUTING.rst`. 2. Run the `do_cmake.sh` script: - ``./do_cmake.sh`` + ./do_cmake.sh ``do_cmake.sh`` by default creates a "debug build" of Ceph, which can be up to five times slower than a non-debug build. Pass @@ -89,7 +89,7 @@ To build Ceph, follow this procedure: non-debug build. 3. Move into the `build` directory: - ``cd build`` + cd build 4. Use the `ninja` buildsystem to build the development environment: ninja -j3 @@ -120,11 +120,11 @@ To build Ceph, follow this procedure: To build only certain targets, run a command of the following form: - ``ninja [target name]`` + ninja [target name] 5. Install the vstart cluster: - ``ninja install`` + ninja install ### CMake Options diff --git a/doc/cephfs/fs-volumes.rst b/doc/cephfs/fs-volumes.rst index 71d665f2cefd6..3adac4a51a438 100644 --- a/doc/cephfs/fs-volumes.rst +++ b/doc/cephfs/fs-volumes.rst @@ -276,7 +276,7 @@ Use a command of the following form to create a subvolume: .. prompt:: bash # - ceph fs subvolume create [--size ] [--group_name ] [--pool_layout ] [--uid ] [--gid ] [--mode ] [--namespace-isolated] + ceph fs subvolume create [--size ] [--group_name ] [--pool_layout ] [--uid ] [--gid ] [--mode ] [--namespace-isolated] [--earmark ] The command succeeds even if the subvolume already exists. @@ -289,6 +289,15 @@ The subvolume can be created in a separate RADOS namespace by specifying the default subvolume group with an octal file mode of ``755``, a uid of its subvolume group, a gid of its subvolume group, a data pool layout of its parent directory, and no size limit. +You can also assign an earmark to a subvolume using the ``--earmark`` option. +The earmark is a unique identifier that tags the subvolume for specific purposes, +such as NFS or SMB services. By default, no earmark is set, allowing for flexible +assignment based on administrative needs. An empty string ("") can be used to remove +any existing earmark from a subvolume. + +The earmarking mechanism ensures that subvolumes are correctly tagged and managed, +helping to avoid conflicts and ensuring that each subvolume is associated +with the intended service or use case. Removing a subvolume ~~~~~~~~~~~~~~~~~~~~ @@ -418,6 +427,7 @@ The output format is JSON and contains the following fields. * ``pool_namespace``: RADOS namespace of the subvolume * ``features``: features supported by the subvolume * ``state``: current state of the subvolume +* ``earmark``: earmark of the subvolume If a subvolume has been removed but its snapshots have been retained, the output contains only the following fields. @@ -522,6 +532,33 @@ subvolume using the metadata key: Using the ``--force`` flag allows the command to succeed when it would otherwise fail (if the metadata key did not exist). +Getting earmark of a subvolume +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use a command of the following form to get the earmark of a subvolume: + +.. prompt:: bash # + + ceph fs subvolume earmark get [--group_name ] + +Setting earmark of a subvolume +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use a command of the following form to set the earmark of a subvolume: + +.. prompt:: bash # + + ceph fs subvolume earmark set [--group_name ] + +Removing earmark of a subvolume +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use a command of the following form to remove the earmark of a subvolume: + +.. prompt:: bash # + + ceph fs subvolume earmark rm [--group_name ] + Creating a Snapshot of a Subvolume ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst index c4d03aa7b776b..fed490a0fae83 100644 --- a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst +++ b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro.rst @@ -317,16 +317,16 @@ directory tree within the ``suites/`` subdirectory of the `ceph/qa sub-directory The set of all tests defined by a given subdirectory of ``suites/`` is called an "integration test suite", or a "teuthology suite". -Combination of yaml facets is controlled by special files (``%`` and -``+``) that are placed within the directory tree and can be thought of as -operators. The ``%`` file is the "convolution" operator and ``+`` -signifies concatenation. +Combination of YAML facets is controlled by special files (``%``, ``+`` and ``$``) +that are placed within the directory tree and can be thought of as +operators. The ``%`` file is the "convolution" operator, ``+`` signifies +concatenation and ``$`` is the "random selection" operator. -Convolution operator -^^^^^^^^^^^^^^^^^^^^ +Convolution operator - ``%`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The convolution operator, implemented as a (typically empty) file called ``%``, -tells teuthology to construct a test matrix from yaml facets found in +tells teuthology to construct a test matrix from YAML facets found in subdirectories below the directory containing the operator. For example, the `ceph-deploy suite @@ -421,8 +421,8 @@ tests will still preserve the correct numerator (subset of subsets). You can disable nested subsets using the ``--no-nested-subset`` argument to ``teuthology-suite``. -Concatenation operator -^^^^^^^^^^^^^^^^^^^^^^ +Concatenation operator - ``+`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For even greater flexibility in sharing yaml files between suites, the special file plus (``+``) can be used to concatenate files within a @@ -561,6 +561,15 @@ rest of the cluster (``5-finish-upgrade.yaml``). The last stage is requiring the updated release (``ceph require-osd-release quincy``, ``ceph osd set-require-min-compat-client quincy``) and running the ``final-workload``. +Random Selection Operator - ``$`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The presence of a file named ``$`` provides a hint to teuthology to randomly +include one of the YAML fragments in the test. Such a scenario is typically +seen when we need to choose one of the flavors of the features/options to be +tested randomly. + + Position Independent Linking ---------------------------- diff --git a/doc/man/8/rbd.rst b/doc/man/8/rbd.rst index 0f490506aa6b4..4039e78fad3ad 100644 --- a/doc/man/8/rbd.rst +++ b/doc/man/8/rbd.rst @@ -543,8 +543,9 @@ Commands :command:`mirror pool info` [*pool-name*] Show information about the pool or namespace mirroring configuration. - For a pool, it includes mirroring mode, peer UUID, remote cluster name, - and remote client name. For a namespace, it includes only mirroring mode. + For both pools and namespaces, it includes the mirroring mode, mirror UUID + and remote namespace. For pools, it additionally includes the site name, + peer UUID, remote cluster name, and remote client name. :command:`mirror pool peer add` [*pool-name*] *remote-cluster-spec* Add a mirroring peer to a pool. @@ -555,7 +556,7 @@ Commands This requires mirroring to be enabled on the pool. :command:`mirror pool peer remove` [*pool-name*] *uuid* - Remove a mirroring peer from a pool. The peer uuid is available + Remove a mirroring peer from a pool. The peer UUID is available from ``mirror pool info`` command. :command:`mirror pool peer set` [*pool-name*] *uuid* *key* *value* diff --git a/doc/mgr/nfs.rst b/doc/mgr/nfs.rst index 746ab4247f39f..3077ea40aa6f1 100644 --- a/doc/mgr/nfs.rst +++ b/doc/mgr/nfs.rst @@ -283,7 +283,7 @@ Create CephFS Export .. code:: bash - $ ceph nfs export create cephfs --cluster-id --pseudo-path --fsname [--readonly] [--path=/path/in/cephfs] [--client_addr ...] [--squash ] [--sectype ...] + $ ceph nfs export create cephfs --cluster-id --pseudo-path --fsname [--readonly] [--path=/path/in/cephfs] [--client_addr ...] [--squash ] [--sectype ...] [--cmount_path ] This creates export RADOS objects containing the export block, where @@ -318,6 +318,12 @@ values may be separated by a comma (example: ``--sectype krb5p,krb5i``). The server will negotatiate a supported security type with the client preferring the supplied methods left-to-right. +```` specifies the path within the CephFS to mount this export on. It is +allowed to be any complete path hierarchy between ``/`` and the ``EXPORT {path}``. (i.e. if ``EXPORT { Path }`` parameter is ``/foo/bar`` then cmount_path could be ``/``, ``/foo`` or ``/foo/bar``). + +.. note:: If this and the other ``EXPORT { FSAL {} }`` options are the same between multiple exports, those exports will share a single CephFS client. + If not specified, the default is ``/``. + .. note:: Specifying values for sectype that require Kerberos will only function on servers that are configured to support Kerberos. Setting up NFS-Ganesha to support Kerberos is outside the scope of this document. @@ -477,9 +483,9 @@ For example,:: ], "fsal": { "name": "CEPH", - "user_id": "nfs.mynfs.1", "fs_name": "a", - "sec_label_xattr": "" + "sec_label_xattr": "", + "cmount_path": "/" }, "clients": [] } @@ -494,6 +500,9 @@ as when creating a new export), with the exception of the authentication credentials, which will be carried over from the previous state of the export where possible. +!! NOTE: The ``user_id`` in the ``fsal`` block should not be modified or mentioned in the JSON file as it is auto-generated for CephFS exports. +It's auto-generated in the format ``nfs...``. + :: $ ceph nfs export apply mynfs -i update_cephfs_export.json @@ -514,9 +523,9 @@ previous state of the export where possible. ], "fsal": { "name": "CEPH", - "user_id": "nfs.mynfs.1", "fs_name": "a", - "sec_label_xattr": "" + "sec_label_xattr": "", + "cmount_path": "/" }, "clients": [] } diff --git a/qa/suites/fs/functional/subvol_versions/.qa b/qa/suites/fs/functional/subvol_versions/.qa deleted file mode 120000 index fea2489fdf6d9..0000000000000 --- a/qa/suites/fs/functional/subvol_versions/.qa +++ /dev/null @@ -1 +0,0 @@ -../.qa \ No newline at end of file diff --git a/qa/suites/fs/functional/subvol_versions/create_subvol_version_v1.yaml b/qa/suites/fs/functional/subvol_versions/create_subvol_version_v1.yaml deleted file mode 120000 index 09cfdb59edaed..0000000000000 --- a/qa/suites/fs/functional/subvol_versions/create_subvol_version_v1.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/cephfs/overrides/subvol_versions/create_subvol_version_v1.yaml \ No newline at end of file diff --git a/qa/suites/fs/functional/subvol_versions/create_subvol_version_v2.yaml b/qa/suites/fs/functional/subvol_versions/create_subvol_version_v2.yaml deleted file mode 120000 index 5a4de14e7e008..0000000000000 --- a/qa/suites/fs/functional/subvol_versions/create_subvol_version_v2.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/cephfs/overrides/subvol_versions/create_subvol_version_v2.yaml \ No newline at end of file diff --git a/qa/suites/fs/functional/tasks/test_snap_schedule/% b/qa/suites/fs/functional/tasks/test_snap_schedule/% new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/qa/suites/fs/functional/tasks/test_snap_schedule/overrides/$ b/qa/suites/fs/functional/tasks/test_snap_schedule/overrides/$ new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/qa/cephfs/overrides/subvol_versions/create_subvol_version_v1.yaml b/qa/suites/fs/functional/tasks/test_snap_schedule/overrides/v1.yaml similarity index 100% rename from qa/cephfs/overrides/subvol_versions/create_subvol_version_v1.yaml rename to qa/suites/fs/functional/tasks/test_snap_schedule/overrides/v1.yaml diff --git a/qa/cephfs/overrides/subvol_versions/create_subvol_version_v2.yaml b/qa/suites/fs/functional/tasks/test_snap_schedule/overrides/v2.yaml similarity index 100% rename from qa/cephfs/overrides/subvol_versions/create_subvol_version_v2.yaml rename to qa/suites/fs/functional/tasks/test_snap_schedule/overrides/v2.yaml diff --git a/qa/suites/fs/functional/tasks/snap-schedule.yaml b/qa/suites/fs/functional/tasks/test_snap_schedule/snap-schedule.yaml similarity index 100% rename from qa/suites/fs/functional/tasks/snap-schedule.yaml rename to qa/suites/fs/functional/tasks/test_snap_schedule/snap-schedule.yaml diff --git a/qa/suites/orch/cephadm/workunits/task/test_monitoring_stack_basic.yaml b/qa/suites/orch/cephadm/workunits/task/test_monitoring_stack_basic.yaml index 89733dabeadd1..515293ea83a71 100644 --- a/qa/suites/orch/cephadm/workunits/task/test_monitoring_stack_basic.yaml +++ b/qa/suites/orch/cephadm/workunits/task/test_monitoring_stack_basic.yaml @@ -61,6 +61,6 @@ tasks: curl -s http://${PROM_IP}:9095/api/v1/alerts curl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e '.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"' # check alertmanager endpoints are responsive and mon down alert is active - curl -s http://${ALERTM_IP}:9093/api/v1/status - curl -s http://${ALERTM_IP}:9093/api/v1/alerts - curl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e '.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"' + curl -s http://${ALERTM_IP}:9093/api/v2/status + curl -s http://${ALERTM_IP}:9093/api/v2/alerts + curl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e '.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"' diff --git a/qa/tasks/cephfs/test_backtrace.py b/qa/tasks/cephfs/test_backtrace.py index 6b094569b7b17..cd23c114bfb86 100644 --- a/qa/tasks/cephfs/test_backtrace.py +++ b/qa/tasks/cephfs/test_backtrace.py @@ -100,3 +100,29 @@ def test_backtrace(self): # we don't update the layout in all the old pools whenever it changes old_pool_layout = self.fs.read_layout(file_ino, pool=old_data_pool_name) self.assertEqual(old_pool_layout['object_size'], 4194304) + + def test_backtrace_flush_on_deleted_data_pool(self): + """ + that the MDS does not go read-only when handling backtrace update errors + when backtrace updates are batched and flushed to RADOS (during journal trim) + and some of the pool have been removed. + """ + data_pool = self.fs.get_data_pool_name() + extra_data_pool_name_1 = data_pool + '_extra1' + self.fs.add_data_pool(extra_data_pool_name_1) + + self.mount_a.run_shell(["mkdir", "dir_x"]) + self.mount_a.setfattr("dir_x", "ceph.dir.layout.pool", extra_data_pool_name_1) + self.mount_a.run_shell(["touch", "dir_x/file_x"]) + self.fs.flush() + + extra_data_pool_name_2 = data_pool + '_extra2' + self.fs.add_data_pool(extra_data_pool_name_2) + self.mount_a.setfattr("dir_x/file_x", "ceph.file.layout.pool", extra_data_pool_name_2) + self.mount_a.run_shell(["setfattr", "-x", "ceph.dir.layout", "dir_x"]) + self.run_ceph_cmd("fs", "rm_data_pool", self.fs.name, extra_data_pool_name_1) + self.fs.flush() + + # quick test to check if the mds has handled backtrace update failure + # on the deleted data pool without going read-only. + self.mount_a.run_shell(["mkdir", "dir_y"]) diff --git a/qa/tasks/cephfs/test_mirroring.py b/qa/tasks/cephfs/test_mirroring.py index 7247ec5db7cc2..55de1c7b92862 100644 --- a/qa/tasks/cephfs/test_mirroring.py +++ b/qa/tasks/cephfs/test_mirroring.py @@ -1505,9 +1505,22 @@ def test_get_set_mirror_dirty_snap_id(self): """ That get/set ceph.mirror.dirty_snap_id attribute succeeds in a remote filesystem. """ + log.debug('reconfigure client auth caps') + self.get_ceph_cmd_result( + 'auth', 'caps', "client.{0}".format(self.mount_b.client_id), + 'mds', 'allow rw', + 'mon', 'allow r', + 'osd', 'allow rw pool={0}, allow rw pool={1}'.format( + self.backup_fs.get_data_pool_name(), + self.backup_fs.get_data_pool_name())) + log.debug(f'mounting filesystem {self.secondary_fs_name}') + self.mount_b.umount_wait() + self.mount_b.mount_wait(cephfs_name=self.secondary_fs_name) + log.debug('setting ceph.mirror.dirty_snap_id attribute') self.mount_b.run_shell(["mkdir", "-p", "d1/d2/d3"]) attr = str(random.randint(1, 10)) self.mount_b.setfattr("d1/d2/d3", "ceph.mirror.dirty_snap_id", attr) + log.debug('getting ceph.mirror.dirty_snap_id attribute') val = self.mount_b.getfattr("d1/d2/d3", "ceph.mirror.dirty_snap_id") self.assertEqual(attr, val, f"Mismatch for ceph.mirror.dirty_snap_id value: {attr} vs {val}") diff --git a/qa/tasks/cephfs/test_nfs.py b/qa/tasks/cephfs/test_nfs.py index 6d1c65dfb7dfd..932d504d47f3e 100644 --- a/qa/tasks/cephfs/test_nfs.py +++ b/qa/tasks/cephfs/test_nfs.py @@ -14,6 +14,8 @@ NFS_POOL_NAME = '.nfs' # should match mgr_module.py # TODO Add test for cluster update when ganesha can be deployed on multiple ports. + + class TestNFS(MgrTestCase): def _cmd(self, *args): return self.get_ceph_cmd_stdout(args) @@ -59,8 +61,9 @@ def setUp(self): ], "fsal": { "name": "CEPH", - "user_id": "nfs.test.1", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": self.fs_name, + "cmount_path": "/", }, "clients": [] } @@ -118,7 +121,7 @@ def _check_nfs_cluster_status(self, expected_status, fail_msg): return self.fail(fail_msg) - def _check_auth_ls(self, export_id=1, check_in=False): + def _check_auth_ls(self, fs_name, check_in=False, user_id=None): ''' Tests export user id creation or deletion. :param export_id: Denotes export number @@ -126,10 +129,12 @@ def _check_auth_ls(self, export_id=1, check_in=False): ''' output = self._cmd('auth', 'ls') client_id = f'client.nfs.{self.cluster_id}' + search_id = f'client.{user_id}' if user_id else f'{client_id}.{fs_name}' + if check_in: - self.assertIn(f'{client_id}.{export_id}', output) + self.assertIn(search_id, output) else: - self.assertNotIn(f'{client_id}.{export_id}', output) + self.assertNotIn(search_id, output) def _test_idempotency(self, cmd_func, cmd_args): ''' @@ -216,7 +221,7 @@ def _create_export(self, export_id, create_fs=False, extra_cmd=None): # Runs the nfs export create command self._cmd(*export_cmd) # Check if user id for export is created - self._check_auth_ls(export_id, check_in=True) + self._check_auth_ls(self.fs_name, check_in=True) res = self._sys_cmd(['rados', '-p', NFS_POOL_NAME, '-N', self.cluster_id, 'get', f'export-{export_id}', '-']) # Check if export object is created @@ -230,12 +235,12 @@ def _create_default_export(self): self._test_create_cluster() self._create_export(export_id='1', create_fs=True) - def _delete_export(self): + def _delete_export(self, pseduo_path=None, check_in=False, user_id=None): ''' Delete an export. ''' - self._nfs_cmd('export', 'rm', self.cluster_id, self.pseudo_path) - self._check_auth_ls() + self._nfs_cmd('export', 'rm', self.cluster_id, pseduo_path if pseduo_path else self.pseudo_path) + self._check_auth_ls(self.fs_name, check_in, user_id) def _test_list_export(self): ''' @@ -256,26 +261,27 @@ def _test_list_detailed(self, sub_vol_path): self.sample_export['export_id'] = 2 self.sample_export['pseudo'] = self.pseudo_path + '1' self.sample_export['access_type'] = 'RO' - self.sample_export['fsal']['user_id'] = f'{self.expected_name}.2' + self.sample_export['fsal']['user_id'] = f'{self.expected_name}.{self.fs_name}.3746f603' self.assertDictEqual(self.sample_export, nfs_output[1]) # Export-3 for subvolume with r only self.sample_export['export_id'] = 3 self.sample_export['path'] = sub_vol_path self.sample_export['pseudo'] = self.pseudo_path + '2' - self.sample_export['fsal']['user_id'] = f'{self.expected_name}.3' + self.sample_export['fsal']['user_id'] = f'{self.expected_name}.{self.fs_name}.3746f603' self.assertDictEqual(self.sample_export, nfs_output[2]) # Export-4 for subvolume self.sample_export['export_id'] = 4 self.sample_export['pseudo'] = self.pseudo_path + '3' self.sample_export['access_type'] = 'RW' - self.sample_export['fsal']['user_id'] = f'{self.expected_name}.4' + self.sample_export['fsal']['user_id'] = f'{self.expected_name}.{self.fs_name}.3746f603' self.assertDictEqual(self.sample_export, nfs_output[3]) - def _get_export(self): + def _get_export(self, pseudo_path=None): ''' Returns export block in json format ''' - return json.loads(self._nfs_cmd('export', 'info', self.cluster_id, self.pseudo_path)) + return json.loads(self._nfs_cmd('export', 'info', self.cluster_id, + pseudo_path if pseudo_path else self.pseudo_path)) def _test_get_export(self): ''' @@ -506,7 +512,7 @@ def test_create_multiple_exports(self): self._test_delete_cluster() # Check if rados ganesha conf object is deleted self._check_export_obj_deleted(conf_obj=True) - self._check_auth_ls() + self._check_auth_ls(self.fs_name) def test_exports_on_mgr_restart(self): ''' @@ -935,7 +941,7 @@ def test_nfs_export_apply_multiple_exports(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.1", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": self.fs_name } }, @@ -948,7 +954,7 @@ def test_nfs_export_apply_multiple_exports(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.2", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": "invalid_fs_name" # invalid fs } }, @@ -961,7 +967,7 @@ def test_nfs_export_apply_multiple_exports(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.3", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": self.fs_name } } @@ -1008,7 +1014,7 @@ def test_nfs_export_apply_single_export(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.1", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": "invalid_fs_name" # invalid fs } } @@ -1048,7 +1054,7 @@ def test_nfs_export_apply_json_output_states(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.1", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": self.fs_name } }, @@ -1061,7 +1067,7 @@ def test_nfs_export_apply_json_output_states(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.2", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": self.fs_name } }, @@ -1075,7 +1081,7 @@ def test_nfs_export_apply_json_output_states(self): "protocols": [4], "fsal": { "name": "CEPH", - "user_id": "nfs.test.3", + "user_id": "nfs.test.nfs-cephfs.3746f603", "fs_name": "invalid_fs_name" } } @@ -1211,3 +1217,65 @@ def test_cephfs_export_update_at_non_dir_path(self): finally: self.ctx.cluster.run(args=['rm', '-rf', f'{mnt_pt}/*']) self._delete_cluster_with_fs(self.fs_name, mnt_pt, preserve_mode) + + def test_nfs_export_creation_without_cmount_path(self): + """ + Test that ensure cmount_path is present in FSAL block + """ + self._create_cluster_with_fs(self.fs_name) + + pseudo_path = '/test_without_cmount' + self._create_export(export_id='1', + extra_cmd=['--pseudo-path', pseudo_path]) + nfs_output = self._get_export(pseudo_path) + self.assertIn('cmount_path', nfs_output['fsal']) + + self._delete_export(pseudo_path) + + def test_nfs_exports_with_same_and_diff_user_id(self): + """ + Test that exports with same FSAL share same user_id + """ + self._create_cluster_with_fs(self.fs_name) + + pseudo_path_1 = '/test1' + pseudo_path_2 = '/test2' + pseudo_path_3 = '/test3' + + # Create subvolumes + self._cmd('fs', 'subvolume', 'create', self.fs_name, 'sub_vol_1') + self._cmd('fs', 'subvolume', 'create', self.fs_name, 'sub_vol_2') + + fs_path_1 = self._cmd('fs', 'subvolume', 'getpath', self.fs_name, 'sub_vol_1').strip() + fs_path_2 = self._cmd('fs', 'subvolume', 'getpath', self.fs_name, 'sub_vol_2').strip() + # Both exports should have same user_id(since cmount_path=/ & fs_name is same) + self._create_export(export_id='1', + extra_cmd=['--pseudo-path', pseudo_path_1, + '--path', fs_path_1]) + self._create_export(export_id='2', + extra_cmd=['--pseudo-path', pseudo_path_2, + '--path', fs_path_2]) + + nfs_output_1 = self._get_export(pseudo_path_1) + nfs_output_2 = self._get_export(pseudo_path_2) + # Check if both exports have same user_id + self.assertEqual(nfs_output_2['fsal']['user_id'], nfs_output_1['fsal']['user_id']) + self.assertEqual(nfs_output_1['fsal']['user_id'], 'nfs.test.nfs-cephfs.3746f603') + + cmount_path = '/volumes' + self._create_export(export_id='3', + extra_cmd=['--pseudo-path', pseudo_path_3, + '--path', fs_path_1, + '--cmount-path', cmount_path]) + + nfs_output_3 = self._get_export(pseudo_path_3) + self.assertNotEqual(nfs_output_3['fsal']['user_id'], nfs_output_1['fsal']['user_id']) + self.assertEqual(nfs_output_3['fsal']['user_id'], 'nfs.test.nfs-cephfs.32cd8545') + + # Deleting export with same user_id should not delete the user_id + self._delete_export(pseudo_path_1, True, nfs_output_1['fsal']['user_id']) + # Deleting export 22 should delete the user_id since it's only export left with that user_id + self._delete_export(pseudo_path_2, False, nfs_output_2['fsal']['user_id']) + + # Deleting export 23 should delete the user_id since it's only export with that user_id + self._delete_export(pseudo_path_3, False, nfs_output_3['fsal']['user_id']) diff --git a/qa/tasks/cephfs/test_volumes.py b/qa/tasks/cephfs/test_volumes.py index eb10d836c7c6c..2baefd72c3fbc 100644 --- a/qa/tasks/cephfs/test_volumes.py +++ b/qa/tasks/cephfs/test_volumes.py @@ -2367,6 +2367,124 @@ def test_subvolume_create_and_ls_providing_group_as_nogroup(self): # verify trash dir is clean. self._wait_for_trash_empty() + + def test_subvolume_create_with_earmark(self): + # create subvolume with earmark + subvolume = self._gen_subvol_name() + earmark = "nfs.test" + self._fs_cmd("subvolume", "create", self.volname, subvolume, "--earmark", earmark) + + # make sure it exists + subvolpath = self._get_subvolume_path(self.volname, subvolume) + self.assertNotEqual(subvolpath, None) + + # verify the earmark + get_earmark = self._fs_cmd("subvolume", "earmark", "get", self.volname, subvolume) + self.assertEqual(get_earmark.rstrip('\n'), earmark) + + def test_subvolume_set_and_get_earmark(self): + # create subvolume + subvolume = self._gen_subvol_name() + self._fs_cmd("subvolume", "create", self.volname, subvolume) + + # set earmark + earmark = "smb.test" + self._fs_cmd("subvolume", "earmark", "set", self.volname, subvolume, "--earmark", earmark) + + # get earmark + get_earmark = self._fs_cmd("subvolume", "earmark", "get", self.volname, subvolume) + self.assertEqual(get_earmark.rstrip('\n'), earmark) + + def test_subvolume_clear_earmark(self): + # create subvolume + subvolume = self._gen_subvol_name() + self._fs_cmd("subvolume", "create", self.volname, subvolume) + + # set earmark + earmark = "smb.test" + self._fs_cmd("subvolume", "earmark", "set", self.volname, subvolume, "--earmark", earmark) + + # remove earmark + self._fs_cmd("subvolume", "earmark", "rm", self.volname, subvolume) + + # get earmark + get_earmark = self._fs_cmd("subvolume", "earmark", "get", self.volname, subvolume) + self.assertEqual(get_earmark, "") + + def test_earmark_on_non_existing_subvolume(self): + subvolume = "non_existing_subvol" + earmark = "nfs.test" + commands = [ + ("set", earmark), + ("get", None), + ("rm", None), + ] + + for action, arg in commands: + try: + # Build the command arguments + cmd_args = ["subvolume", "earmark", action, self.volname, subvolume] + if arg is not None: + cmd_args.extend(["--earmark", arg]) + + # Execute the command with built arguments + self._fs_cmd(*cmd_args) + except CommandFailedError as ce: + self.assertEqual(ce.exitstatus, errno.ENOENT) + + def test_get_remove_earmark_when_not_set(self): + # Create a subvolume without setting an earmark + subvolume = self._gen_subvol_name() + self._fs_cmd("subvolume", "create", self.volname, subvolume) + + # Attempt to get an earmark when it's not set + get_earmark = self._fs_cmd("subvolume", "earmark", "get", self.volname, subvolume) + self.assertEqual(get_earmark, "") + + # Attempt to remove an earmark when it's not set + self._fs_cmd("subvolume", "earmark", "rm", self.volname, subvolume) + + def test_set_invalid_earmark(self): + # Create a subvolume + subvolume = self._gen_subvol_name() + self._fs_cmd("subvolume", "create", self.volname, subvolume) + + # Attempt to set an invalid earmark + invalid_earmark = "invalid_format" + expected_message = ( + f"Invalid earmark specified: '{invalid_earmark}'. A valid earmark should " + "either be empty or start with 'nfs' or 'smb', followed by dot-separated " + "non-empty components." + ) + try: + self._fs_cmd("subvolume", "earmark", "set", self.volname, subvolume, "--earmark", invalid_earmark) + except CommandFailedError as ce: + self.assertEqual(ce.exitstatus, errno.EINVAL, expected_message) + + def test_earmark_on_deleted_subvolume_with_retained_snapshot(self): + subvolume = self._gen_subvol_name() + snapshot = self._gen_subvol_snap_name() + + # Create subvolume and snapshot + self._fs_cmd("subvolume", "create", self.volname, subvolume) + self._fs_cmd("subvolume", "snapshot", "create", self.volname, subvolume, snapshot) + + # Delete subvolume while retaining the snapshot + self._fs_cmd("subvolume", "rm", self.volname, subvolume, "--retain-snapshots") + + # Define the expected error message + error_message = f'subvolume "{subvolume}" is removed and has only snapshots retained' + + # Test cases for setting, getting, and removing earmarks + for operation in ["get", "rm", "set"]: + try: + extra_arg = "smb" if operation == "set" else None + if operation == "set": + self._fs_cmd("subvolume", "earmark", operation, self.volname, subvolume, "--earmark", extra_arg) + else: + self._fs_cmd("subvolume", "earmark", operation, self.volname, subvolume) + except CommandFailedError as ce: + self.assertEqual(ce.exitstatus, errno.ENOENT, error_message) def test_subvolume_expand(self): """ @@ -2440,6 +2558,14 @@ def test_subvolume_info(self): for feature in ['snapshot-clone', 'snapshot-autoprotect', 'snapshot-retention']: self.assertIn(feature, subvol_info["features"], msg="expected feature '{0}' in subvolume".format(feature)) + # set earmark + earmark = "smb.test" + self._fs_cmd("subvolume", "earmark", "set", self.volname, subvolume, "--earmark", earmark) + + subvol_info = json.loads(self._get_subvolume_info(self.volname, subvolume)) + + self.assertEqual(subvol_info["earmark"], earmark) + # remove subvolumes self._fs_cmd("subvolume", "rm", self.volname, subvolume) diff --git a/qa/tasks/mgr/dashboard/test_auth.py b/qa/tasks/mgr/dashboard/test_auth.py index a2266229bef7f..2b9240b635ec6 100644 --- a/qa/tasks/mgr/dashboard/test_auth.py +++ b/qa/tasks/mgr/dashboard/test_auth.py @@ -152,7 +152,8 @@ def test_logout(self): self._post("/api/auth/logout") self.assertStatus(200) self.assertJsonBody({ - "redirect_url": "#/login" + "redirect_url": "#/login", + "protocol": 'local' }) self._get("/api/host", version='1.1') self.assertStatus(401) @@ -167,7 +168,8 @@ def test_logout(self): self._post("/api/auth/logout", set_cookies=True) self.assertStatus(200) self.assertJsonBody({ - "redirect_url": "#/login" + "redirect_url": "#/login", + "protocol": 'local' }) self._get("/api/host", set_cookies=True, version='1.1') self.assertStatus(401) diff --git a/qa/workunits/rbd/rbd_mirror.sh b/qa/workunits/rbd/rbd_mirror.sh index 1cda355039eb8..90d5204b92feb 100755 --- a/qa/workunits/rbd/rbd_mirror.sh +++ b/qa/workunits/rbd/rbd_mirror.sh @@ -37,12 +37,12 @@ set_image_meta ${CLUSTER2} ${POOL} ${image} "key1" "value1" set_image_meta ${CLUSTER2} ${POOL} ${image} "key2" "value2" wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} write_image ${CLUSTER2} ${POOL} ${image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} if [ -z "${RBD_MIRROR_USE_RBD_MIRROR}" ]; then wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'down+unknown' fi -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} compare_image_meta ${CLUSTER1} ${POOL} ${image} "key1" "value1" compare_image_meta ${CLUSTER1} ${POOL} ${image} "key2" "value2" @@ -53,19 +53,19 @@ create_image_and_enable_mirror ${CLUSTER2} ${POOL} ${image1} ${RBD_MIRROR_MODE} write_image ${CLUSTER2} ${POOL} ${image1} 100 start_mirrors ${CLUSTER1} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image1} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image1} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image1} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image1} if [ -z "${RBD_MIRROR_USE_RBD_MIRROR}" ]; then wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image1} 'down+unknown' fi -compare_images ${POOL} ${image1} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image1} testlog "TEST: test the first image is replaying after restart" write_image ${CLUSTER2} ${POOL} ${image} 100 wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} if [ -z "${RBD_MIRROR_USE_RBD_MIRROR}" ]; then testlog "TEST: stop/start/restart mirror via admin socket" @@ -173,7 +173,7 @@ wait_for_image_in_omap ${CLUSTER2} ${POOL} create_image_and_enable_mirror ${CLUSTER2} ${POOL} ${image} ${RBD_MIRROR_MODE} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} write_image ${CLUSTER2} ${POOL} ${image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} 'up+replaying' testlog "TEST: failover and failback" @@ -187,10 +187,10 @@ wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+unknown' promote_image ${CLUSTER2} ${POOL} ${image} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} write_image ${CLUSTER2} ${POOL} ${image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+stopped' wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} # failover (unmodified) demote_image ${CLUSTER2} ${POOL} ${image} @@ -207,10 +207,10 @@ wait_for_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} 'up+unknown' wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+unknown' promote_image ${CLUSTER2} ${POOL} ${image} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+stopped' -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} # failover demote_image ${CLUSTER2} ${POOL} ${image} @@ -220,10 +220,10 @@ wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+unknown' promote_image ${CLUSTER1} ${POOL} ${image} wait_for_image_replay_started ${CLUSTER2} ${POOL} ${image} write_image ${CLUSTER1} ${POOL} ${image} 100 -wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${POOL} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} 'up+stopped' wait_for_replaying_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} # failback demote_image ${CLUSTER1} ${POOL} ${image} @@ -233,10 +233,10 @@ wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+unknown' promote_image ${CLUSTER2} ${POOL} ${image} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} write_image ${CLUSTER2} ${POOL} ${image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+stopped' -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} testlog "TEST: failover / failback loop" for i in `seq 1 20`; do @@ -246,7 +246,7 @@ for i in `seq 1 20`; do wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+unknown' promote_image ${CLUSTER1} ${POOL} ${image} wait_for_image_replay_started ${CLUSTER2} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${POOL} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} 'up+stopped' wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+replaying' demote_image ${CLUSTER1} ${POOL} ${image} @@ -255,7 +255,7 @@ for i in `seq 1 20`; do wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+unknown' promote_image ${CLUSTER2} ${POOL} ${image} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} 'up+stopped' wait_for_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} 'up+replaying' done @@ -271,7 +271,7 @@ create_image_and_enable_mirror ${CLUSTER2} ${POOL} ${force_promote_image} ${RBD_ write_image ${CLUSTER2} ${POOL} ${force_promote_image} 100 wait_for_image_replay_stopped ${CLUSTER2} ${POOL} ${force_promote_image} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${force_promote_image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${force_promote_image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${force_promote_image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${force_promote_image} wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${force_promote_image} 'up+stopped' promote_image ${CLUSTER1} ${POOL} ${force_promote_image} '--force' @@ -302,14 +302,14 @@ else enable_mirror ${CLUSTER2} ${PARENT_POOL} ${parent_image} ${RBD_MIRROR_MODE} fi wait_for_image_replay_started ${CLUSTER1} ${PARENT_POOL} ${parent_image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${PARENT_POOL} ${parent_image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${PARENT_POOL} ${PARENT_POOL} ${parent_image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${PARENT_POOL} ${parent_image} -compare_images ${PARENT_POOL} ${parent_image} +compare_images ${CLUSTER1} ${CLUSTER2} ${PARENT_POOL} ${PARENT_POOL} ${parent_image} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${clone_image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${clone_image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${clone_image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${clone_image} -compare_images ${POOL} ${clone_image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${clone_image} remove_image_retry ${CLUSTER2} ${POOL} ${clone_image} testlog " - clone v1" @@ -383,11 +383,11 @@ create_snapshot ${CLUSTER2} ${POOL} ${dp_image} 'snap1' write_image ${CLUSTER2} ${POOL} ${dp_image} 100 create_snapshot ${CLUSTER2} ${POOL} ${dp_image} 'snap2' write_image ${CLUSTER2} ${POOL} ${dp_image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${dp_image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${dp_image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${dp_image} -compare_images ${POOL} ${dp_image}@snap1 -compare_images ${POOL} ${dp_image}@snap2 -compare_images ${POOL} ${dp_image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${dp_image}@snap1 +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${dp_image}@snap2 +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${dp_image} remove_image_retry ${CLUSTER2} ${POOL} ${dp_image} testlog "TEST: disable mirroring / delete non-primary image" @@ -436,8 +436,8 @@ if [ "${RBD_MIRROR_MODE}" = "journal" ]; then wait_for_image_present ${CLUSTER1} ${POOL} ${i} 'present' wait_for_snap_present ${CLUSTER1} ${POOL} ${i} 'snap2' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${i} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${i} - compare_images ${POOL} ${i} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${i} + compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${i} done testlog "TEST: remove mirroring pool" @@ -454,9 +454,9 @@ if [ "${RBD_MIRROR_MODE}" = "journal" ]; then create_image ${CLUSTER2} ${POOL} ${rdp_image} 128 --data-pool ${pool} write_image ${CLUSTER2} ${pool} ${image} 100 write_image ${CLUSTER2} ${POOL} ${rdp_image} 100 - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${pool} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${pool} ${pool} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${pool} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${rdp_image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${rdp_image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${rdp_image} for cluster in ${CLUSTER1} ${CLUSTER2}; do CEPH_ARGS='' ceph --cluster ${cluster} osd pool rm ${pool} ${pool} --yes-i-really-really-mean-it @@ -519,12 +519,12 @@ wait_for_image_replay_started ${CLUSTER1} ${POOL}/${NS1} ${image} wait_for_image_replay_started ${CLUSTER1} ${POOL}/${NS2} ${image} write_image ${CLUSTER2} ${POOL}/${NS1} ${image} 100 write_image ${CLUSTER2} ${POOL}/${NS2} ${image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS1} ${image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS2} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS1} ${POOL}/${NS1} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS2} ${POOL}/${NS2} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL}/${NS1} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL}/${NS2} ${image} -compare_images ${POOL}/${NS1} ${image} -compare_images ${POOL}/${NS2} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS1} ${POOL}/${NS1} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS2} ${POOL}/${NS2} ${image} testlog " - disable mirroring / delete image" remove_image_retry ${CLUSTER2} ${POOL}/${NS1} ${image} @@ -533,6 +533,40 @@ wait_for_image_present ${CLUSTER1} ${POOL}/${NS1} ${image} 'deleted' wait_for_image_present ${CLUSTER1} ${POOL}/${NS2} ${image} 'deleted' remove_image_retry ${CLUSTER2} ${POOL}/${NS2} ${image} +testlog "TEST: mirror to a different remote namespace" +testlog " - replay" +NS3=ns3 +NS4=ns4 +rbd --cluster ${CLUSTER1} namespace create ${POOL}/${NS3} +rbd --cluster ${CLUSTER2} namespace create ${POOL}/${NS4} +rbd --cluster ${CLUSTER1} mirror pool enable ${POOL}/${NS3} ${MIRROR_POOL_MODE} --remote-namespace ${NS4} +rbd --cluster ${CLUSTER2} mirror pool enable ${POOL}/${NS4} ${MIRROR_POOL_MODE} --remote-namespace ${NS3} +create_image_and_enable_mirror ${CLUSTER2} ${POOL}/${NS4} ${image} ${RBD_MIRROR_MODE} +wait_for_image_replay_started ${CLUSTER1} ${POOL}/${NS3} ${image} +write_image ${CLUSTER2} ${POOL}/${NS4} ${image} 100 +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS3} ${POOL}/${NS4} ${image} +wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL}/${NS3} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS3} ${POOL}/${NS4} ${image} + +testlog " - disable mirroring and re-enable without remote-namespace" +remove_image_retry ${CLUSTER2} ${POOL}/${NS4} ${image} +wait_for_image_present ${CLUSTER1} ${POOL}/${NS3} ${image} 'deleted' +rbd --cluster ${CLUSTER1} mirror pool disable ${POOL}/${NS3} +rbd --cluster ${CLUSTER2} mirror pool disable ${POOL}/${NS4} +rbd --cluster ${CLUSTER2} namespace create ${POOL}/${NS3} +rbd --cluster ${CLUSTER2} mirror pool enable ${POOL}/${NS3} ${MIRROR_POOL_MODE} +rbd --cluster ${CLUSTER1} mirror pool enable ${POOL}/${NS3} ${MIRROR_POOL_MODE} +create_image_and_enable_mirror ${CLUSTER2} ${POOL}/${NS3} ${image} ${RBD_MIRROR_MODE} +wait_for_image_replay_started ${CLUSTER1} ${POOL}/${NS3} ${image} +write_image ${CLUSTER2} ${POOL}/${NS3} ${image} 100 +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS3} ${POOL}/${NS3} ${image} +wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL}/${NS3} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS3} ${POOL}/${NS3} ${image} +remove_image_retry ${CLUSTER2} ${POOL}/${NS3} ${image} +wait_for_image_present ${CLUSTER1} ${POOL}/${NS3} ${image} 'deleted' +rbd --cluster ${CLUSTER1} mirror pool disable ${POOL}/${NS3} +rbd --cluster ${CLUSTER2} mirror pool disable ${POOL}/${NS3} + testlog " - data pool" dp_image=test_data_pool create_image_and_enable_mirror ${CLUSTER2} ${POOL}/${NS1} ${dp_image} ${RBD_MIRROR_MODE} 128 --data-pool ${PARENT_POOL} @@ -542,9 +576,9 @@ wait_for_image_replay_started ${CLUSTER1} ${POOL}/${NS1} ${dp_image} data_pool=$(get_image_data_pool ${CLUSTER1} ${POOL}/${NS1} ${dp_image}) test "${data_pool}" = "${PARENT_POOL}" write_image ${CLUSTER2} ${POOL}/${NS1} ${dp_image} 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS1} ${dp_image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS1} ${POOL}/${NS1} ${dp_image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL}/${NS1} ${dp_image} -compare_images ${POOL}/${NS1} ${dp_image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL}/${NS1} ${POOL}/${NS1} ${dp_image} remove_image_retry ${CLUSTER2} ${POOL}/${NS1} ${dp_image} testlog "TEST: simple image resync" @@ -553,7 +587,7 @@ wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'deleted' ${image_id} wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'present' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} if [ -z "${RBD_MIRROR_USE_RBD_MIRROR}" ]; then testlog "TEST: image resync while replayer is stopped" @@ -566,7 +600,7 @@ if [ -z "${RBD_MIRROR_USE_RBD_MIRROR}" ]; then wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'present' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} - compare_images ${POOL} ${image} + compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} fi testlog "TEST: request image resync while daemon is offline" @@ -577,7 +611,7 @@ wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'deleted' ${image_id} wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'present' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} -compare_images ${POOL} ${image} +compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} remove_image_retry ${CLUSTER2} ${POOL} ${image} if [ "${RBD_MIRROR_MODE}" = "journal" ]; then @@ -588,7 +622,7 @@ if [ "${RBD_MIRROR_MODE}" = "journal" ]; then testlog " - replay stopped after disconnect" wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} test -n "$(get_mirror_journal_position ${CLUSTER2} ${POOL} ${image})" disconnect_image ${CLUSTER2} ${POOL} ${image} test -z "$(get_mirror_journal_position ${CLUSTER2} ${POOL} ${image})" @@ -600,9 +634,9 @@ if [ "${RBD_MIRROR_MODE}" = "journal" ]; then wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'deleted' ${image_id} wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'present' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} test -n "$(get_mirror_journal_position ${CLUSTER2} ${POOL} ${image})" - compare_images ${POOL} ${image} + compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} testlog " - disconnected after max_concurrent_object_sets reached" if [ -z "${RBD_MIRROR_USE_RBD_MIRROR}" ]; then @@ -628,25 +662,25 @@ if [ "${RBD_MIRROR_MODE}" = "journal" ]; then wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'deleted' ${image_id} wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'present' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} test -n "$(get_mirror_journal_position ${CLUSTER2} ${POOL} ${image})" - compare_images ${POOL} ${image} + compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} testlog " - rbd_mirroring_resync_after_disconnect config option" set_image_meta ${CLUSTER2} ${POOL} ${image} \ conf_rbd_mirroring_resync_after_disconnect true - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} image_id=$(get_image_id ${CLUSTER1} ${POOL} ${image}) disconnect_image ${CLUSTER2} ${POOL} ${image} wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'deleted' ${image_id} wait_for_image_present ${CLUSTER1} ${POOL} ${image} 'present' wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} test -n "$(get_mirror_journal_position ${CLUSTER2} ${POOL} ${image})" - compare_images ${POOL} ${image} + compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} set_image_meta ${CLUSTER2} ${POOL} ${image} \ conf_rbd_mirroring_resync_after_disconnect false - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} disconnect_image ${CLUSTER2} ${POOL} ${image} test -z "$(get_mirror_journal_position ${CLUSTER2} ${POOL} ${image})" wait_for_image_replay_stopped ${CLUSTER1} ${POOL} ${image} diff --git a/qa/workunits/rbd/rbd_mirror_bootstrap.sh b/qa/workunits/rbd/rbd_mirror_bootstrap.sh index 412e84c88a64b..3ddb0aa219b79 100755 --- a/qa/workunits/rbd/rbd_mirror_bootstrap.sh +++ b/qa/workunits/rbd/rbd_mirror_bootstrap.sh @@ -38,7 +38,7 @@ create_image_and_enable_mirror ${CLUSTER1} ${POOL} image1 wait_for_image_replay_started ${CLUSTER2} ${POOL} image1 write_image ${CLUSTER1} ${POOL} image1 100 -wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${POOL} image1 +wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${POOL} ${POOL} image1 wait_for_replaying_status_in_pool_dir ${CLUSTER2} ${POOL} image1 testlog "TEST: verify rx-tx direction" @@ -54,12 +54,12 @@ enable_mirror ${CLUSTER2} ${PARENT_POOL} image2 wait_for_image_replay_started ${CLUSTER2} ${PARENT_POOL} image1 write_image ${CLUSTER1} ${PARENT_POOL} image1 100 -wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${PARENT_POOL} image1 +wait_for_replay_complete ${CLUSTER2} ${CLUSTER1} ${PARENT_POOL} ${PARENT_POOL} image1 wait_for_replaying_status_in_pool_dir ${CLUSTER2} ${PARENT_POOL} image1 wait_for_image_replay_started ${CLUSTER1} ${PARENT_POOL} image2 write_image ${CLUSTER2} ${PARENT_POOL} image2 100 -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${PARENT_POOL} image2 +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${PARENT_POOL} ${PARENT_POOL} image2 wait_for_replaying_status_in_pool_dir ${CLUSTER1} ${PARENT_POOL} image2 testlog "TEST: pool replayer and callout cleanup when peer is updated" diff --git a/qa/workunits/rbd/rbd_mirror_ha.sh b/qa/workunits/rbd/rbd_mirror_ha.sh index 1e43712a63152..e5a086b82ab86 100755 --- a/qa/workunits/rbd/rbd_mirror_ha.sh +++ b/qa/workunits/rbd/rbd_mirror_ha.sh @@ -71,7 +71,7 @@ test_replay() wait_for_image_replay_started ${CLUSTER1}:${LEADER} ${POOL} ${image} write_image ${CLUSTER2} ${POOL} ${image} 100 wait_for_replay_complete ${CLUSTER1}:${LEADER} ${CLUSTER2} ${POOL} \ - ${image} + ${POOL} ${image} wait_for_status_in_pool_dir ${CLUSTER1} ${POOL} ${image} 'up+replaying' \ 'primary_position' \ "${MIRROR_USER_ID_PREFIX}${LEADER} on $(hostname -s)" @@ -79,7 +79,7 @@ test_replay() wait_for_status_in_pool_dir ${CLUSTER2} ${POOL} ${image} \ 'down+unknown' fi - compare_images ${POOL} ${image} + compare_images ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} done } diff --git a/qa/workunits/rbd/rbd_mirror_helpers.sh b/qa/workunits/rbd/rbd_mirror_helpers.sh index 1c3062891768d..1b1436db74d70 100755 --- a/qa/workunits/rbd/rbd_mirror_helpers.sh +++ b/qa/workunits/rbd/rbd_mirror_helpers.sh @@ -743,17 +743,18 @@ wait_for_journal_replay_complete() { local local_cluster=$1 local cluster=$2 - local pool=$3 - local image=$4 + local local_pool=$3 + local remote_pool=$4 + local image=$5 local s master_pos mirror_pos last_mirror_pos local master_tag master_entry mirror_tag mirror_entry while true; do for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32; do sleep ${s} - flush "${local_cluster}" "${pool}" "${image}" - master_pos=$(get_master_journal_position "${cluster}" "${pool}" "${image}") - mirror_pos=$(get_mirror_journal_position "${cluster}" "${pool}" "${image}") + flush "${local_cluster}" "${local_pool}" "${image}" + master_pos=$(get_master_journal_position "${cluster}" "${remote_pool}" "${image}") + mirror_pos=$(get_mirror_journal_position "${cluster}" "${remote_pool}" "${image}") test -n "${master_pos}" -a "${master_pos}" = "${mirror_pos}" && return 0 test "${mirror_pos}" != "${last_mirror_pos}" && break done @@ -796,21 +797,22 @@ wait_for_snapshot_sync_complete() { local local_cluster=$1 local cluster=$2 - local pool=$3 - local image=$4 + local local_pool=$3 + local remote_pool=$4 + local image=$5 - local status_log=${TEMPDIR}/$(mkfname ${cluster}-${pool}-${image}.status) - local local_status_log=${TEMPDIR}/$(mkfname ${local_cluster}-${pool}-${image}.status) + local status_log=${TEMPDIR}/$(mkfname ${cluster}-${remote_pool}-${image}.status) + local local_status_log=${TEMPDIR}/$(mkfname ${local_cluster}-${local_pool}-${image}.status) - mirror_image_snapshot "${cluster}" "${pool}" "${image}" - get_newest_mirror_snapshot "${cluster}" "${pool}" "${image}" "${status_log}" + mirror_image_snapshot "${cluster}" "${remote_pool}" "${image}" + get_newest_mirror_snapshot "${cluster}" "${remote_pool}" "${image}" "${status_log}" local snapshot_id=$(xmlstarlet sel -t -v "//snapshot/id" < ${status_log}) while true; do for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32; do sleep ${s} - get_newest_mirror_snapshot "${local_cluster}" "${pool}" "${image}" "${local_status_log}" + get_newest_mirror_snapshot "${local_cluster}" "${local_pool}" "${image}" "${local_status_log}" local primary_snapshot_id=$(xmlstarlet sel -t -v "//snapshot/namespace/primary_snap_id" < ${local_status_log}) test "${snapshot_id}" = "${primary_snapshot_id}" && return 0 @@ -825,13 +827,14 @@ wait_for_replay_complete() { local local_cluster=$1 local cluster=$2 - local pool=$3 - local image=$4 + local local_pool=$3 + local remote_pool=$4 + local image=$5 if [ "${RBD_MIRROR_MODE}" = "journal" ]; then - wait_for_journal_replay_complete ${local_cluster} ${cluster} ${pool} ${image} + wait_for_journal_replay_complete ${local_cluster} ${cluster} ${local_pool} ${remote_pool} ${image} elif [ "${RBD_MIRROR_MODE}" = "snapshot" ]; then - wait_for_snapshot_sync_complete ${local_cluster} ${cluster} ${pool} ${image} + wait_for_snapshot_sync_complete ${local_cluster} ${cluster} ${local_pool} ${remote_pool} ${image} else return 1 fi @@ -1298,16 +1301,19 @@ show_diff() compare_images() { - local pool=$1 - local image=$2 local ret=0 + local local_cluster=$1 + local cluster=$2 + local local_pool=$3 + local remote_pool=$4 + local image=$5 - local rmt_export=${TEMPDIR}/$(mkfname ${CLUSTER2}-${pool}-${image}.export) - local loc_export=${TEMPDIR}/$(mkfname ${CLUSTER1}-${pool}-${image}.export) + local rmt_export=${TEMPDIR}/$(mkfname ${cluster}-${remote_pool}-${image}.export) + local loc_export=${TEMPDIR}/$(mkfname ${local_cluster}-${local_pool}-${image}.export) rm -f ${rmt_export} ${loc_export} - rbd --cluster ${CLUSTER2} export ${pool}/${image} ${rmt_export} - rbd --cluster ${CLUSTER1} export ${pool}/${image} ${loc_export} + rbd --cluster ${cluster} export ${remote_pool}/${image} ${rmt_export} + rbd --cluster ${local_cluster} export ${local_pool}/${image} ${loc_export} if ! cmp ${rmt_export} ${loc_export} then show_diff ${rmt_export} ${loc_export} diff --git a/qa/workunits/rbd/rbd_mirror_stress.sh b/qa/workunits/rbd/rbd_mirror_stress.sh index baf0c9f1a8f8f..b0a85e8a48a55 100755 --- a/qa/workunits/rbd/rbd_mirror_stress.sh +++ b/qa/workunits/rbd/rbd_mirror_stress.sh @@ -111,7 +111,7 @@ do snap_name="snap${i}" create_snap ${CLUSTER2} ${POOL} ${image} ${snap_name} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_snap_present ${CLUSTER1} ${POOL} ${image} ${snap_name} if [ -n "${clean_snap_name}" ]; then @@ -124,7 +124,7 @@ do done wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} -wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} +wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_snap_present ${CLUSTER1} ${POOL} ${image} ${clean_snap_name} for i in `seq 1 10` @@ -173,7 +173,7 @@ do image="image_${i}" create_snap ${CLUSTER2} ${POOL} ${image} ${snap_name} wait_for_image_replay_started ${CLUSTER1} ${POOL} ${image} - wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${image} + wait_for_replay_complete ${CLUSTER1} ${CLUSTER2} ${POOL} ${POOL} ${image} wait_for_snap_present ${CLUSTER1} ${POOL} ${image} ${snap_name} compare_image_snaps ${POOL} ${image} ${snap_name} done diff --git a/src/cls/rbd/cls_rbd.cc b/src/cls/rbd/cls_rbd.cc index b84630e14860b..d0d6bd1184042 100644 --- a/src/cls/rbd/cls_rbd.cc +++ b/src/cls/rbd/cls_rbd.cc @@ -4624,6 +4624,7 @@ static const std::string STATUS_GLOBAL_KEY_PREFIX("status_global_"); static const std::string REMOTE_STATUS_GLOBAL_KEY_PREFIX("remote_status_global_"); static const std::string INSTANCE_KEY_PREFIX("instance_"); static const std::string MIRROR_IMAGE_MAP_KEY_PREFIX("image_map_"); +static const std::string REMOTE_NAMESPACE("remote_namespace"); std::string peer_key(const std::string &uuid) { return PEER_KEY_PREFIX + uuid; @@ -5920,6 +5921,56 @@ int mirror_mode_set(cls_method_context_t hctx, bufferlist *in, if (r < 0) { return r; } + + r = remove_key(hctx, mirror::REMOTE_NAMESPACE); + if (r < 0) { + return r; + } + } + return 0; +} + +int mirror_remote_namespace_get(cls_method_context_t hctx, bufferlist *in, + bufferlist *out) { + std::string mirror_ns_decode; + int r = read_key(hctx, mirror::REMOTE_NAMESPACE, &mirror_ns_decode); + if (r < 0) { + CLS_ERR("error getting mirror remote namespace: %s", + cpp_strerror(r).c_str()); + return r; + } + + encode(mirror_ns_decode, *out); + return 0; +} + +int mirror_remote_namespace_set(cls_method_context_t hctx, bufferlist *in, + bufferlist *out) { + std::string mirror_namespace; + try { + auto bl_it = in->cbegin(); + decode(mirror_namespace, bl_it); + } catch (const ceph::buffer::error &err) { + return -EINVAL; + } + + uint32_t mirror_mode; + int r = read_key(hctx, mirror::MODE, &mirror_mode); + if (r < 0 && r != -ENOENT) { + return r; + } else if (r == 0 && mirror_mode != cls::rbd::MIRROR_MODE_DISABLED) { + CLS_ERR("cannot set mirror remote namespace while mirroring enabled"); + return -EINVAL; + } + + bufferlist bl; + encode(mirror_namespace, bl); + + r = cls_cxx_map_set_val(hctx, mirror::REMOTE_NAMESPACE, &bl); + if (r < 0) { + CLS_ERR("error setting mirror remote namespace: %s", + cpp_strerror(r).c_str()); + return r; } return 0; } @@ -8278,6 +8329,8 @@ CLS_INIT(rbd) cls_method_handle_t h_mirror_uuid_set; cls_method_handle_t h_mirror_mode_get; cls_method_handle_t h_mirror_mode_set; + cls_method_handle_t h_mirror_remote_namespace_get; + cls_method_handle_t h_mirror_remote_namespace_set; cls_method_handle_t h_mirror_peer_ping; cls_method_handle_t h_mirror_peer_list; cls_method_handle_t h_mirror_peer_add; @@ -8575,6 +8628,13 @@ CLS_INIT(rbd) cls_register_cxx_method(h_class, "mirror_mode_set", CLS_METHOD_RD | CLS_METHOD_WR, mirror_mode_set, &h_mirror_mode_set); + cls_register_cxx_method(h_class, "mirror_remote_namespace_get", + CLS_METHOD_RD, mirror_remote_namespace_get, + &h_mirror_remote_namespace_get); + cls_register_cxx_method(h_class, "mirror_remote_namespace_set", + CLS_METHOD_RD | CLS_METHOD_WR, + mirror_remote_namespace_set, + &h_mirror_remote_namespace_set); cls_register_cxx_method(h_class, "mirror_peer_ping", CLS_METHOD_RD | CLS_METHOD_WR, mirror_peer_ping, &h_mirror_peer_ping); diff --git a/src/cls/rbd/cls_rbd_client.cc b/src/cls/rbd/cls_rbd_client.cc index 458bfd985c390..559ac221f89ad 100644 --- a/src/cls/rbd/cls_rbd_client.cc +++ b/src/cls/rbd/cls_rbd_client.cc @@ -1882,6 +1882,40 @@ int mirror_mode_set(librados::IoCtx *ioctx, return 0; } +int mirror_remote_namespace_get(librados::IoCtx *ioctx, + std::string *mirror_namespace) { + bufferlist in_bl; + bufferlist out_bl; + + int r = ioctx->exec(RBD_MIRRORING, "rbd", "mirror_remote_namespace_get", + in_bl, out_bl); + if (r < 0) { + return r; + } + + auto it = out_bl.cbegin(); + try { + decode(*mirror_namespace, it); + } catch (const ceph::buffer::error &err) { + return -EBADMSG; + } + return 0; +} + +int mirror_remote_namespace_set(librados::IoCtx *ioctx, + const std::string &mirror_namespace) { + bufferlist in_bl; + encode(mirror_namespace, in_bl); + + bufferlist out_bl; + int r = ioctx->exec(RBD_MIRRORING, "rbd", "mirror_remote_namespace_set", + in_bl, out_bl); + if (r < 0) { + return r; + } + return 0; +} + void mirror_peer_list_start(librados::ObjectReadOperation *op) { bufferlist bl; op->exec("rbd", "mirror_peer_list", bl); diff --git a/src/cls/rbd/cls_rbd_client.h b/src/cls/rbd/cls_rbd_client.h index b1553bd1f17d9..37992203affb5 100644 --- a/src/cls/rbd/cls_rbd_client.h +++ b/src/cls/rbd/cls_rbd_client.h @@ -389,6 +389,11 @@ int mirror_mode_get(librados::IoCtx *ioctx, int mirror_mode_set(librados::IoCtx *ioctx, cls::rbd::MirrorMode mirror_mode); +int mirror_remote_namespace_get(librados::IoCtx *ioctx, + std::string *mirror_namespace); +int mirror_remote_namespace_set(librados::IoCtx *ioctx, + const std::string &mirror_namespace); + int mirror_peer_ping(librados::IoCtx *ioctx, const std::string& site_name, const std::string& fsid); diff --git a/src/common/CMakeLists.txt b/src/common/CMakeLists.txt index c3fd51d26468f..41efeebe7be30 100644 --- a/src/common/CMakeLists.txt +++ b/src/common/CMakeLists.txt @@ -12,6 +12,7 @@ if(WIN32) add_library(dlfcn_win32 STATIC win32/dlfcn.cc win32/errno.cc) endif() +add_subdirectory(io_exerciser) add_subdirectory(options) set(common_srcs diff --git a/src/common/io_exerciser/CMakeLists.txt b/src/common/io_exerciser/CMakeLists.txt new file mode 100644 index 0000000000000..07091df86e100 --- /dev/null +++ b/src/common/io_exerciser/CMakeLists.txt @@ -0,0 +1,13 @@ +add_library(object_io_exerciser STATIC + DataGenerator.cc + IoOp.cc + IoSequence.cc + Model.cc + ObjectModel.cc + RadosIo.cc +) + +target_link_libraries(object_io_exerciser + librados + global +) \ No newline at end of file diff --git a/src/common/io_exerciser/DataGenerator.cc b/src/common/io_exerciser/DataGenerator.cc new file mode 100644 index 0000000000000..9aa77eeb6e98b --- /dev/null +++ b/src/common/io_exerciser/DataGenerator.cc @@ -0,0 +1,753 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab +#include "DataGenerator.h" + +#include "ObjectModel.h" + +#include "common/debug.h" +#include "common/dout.h" + +#include "fmt/format.h" +#include "fmt/ranges.h" + +#include +#include +#include + +#define dout_subsys ceph_subsys_rados +#define dout_context g_ceph_context + +using DataGenerator = ceph::io_exerciser::data_generation::DataGenerator; +using SeededRandomGenerator = ceph::io_exerciser::data_generation + ::SeededRandomGenerator; +using HeaderedSeededRandomGenerator = ceph::io_exerciser::data_generation + ::HeaderedSeededRandomGenerator; + +std::unique_ptr DataGenerator::create_generator( + GenerationType generationType, const ObjectModel& model) +{ + switch(generationType) + { + case GenerationType::SeededRandom: + return std::make_unique(model); + case GenerationType::HeaderedSeededRandom: + return std::make_unique(model); + default: + throw std::invalid_argument("Not yet implemented"); + } + + return nullptr; +} + +bufferlist DataGenerator::generate_wrong_data(uint64_t offset, uint64_t length) +{ + bufferlist retlist; + uint64_t block_size = m_model.get_block_size(); + char buffer[block_size]; + for (uint64_t block_offset = offset; + block_offset < offset + length; + block_offset++) + { + std::memset(buffer, 0, block_size); + retlist.append(ceph::bufferptr(buffer, block_size)); + } + return retlist; +} + +bool DataGenerator::validate(bufferlist& bufferlist, uint64_t offset, uint64_t length) +{ + return bufferlist.contents_equal(generate_data(offset, length)); +} + +ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) +{ + uint64_t block_size = m_model.get_block_size(); + char buffer[block_size]; + + std::mt19937_64 random_generator(m_model.get_seed(block_offset)); + uint64_t rand1 = random_generator(); + uint64_t rand2 = random_generator(); + + constexpr size_t generation_length = sizeof(uint64_t); + + for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--) + { + std::memcpy(buffer + i, &rand1, generation_length); + std::memcpy(buffer + i + generation_length, &rand2, generation_length); + } + + size_t remainingBytes = block_size % (generation_length * 2); + if (remainingBytes > generation_length) + { + size_t remainingBytes2 = remainingBytes - generation_length; + std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); + std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2); + } + else if (remainingBytes > 0) + { + std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); + } + + return ceph::bufferptr(buffer, block_size); +} + +ceph::bufferptr SeededRandomGenerator::generate_wrong_block(uint64_t block_offset) +{ + uint64_t block_size = m_model.get_block_size(); + char buffer[block_size]; + + std::mt19937_64 random_generator(m_model.get_seed(block_offset)); + uint64_t rand1 = random_generator() - 1; + uint64_t rand2 = random_generator() + 1; + + constexpr size_t generation_length = sizeof(uint64_t); + + for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--) + { + std::memcpy(buffer + i, &rand1, generation_length); + std::memcpy(buffer + i + generation_length, &rand2, generation_length); + } + + size_t remainingBytes = block_size % (generation_length * 2); + if (remainingBytes > generation_length) + { + size_t remainingBytes2 = remainingBytes - generation_length; + std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); + std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2); + } + else if (remainingBytes > 0) + { + std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); + } + + return ceph::bufferptr(buffer, block_size); +} + +bufferlist SeededRandomGenerator::generate_data(uint64_t offset, uint64_t length) +{ + bufferlist retlist; + + for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) + { + retlist.append(generate_block(block_offset)); + } + + return retlist; +} + +bufferlist SeededRandomGenerator::generate_wrong_data(uint64_t offset, uint64_t length) +{ + bufferlist retlist; + + for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) + { + retlist.append(generate_wrong_block(block_offset)); + } + + return retlist; +} + +HeaderedSeededRandomGenerator + ::HeaderedSeededRandomGenerator(const ObjectModel& model, + std::optional unique_run_id) : + SeededRandomGenerator(model), + unique_run_id(unique_run_id.value_or(generate_unique_run_id())) +{ + +} + +uint64_t HeaderedSeededRandomGenerator::generate_unique_run_id() +{ + std::mt19937_64 random_generator = + std::mt19937_64(duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count()); + + return random_generator(); +} + +ceph::bufferptr HeaderedSeededRandomGenerator::generate_block(uint64_t block_offset) +{ + SeedBytes seed = m_model.get_seed(block_offset); + TimeBytes current_time = duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + + ceph::bufferptr bufferptr = SeededRandomGenerator::generate_block(block_offset); + + std::memcpy(bufferptr.c_str() + uniqueIdStart(), &unique_run_id, uniqueIdLength()); + std::memcpy(bufferptr.c_str() + seedStart(), &seed, seedLength()); + std::memcpy(bufferptr.c_str() + timeStart(), ¤t_time, timeLength()); + + return bufferptr; +} + +ceph::bufferptr HeaderedSeededRandomGenerator::generate_wrong_block(uint64_t block_offset) +{ + return HeaderedSeededRandomGenerator::generate_block(block_offset % 8); +} + +const HeaderedSeededRandomGenerator::UniqueIdBytes + HeaderedSeededRandomGenerator::readUniqueRunId(uint64_t block_offset, + const bufferlist& bufferlist) +{ + UniqueIdBytes read_unique_run_id = 0; + std::memcpy(&read_unique_run_id, + &bufferlist[(block_offset * m_model.get_block_size()) + uniqueIdStart()], + uniqueIdLength()); + return read_unique_run_id; +} + +const HeaderedSeededRandomGenerator::SeedBytes + HeaderedSeededRandomGenerator::readSeed(uint64_t block_offset, + const bufferlist& bufferlist) +{ + SeedBytes read_seed = 0; + std::memcpy(&read_seed, + &bufferlist[(block_offset * m_model.get_block_size()) + seedStart()], + seedLength()); + return read_seed; +} + +const HeaderedSeededRandomGenerator::TimeBytes + HeaderedSeededRandomGenerator::readDateTime(uint64_t block_offset, + const bufferlist& bufferlist) +{ + TimeBytes read_time = 0; + std::memcpy(&read_time, + &bufferlist[(block_offset * m_model.get_block_size()) + timeStart()], + timeLength()); + return read_time; +} + +bool HeaderedSeededRandomGenerator::validate(bufferlist& bufferlist, + uint64_t offset, uint64_t length) +{ + std::vector invalid_block_offsets; + + for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) + { + bool valid_block + = validate_block(block_offset, + (bufferlist.c_str() + ((block_offset - offset) * + m_model.get_block_size()))); + if (!valid_block) + { + invalid_block_offsets.push_back(block_offset); + } + } + + if (!invalid_block_offsets.empty()) + { + printDebugInformationForOffsets(offset, invalid_block_offsets, bufferlist); + } + + return invalid_block_offsets.empty(); +} + +bool HeaderedSeededRandomGenerator::validate_block(uint64_t block_offset, + const char* buffer_start) +{ + // We validate the block matches what we generate byte for byte + // however we ignore the time section of the header + ceph::bufferptr bufferptr = generate_block(block_offset); + bool valid = strncmp(bufferptr.c_str(), buffer_start, timeStart()) == 0; + valid = valid ? strncmp(bufferptr.c_str() + timeEnd(), + buffer_start + timeEnd(), + m_model.get_block_size() - timeEnd()) == 0 : valid; + return valid; +} + +const HeaderedSeededRandomGenerator::ErrorType + HeaderedSeededRandomGenerator::getErrorTypeForBlock(uint64_t read_offset, + uint64_t block_offset, + const bufferlist& bufferlist) +{ + try + { + UniqueIdBytes read_unique_run_id = readUniqueRunId(block_offset - read_offset, + bufferlist); + if (unique_run_id != read_unique_run_id) + { + return ErrorType::RUN_ID_MISMATCH; + } + + SeedBytes read_seed = readSeed(block_offset - read_offset, bufferlist); + if (m_model.get_seed(block_offset) != read_seed) + { + return ErrorType::SEED_MISMATCH; + } + + if (std::strncmp(&bufferlist[((block_offset - read_offset) * + m_model.get_block_size()) + bodyStart()], + generate_block(block_offset).c_str() + bodyStart(), + m_model.get_block_size() - bodyStart()) != 0) + { + return ErrorType::DATA_MISMATCH; + } + } + catch(const std::exception& e) + { + return ErrorType::DATA_NOT_FOUND; + } + + return ErrorType::UNKNOWN; +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationForBlock(uint64_t read_offset, uint64_t block_offset, + const bufferlist& bufferlist) +{ + ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, bufferlist); + + TimeBytes read_time = 0; + std::time_t ttp; + + char read_bytes[m_model.get_block_size()]; + char generated_bytes[m_model.get_block_size()]; + + if (blockError == ErrorType::DATA_MISMATCH || blockError == ErrorType::UNKNOWN) + { + read_time = readDateTime(block_offset - read_offset, bufferlist); + std::chrono::system_clock::time_point time_point{std::chrono::milliseconds{read_time}}; + ttp = std::chrono::system_clock::to_time_t(time_point); + + std::memcpy(&read_bytes, + &bufferlist[((block_offset - read_offset) * m_model.get_block_size())], + m_model.get_block_size() - bodyStart()); + std::memcpy(&generated_bytes, + generate_block(block_offset).c_str(), + m_model.get_block_size() - bodyStart()); + } + + std::string error_string; + switch(blockError) + { + case ErrorType::RUN_ID_MISMATCH: + { + UniqueIdBytes read_unique_run_id = readUniqueRunId((block_offset - read_offset), + bufferlist); + error_string = fmt::format("Header (Run ID) mismatch detected at block {} " + "(byte offset {}) Header expected run id {} but found id {}. " + "Block data corrupt or not written from this instance of this application.", + block_offset, + block_offset * m_model.get_block_size(), + unique_run_id, + read_unique_run_id); + } + break; + + case ErrorType::SEED_MISMATCH: + { + SeedBytes read_seed = readSeed((block_offset - read_offset), bufferlist); + + if (m_model.get_seed_offsets(read_seed).size() == 0) + { + error_string = fmt::format("Data (Seed) mismatch detected at block {}" + " (byte offset {}). Header expected seed {} but found seed {}. " + "Read data was not from any other recognised block in the object.", + block_offset, + block_offset * m_model.get_block_size(), + m_model.get_seed(block_offset), + read_seed); + } + else + { + std::vector seed_offsets = m_model.get_seed_offsets(read_seed); + error_string = fmt::format("Data (Seed) mismatch detected at block {}" + " (byte offset {}). Header expected seed {} but found seed {}." + " Read data was from a different block(s): {}", + block_offset, + block_offset * m_model.get_block_size(), + m_model.get_seed(block_offset), + read_seed, + fmt::join(seed_offsets.begin(), seed_offsets.end(), "")); + } + } + break; + + case ErrorType::DATA_MISMATCH: + { + error_string = fmt::format("Data (Body) mismatch detected at block {}" + " (byte offset {}). Header data matches, data body does not." + " Data written at {}\nExpected data: \n{:02x}\nRead data:{:02x}", + block_offset, + block_offset * m_model.get_block_size(), + std::ctime(&ttp), + fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), ""), + fmt::join(read_bytes, read_bytes + m_model.get_block_size(), "")); + } + break; + + case ErrorType::DATA_NOT_FOUND: + { + uint64_t bufferlist_length = bufferlist.to_str().size(); + error_string = fmt::format("Data (Body) could not be read at block {}" + " (byte offset {}) offset in bufferlist returned from read: {}" + " ({} bytes). Returned bufferlist length: {}.", + block_offset, + block_offset * m_model.get_block_size(), + (block_offset - read_offset), + (block_offset - read_offset) * m_model.get_block_size(), + bufferlist_length); + } + break; + + case ErrorType::UNKNOWN: + [[ fallthrough ]]; + + default: + { + error_string = fmt::format("Data mismatch detected at block {}" + " (byte offset {}).\nExpected data:\n{:02x}\nRead data:\n{:02x}", + block_offset, + block_offset * m_model.get_block_size(), + fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), ""), + fmt::join(read_bytes, read_bytes + m_model.get_block_size(), "")); + } + break; + } + dout(0) << error_string << dendl; +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationForRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + ErrorType rangeError, + const bufferlist& bufferlist) +{ + switch(rangeError) + { + case ErrorType::RUN_ID_MISMATCH: + printDebugInformationForRunIdMismatchRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; + case ErrorType::SEED_MISMATCH: + printDebugInformationForSeedMismatchRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; + case ErrorType::DATA_MISMATCH: + printDebugInformationDataBodyMismatchRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; + case ErrorType::DATA_NOT_FOUND: + printDebugInformationDataNotFoundRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; + case ErrorType::UNKNOWN: + [[ fallthrough ]]; + default: + printDebugInformationCorruptRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; + } +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationForRunIdMismatchRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist) +{ + uint64_t range_start = start_block_offset; + uint64_t range_length = 0; + UniqueIdBytes initial_read_unique_run_id = readUniqueRunId(start_block_offset - read_offset, + bufferlist); + for (uint64_t i = start_block_offset; + i < start_block_offset + range_length_in_blocks; i++) + { + ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) + == ErrorType::RUN_ID_MISMATCH); + + UniqueIdBytes read_unique_run_id = readUniqueRunId(i - read_offset, bufferlist); + if (initial_read_unique_run_id != read_unique_run_id || + i == (start_block_offset + range_length_in_blocks - 1)) + { + if (range_length == 1) + { + printDebugInformationForBlock(read_offset, i, bufferlist); + } + else if (range_length > 1) + { + dout(0) << fmt::format("Data (Run ID) Mismatch detected from block {} ({} bytes)" + " and spanning a range of {} blocks ({} bytes). " + "Expected run id {} for range but found id {}" + " for all blocks in range. " + "Block data corrupt or not written from this instance of this application.", + range_start, + range_start * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size(), + unique_run_id, + initial_read_unique_run_id) << dendl; + } + + range_start = i; + range_length = 1; + initial_read_unique_run_id = read_unique_run_id; + } + else + { + range_length++; + } + } + + if (range_length == 1) + { + printDebugInformationForBlock(read_offset, + start_block_offset + range_length_in_blocks - 1, + bufferlist); + } + else if (range_length > 1) + { + dout(0) << fmt::format("Data (Run ID) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Expected run id {} for range but found id for all blocks in range. " + "Block data corrupt or not written from this instance of this application.", + range_start, + range_start * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size(), + unique_run_id, + initial_read_unique_run_id) + << dendl; + } +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationForSeedMismatchRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist) +{ + uint64_t range_start = start_block_offset; + uint64_t range_length = 0; + + // Assert here if needed, as we can't support values + // that can't be converted to a signed integer. + ceph_assert(m_model.get_block_size() < (std::numeric_limits::max() / 2)); + std::optional range_offset = 0; + + for (uint64_t i = start_block_offset; + i < start_block_offset + range_length_in_blocks; i++) + { + ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) + == ErrorType::SEED_MISMATCH); + SeedBytes read_seed = readSeed(i - read_offset, bufferlist); + + std::vector seed_found_offsets = m_model.get_seed_offsets(read_seed); + + if ((seed_found_offsets.size() == 1 && + (static_cast(seed_found_offsets.front() - i) == range_offset)) || + range_length == 0) + { + if (range_length == 0) + { + range_start = i; + if (seed_found_offsets.size() > 0) + { + range_offset = seed_found_offsets.front() - i; + } + else + { + range_offset = std::nullopt; + } + } + range_length++; + } + else + { + if (range_length == 1) + { + printDebugInformationForBlock(read_offset, i - 1, bufferlist); + } + else if (range_length > 1 && range_offset.has_value()) + { + dout(0) << fmt::format("Data (Seed) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Returned data located starting from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes).", + range_start, + range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + static_cast(*range_offset) + range_start, + (static_cast(*range_offset) + range_start) + * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size()) + << dendl; + } + else + { + dout(0) << fmt::format("Data (Seed) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Data seed mismatch spanning a range of {} blocks ({} bytes).", + range_start, + range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size()) + << dendl; + } + range_length = 1; + range_start = i; + if (seed_found_offsets.size() > 0) + { + range_offset = seed_found_offsets.front() - i; + } + else + { + range_offset = std::nullopt; + } + } + } + + if (range_length == 1) + { + printDebugInformationForBlock(read_offset, + start_block_offset + range_length_in_blocks - 1, + bufferlist); + } + else if (range_length > 1 && range_offset.has_value()) + { + dout(0) << fmt::format("Data (Seed) Mismatch detected from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes). " + "Returned data located starting from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes).", + range_start, + range_start * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size(), + *range_offset + range_start, + (*range_offset + range_start) * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size()) + << dendl; + } + else + { + dout(0) << fmt::format("Data (Seed) Mismatch detected from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes). " + "and spanning a range of {} blocks ({} bytes).", + range_start, + range_start * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size(), + range_length, + range_length * m_model.get_block_size()) + << dendl; + } +} + +void HeaderedSeededRandomGenerator +::printDebugInformationDataBodyMismatchRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist) +{ + dout(0) << fmt::format("Data Mismatch detected in blocks from {} to {}. " + "Headers look as expected for range, " + "but generated data body does not match. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) + << dendl; + + for (uint64_t i = start_block_offset; + i < start_block_offset + range_length_in_blocks; i++) + { + printDebugInformationForBlock(read_offset, i, bufferlist); + } +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationCorruptRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist) +{ + dout(0) << fmt::format("Data Mismatch detected in blocks from {} to {}. " + "Headers look as expected for range, " + "but generated data body does not match. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) + << dendl; + + for (uint64_t i = start_block_offset; + i < start_block_offset + range_length_in_blocks; i++) + { + printDebugInformationForBlock(read_offset, i, bufferlist); + } +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationDataNotFoundRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist) +{ + dout(0) << fmt::format("Data not found for blocks from {} to {}. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) + << dendl; + + for (uint64_t i = start_block_offset; i < start_block_offset + range_length_in_blocks; i++) + { + printDebugInformationForBlock(read_offset, i, bufferlist); + } +} + +void HeaderedSeededRandomGenerator + ::printDebugInformationForOffsets(uint64_t read_offset, + std::vector offsets, + const bufferlist& bufferlist) +{ + uint64_t range_start = 0; + uint64_t range_length = 0; + ErrorType rangeError = ErrorType::UNKNOWN; + + for (const uint64_t& block_offset : offsets) + { + ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, + bufferlist); + + if (range_start == 0 && range_length == 0) + { + range_start = block_offset; + range_length = 1; + rangeError = blockError; + } + else if (blockError == rangeError && + range_start + range_length == block_offset) +{ + range_length++; + } + else + { + if (range_length == 1) + { + printDebugInformationForBlock(read_offset, range_start, bufferlist); + } + else if (range_length > 1) + { + printDebugInformationForRange(read_offset, range_start, range_length, + rangeError, bufferlist); + } + + range_start = block_offset; + range_length = 1; + rangeError = blockError; + } + } + + if (range_length == 1) + { + printDebugInformationForBlock(read_offset, range_start, bufferlist); + } + else if (range_length > 1) + { + printDebugInformationForRange(read_offset, range_start, range_length, + rangeError, bufferlist); + } +} \ No newline at end of file diff --git a/src/common/io_exerciser/DataGenerator.h b/src/common/io_exerciser/DataGenerator.h new file mode 100644 index 0000000000000..1e5784a54ccd7 --- /dev/null +++ b/src/common/io_exerciser/DataGenerator.h @@ -0,0 +1,171 @@ +#pragma once + +#include +#include + +#include "include/buffer.h" +#include "ObjectModel.h" + +/* Overview + * + * class DataGenerator + * Generates data buffers for write I/Os using state queried + * from ObjectModel. Validates data buffers for read I/Os + * against the state in the ObjectModel. If a data miscompare + * is detected provide debug information about the state of the + * object, the buffer that was read and the expected buffer. + * + * + * class SeededRandomGenerator + * Inherits from DataGenerator. Generates entirely random patterns + * based on the seed retrieved by the model. + * + * + * class HeaderedSeededRandomGenerator + * Inherits from SeededDataGenerator. Generates entirely random patterns + * based on the seed retrieved by the model, however also appends a + * header to the start of each block. This generator also provides + * a range of verbose debug options to help disagnose a miscompare + * whenever it detects unexpected data. + */ + +namespace ceph { + namespace io_exerciser { + namespace data_generation { + enum class GenerationType { + SeededRandom, + HeaderedSeededRandom + // CompressedGenerator + // MixedGenerator + }; + + class DataGenerator { + public: + virtual ~DataGenerator() = default; + static std::unique_ptr + create_generator(GenerationType generatorType, + const ObjectModel& model); + virtual bufferlist generate_data(uint64_t length, uint64_t offset)=0; + virtual bool validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length); + + // Used for testing debug outputs from data generation + virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length); + + protected: + const ObjectModel& m_model; + + DataGenerator(const ObjectModel& model) : m_model(model) {} + }; + + class SeededRandomGenerator : public DataGenerator + { + public: + SeededRandomGenerator(const ObjectModel& model) + : DataGenerator(model) {} + + virtual bufferptr generate_block(uint64_t offset); + virtual bufferlist generate_data(uint64_t length, uint64_t offset); + virtual bufferptr generate_wrong_block(uint64_t offset); + virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length) override; + }; + + class HeaderedSeededRandomGenerator : public SeededRandomGenerator + { + public: + HeaderedSeededRandomGenerator(const ObjectModel& model, + std::optional unique_run_id = std::nullopt); + + bufferptr generate_block(uint64_t offset) override; + bufferptr generate_wrong_block(uint64_t offset) override; + bool validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length) override; + + private: + using UniqueIdBytes = uint64_t; + using SeedBytes = int; + using TimeBytes = uint64_t; + + enum class ErrorType { + RUN_ID_MISMATCH, + SEED_MISMATCH, + DATA_MISMATCH, + DATA_NOT_FOUND, + UNKNOWN + }; + + constexpr uint8_t headerStart() const + { return 0; }; + constexpr uint8_t uniqueIdStart() const + { return headerStart(); }; + constexpr uint8_t uniqueIdLength() const + { return sizeof(UniqueIdBytes); }; + constexpr uint8_t seedStart() const + { return uniqueIdStart() + uniqueIdLength(); }; + constexpr uint8_t seedLength() const + { return sizeof(SeedBytes); }; + constexpr uint8_t timeStart() const + { return seedStart() + seedLength(); }; + constexpr uint8_t timeLength() const + { return sizeof(TimeBytes); }; + constexpr uint8_t timeEnd() const + { return timeStart() + timeLength(); }; + constexpr uint8_t headerLength() const + { return uniqueIdLength() + seedLength() + timeLength(); }; + constexpr uint8_t bodyStart() const + { return headerStart() + headerLength(); }; + + const UniqueIdBytes readUniqueRunId(uint64_t block_offset, + const bufferlist& bufferlist); + const SeedBytes readSeed(uint64_t block_offset, + const bufferlist& bufferlist); + const TimeBytes readDateTime(uint64_t block_offset, + const bufferlist& bufferlist); + + const UniqueIdBytes unique_run_id; + + uint64_t generate_unique_run_id(); + + bool validate_block(uint64_t block_offset, const char* buffer_start); + + const ErrorType getErrorTypeForBlock(uint64_t read_offset, + uint64_t block_offset, + const bufferlist& bufferlist); + + void printDebugInformationForBlock(uint64_t read_offset, + uint64_t block_offset, + const bufferlist& bufferlist); + void printDebugInformationForRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + ErrorType rangeError, + const bufferlist& bufferlist); + + void printDebugInformationForRunIdMismatchRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + void printDebugInformationForSeedMismatchRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + void printDebugInformationDataBodyMismatchRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + void printDebugInformationDataNotFoundRange(uint64_t ßread_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + void printDebugInformationCorruptRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + + void printDebugInformationForOffsets(uint64_t read_offset, + std::vector offsets, + const bufferlist& bufferlist); + }; + } + } +} diff --git a/src/common/io_exerciser/IoOp.cc b/src/common/io_exerciser/IoOp.cc new file mode 100644 index 0000000000000..cd855ba6fff8a --- /dev/null +++ b/src/common/io_exerciser/IoOp.cc @@ -0,0 +1,188 @@ +#include "IoOp.h" + +using IoOp = ceph::io_exerciser::IoOp; + +IoOp::IoOp( OpType op, + uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) : + op(op), + offset1(offset1), length1(length1), + offset2(offset2), length2(length2), + offset3(offset3), length3(length3) +{ + +} + +std::string IoOp::value_to_string(uint64_t v) const +{ + if (v < 1024 || (v % 1024) != 0) { + return std::to_string(v); + }else if (v < 1024*1024 || (v % (1024 * 1024)) != 0 ) { + return std::to_string(v / 1024) + "K"; + }else{ + return std::to_string(v / 1024 / 1024) + "M"; + } +} + +std::unique_ptr IoOp + ::generate_done() { + + return std::make_unique(OpType::Done); +} + +std::unique_ptr IoOp + ::generate_barrier() { + + return std::make_unique(OpType::BARRIER); +} + +std::unique_ptr IoOp + ::generate_create(uint64_t size) { + + return std::make_unique(OpType::CREATE,0,size); +} + +std::unique_ptr IoOp + ::generate_remove() { + + return std::make_unique(OpType::REMOVE); +} + +std::unique_ptr IoOp + ::generate_read(uint64_t offset, uint64_t length) { + + return std::make_unique(OpType::READ, offset, length); +} + +std::unique_ptr IoOp + ::generate_read2(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2) { + + if (offset1 < offset2) { + ceph_assert( offset1 + length1 <= offset2 ); + } else { + ceph_assert( offset2 + length2 <= offset1 ); + } + + return std::make_unique(OpType::READ2, + offset1, length1, + offset2, length2); +} + +std::unique_ptr IoOp + ::generate_read3(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + + if (offset1 < offset2) { + ceph_assert( offset1 + length1 <= offset2 ); + } else { + ceph_assert( offset2 + length2 <= offset1 ); + } + if (offset1 < offset3) { + ceph_assert( offset1 + length1 <= offset3 ); + } else { + ceph_assert( offset3 + length3 <= offset1 ); + } + if (offset2 < offset3) { + ceph_assert( offset2 + length2 <= offset3 ); + } else { + ceph_assert( offset3 + length3 <= offset2 ); + } + return std::make_unique(OpType::READ3, + offset1, length1, + offset2, length2, + offset3, length3); +} + +std::unique_ptr IoOp::generate_write(uint64_t offset, uint64_t length) { + return std::make_unique(OpType::WRITE, offset, length); +} + +std::unique_ptr IoOp::generate_write2(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2) { + if (offset1 < offset2) { + ceph_assert( offset1 + length1 <= offset2 ); + } else { + ceph_assert( offset2 + length2 <= offset1 ); + } + return std::make_unique(OpType::WRITE2, + offset1, length1, + offset2, length2); +} + +std::unique_ptr IoOp::generate_write3(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + if (offset1 < offset2) { + ceph_assert( offset1 + length1 <= offset2 ); + } else { + ceph_assert( offset2 + length2 <= offset1 ); + } + if (offset1 < offset3) { + ceph_assert( offset1 + length1 <= offset3 ); + } else { + ceph_assert( offset3 + length3 <= offset1 ); + } + if (offset2 < offset3) { + ceph_assert( offset2 + length2 <= offset3 ); + } else { + ceph_assert( offset3 + length3 <= offset2 ); + } + return std::make_unique(OpType::WRITE3, + offset1, length1, + offset2, length2, + offset3, length3); +} + +bool IoOp::done() { + return (op == OpType::Done); +} + +std::string IoOp::to_string(uint64_t block_size) const +{ + switch (op) { + case OpType::Done: + return "Done"; + case OpType::BARRIER: + return "Barrier"; + case OpType::CREATE: + return "Create (size=" + value_to_string(length1 * block_size) + ")"; + case OpType::REMOVE: + return "Remove"; + case OpType::READ: + return "Read (offset=" + value_to_string(offset1 * block_size) + + ",length=" + value_to_string(length1 * block_size) + ")"; + case OpType::READ2: + return "Read2 (offset1=" + value_to_string(offset1 * block_size) + + ",length1=" + value_to_string(length1 * block_size) + + ",offset2=" + value_to_string(offset2 * block_size) + + ",length2=" + value_to_string(length2 * block_size) + ")"; + case OpType::READ3: + return "Read3 (offset1=" + value_to_string(offset1 * block_size) + + ",length1=" + value_to_string(length1 * block_size) + + ",offset2=" + value_to_string(offset2 * block_size) + + ",length2=" + value_to_string(length2 * block_size) + + ",offset3=" + value_to_string(offset3 * block_size) + + ",length3=" + value_to_string(length3 * block_size) + ")"; + case OpType::WRITE: + return "Write (offset=" + value_to_string(offset1 * block_size) + + ",length=" + value_to_string(length1 * block_size) + ")"; + case OpType::WRITE2: + return "Write2 (offset1=" + value_to_string(offset1 * block_size) + + ",length1=" + value_to_string(length1 * block_size) + + ",offset2=" + value_to_string(offset2 * block_size) + + ",length2=" + value_to_string(length2 * block_size) + ")"; + case OpType::WRITE3: + return "Write3 (offset1=" + value_to_string(offset1 * block_size) + + ",length1=" + value_to_string(length1 * block_size) + + ",offset2=" + value_to_string(offset2 * block_size) + + ",length2=" + value_to_string(length2 * block_size) + + ",offset3=" + value_to_string(offset3 * block_size) + + ",length3=" + value_to_string(length3 * block_size) + ")"; + default: + break; + } + return "Unknown"; +} \ No newline at end of file diff --git a/src/common/io_exerciser/IoOp.h b/src/common/io_exerciser/IoOp.h new file mode 100644 index 0000000000000..60c02a93d4e22 --- /dev/null +++ b/src/common/io_exerciser/IoOp.h @@ -0,0 +1,94 @@ +#pragma once + +#include +#include +#include "include/ceph_assert.h" + +/* Overview + * + * enum OpType + * Enumeration of different types of I/O operation + * + * class IoOp + * Stores details for an I/O operation. Generated by IoSequences + * and applied by IoExerciser's + */ + +namespace ceph { + namespace io_exerciser { + + enum class OpType { + Done, // End of I/O sequence + BARRIER, // Barrier - all prior I/Os must complete + CREATE, // Create object and pattern with data + REMOVE, // Remove object + READ, // Read + READ2, // 2 Reads in one op + READ3, // 3 Reads in one op + WRITE, // Write + WRITE2, // 2 Writes in one op + WRITE3 // 3 Writes in one op + }; + + class IoOp { + protected: + std::string value_to_string(uint64_t v) const; + + public: + OpType op; + uint64_t offset1; + uint64_t length1; + uint64_t offset2; + uint64_t length2; + uint64_t offset3; + uint64_t length3; + + IoOp( OpType op, + uint64_t offset1 = 0, uint64_t length1 = 0, + uint64_t offset2 = 0, uint64_t length2 = 0, + uint64_t offset3 = 0, uint64_t length3 = 0 ); + + static std::unique_ptr generate_done(); + + static std::unique_ptr generate_barrier(); + + static std::unique_ptr generate_create(uint64_t size); + + static std::unique_ptr generate_remove(); + + static std::unique_ptr generate_read(uint64_t offset, + uint64_t length); + + static std::unique_ptr generate_read2(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); + + static std::unique_ptr generate_read3(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2, + uint64_t offset3, + uint64_t length3); + + static std::unique_ptr generate_write(uint64_t offset, + uint64_t length); + + static std::unique_ptr generate_write2(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); + + static std::unique_ptr generate_write3(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2, + uint64_t offset3, + uint64_t length3); + + bool done(); + + std::string to_string(uint64_t block_size) const; + }; + } +} \ No newline at end of file diff --git a/src/common/io_exerciser/IoSequence.cc b/src/common/io_exerciser/IoSequence.cc new file mode 100644 index 0000000000000..4a7ca0593d1d2 --- /dev/null +++ b/src/common/io_exerciser/IoSequence.cc @@ -0,0 +1,500 @@ +#include "IoSequence.h" + +using Sequence = ceph::io_exerciser::Sequence; +using IoSequence = ceph::io_exerciser::IoSequence; + +std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& seq) +{ + switch (seq) + { + case Sequence::SEQUENCE_SEQ0: + os << "SEQUENCE_SEQ0"; + break; + case Sequence::SEQUENCE_SEQ1: + os << "SEQUENCE_SEQ1"; + break; + case Sequence::SEQUENCE_SEQ2: + os << "SEQUENCE_SEQ2"; + break; + case Sequence::SEQUENCE_SEQ3: + os << "SEQUENCE_SEQ3"; + break; + case Sequence::SEQUENCE_SEQ4: + os << "SEQUENCE_SEQ4"; + break; + case Sequence::SEQUENCE_SEQ5: + os << "SEQUENCE_SEQ5"; + break; + case Sequence::SEQUENCE_SEQ6: + os << "SEQUENCE_SEQ6"; + break; + case Sequence::SEQUENCE_SEQ7: + os << "SEQUENCE_SEQ7"; + break; + case Sequence::SEQUENCE_SEQ8: + os << "SEQUENCE_SEQ8"; + break; + case Sequence::SEQUENCE_SEQ9: + os << "SEQUENCE_SEQ9"; + break; + case Sequence::SEQUENCE_END: + os << "SEQUENCE_END"; + break; + } + return os; +} + +IoSequence::IoSequence(std::pair obj_size_range, + int seed) : + min_obj_size(obj_size_range.first), max_obj_size(obj_size_range.second), + create(true), barrier(false), done(false), remove(false), + obj_size(min_obj_size), step(-1), seed(seed) +{ + rng.seed(seed); +} + +std::unique_ptr IoSequence::generate_sequence(Sequence s, + std::pair obj_size_range, + int seed) +{ + switch (s) { + case Sequence::SEQUENCE_SEQ0: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ1: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ2: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ3: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ4: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ5: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ6: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ7: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ8: + return std::make_unique(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ9: + return std::make_unique(obj_size_range, seed); + default: + break; + } + return nullptr; +} + +int IoSequence::get_step() const +{ + return step; +} + +int IoSequence::get_seed() const +{ + return seed; +} + +void IoSequence::set_min_object_size(uint64_t size) +{ + min_obj_size = size; + if (obj_size < size) { + obj_size = size; + if (obj_size > max_obj_size) { + done = true; + } + } +} + +void IoSequence::set_max_object_size(uint64_t size) +{ + max_obj_size = size; + if (obj_size > size) { + done = true; + } +} + +void IoSequence::select_random_object_size() +{ + if (max_obj_size != min_obj_size) { + obj_size = min_obj_size + rng(max_obj_size - min_obj_size); + } +} + +std::unique_ptr IoSequence::increment_object_size() +{ + obj_size++; + if (obj_size > max_obj_size) { + done = true; + } + create = true; + barrier = true; + remove = true; + return IoOp::generate_barrier(); +} + +std::unique_ptr IoSequence::next() +{ + step++; + if (remove) { + remove = false; + return IoOp::generate_remove(); + } + if (barrier) { + barrier = false; + return IoOp::generate_barrier(); + } + if (done) { + return IoOp::generate_done(); + } + if (create) { + create = false; + barrier = true; + return IoOp::generate_create(obj_size); + } + return _next(); +} + + + +ceph::io_exerciser::Seq0::Seq0(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset(0) +{ + select_random_object_size(); + length = 1 + rng(obj_size - 1); +} + +std::string ceph::io_exerciser::Seq0::get_name() const +{ + return "Sequential reads of length " + std::to_string(length) + + " with queue depth 1 (seqseed " + std::to_string(get_seed()) + ")"; +} + +std::unique_ptr ceph::io_exerciser::Seq0::_next() +{ + std::unique_ptr r; + if (offset >= obj_size) { + done = true; + barrier = true; + remove = true; + return IoOp::generate_barrier(); + } + if (offset + length > obj_size) { + r = IoOp::generate_read(offset, obj_size - offset); + } else { + r = IoOp::generate_read(offset, length); + } + offset += length; + return r; +} + + + +ceph::io_exerciser::Seq1::Seq1(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed) +{ + select_random_object_size(); + count = 3 * obj_size; +} + +std::string ceph::io_exerciser::Seq1::get_name() const +{ + return "Random offset, random length read/write I/O with queue depth 1 (seqseed " + + std::to_string(get_seed()) + ")"; +} + +std::unique_ptr ceph::io_exerciser::Seq1::_next() +{ + barrier = true; + if (count-- == 0) { + done = true; + remove = true; + return IoOp::generate_barrier(); + } + + uint64_t offset = rng(obj_size - 1); + uint64_t length = 1 + rng(obj_size - 1 - offset); + return (rng(2) != 0) ? IoOp::generate_write(offset, length) : + IoOp::generate_read(offset, length); +} + + + +ceph::io_exerciser::Seq2::Seq2(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset(0), length(0) {} + +std::string ceph::io_exerciser::Seq2::get_name() const +{ + return "Permutations of offset and length read I/O"; +} + +std::unique_ptr ceph::io_exerciser::Seq2::_next() +{ + length++; + if (length > obj_size - offset) { + length = 1; + offset++; + if (offset >= obj_size) { + offset = 0; + length = 0; + return increment_object_size(); + } + } + return IoOp::generate_read(offset, length); +} + + + +ceph::io_exerciser::Seq3::Seq3(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset1(0), offset2(0) +{ + set_min_object_size(2); +} + +std::string ceph::io_exerciser::Seq3::get_name() const +{ + return "Permutations of offset 2-region 1-block read I/O"; +} + +std::unique_ptr ceph::io_exerciser::Seq3::_next() +{ + offset2++; + if (offset2 >= obj_size - offset1) { + offset2 = 1; + offset1++; + if (offset1 + 1 >= obj_size) { + offset1 = 0; + offset2 = 0; + return increment_object_size(); + } + } + return IoOp::generate_read2(offset1, 1, offset1 + offset2, 1); +} + + + +ceph::io_exerciser::Seq4::Seq4(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset1(0), offset2(1) +{ + set_min_object_size(3); +} + +std::string ceph::io_exerciser::Seq4::get_name() const +{ + return "Permutations of offset 3-region 1-block read I/O"; +} + +std::unique_ptr ceph::io_exerciser::Seq4::_next() +{ + offset2++; + if (offset2 >= obj_size - offset1) { + offset2 = 2; + offset1++; + if (offset1 + 2 >= obj_size) { + offset1 = 0; + offset2 = 1; + return increment_object_size(); + } + } + return IoOp::generate_read3(offset1, 1, + offset1 + offset2, 1, + (offset1 * 2 + offset2)/2, 1); +} + + + +ceph::io_exerciser::Seq5::Seq5(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset(0), length(1), + doneread(false), donebarrier(false) {} + +std::string ceph::io_exerciser::Seq5::get_name() const +{ + return "Permutation of length sequential writes"; +} + +std::unique_ptr ceph::io_exerciser::Seq5::_next() +{ + if (offset >= obj_size) { + if (!doneread) { + if (!donebarrier) { + donebarrier = true; + return IoOp::generate_barrier(); + } + doneread = true; + barrier = true; + return IoOp::generate_read(0, obj_size); + } + doneread = false; + donebarrier = false; + offset = 0; + length++; + if (length > obj_size) { + length = 1; + return increment_object_size(); + } + } + uint64_t io_len = (offset + length > obj_size) ? (obj_size - offset) : length; + std::unique_ptr r = IoOp::generate_write(offset, io_len); + offset += io_len; + return r; +} + + + +ceph::io_exerciser::Seq6::Seq6(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset(0), length(1), + doneread(false), donebarrier(false) {} + +std::string ceph::io_exerciser::Seq6::get_name() const +{ + return "Permutation of length sequential writes, different alignment"; +} + +std::unique_ptr ceph::io_exerciser::Seq6::_next() +{ + if (offset >= obj_size) { + if (!doneread) { + if (!donebarrier) { + donebarrier = true; + return IoOp::generate_barrier(); + } + doneread = true; + barrier = true; + return IoOp::generate_read(0, obj_size); + } + doneread = false; + donebarrier = false; + offset = 0; + length++; + if (length > obj_size) { + length = 1; + return increment_object_size(); + } + } + uint64_t io_len = (offset == 0) ? (obj_size % length) : length; + if (io_len == 0) { + io_len = length; + } + std::unique_ptr r = IoOp::generate_write(offset, io_len); + offset += io_len; + return r; +} + + + +ceph::io_exerciser::Seq7::Seq7(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed) +{ + set_min_object_size(2); + offset = obj_size; +} + +std::string ceph::io_exerciser::Seq7::get_name() const +{ + return "Permutations of offset 2-region 1-block writes"; +} + +std::unique_ptr ceph::io_exerciser::Seq7::_next() +{ + if (!doneread) { + if (!donebarrier) { + donebarrier = true; + return IoOp::generate_barrier(); + } + doneread = true; + barrier = true; + return IoOp::generate_read(0, obj_size); + } + if (offset == 0) { + doneread = false; + donebarrier = false; + offset = obj_size+1; + return increment_object_size(); + } + offset--; + if (offset == obj_size/2) { + return _next(); + } + doneread = false; + donebarrier = false; + return IoOp::generate_write2(offset, 1, obj_size/2, 1); +} + + + +ceph::io_exerciser::Seq8::Seq8(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset1(0), offset2(1) +{ + set_min_object_size(3); +} + +std::string ceph::io_exerciser::Seq8::get_name() const +{ + return "Permutations of offset 3-region 1-block write I/O"; +} + +std::unique_ptr ceph::io_exerciser::Seq8::_next() +{ + if (!doneread) { + if (!donebarrier) { + donebarrier = true; + return IoOp::generate_barrier(); + } + doneread = true; + barrier = true; + return IoOp::generate_read(0, obj_size); + } + offset2++; + if (offset2 >= obj_size - offset1) { + offset2 = 2; + offset1++; + if (offset1 + 2 >= obj_size) { + offset1 = 0; + offset2 = 1; + return increment_object_size(); + } + } + doneread = false; + donebarrier = false; + return IoOp::generate_write3(offset1, 1, + offset1 + offset2, 1, + (offset1 * 2 + offset2)/2, 1); +} + + + +ceph::io_exerciser::Seq9::Seq9(std::pair obj_size_range, int seed) : + IoSequence(obj_size_range, seed), offset(0), length(0) +{ + +} + +std::string ceph::io_exerciser::Seq9::get_name() const +{ + return "Permutations of offset and length write I/O"; +} + +std::unique_ptr ceph::io_exerciser::Seq9::_next() +{ + if (!doneread) { + if (!donebarrier) { + donebarrier = true; + return IoOp::generate_barrier(); + } + doneread = true; + barrier = true; + return IoOp::generate_read(0, obj_size); + } + length++; + if (length > obj_size - offset) { + length = 1; + offset++; + if (offset >= obj_size) { + offset = 0; + length = 0; + return increment_object_size(); + } + } + doneread = false; + donebarrier = false; + return IoOp::generate_write(offset, length); +} \ No newline at end of file diff --git a/src/common/io_exerciser/IoSequence.h b/src/common/io_exerciser/IoSequence.h new file mode 100644 index 0000000000000..114ff76303f46 --- /dev/null +++ b/src/common/io_exerciser/IoSequence.h @@ -0,0 +1,223 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab +#pragma once + +#include "IoOp.h" + +#include "include/random.h" + +/* Overview + * + * enum Sequence + * Enumeration of the different sequences + * + * class IoSequence + * Virtual class. IoSequences generate a stream of IoOPs. + * Sequences typically exhastively test permutations of + * offset and length to allow validation of code such as + * Erasure Coding. An IoSequence does not determine + * whether I/Os are issued sequentially or in parallel, + * it must generate barrier I/Os where operations must + * be serialized. + * + * class Seq* + * Implementations of IoSequence. Each class generates + * a different sequence of I/O. + * + * generate_sequence + * Create an IoSequence + */ + +namespace ceph { + namespace io_exerciser { + + enum class Sequence { + SEQUENCE_SEQ0, + SEQUENCE_SEQ1, + SEQUENCE_SEQ2, + SEQUENCE_SEQ3, + SEQUENCE_SEQ4, + SEQUENCE_SEQ5, + SEQUENCE_SEQ6, + SEQUENCE_SEQ7, + SEQUENCE_SEQ8, + SEQUENCE_SEQ9, + // + SEQUENCE_END, + SEQUENCE_BEGIN = SEQUENCE_SEQ0 + }; + + inline Sequence operator++( Sequence& s ) + { + return s = (Sequence)(((int)(s) + 1)); + } + + std::ostream& operator<<(std::ostream& os, const Sequence& seq); + + /* I/O Sequences */ + + class IoSequence { + public: + virtual ~IoSequence() = default; + + virtual std::string get_name() const = 0; + int get_step() const; + int get_seed() const; + + std::unique_ptr next(); + + static std::unique_ptr + generate_sequence(Sequence s, std::pair obj_size_range, int seed ); + + protected: + uint64_t min_obj_size; + uint64_t max_obj_size; + bool create; + bool barrier; + bool done; + bool remove; + uint64_t obj_size; + int step; + int seed; + ceph::util::random_number_generator rng = + ceph::util::random_number_generator(); + + IoSequence(std::pair obj_size_range, int seed); + + virtual std::unique_ptr _next() = 0; + + void set_min_object_size(uint64_t size); + void set_max_object_size(uint64_t size); + void select_random_object_size(); + std::unique_ptr increment_object_size(); + + }; + + class Seq0: public IoSequence { + public: + Seq0(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + + private: + uint64_t offset; + uint64_t length; + }; + + class Seq1: public IoSequence { + public: + Seq1(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next(); + + private: + int count; + }; + + class Seq2: public IoSequence { + public: + Seq2(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + + private: + uint64_t offset; + uint64_t length; + }; + + class Seq3: public IoSequence { + public: + Seq3(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + private: + uint64_t offset1; + uint64_t offset2; + }; + + class Seq4: public IoSequence { + public: + Seq4(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + + private: + uint64_t offset1; + uint64_t offset2; + }; + + class Seq5: public IoSequence { + public: + Seq5(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + + private: + uint64_t offset; + uint64_t length; + bool doneread; + bool donebarrier; + }; + + class Seq6: public IoSequence { + public: + Seq6(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + + private: + uint64_t offset; + uint64_t length; + bool doneread; + bool donebarrier; + }; + + class Seq7: public IoSequence { + public: + Seq7(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + + private: + uint64_t offset; + bool doneread = true; + bool donebarrier = false; + }; + + class Seq8: public IoSequence { + public: + Seq8(std::pair obj_size_range, int seed); + + std::string get_name() const override; + std::unique_ptr _next() override; + private: + uint64_t offset1; + uint64_t offset2; + bool doneread = true; + bool donebarrier = false; + }; + + class Seq9: public IoSequence { + private: + uint64_t offset; + uint64_t length; + bool doneread = true; + bool donebarrier = false; + + public: + Seq9(std::pair obj_size_range, int seed); + + std::string get_name() const override; + + std::unique_ptr _next() override; + }; + } +} \ No newline at end of file diff --git a/src/common/io_exerciser/Model.cc b/src/common/io_exerciser/Model.cc new file mode 100644 index 0000000000000..50812ecbb155d --- /dev/null +++ b/src/common/io_exerciser/Model.cc @@ -0,0 +1,28 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab +#include "Model.h" + +using Model = ceph::io_exerciser::Model; + +Model::Model(const std::string& oid, uint64_t block_size) : +num_io(0), +oid(oid), +block_size(block_size) +{ + +} + +const uint64_t Model::get_block_size() const +{ + return block_size; +} + +const std::string Model::get_oid() const +{ + return oid; +} + +int Model::get_num_io() const +{ + return num_io; +} \ No newline at end of file diff --git a/src/common/io_exerciser/Model.h b/src/common/io_exerciser/Model.h new file mode 100644 index 0000000000000..58d107409a654 --- /dev/null +++ b/src/common/io_exerciser/Model.h @@ -0,0 +1,49 @@ +#pragma once + +#include "IoOp.h" + +#include + +#include "librados/librados_asio.h" + +#include "include/interval_set.h" +#include "global/global_init.h" +#include "global/global_context.h" +#include "common/Thread.h" + +/* Overview + * + * class Model + * Virtual class. Models apply IoOps generated by an + * IoSequence, they can choose how many I/Os to execute in + * parallel and scale up the size of I/Os by the blocksize + * + */ + +namespace ceph { + namespace io_exerciser { + + class Model + { + protected: + int num_io{0}; + std::string oid; + uint64_t block_size; + + public: + Model(const std::string& oid, uint64_t block_size); + virtual ~Model() = default; + + virtual bool readyForIoOp(IoOp& op) = 0; + virtual void applyIoOp(IoOp& op) = 0; + + const std::string get_oid() const; + const uint64_t get_block_size() const; + int get_num_io() const; + }; + + /* Simple RADOS I/O generator */ + + + } +} \ No newline at end of file diff --git a/src/common/io_exerciser/ObjectModel.cc b/src/common/io_exerciser/ObjectModel.cc new file mode 100644 index 0000000000000..589f6434282b3 --- /dev/null +++ b/src/common/io_exerciser/ObjectModel.cc @@ -0,0 +1,174 @@ +#include "ObjectModel.h" + +#include +#include +#include + +using ObjectModel = ceph::io_exerciser::ObjectModel; + +ObjectModel::ObjectModel(const std::string& oid, uint64_t block_size, int seed) : + Model(oid, block_size), created(false) +{ + rng.seed(seed); +} + +int ObjectModel::get_seed(uint64_t offset) const +{ + ceph_assert(offset < contents.size()); + return contents[offset]; +} + +std::vector ObjectModel::get_seed_offsets(int seed) const +{ + std::vector offsets; + for (size_t i = 0; i < contents.size(); i++) + { + if (contents[i] == seed) + { + offsets.push_back(i); + } + } + + return offsets; +} + +std::string ObjectModel::to_string(int mask) const +{ + if (!created) { + return "Object does not exist"; + } + std::string result = "{"; + for (uint64_t i = 0; i < contents.size(); i++) { + if (i != 0) { + result += ","; + } + result += std::to_string(contents[i] & mask); + } + result += "}"; + return result; +} + +bool ObjectModel::readyForIoOp(IoOp& op) +{ + return true; +} + +void ObjectModel::applyIoOp(IoOp& op) +{ + auto generate_random = [&rng = rng]() { + return rng(); + }; + + switch (op.op) { + case OpType::BARRIER: + reads.clear(); + writes.clear(); + break; + + case OpType::CREATE: + ceph_assert(!created); + ceph_assert(reads.empty()); + ceph_assert(writes.empty()); + created = true; + contents.resize(op.length1); + std::generate(std::execution::seq, contents.begin(), contents.end(), + generate_random); + break; + + case OpType::REMOVE: + ceph_assert(created); + ceph_assert(reads.empty()); + ceph_assert(writes.empty()); + created = false; + contents.resize(0); + break; + + case OpType::READ3: + ceph_assert(created); + ceph_assert(op.offset3 + op.length3 <= contents.size()); + // Not allowed: read overlapping with parallel write + ceph_assert(!writes.intersects(op.offset3, op.length3)); + reads.union_insert(op.offset3, op.length3); + [[fallthrough]]; + + case OpType::READ2: + ceph_assert(created); + ceph_assert(op.offset2 + op.length2 <= contents.size()); + // Not allowed: read overlapping with parallel write + ceph_assert(!writes.intersects(op.offset2, op.length2)); + reads.union_insert(op.offset2, op.length2); + [[fallthrough]]; + + case OpType::READ: + ceph_assert(created); + ceph_assert(op.offset1 + op.length1 <= contents.size()); + // Not allowed: read overlapping with parallel write + ceph_assert(!writes.intersects(op.offset1, op.length1)); + reads.union_insert(op.offset1, op.length1); + num_io++; + break; + + case OpType::WRITE3: + ceph_assert(created); + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(op.offset3, op.length3)); + ceph_assert(!writes.intersects(op.offset3, op.length3)); + writes.union_insert(op.offset3, op.length3); + ceph_assert(op.offset3 + op.length3 <= contents.size()); + std::generate(std::execution::seq, + std::next(contents.begin(), op.offset3), + std::next(contents.begin(), op.offset3 + op.length3), + generate_random); + [[fallthrough]]; + + case OpType::WRITE2: + ceph_assert(created); + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(op.offset2, op.length2)); + ceph_assert(!writes.intersects(op.offset2, op.length2)); + writes.union_insert(op.offset2, op.length2); + ceph_assert(op.offset2 + op.length2 <= contents.size()); + std::generate(std::execution::seq, + std::next(contents.begin(), op.offset2), + std::next(contents.begin(), op.offset2 + op.length2), + generate_random); + [[fallthrough]]; + + case OpType::WRITE: + ceph_assert(created); + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(op.offset1, op.length1)); + ceph_assert(!writes.intersects(op.offset1, op.length1)); + writes.union_insert(op.offset1, op.length1); + ceph_assert(op.offset1 + op.length1 <= contents.size()); + std::generate(std::execution::seq, + std::next(contents.begin(), op.offset1), + std::next(contents.begin(), op.offset1 + op.length1), + generate_random); + num_io++; + break; + default: + break; + } +} + +void ObjectModel::encode(ceph::buffer::list& bl) const { + ENCODE_START(1, 1, bl); + encode(created, bl); + if (created) { + encode(contents, bl); + } + ENCODE_FINISH(bl); +} + +void ObjectModel::decode(ceph::buffer::list::const_iterator& bl) { + DECODE_START(1, bl); + DECODE_OLDEST(1); + decode(created, bl); + if (created) { + decode(contents, bl); + } else { + contents.resize(0); + } + DECODE_FINISH(bl); +} diff --git a/src/common/io_exerciser/ObjectModel.h b/src/common/io_exerciser/ObjectModel.h new file mode 100644 index 0000000000000..93c70f4142978 --- /dev/null +++ b/src/common/io_exerciser/ObjectModel.h @@ -0,0 +1,53 @@ +#pragma once + +#include "Model.h" + +/* Overview + * + * class ObjectModel + * An IoExerciser. Tracks the data stored in an object, applies + * IoOp's to update the model. Polices that I/Os that are + * permitted to run in parallel do not break rules. Provides + * interface to query state of object. State can be encoded + * and decoded + * + */ + +namespace ceph { + namespace io_exerciser { + /* Model of an object to track its data contents */ + + class ObjectModel : public Model { + private: + bool created; + std::vector contents; + ceph::util::random_number_generator rng = + ceph::util::random_number_generator(); + + // Track read and write I/Os that can be submitted in + // parallel to detect violations: + // + // * Read may not overlap with a parallel write + // * Write may not overlap with a parallel read or write + // * Create / remove may not be in parallel with read or write + // + // Fix broken test cases by adding barrier ops to restrict + // I/O exercisers from issuing conflicting ops in parallel + interval_set reads; + interval_set writes; + public: + ObjectModel(const std::string& oid, uint64_t block_size, int seed); + + int get_seed(uint64_t offset) const; + std::vector get_seed_offsets(int seed) const; + + std::string to_string(int mask = -1) const; + + bool readyForIoOp(IoOp& op); + void applyIoOp(IoOp& op); + + void encode(ceph::buffer::list& bl) const; + void decode(ceph::buffer::list::const_iterator& bl); + }; + } +} \ No newline at end of file diff --git a/src/common/io_exerciser/RadosIo.cc b/src/common/io_exerciser/RadosIo.cc new file mode 100644 index 0000000000000..3f907ccf47416 --- /dev/null +++ b/src/common/io_exerciser/RadosIo.cc @@ -0,0 +1,279 @@ +#include "RadosIo.h" + +#include "DataGenerator.h" + +using RadosIo = ceph::io_exerciser::RadosIo; + +RadosIo::RadosIo(librados::Rados& rados, + boost::asio::io_context& asio, + const std::string& pool, + const std::string& oid, + uint64_t block_size, + int seed, + int threads, + ceph::mutex& lock, + ceph::condition_variable& cond) : + Model(oid, block_size), + rados(rados), + asio(asio), + om(std::make_unique(oid, block_size, seed)), + db(data_generation::DataGenerator::create_generator( + data_generation::GenerationType::HeaderedSeededRandom, *om)), + pool(pool), + threads(threads), + lock(lock), + cond(cond), + outstanding_io(0) +{ + int rc; + rc = rados.ioctx_create(pool.c_str(), io); + ceph_assert(rc == 0); + allow_ec_overwrites(true); +} + +RadosIo::~RadosIo() +{ +} + +void RadosIo::start_io() +{ + std::lock_guard l(lock); + outstanding_io++; +} + +void RadosIo::finish_io() +{ + std::lock_guard l(lock); + ceph_assert(outstanding_io > 0); + outstanding_io--; + cond.notify_all(); +} + +void RadosIo::wait_for_io(int count) +{ + std::unique_lock l(lock); + while (outstanding_io > count) { + cond.wait(l); + } +} + +void RadosIo::allow_ec_overwrites(bool allow) +{ + int rc; + bufferlist inbl, outbl; + std::string cmdstr = + "{\"prefix\": \"osd pool set\", \"pool\": \"" + pool + "\", \ + \"var\": \"allow_ec_overwrites\", \"val\": \"" + + (allow ? "true" : "false") + "\"}"; + rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr); + ceph_assert(rc == 0); +} + +RadosIo::AsyncOpInfo::AsyncOpInfo(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3 ) : + offset1(offset1), length1(length1), + offset2(offset2), length2(length2), + offset3(offset3), length3(length3) +{ + +} + +bool RadosIo::readyForIoOp(IoOp &op) +{ + ceph_assert(lock.is_locked_by_me()); //Must be called with lock held + if (!om->readyForIoOp(op)) { + return false; + } + switch (op.op) { + case OpType::Done: + case OpType::BARRIER: + return outstanding_io == 0; + default: + return outstanding_io < threads; + } +} + +void RadosIo::applyIoOp(IoOp &op) +{ + std::shared_ptr op_info; + + om->applyIoOp(op); + + // If there are thread concurrent I/Os in flight then wait for + // at least one I/O to complete + wait_for_io(threads-1); + + switch (op.op) { + case OpType::Done: + [[ fallthrough ]]; + case OpType::BARRIER: + // Wait for all outstanding I/O to complete + wait_for_io(0); + break; + + case OpType::CREATE: + { + start_io(); + op_info = std::make_shared(0, op.length1); + op_info->bl1 = db->generate_data(0, op.length1); + op_info->wop.write_full(op_info->bl1); + auto create_cb = [this] (boost::system::error_code ec) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->wop, 0, nullptr, create_cb); + } + break; + + case OpType::REMOVE: + { + start_io(); + op_info = std::make_shared(); + op_info->wop.remove(); + auto remove_cb = [this] (boost::system::error_code ec) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->wop, 0, nullptr, remove_cb); + } + break; + + case OpType::READ: + { + start_io(); + op_info = std::make_shared(op.offset1, op.length1); + op_info->rop.read(op.offset1 * block_size, + op.length1 * block_size, + &op_info->bl1, nullptr); + auto read_cb = [this, op_info] (boost::system::error_code ec, bufferlist bl) { + ceph_assert(ec == boost::system::errc::success); + db->validate(op_info->bl1, op_info->offset1, op_info->length1); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->rop, 0, nullptr, read_cb); + num_io++; + } + break; + + case OpType::READ2: + { + start_io(); + op_info = std::make_shared(op.offset1, + op.length1, + op.offset2, + op.length2); + + op_info->rop.read(op.offset1 * block_size, + op.length1 * block_size, + &op_info->bl1, nullptr); + op_info->rop.read(op.offset2 * block_size, + op.length2 * block_size, + &op_info->bl2, nullptr); + auto read2_cb = [this, op_info] (boost::system::error_code ec, + bufferlist bl) { + ceph_assert(ec == boost::system::errc::success); + db->validate(op_info->bl1, op_info->offset1, op_info->length1); + db->validate(op_info->bl2, op_info->offset2, op_info->length2); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->rop, 0, nullptr, read2_cb); + num_io++; + } + break; + + case OpType::READ3: + { + start_io(); + op_info = std::make_shared(op.offset1, op.length1, + op.offset2, op.length2, + op.offset3, op.length3); + op_info->rop.read(op.offset1 * block_size, + op.length1 * block_size, + &op_info->bl1, nullptr); + op_info->rop.read(op.offset2 * block_size, + op.length2 * block_size, + &op_info->bl2, nullptr); + op_info->rop.read(op.offset3 * block_size, + op.length3 * block_size, + &op_info->bl3, nullptr); + auto read3_cb = [this, op_info] (boost::system::error_code ec, + bufferlist bl) { + ceph_assert(ec == boost::system::errc::success); + db->validate(op_info->bl1, op_info->offset1, op_info->length1); + db->validate(op_info->bl2, op_info->offset2, op_info->length2); + db->validate(op_info->bl3, op_info->offset3, op_info->length3); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->rop, 0, nullptr, read3_cb); + num_io++; + } + break; + + case OpType::WRITE: + { + start_io(); + op_info = std::make_shared(op.offset1, op.length1); + op_info->bl1 = db->generate_data(op.offset1, op.length1); + + op_info->wop.write(op.offset1 * block_size, op_info->bl1); + auto write_cb = [this] (boost::system::error_code ec) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->wop, 0, nullptr, write_cb); + num_io++; + } + break; + + case OpType::WRITE2: + { + start_io(); + op_info = std::make_shared(op.offset1, op.length1, + op.offset2, op.length2); + op_info->bl1 = db->generate_data(op.offset1, op.length1); + op_info->bl2 = db->generate_data(op.offset2, op.length2); + op_info->wop.write(op.offset1 * block_size, op_info->bl1); + op_info->wop.write(op.offset2 * block_size, op_info->bl2); + auto write2_cb = [this] (boost::system::error_code ec) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->wop, 0, nullptr, write2_cb); + num_io++; + } + break; + + case OpType::WRITE3: + { + start_io(); + op_info = std::make_shared(op.offset1, op.length1, + op.offset2, op.length2, + op.offset3, op.length3); + op_info->bl1 = db->generate_data(op.offset1, op.length1); + op_info->bl2 = db->generate_data(op.offset2, op.length2); + op_info->bl3 = db->generate_data(op.offset3, op.length3); + op_info->wop.write(op.offset1 * block_size, op_info->bl1); + op_info->wop.write(op.offset2 * block_size, op_info->bl2); + op_info->wop.write(op.offset3 * block_size, op_info->bl3); + auto write3_cb = [this] (boost::system::error_code ec) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, + &op_info->wop, 0, nullptr, write3_cb); + num_io++; + } + break; + + default: + break; + } +} diff --git a/src/common/io_exerciser/RadosIo.h b/src/common/io_exerciser/RadosIo.h new file mode 100644 index 0000000000000..179c5bba3aea0 --- /dev/null +++ b/src/common/io_exerciser/RadosIo.h @@ -0,0 +1,80 @@ +#pragma once + +#include "ObjectModel.h" + +/* Overview + * + * class RadosIo + * An IoExerciser. A simple RADOS client that generates I/Os + * from IoOps. Uses an ObjectModel to track the data stored + * in the object. Uses DataBuffer to create and validate + * data buffers. When there are not barrier I/Os this may + * issue multiple async I/Os in parallel. + * + */ + +namespace ceph { + namespace io_exerciser { + namespace data_generation { + class DataGenerator; + } + + class RadosIo: public Model { + protected: + librados::Rados& rados; + boost::asio::io_context& asio; + std::unique_ptr om; + std::unique_ptr db; + std::string pool; + int threads; + ceph::mutex& lock; + ceph::condition_variable& cond; + librados::IoCtx io; + int outstanding_io; + + void start_io(); + void finish_io(); + void wait_for_io(int count); + + public: + RadosIo(librados::Rados& rados, + boost::asio::io_context& asio, + const std::string& pool, + const std::string& oid, + uint64_t block_size, + int seed, + int threads, + ceph::mutex& lock, + ceph::condition_variable& cond); + + ~RadosIo(); + + void allow_ec_overwrites(bool allow); + + class AsyncOpInfo { + public: + librados::ObjectReadOperation rop; + librados::ObjectWriteOperation wop; + ceph::buffer::list bl1; + ceph::buffer::list bl2; + ceph::buffer::list bl3; + uint64_t offset1; + uint64_t length1; + uint64_t offset2; + uint64_t length2; + uint64_t offset3; + uint64_t length3; + + AsyncOpInfo(uint64_t offset1 = 0, uint64_t length1 = 0, + uint64_t offset2 = 0, uint64_t length2 = 0, + uint64_t offset3 = 0, uint64_t length3 = 0 ); + ~AsyncOpInfo() = default; + }; + + // Must be called with lock held + bool readyForIoOp(IoOp& op); + + void applyIoOp(IoOp& op); + }; + } +} \ No newline at end of file diff --git a/src/crimson/osd/object_context.h b/src/crimson/osd/object_context.h index 4148e3b592c03..e17af91e3ade8 100644 --- a/src/crimson/osd/object_context.h +++ b/src/crimson/osd/object_context.h @@ -76,6 +76,11 @@ class ObjectContext : public ceph::common::intrusive_lru_base< ObjectContext(hobject_t hoid) : lock(hoid), obs(std::move(hoid)) {} + void update_from(const ObjectContext &obc) { + obs = obc.obs; + ssc = obc.ssc; + } + const hobject_t &get_oid() const { return obs.oi.soid; } diff --git a/src/crimson/osd/object_context_loader.cc b/src/crimson/osd/object_context_loader.cc index 8ecb1d4b8efbc..12aa40b925aea 100644 --- a/src/crimson/osd/object_context_loader.cc +++ b/src/crimson/osd/object_context_loader.cc @@ -1,5 +1,6 @@ #include "crimson/osd/object_context_loader.h" #include "osd/osd_types_fmt.h" +#include "osd/object_state_fmt.h" SET_SUBSYS(osd); @@ -111,7 +112,7 @@ using crimson::common::local_conf; return std::invoke(std::move(func), obc); } ).finally([FNAME, this, obc=ObjectContextRef(obc)] { - DEBUGDPP("released object {}", dpp, obc->get_oid()); + DEBUGDPP("released object {}, {}", dpp, obc->get_oid(), obc->obs); if constexpr (track) { obc->remove_from(obc_set_accessing); } @@ -125,7 +126,7 @@ using crimson::common::local_conf; return std::invoke(std::move(func), obc); } ).finally([FNAME, this, obc=ObjectContextRef(obc)] { - DEBUGDPP("released object {}", dpp, obc->get_oid()); + DEBUGDPP("released object {}, {}", dpp, obc->get_oid(), obc->obs); if constexpr (track) { obc->remove_from(obc_set_accessing); } diff --git a/src/crimson/osd/ops_executer.h b/src/crimson/osd/ops_executer.h index e0e5e10e0a9b4..0dea7d0515e93 100644 --- a/src/crimson/osd/ops_executer.h +++ b/src/crimson/osd/ops_executer.h @@ -552,11 +552,7 @@ OpsExecuter::flush_changes_n_do_ops_effects( template struct OpsExecuter::RollbackHelper { - interruptible_future<> rollback_obc_if_modified(const std::error_code& e); - ObjectContextRef get_obc() const { - assert(ox); - return ox->obc; - } + void rollback_obc_if_modified(const std::error_code& e); seastar::lw_shared_ptr ox; Func func; }; @@ -569,8 +565,7 @@ OpsExecuter::create_rollbacker(Func&& func) { template -OpsExecuter::interruptible_future<> -OpsExecuter::RollbackHelper::rollback_obc_if_modified( +void OpsExecuter::RollbackHelper::rollback_obc_if_modified( const std::error_code& e) { // Oops, an operation had failed. do_osd_ops() altogether with @@ -580,12 +575,6 @@ OpsExecuter::RollbackHelper::rollback_obc_if_modified( // we maintain and we did it for both reading and writing. // Now all modifications must be reverted. // - // Let's just reload from the store. Evicting from the shared - // LRU would be tricky as next MOSDOp (the one at `get_obc` - // phase) could actually already finished the lookup. Fortunately, - // this is supposed to live on cold paths, so performance is not - // a concern -- simplicity wins. - // // The conditional's purpose is to efficiently handle hot errors // which may appear as a result of e.g. CEPH_OSD_OP_CMPXATTR or // CEPH_OSD_OP_OMAP_CMP. These are read-like ops and clients @@ -600,7 +589,9 @@ OpsExecuter::RollbackHelper::rollback_obc_if_modified( ox->obc->get_oid(), e, need_rollback); - return need_rollback ? func(*ox->obc) : interruptor::now(); + if (need_rollback) { + func(ox->obc); + } } // PgOpsExecuter -- a class for executing ops targeting a certain PG. diff --git a/src/crimson/osd/pg.cc b/src/crimson/osd/pg.cc index 644cc84513d49..d210773ca3031 100644 --- a/src/crimson/osd/pg.cc +++ b/src/crimson/osd/pg.cc @@ -964,6 +964,15 @@ PG::BackgroundProcessLock::lock() noexcept return interruptor::make_interruptible(mutex.lock()); } +// We may need to rollback the ObjectContext on failed op execution. +// Copy the current obc before mutating it in order to recover on failures. +ObjectContextRef duplicate_obc(const ObjectContextRef &obc) { + ObjectContextRef object_context = new ObjectContext(obc->obs.oi.soid); + object_context->obs = obc->obs; + object_context->ssc = new SnapSetContext(*obc->ssc); + return object_context; +} + template PG::do_osd_ops_iertr::future> PG::do_osd_ops_execute( @@ -976,9 +985,9 @@ PG::do_osd_ops_execute( FailureFunc&& failure_func) { assert(ox); - auto rollbacker = ox->create_rollbacker([this] (auto& obc) { - return obc_loader.reload_obc(obc).handle_error_interruptible( - load_obc_ertr::assert_all{"can't live with object state messed up"}); + auto rollbacker = ox->create_rollbacker( + [object_context=duplicate_obc(obc)] (auto& obc) mutable { + obc->update_from(*object_context); }); auto failure_func_ptr = seastar::make_lw_shared(std::move(failure_func)); return interruptor::do_for_each(ops, [ox](OSDOp& osd_op) { @@ -1040,23 +1049,21 @@ PG::do_osd_ops_execute( std::move(log_entries)); }); }).safe_then_unpack_interruptible( - [success_func=std::move(success_func), rollbacker, this, failure_func_ptr] + [success_func=std::move(success_func), rollbacker, this, failure_func_ptr, obc] (auto submitted_fut, auto _all_completed_fut) mutable { auto all_completed_fut = _all_completed_fut.safe_then_interruptible_tuple( std::move(success_func), crimson::ct_error::object_corrupted::handle( - [rollbacker, this] (const std::error_code& e) mutable { + [rollbacker, this, obc] (const std::error_code& e) mutable { // this is a path for EIO. it's special because we want to fix the obejct // and try again. that is, the layer above `PG::do_osd_ops` is supposed to // restart the execution. - return rollbacker.rollback_obc_if_modified(e).then_interruptible( - [obc=rollbacker.get_obc(), this] { - return repair_object(obc->obs.oi.soid, - obc->obs.oi.version - ).then_interruptible([] { - return do_osd_ops_iertr::future{crimson::ct_error::eagain::make()}; - }); + rollbacker.rollback_obc_if_modified(e); + return repair_object(obc->obs.oi.soid, + obc->obs.oi.version + ).then_interruptible([] { + return do_osd_ops_iertr::future{crimson::ct_error::eagain::make()}; }); }), OpsExecuter::osd_op_errorator::all_same_way( [rollbacker, failure_func_ptr] @@ -1065,11 +1072,8 @@ PG::do_osd_ops_execute( ceph_assert(e.value() == EDQUOT || e.value() == ENOSPC || e.value() == EAGAIN); - return rollbacker.rollback_obc_if_modified(e).then_interruptible( - [e, failure_func_ptr] { - // no need to record error log - return (*failure_func_ptr)(e); - }); + rollbacker.rollback_obc_if_modified(e); + return (*failure_func_ptr)(e); })); return PG::do_osd_ops_iertr::make_ready_future>( @@ -1081,45 +1085,42 @@ PG::do_osd_ops_execute( rollbacker, failure_func_ptr] (const std::error_code& e) mutable { ceph_tid_t rep_tid = shard_services.get_tid(); - return rollbacker.rollback_obc_if_modified(e).then_interruptible( - [&, op_info, m, obc, - this, e, rep_tid, failure_func_ptr] { - // record error log - auto maybe_submit_error_log = - seastar::make_ready_future>(std::nullopt); - // call submit_error_log only for non-internal clients - if constexpr (!std::is_same_v) { - if(op_info.may_write()) { - maybe_submit_error_log = - submit_error_log(m, op_info, obc, e, rep_tid); - } + rollbacker.rollback_obc_if_modified(e); + // record error log + auto maybe_submit_error_log = + interruptor::make_ready_future>(std::nullopt); + // call submit_error_log only for non-internal clients + if constexpr (!std::is_same_v) { + if(op_info.may_write()) { + maybe_submit_error_log = + submit_error_log(m, op_info, obc, e, rep_tid); } - return maybe_submit_error_log.then( - [this, failure_func_ptr, e, rep_tid] (auto version) { - auto all_completed = - [this, failure_func_ptr, e, rep_tid, version] { - if (version.has_value()) { - return complete_error_log(rep_tid, version.value()).then( - [failure_func_ptr, e] { - return (*failure_func_ptr)(e); - }); - } else { + } + return maybe_submit_error_log.then_interruptible( + [this, failure_func_ptr, e, rep_tid] (auto version) { + auto all_completed = + [this, failure_func_ptr, e, rep_tid, version] { + if (version.has_value()) { + return complete_error_log(rep_tid, version.value() + ).then_interruptible([failure_func_ptr, e] { return (*failure_func_ptr)(e); - } - }; - return PG::do_osd_ops_iertr::make_ready_future>( - std::move(seastar::now()), - std::move(all_completed()) - ); - }); + }); + } else { + return (*failure_func_ptr)(e); + } + }; + return PG::do_osd_ops_iertr::make_ready_future>( + std::move(seastar::now()), + std::move(all_completed()) + ); }); })); } -seastar::future<> PG::complete_error_log(const ceph_tid_t& rep_tid, +PG::interruptible_future<> PG::complete_error_log(const ceph_tid_t& rep_tid, const eversion_t& version) { - auto result = seastar::now(); + auto result = interruptor::now(); auto last_complete = peering_state.get_info().last_complete; ceph_assert(log_entry_update_waiting_on.contains(rep_tid)); auto& log_update = log_entry_update_waiting_on[rep_tid]; @@ -1133,8 +1134,9 @@ seastar::future<> PG::complete_error_log(const ceph_tid_t& rep_tid, } else { logger().debug("complete_error_log: rep_tid {} awaiting update from {}", rep_tid, log_update.waiting_on); - result = log_update.all_committed.get_shared_future().then( - [this, last_complete, rep_tid, version] { + result = interruptor::make_interruptible( + log_update.all_committed.get_shared_future() + ).then_interruptible([this, last_complete, rep_tid, version] { logger().debug("complete_error_log: rep_tid {} awaited ", rep_tid); peering_state.complete_write(version, last_complete); ceph_assert(!log_entry_update_waiting_on.contains(rep_tid)); @@ -1144,7 +1146,7 @@ seastar::future<> PG::complete_error_log(const ceph_tid_t& rep_tid, return result; } -seastar::future> PG::submit_error_log( +PG::interruptible_future> PG::submit_error_log( Ref m, const OpInfo &op_info, ObjectContextRef obc, @@ -1175,7 +1177,7 @@ seastar::future> PG::submit_error_log( return seastar::do_with(log_entries, set{}, [this, t=std::move(t), rep_tid](auto& log_entries, auto& waiting_on) mutable { - return seastar::do_for_each(get_acting_recovery_backfill(), + return interruptor::do_for_each(get_acting_recovery_backfill(), [this, log_entries, waiting_on, rep_tid] (auto& i) mutable { pg_shard_t peer(i); @@ -1200,7 +1202,7 @@ seastar::future> PG::submit_error_log( return shard_services.send_to_osd(peer.osd, std::move(log_m), get_osdmap_epoch()); - }).then([this, waiting_on, t=std::move(t), rep_tid] () mutable { + }).then_interruptible([this, waiting_on, t=std::move(t), rep_tid] () mutable { waiting_on.insert(pg_whoami); logger().debug("submit_error_log: inserting rep_tid {}", rep_tid); log_entry_update_waiting_on.insert( diff --git a/src/crimson/osd/pg.h b/src/crimson/osd/pg.h index 11c0e3668b142..93279a18c565b 100644 --- a/src/crimson/osd/pg.h +++ b/src/crimson/osd/pg.h @@ -619,9 +619,9 @@ class PG : public boost::intrusive_ref_counter< void print(std::ostream& os) const; void dump_primary(Formatter*); - seastar::future<> complete_error_log(const ceph_tid_t& rep_tid, + interruptible_future<> complete_error_log(const ceph_tid_t& rep_tid, const eversion_t& version); - seastar::future> submit_error_log( + interruptible_future> submit_error_log( Ref m, const OpInfo &op_info, ObjectContextRef obc, diff --git a/src/crimson/osd/pg_backend.cc b/src/crimson/osd/pg_backend.cc index 1e4acf95acbc9..fa8201b61c28d 100644 --- a/src/crimson/osd/pg_backend.cc +++ b/src/crimson/osd/pg_backend.cc @@ -30,6 +30,7 @@ #include "replicated_recovery_backend.h" #include "ec_backend.h" #include "exceptions.h" +#include "osd/object_state_fmt.h" namespace { seastar::logger& logger() { @@ -928,6 +929,7 @@ PGBackend::create_iertr::future<> PGBackend::create( ceph::os::Transaction& txn, object_stat_sum_t& delta_stats) { + logger().debug("{} obc existed: {}, osd_op {}", __func__, os, osd_op); if (os.exists && !os.oi.is_whiteout() && (osd_op.op.flags & CEPH_OSD_OP_FLAG_EXCL)) { // this is an exclusive create diff --git a/src/include/rbd/librbd.h b/src/include/rbd/librbd.h index f9af9262b2aa3..7144e2038c1ca 100644 --- a/src/include/rbd/librbd.h +++ b/src/include/rbd/librbd.h @@ -577,6 +577,16 @@ CEPH_RBD_API int rbd_mirror_mode_get(rados_ioctx_t io_ctx, rbd_mirror_mode_t *mirror_mode); CEPH_RBD_API int rbd_mirror_mode_set(rados_ioctx_t io_ctx, rbd_mirror_mode_t mirror_mode); +CEPH_RBD_API int rbd_mirror_remote_namespace_get(rados_ioctx_t io_ctx, + char *remote_namespace, + size_t *max_len); +/** + * The value can be set only if mirroring on io_ctx is disabled. The previously + * set value will be automatically reset to io_ctx's namespace when mirroring on + * io_ctx is disabled. + */ +CEPH_RBD_API int rbd_mirror_remote_namespace_set(rados_ioctx_t io_ctx, + const char *remote_namespace); CEPH_RBD_API int rbd_mirror_uuid_get(rados_ioctx_t io_ctx, char *uuid, size_t *max_len); diff --git a/src/include/rbd/librbd.hpp b/src/include/rbd/librbd.hpp index 50a6c623d3a00..5d64943fe56e5 100644 --- a/src/include/rbd/librbd.hpp +++ b/src/include/rbd/librbd.hpp @@ -357,6 +357,16 @@ class CEPH_RBD_API RBD int mirror_mode_get(IoCtx& io_ctx, rbd_mirror_mode_t *mirror_mode); int mirror_mode_set(IoCtx& io_ctx, rbd_mirror_mode_t mirror_mode); + int mirror_remote_namespace_get(IoCtx& io_ctx, + std::string* remote_namespace); + + /** + * The value can be set only if mirroring on io_ctx is disabled. The + * previously set value will be automatically reset to io_ctx's namespace when + * mirroring on io_ctx is disabled. + */ + int mirror_remote_namespace_set(IoCtx& io_ctx, + const std::string& remote_namespace); int mirror_uuid_get(IoCtx& io_ctx, std::string* mirror_uuid); int mirror_peer_bootstrap_create(IoCtx& io_ctx, std::string* token); diff --git a/src/kv/RocksDBStore.cc b/src/kv/RocksDBStore.cc index c2b0da79ef797..ca63ea0648414 100644 --- a/src/kv/RocksDBStore.cc +++ b/src/kv/RocksDBStore.cc @@ -1417,8 +1417,13 @@ int64_t RocksDBStore::estimate_prefix_size(const string& prefix, void RocksDBStore::get_statistics(Formatter *f) { if (!cct->_conf->rocksdb_perf) { - dout(20) << __func__ << " RocksDB perf is disabled, can't probe for stats" - << dendl; + f->write_raw_data("error: RocksDB perf is disabled, can't probe for stats.\n"); + return; + } + if (!cct->_conf->rocksdb_collect_compaction_stats && + !cct->_conf->rocksdb_collect_extended_stats && + !cct->_conf->rocksdb_collect_memory_stats) { + f->write_raw_data("error: None of rocksdb_collect_* setting is enabled, hence no output.\n"); return; } diff --git a/src/librbd/api/Mirror.cc b/src/librbd/api/Mirror.cc index 2cfad0d327532..06a5e836fafc7 100644 --- a/src/librbd/api/Mirror.cc +++ b/src/librbd/api/Mirror.cc @@ -1115,9 +1115,8 @@ int Mirror::mode_set(librados::IoCtx& io_ctx, << dendl; return r; } - if (current_mirror_mode == next_mirror_mode) { - return 0; + return 0; // Nothing more to be done } else if (current_mirror_mode == cls::rbd::MIRROR_MODE_DISABLED) { uuid_d uuid_gen; uuid_gen.generate_random(); @@ -1271,6 +1270,55 @@ int Mirror::mode_set(librados::IoCtx& io_ctx, return 0; } +template +int Mirror::remote_namespace_get(librados::IoCtx& io_ctx, + std::string* remote_namespace) { + + CephContext *cct = reinterpret_cast(io_ctx.cct()); + ldout(cct, 20) << dendl; + + int r = cls_client::mirror_remote_namespace_get(&io_ctx, remote_namespace); + if (r < 0) { + if (r != -ENOENT && r != -EOPNOTSUPP) { + lderr(cct) << "failed to retrieve remote mirror namespace: " + << cpp_strerror(r) << dendl; + return r; + } + *remote_namespace = io_ctx.get_namespace(); + } + return 0; +} + + +template +int Mirror::remote_namespace_set(librados::IoCtx& io_ctx, + const std::string& remote_namespace) { + CephContext *cct = reinterpret_cast(io_ctx.cct()); + ldout(cct, 20) << dendl; + + std::string local_namespace = io_ctx.get_namespace(); + + if (local_namespace.empty() && !remote_namespace.empty()) { + lderr(cct) << "cannot mirror the default namespace to a " + << "non-default namespace." << dendl; + return -EINVAL; + } + + if (!local_namespace.empty() && remote_namespace.empty()) { + lderr(cct) << "cannot mirror a non-default namespace to the default " + << "namespace." << dendl; + return -EINVAL; + } + + int r = cls_client::mirror_remote_namespace_set(&io_ctx, remote_namespace); + if (r < 0) { + lderr(cct) << "failed to set remote mirror namespace: " + << cpp_strerror(r) << dendl; + return r; + } + return 0; +} + template int Mirror::uuid_get(librados::IoCtx& io_ctx, std::string* mirror_uuid) { CephContext *cct = reinterpret_cast(io_ctx.cct()); diff --git a/src/librbd/api/Mirror.h b/src/librbd/api/Mirror.h index b3a552b13b7ee..6e84247b67846 100644 --- a/src/librbd/api/Mirror.h +++ b/src/librbd/api/Mirror.h @@ -31,6 +31,11 @@ struct Mirror { static int mode_get(librados::IoCtx& io_ctx, rbd_mirror_mode_t *mirror_mode); static int mode_set(librados::IoCtx& io_ctx, rbd_mirror_mode_t mirror_mode); + static int remote_namespace_get(librados::IoCtx& io_ctx, + std::string* remote_namespace); + static int remote_namespace_set(librados::IoCtx& io_ctx, + const std::string& remote_namespace); + static int uuid_get(librados::IoCtx& io_ctx, std::string* mirror_uuid); static void uuid_get(librados::IoCtx& io_ctx, std::string* mirror_uuid, Context* on_finish); diff --git a/src/librbd/librbd.cc b/src/librbd/librbd.cc index c389282c0cc88..cc63595257fe8 100644 --- a/src/librbd/librbd.cc +++ b/src/librbd/librbd.cc @@ -1101,6 +1101,18 @@ namespace librbd { return librbd::api::Mirror<>::mode_set(io_ctx, mirror_mode); } + int RBD::mirror_remote_namespace_get(IoCtx& io_ctx, + std::string* remote_namespace) { + return librbd::api::Mirror<>::remote_namespace_get(io_ctx, + remote_namespace); + } + + int RBD::mirror_remote_namespace_set(IoCtx& io_ctx, + const std::string& remote_namespace) { + return librbd::api::Mirror<>::remote_namespace_set(io_ctx, + remote_namespace); + } + int RBD::mirror_uuid_get(IoCtx& io_ctx, std::string* mirror_uuid) { return librbd::api::Mirror<>::uuid_get(io_ctx, mirror_uuid); } @@ -3397,6 +3409,37 @@ extern "C" int rbd_mirror_mode_set(rados_ioctx_t p, return librbd::api::Mirror<>::mode_set(io_ctx, mirror_mode); } +extern "C" int rbd_mirror_remote_namespace_get(rados_ioctx_t p, + char *remote_namespace, + size_t *max_len) { + librados::IoCtx io_ctx; + librados::IoCtx::from_rados_ioctx_t(p, io_ctx); + + std::string remote_namespace_str; + int r = librbd::api::Mirror<>::remote_namespace_get(io_ctx, + &remote_namespace_str); + if (r < 0) { + return r; + } + + auto total_len = remote_namespace_str.size() + 1; + if (*max_len < total_len) { + *max_len = total_len; + return -ERANGE; + } + *max_len = total_len; + + strcpy(remote_namespace, remote_namespace_str.c_str()); + return 0; +} + +extern "C" int rbd_mirror_remote_namespace_set(rados_ioctx_t p, + const char *remote_namespace) { + librados::IoCtx io_ctx; + librados::IoCtx::from_rados_ioctx_t(p, io_ctx); + return librbd::api::Mirror<>::remote_namespace_set(io_ctx, remote_namespace); +} + extern "C" int rbd_mirror_uuid_get(rados_ioctx_t p, char *mirror_uuid, size_t *max_len) { librados::IoCtx io_ctx; diff --git a/src/mds/CInode.cc b/src/mds/CInode.cc index faf9f408688dd..0e9b6996ad2c5 100644 --- a/src/mds/CInode.cc +++ b/src/mds/CInode.cc @@ -1386,7 +1386,7 @@ void CInode::_commit_ops(int r, C_GatherBuilder &gather_bld, } void CInode::_store_backtrace(std::vector &ops_vec, - inode_backtrace_t &bt, int op_prio) + inode_backtrace_t &bt, int op_prio, bool ignore_old_pools) { dout(10) << __func__ << " on " << *this << dendl; ceph_assert(is_dirty_parent()); @@ -1407,8 +1407,8 @@ void CInode::_store_backtrace(std::vector &ops_vec, ops_vec.emplace_back(op_prio, pool, get_inode()->layout, mdcache->mds->mdsmap->get_up_features(), slink); - if (!state_test(STATE_DIRTYPOOL) || get_inode()->old_pools.empty()) { - dout(20) << __func__ << ": no dirtypool or no old pools" << dendl; + if (!state_test(STATE_DIRTYPOOL) || get_inode()->old_pools.empty() || ignore_old_pools) { + dout(20) << __func__ << ": no dirtypool or no old pools or ignore_old_pools" << dendl; return; } @@ -1431,7 +1431,7 @@ void CInode::store_backtrace(MDSContext *fin, int op_prio) inode_backtrace_t bt; auto version = get_inode()->backtrace_version; - _store_backtrace(ops_vec, bt, op_prio); + _store_backtrace(ops_vec, bt, op_prio, false); C_GatherBuilder gather(g_ceph_context, new C_OnFinisher( @@ -1442,12 +1442,14 @@ void CInode::store_backtrace(MDSContext *fin, int op_prio) gather.activate(); } -void CInode::store_backtrace(CInodeCommitOperations &op, int op_prio) +void CInode::store_backtrace(CInodeCommitOperations &op, int op_prio, + bool ignore_old_pools) { op.version = get_inode()->backtrace_version; op.in = this; - _store_backtrace(op.ops_vec, op.bt, op_prio); + // update backtraces in old pools + _store_backtrace(op.ops_vec, op.bt, op_prio, ignore_old_pools); } void CInode::_stored_backtrace(int r, version_t v, Context *fin) diff --git a/src/mds/CInode.h b/src/mds/CInode.h index d55b644210767..8ae1d5f71687b 100644 --- a/src/mds/CInode.h +++ b/src/mds/CInode.h @@ -754,8 +754,9 @@ class CInode : public MDSCacheObject, public InodeStoreBase, public Counter &ops_vec, - inode_backtrace_t &bt, int op_prio); - void store_backtrace(CInodeCommitOperations &op, int op_prio); + inode_backtrace_t &bt, int op_prio, bool ignore_old_pools); + void store_backtrace(CInodeCommitOperations &op, int op_prio, + bool ignore_old_pools=false); void store_backtrace(MDSContext *fin, int op_prio=-1); void _stored_backtrace(int r, version_t v, Context *fin); void fetch_backtrace(Context *fin, ceph::buffer::list *backtrace); @@ -1142,6 +1143,14 @@ class CInode : public MDSCacheObject, public InodeStoreBase, public Counter=0 + */ + int64_t get_backtrace_pool() const; + protected: ceph_lock_state_t *get_fcntl_lock_state() { if (!fcntl_locks) @@ -1192,14 +1201,6 @@ class CInode : public MDSCacheObject, public InodeStoreBase, public Counter=0 - */ - int64_t get_backtrace_pool() const; - // parent dentries in cache CDentry *parent = nullptr; // primary link mempool::mds_co::compact_set remote_parents; // if hard linked diff --git a/src/mds/LogSegment.h b/src/mds/LogSegment.h index e6d8a2ca8830c..04427ad8be8ea 100644 --- a/src/mds/LogSegment.h +++ b/src/mds/LogSegment.h @@ -108,7 +108,7 @@ class LogSegment { static inline std::ostream& operator<<(std::ostream& out, const LogSegment& ls) { return out << "LogSegment(" << ls.seq << "/0x" << std::hex << ls.offset - << std::dec << " events=" << ls.num_events << ")"; + << "~" << ls.end << std::dec << " events=" << ls.num_events << ")"; } #endif diff --git a/src/mds/MDLog.cc b/src/mds/MDLog.cc index cd274f8edc4e6..40d893d62623f 100644 --- a/src/mds/MDLog.cc +++ b/src/mds/MDLog.cc @@ -749,6 +749,7 @@ class C_MaybeExpiredSegment : public MDSInternalContext { C_MaybeExpiredSegment(MDLog *mdl, LogSegment *s, int p) : MDSInternalContext(mdl->mds), mdlog(mdl), ls(s), op_prio(p) {} void finish(int res) override { + dout(10) << __func__ << ": ls=" << *ls << ", r=" << res << dendl; if (res < 0) mdlog->mds->handle_write_error(res); mdlog->_maybe_expired(ls, op_prio); diff --git a/src/mds/journal.cc b/src/mds/journal.cc index e080b11761050..40400ff4054ca 100644 --- a/src/mds/journal.cc +++ b/src/mds/journal.cc @@ -237,27 +237,53 @@ void LogSegment::try_to_expire(MDSRank *mds, MDSGatherBuilder &gather_bld, int o ceph_assert(g_conf()->mds_kill_journal_expire_at != 3); - size_t count = 0; - for (elist::iterator it = dirty_parent_inodes.begin(); !it.end(); ++it) - count++; - - std::vector ops_vec; - ops_vec.reserve(count); + std::map> ops_vec_map; // backtraces to be stored/updated for (elist::iterator p = dirty_parent_inodes.begin(); !p.end(); ++p) { CInode *in = *p; ceph_assert(in->is_auth()); if (in->can_auth_pin()) { dout(15) << "try_to_expire waiting for storing backtrace on " << *in << dendl; - ops_vec.resize(ops_vec.size() + 1); - in->store_backtrace(ops_vec.back(), op_prio); + auto pool_id = in->get_backtrace_pool(); + + // this is for the default data pool + dout(20) << __func__ << ": updating pool=" << pool_id << dendl; + ops_vec_map[pool_id].push_back(CInodeCommitOperations()); + in->store_backtrace(ops_vec_map[pool_id].back(), op_prio, true); + + + if (!in->state_test(CInode::STATE_DIRTYPOOL)) { + dout(20) << __func__ << ": no dirtypool" << dendl; + continue; + } + + // dispatch separate ops for backtrace updates for old pools + for (auto _pool_id : in->get_inode()->old_pools) { + if (_pool_id == pool_id) { + continue; + } + + in->auth_pin(in); // CInode::_stored_backtrace() does auth_unpin() + dout(20) << __func__ << ": updating old_pool=" << _pool_id << dendl; + + auto cco = CInodeCommitOperations(); + cco.in = in; + // use backtrace from the main pool so as to pickup the main + // pool-id for old pool updates. + cco.bt = ops_vec_map[pool_id].back().bt; + cco.ops_vec.emplace_back(op_prio, _pool_id); + cco.version = in->get_inode()->backtrace_version; + ops_vec_map[_pool_id].push_back(cco); + } } else { dout(15) << "try_to_expire waiting for unfreeze on " << *in << dendl; in->add_waiter(CInode::WAIT_UNFREEZE, gather_bld.new_sub()); } } - if (!ops_vec.empty()) + + for (auto& [pool_id, ops_vec] : ops_vec_map) { mds->finisher->queue(new BatchCommitBacktrace(mds, gather_bld.new_sub(), std::move(ops_vec))); + } ceph_assert(g_conf()->mds_kill_journal_expire_at != 4); diff --git a/src/mon/NVMeofGwMap.cc b/src/mon/NVMeofGwMap.cc index 7d886344244ee..d60d3edefd2d5 100755 --- a/src/mon/NVMeofGwMap.cc +++ b/src/mon/NVMeofGwMap.cc @@ -247,6 +247,30 @@ void NVMeofGwMap::track_deleting_gws(const NvmeGroupKey& group_key, } } +int NVMeofGwMap::process_gw_map_gw_no_subsystems( + const NvmeGwId &gw_id, const NvmeGroupKey& group_key, bool &propose_pending) +{ + int rc = 0; + auto& gws_states = created_gws[group_key]; + auto gw_state = gws_states.find(gw_id); + if (gw_state != gws_states.end()) { + dout(10) << "GW- no subsystems configured " << gw_id << dendl; + auto& st = gw_state->second; + st.availability = gw_availability_t::GW_CREATED; + for (auto& state_itr: created_gws[group_key][gw_id].sm_state) { + fsm_handle_gw_no_subsystems( + gw_id, group_key, state_itr.second,state_itr.first, propose_pending); + } + propose_pending = true; // map should reflect that gw becames Created + if (propose_pending) validate_gw_map(group_key); + } else { + dout(1) << __FUNCTION__ << "ERROR GW-id was not found in the map " + << gw_id << dendl; + rc = -EINVAL; + } + return rc; +} + int NVMeofGwMap::process_gw_map_gw_down( const NvmeGwId &gw_id, const NvmeGroupKey& group_key, bool &propose_pending) { @@ -263,7 +287,7 @@ int NVMeofGwMap::process_gw_map_gw_down( state_itr.first, propose_pending); state_itr.second = gw_states_per_group_t::GW_STANDBY_STATE; } - propose_pending = true; // map should reflect that gw becames unavailable + propose_pending = true; // map should reflect that gw becames Unavailable if (propose_pending) validate_gw_map(group_key); } else { dout(1) << __FUNCTION__ << "ERROR GW-id was not found in the map " @@ -615,6 +639,59 @@ void NVMeofGwMap::fsm_handle_gw_alive( } } +void NVMeofGwMap::fsm_handle_gw_no_subsystems( + const NvmeGwId &gw_id, const NvmeGroupKey& group_key, + gw_states_per_group_t state, NvmeAnaGrpId grpid, bool &map_modified) +{ + switch (state) { + case gw_states_per_group_t::GW_STANDBY_STATE: + case gw_states_per_group_t::GW_IDLE_STATE: + // nothing to do + break; + + case gw_states_per_group_t::GW_WAIT_BLOCKLIST_CMPL: + { + cancel_timer(gw_id, group_key, grpid); + auto& gw_st = created_gws[group_key][gw_id]; + gw_st.standby_state(grpid); + map_modified = true; + } + break; + + case gw_states_per_group_t::GW_WAIT_FAILBACK_PREPARED: + cancel_timer(gw_id, group_key, grpid); + map_modified = true; + for (auto& gw_st: created_gws[group_key]) { + auto& st = gw_st.second; + // found GW that was intended for Failback for this ana grp + if (st.sm_state[grpid] == + gw_states_per_group_t::GW_OWNER_WAIT_FAILBACK_PREPARED) { + dout(4) << "Warning: Outgoing Failback when GW is without subsystems" + << " - to rollback it" <<" GW " << gw_id << "for ANA Group " + << grpid << dendl; + st.standby_state(grpid); + break; + } + } + break; + + case gw_states_per_group_t::GW_OWNER_WAIT_FAILBACK_PREPARED: + case gw_states_per_group_t::GW_ACTIVE_STATE: + { + dout(4) << "Set state to Standby for GW " << gw_id << " group " + << grpid << dendl; + auto& gw_st = created_gws[group_key][gw_id]; + gw_st.standby_state(grpid); + } + break; + + default: + { + dout(4) << "Error : Invalid state " << state << "for GW " << gw_id << dendl; + } + } +} + void NVMeofGwMap::fsm_handle_gw_down( const NvmeGwId &gw_id, const NvmeGroupKey& group_key, gw_states_per_group_t state, NvmeAnaGrpId grpid, bool &map_modified) diff --git a/src/mon/NVMeofGwMap.h b/src/mon/NVMeofGwMap.h index 4c9d796641018..2971037174218 100755 --- a/src/mon/NVMeofGwMap.h +++ b/src/mon/NVMeofGwMap.h @@ -54,6 +54,9 @@ class NVMeofGwMap int process_gw_map_gw_down( const NvmeGwId &gw_id, const NvmeGroupKey& group_key, bool &propose_pending); + int process_gw_map_gw_no_subsystems( + const NvmeGwId &gw_id, const NvmeGroupKey& group_key, + bool &propose_pending); void update_active_timers(bool &propose_pending); void handle_abandoned_ana_groups(bool &propose_pending); void handle_removed_subsystems( @@ -77,6 +80,9 @@ class NVMeofGwMap void fsm_handle_gw_down( const NvmeGwId &gw_id, const NvmeGroupKey& group_key, gw_states_per_group_t state, NvmeAnaGrpId grpid, bool &map_modified); + void fsm_handle_gw_no_subsystems( + const NvmeGwId &gw_id, const NvmeGroupKey& group_key, + gw_states_per_group_t state, NvmeAnaGrpId grpid, bool &map_modified); void fsm_handle_gw_delete( const NvmeGwId &gw_id, const NvmeGroupKey& group_key, gw_states_per_group_t state, NvmeAnaGrpId grpid, bool &map_modified); diff --git a/src/mon/NVMeofGwMon.cc b/src/mon/NVMeofGwMon.cc index b6faeb2e97ce5..544ad67472295 100644 --- a/src/mon/NVMeofGwMon.cc +++ b/src/mon/NVMeofGwMon.cc @@ -432,7 +432,8 @@ bool NVMeofGwMon::prepare_command(MonOpRequestRef op) if (rc == 0) { bool propose = false; // Simulate immediate Failover of this GW - process_gw_down(id, group_key, propose); + process_gw_down(id, group_key, propose, + gw_availability_t::GW_UNAVAILABLE); } else if (rc == -EINVAL) { dout (4) << "Error: GW not found in the database " << id << " " << pool << " " << group << " rc " << rc << dendl; @@ -462,13 +463,19 @@ bool NVMeofGwMon::prepare_command(MonOpRequestRef op) } void NVMeofGwMon::process_gw_down(const NvmeGwId &gw_id, - const NvmeGroupKey& group_key, bool &propose_pending) + const NvmeGroupKey& group_key, bool &propose_pending, + gw_availability_t avail) { LastBeacon lb = {gw_id, group_key}; auto it = last_beacon.find(lb); if (it != last_beacon.end()) { last_beacon.erase(it); - pending_map.process_gw_map_gw_down(gw_id, group_key, propose_pending); + if (avail == gw_availability_t::GW_UNAVAILABLE) { + pending_map.process_gw_map_gw_down(gw_id, group_key, propose_pending); + } else { + pending_map.process_gw_map_gw_no_subsystems(gw_id, group_key, propose_pending); + } + } } @@ -581,7 +588,7 @@ bool NVMeofGwMon::prepare_beacon(MonOpRequestRef op) } if (sub.size() == 0) { - avail = gw_availability_t::GW_UNAVAILABLE; + avail = gw_availability_t::GW_CREATED; } if (pending_map.created_gws[group_key][gw_id].subsystems != sub) { dout(10) << "subsystems of GW changed, propose pending " << gw_id << dendl; @@ -607,8 +614,9 @@ bool NVMeofGwMon::prepare_beacon(MonOpRequestRef op) epoch_t last_osd_epoch = m->get_last_osd_epoch(); pending_map.process_gw_map_ka(gw_id, group_key, last_osd_epoch, propose); // state set by GW client application - } else if (avail == gw_availability_t::GW_UNAVAILABLE) { - process_gw_down(gw_id, group_key, propose); + } else if (avail == gw_availability_t::GW_UNAVAILABLE || + avail == gw_availability_t::GW_CREATED) { + process_gw_down(gw_id, group_key, propose, avail); } // Periodic: check active FSM timers pending_map.update_active_timers(timer_propose); diff --git a/src/mon/NVMeofGwMon.h b/src/mon/NVMeofGwMon.h index f132c87d92af7..7fae8b766a5e7 100644 --- a/src/mon/NVMeofGwMon.h +++ b/src/mon/NVMeofGwMon.h @@ -85,7 +85,8 @@ class NVMeofGwMon: public PaxosService, private: void synchronize_last_beacon(); void process_gw_down(const NvmeGwId &gw_id, - const NvmeGroupKey& group_key, bool &propose_pending); + const NvmeGroupKey& group_key, bool &propose_pending, + gw_availability_t avail); }; #endif /* MON_NVMEGWMONITOR_H_ */ diff --git a/src/osd/osd_types.h b/src/osd/osd_types.h index fe62fad2805d3..e2edaa39dfc2d 100644 --- a/src/osd/osd_types.h +++ b/src/osd/osd_types.h @@ -4268,6 +4268,7 @@ struct OSDOp { } }; std::ostream& operator<<(std::ostream& out, const OSDOp& op); +template <> struct fmt::formatter : fmt::ostream_formatter {}; struct pg_log_op_return_item_t { int32_t rval; diff --git a/src/pybind/mgr/cephadm/services/oauth2_proxy.py b/src/pybind/mgr/cephadm/services/oauth2_proxy.py index c19005c95f3ca..cabb21bce139e 100644 --- a/src/pybind/mgr/cephadm/services/oauth2_proxy.py +++ b/src/pybind/mgr/cephadm/services/oauth2_proxy.py @@ -67,12 +67,12 @@ def generate_random_secret(self) -> str: def generate_config(self, daemon_spec: CephadmDaemonDeploySpec) -> Tuple[Dict[str, Any], List[str]]: assert self.TYPE == daemon_spec.daemon_type svc_spec = cast(OAuth2ProxySpec, self.mgr.spec_store[daemon_spec.service_name].spec) - whitelist_domains = svc_spec.whitelist_domains or [] - whitelist_domains += self.get_service_ips_and_hosts('mgmt-gateway') + allowlist_domains = svc_spec.allowlist_domains or [] + allowlist_domains += self.get_service_ips_and_hosts('mgmt-gateway') context = { 'spec': svc_spec, 'cookie_secret': svc_spec.cookie_secret or self.generate_random_secret(), - 'whitelist_domains': whitelist_domains, + 'allowlist_domains': allowlist_domains, 'redirect_url': svc_spec.redirect_url or self.get_redirect_url() } diff --git a/src/pybind/mgr/cephadm/templates/services/oauth2-proxy/oauth2-proxy.conf.j2 b/src/pybind/mgr/cephadm/templates/services/oauth2-proxy/oauth2-proxy.conf.j2 index 20ca8cb6504c0..c8d9f920adf5a 100644 --- a/src/pybind/mgr/cephadm/templates/services/oauth2-proxy/oauth2-proxy.conf.j2 +++ b/src/pybind/mgr/cephadm/templates/services/oauth2-proxy/oauth2-proxy.conf.j2 @@ -34,4 +34,4 @@ set_xauthrequest= true # Secret value for encrypting cookies. cookie_secret= "{{ cookie_secret }}" email_domains= "*" -whitelist_domains= "{{ whitelist_domains | join(',') }}" +whitelist_domains= "{{ allowlist_domains | join(',') }}" diff --git a/src/pybind/mgr/dashboard/controllers/auth.py b/src/pybind/mgr/dashboard/controllers/auth.py index 2e6cf855c2977..16276af17e4c4 100644 --- a/src/pybind/mgr/dashboard/controllers/auth.py +++ b/src/pybind/mgr/dashboard/controllers/auth.py @@ -10,7 +10,7 @@ from .. import mgr from ..exceptions import InvalidCredentialsError, UserDoesNotExist -from ..services.auth import AuthManager, JwtManager +from ..services.auth import AuthManager, AuthType, BaseAuth, JwtManager, OAuth2 from ..services.cluster import ClusterModel from ..settings import Settings from . import APIDoc, APIRouter, ControllerAuthMixin, EndpointDoc, RESTController, allow_empty_body @@ -132,7 +132,7 @@ def create(self, username, password, ttl: Optional[int] = None): 'username': username, 'permissions': user_perms, 'pwdExpirationDate': pwd_expiration_date, - 'sso': mgr.SSO_DB.protocol == 'saml2', + 'sso': BaseAuth.from_protocol(mgr.SSO_DB.protocol).sso, 'pwdUpdateRequired': pwd_update_required } mgr.ACCESS_CTRL_DB.increment_attempt(username) @@ -156,37 +156,33 @@ def create(self, username, password, ttl: Optional[int] = None): @RESTController.Collection('POST') @allow_empty_body def logout(self): - logger.debug('Logout successful') - token = JwtManager.get_token_from_header() + logger.debug('Logout started') + token = JwtManager.get_token(cherrypy.request) JwtManager.blocklist_token(token) self._delete_token_cookie(token) - redirect_url = '#/login' - if mgr.SSO_DB.protocol == 'saml2': - redirect_url = 'auth/saml2/slo' return { - 'redirect_url': redirect_url + 'redirect_url': BaseAuth.from_db(mgr.SSO_DB).LOGOUT_URL, + 'protocol': BaseAuth.from_db(mgr.SSO_DB).get_auth_name() } - def _get_login_url(self): - if mgr.SSO_DB.protocol == 'saml2': - return 'auth/saml2/login' - return '#/login' - @RESTController.Collection('POST', query_params=['token']) @EndpointDoc("Check token Authentication", parameters={'token': (str, 'Authentication Token')}, responses={201: AUTH_CHECK_SCHEMA}) def check(self, token): if token: - user = JwtManager.get_user(token) + if mgr.SSO_DB.protocol == AuthType.OAUTH2: + user = OAuth2.get_user(token) + else: + user = JwtManager.get_user(token) if user: return { 'username': user.username, 'permissions': user.permissions_dict(), - 'sso': mgr.SSO_DB.protocol == 'saml2', + 'sso': BaseAuth.from_db(mgr.SSO_DB).sso, 'pwdUpdateRequired': user.pwd_update_required } return { - 'login_url': self._get_login_url(), + 'login_url': BaseAuth.from_db(mgr.SSO_DB).LOGIN_URL, 'cluster_status': ClusterModel.from_db().dict()['status'] } diff --git a/src/pybind/mgr/dashboard/controllers/host.py b/src/pybind/mgr/dashboard/controllers/host.py index 645fc4cc44c55..68dd8440b7f4a 100644 --- a/src/pybind/mgr/dashboard/controllers/host.py +++ b/src/pybind/mgr/dashboard/controllers/host.py @@ -515,8 +515,6 @@ def inventory(self, refresh=None): @Endpoint('GET') @ReadPermission - @raise_if_no_orchestrator([OrchFeature.HOST_LIST]) - @handle_orchestrator_error('host') def list(self): """ Get all hosts. diff --git a/src/pybind/mgr/dashboard/controllers/oauth2.py b/src/pybind/mgr/dashboard/controllers/oauth2.py new file mode 100644 index 0000000000000..ae37c4ac1f7f6 --- /dev/null +++ b/src/pybind/mgr/dashboard/controllers/oauth2.py @@ -0,0 +1,32 @@ +import cherrypy + +from dashboard.exceptions import DashboardException +from dashboard.services.auth.oauth2 import OAuth2 + +from . import Endpoint, RESTController, Router + + +@Router('/auth/oauth2', secure=False) +class Oauth2(RESTController): + + @Endpoint(json_response=False, version=None) + def login(self): + if not OAuth2.enabled(): + raise DashboardException(500, msg='Failed to login: SSO OAuth2 is not enabled') + + token = OAuth2.get_token(cherrypy.request) + if not token: + raise cherrypy.HTTPError() + + raise cherrypy.HTTPRedirect(OAuth2.get_login_redirect_url(token)) + + @Endpoint(json_response=False, version=None) + def logout(self): + if not OAuth2.enabled(): + raise DashboardException(500, msg='Failed to logout: SSO OAuth2 is not enabled') + + token = OAuth2.get_token(cherrypy.request) + if not token: + raise cherrypy.HTTPError() + + raise cherrypy.HTTPRedirect(OAuth2.get_logout_redirect_url(token)) diff --git a/src/pybind/mgr/dashboard/controllers/saml2.py b/src/pybind/mgr/dashboard/controllers/saml2.py index c11b18a27bc7e..f834be9587ee4 100644 --- a/src/pybind/mgr/dashboard/controllers/saml2.py +++ b/src/pybind/mgr/dashboard/controllers/saml2.py @@ -37,7 +37,7 @@ def _check_python_saml(): if not python_saml_imported: raise cherrypy.HTTPError(400, 'Required library not found: `python3-saml`') try: - OneLogin_Saml2_Settings(mgr.SSO_DB.saml2.onelogin_settings) + OneLogin_Saml2_Settings(mgr.SSO_DB.config.onelogin_settings) except OneLogin_Saml2_Error: raise cherrypy.HTTPError(400, 'Single Sign-On is not configured.') @@ -46,19 +46,19 @@ def _check_python_saml(): def auth_response(self, **kwargs): Saml2._check_python_saml() req = Saml2._build_req(self._request, kwargs) - auth = OneLogin_Saml2_Auth(req, mgr.SSO_DB.saml2.onelogin_settings) + auth = OneLogin_Saml2_Auth(req, mgr.SSO_DB.config.onelogin_settings) auth.process_response() errors = auth.get_errors() if auth.is_authenticated(): JwtManager.reset_user() - username_attribute = auth.get_attribute(mgr.SSO_DB.saml2.get_username_attribute()) + username_attribute = auth.get_attribute(mgr.SSO_DB.config.get_username_attribute()) if username_attribute is None: raise cherrypy.HTTPError(400, 'SSO error - `{}` not found in auth attributes. ' 'Received attributes: {}' .format( - mgr.SSO_DB.saml2.get_username_attribute(), + mgr.SSO_DB.config.get_username_attribute(), auth.get_attributes())) username = username_attribute[0] url_prefix = prepare_url_prefix(mgr.get_module_option('url_prefix', default='')) @@ -85,21 +85,21 @@ def auth_response(self, **kwargs): @Endpoint(xml=True, version=None) def metadata(self): Saml2._check_python_saml() - saml_settings = OneLogin_Saml2_Settings(mgr.SSO_DB.saml2.onelogin_settings) + saml_settings = OneLogin_Saml2_Settings(mgr.SSO_DB.config.onelogin_settings) return saml_settings.get_sp_metadata() @Endpoint(json_response=False, version=None) def login(self): Saml2._check_python_saml() req = Saml2._build_req(self._request, {}) - auth = OneLogin_Saml2_Auth(req, mgr.SSO_DB.saml2.onelogin_settings) + auth = OneLogin_Saml2_Auth(req, mgr.SSO_DB.config.onelogin_settings) raise cherrypy.HTTPRedirect(auth.login()) @Endpoint(json_response=False, version=None) def slo(self): Saml2._check_python_saml() req = Saml2._build_req(self._request, {}) - auth = OneLogin_Saml2_Auth(req, mgr.SSO_DB.saml2.onelogin_settings) + auth = OneLogin_Saml2_Auth(req, mgr.SSO_DB.config.onelogin_settings) raise cherrypy.HTTPRedirect(auth.logout()) @Endpoint(json_response=False, version=None) @@ -107,7 +107,7 @@ def logout(self, **kwargs): # pylint: disable=unused-argument Saml2._check_python_saml() JwtManager.reset_user() - token = JwtManager.get_token_from_header() + token = JwtManager.get_token(cherrypy.request) self._delete_token_cookie(token) url_prefix = prepare_url_prefix(mgr.get_module_option('url_prefix', default='')) raise cherrypy.HTTPRedirect("{}/#/login".format(url_prefix)) diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/inventory.po.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/inventory.po.ts index 5a9abdc036c91..3da482cfe9095 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/inventory.po.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/inventory.po.ts @@ -10,13 +10,13 @@ export class InventoryPageHelper extends PageHelper { identify() { // Nothing we can do, just verify the form is there this.getFirstTableCell().click(); - cy.contains('cd-table-actions button', 'Identify').click(); - cy.get('cd-modal').within(() => { + cy.contains('[data-testid="primary-action"]', 'Identify').click(); + cy.get('cds-modal').within(() => { cy.get('#duration').select('15 minutes'); cy.get('#duration').select('10 minutes'); cy.get('cd-back-button').click(); }); - cy.get('cd-modal').should('not.exist'); + cy.get('cds-modal').should('not.exist'); cy.get(`${this.pages.index.id}`); } } diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/services.po.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/services.po.ts index 4cb5223d46ac2..c546d5cc513a5 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/services.po.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/services.po.ts @@ -100,6 +100,14 @@ export class ServicesPageHelper extends PageHelper { } break; + case 'oauth2-proxy': + cy.get('#https_address').type('localhost:8443'); + cy.get('#provider_display_name').type('provider'); + cy.get('#client_id').type('foo'); + cy.get('#client_secret').type('bar'); + cy.get('#oidc_issuer_url').type('http://127.0.0.0:8080/realms/ceph'); + break; + default: cy.get('#service_id').type('test'); unmanaged diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/05-services.e2e-spec.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/05-services.e2e-spec.ts index 0f30542f793c0..a3160625067cc 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/05-services.e2e-spec.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/05-services.e2e-spec.ts @@ -40,5 +40,23 @@ describe('Services page', () => { services.deleteService('smb.testsmb'); }); + + it('should create and delete an oauth2-proxy service', () => { + services.navigateTo('create'); + services.addService('oauth2-proxy'); + + services.checkExist('oauth2-proxy', true); + + services.deleteService('oauth2-proxy'); + }); + + it('should create and delete a mgmt-gateway service', () => { + services.navigateTo('create'); + services.addService('mgmt-gateway'); + + services.checkExist('mgmt-gateway', true); + + services.deleteService('mgmt-gateway'); + }); }); }); diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/11-inventory.e2e-spec.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/11-inventory.e2e-spec.ts new file mode 100644 index 0000000000000..0397a335d7b34 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/orchestrator/workflow/11-inventory.e2e-spec.ts @@ -0,0 +1,14 @@ +import { InventoryPageHelper } from '../../cluster/inventory.po'; + +describe('Physical Disks page', () => { + const inventory = new InventoryPageHelper(); + + beforeEach(() => { + cy.login(); + inventory.navigateTo(); + }); + + it('should identify device', () => { + inventory.identify(); + }); +}); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.html index 2578d18ab17b5..cbc11db138fb7 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-detail/cephfs-detail.component.html @@ -1,23 +1,23 @@
-
+
Ranks - Standbys - - -
- -
Pools
+ +
+ Standbys + + +
MDS performance counters diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts index 14587ebca9566..da1a3f355c737 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts @@ -2,6 +2,7 @@ import { ChangeDetectorRef, Component, Inject, OnInit, Optional } from '@angular import { AbstractControl, FormArray, FormControl, FormGroup, Validators } from '@angular/forms'; import { NgbDateStruct, NgbTimeStruct } from '@ng-bootstrap/ng-bootstrap'; import { padStart, uniq } from 'lodash'; +import moment from 'moment'; import { Observable, OperatorFunction, of, timer } from 'rxjs'; import { catchError, @@ -165,29 +166,26 @@ export class CephfsSnapshotscheduleFormComponent extends CdForm implements OnIni this.action = this.actionLabels.EDIT; this.snapScheduleService.getSnapshotSchedule(this.path, this.fsName, false).subscribe({ next: (response: SnapshotSchedule[]) => { - const first = response.find((x) => x.path === this.path); + const schedule = response.find((x) => x.path === this.path); + const offset = moment().utcOffset(); + const startDate = moment + .parseZone(schedule.start) + .utc() + .utcOffset(offset) + .local() + .format('YYYY-MM-DD HH:mm:ss'); this.snapScheduleForm.get('directory').disable(); - this.snapScheduleForm.get('directory').setValue(first.path); + this.snapScheduleForm.get('directory').setValue(schedule.path); this.snapScheduleForm.get('startDate').disable(); - this.snapScheduleForm - .get('startDate') - .setValue( - `${new Date(first.start).getUTCFullYear()}-${ - new Date(first.start).getUTCMonth() + 1 - }-${new Date(first.start).getUTCDate()} ${new Date( - first.start - ).getUTCHours()}:${new Date(first.start).getUTCMinutes()}:${new Date( - first.start - ).getUTCSeconds()}` - ); + this.snapScheduleForm.get('startDate').setValue(startDate); this.snapScheduleForm.get('repeatInterval').disable(); - this.snapScheduleForm.get('repeatInterval').setValue(first.schedule.split('')?.[0]); + this.snapScheduleForm.get('repeatInterval').setValue(schedule.schedule.split('')?.[0]); this.snapScheduleForm.get('repeatFrequency').disable(); - this.snapScheduleForm.get('repeatFrequency').setValue(first.schedule.split('')?.[1]); + this.snapScheduleForm.get('repeatFrequency').setValue(schedule.schedule.split('')?.[1]); // retention policies - first.retention && - Object.entries(first.retention).forEach(([frequency, interval], idx) => { + schedule.retention && + Object.entries(schedule.retention).forEach(([frequency, interval], idx) => { const freqKey = Object.keys(RetentionFrequency)[ Object.values(RetentionFrequency).indexOf(frequency as any) ]; diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.html index da6386c350688..bfe66c864b378 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.html @@ -1,12 +1,12 @@
-
-
+
to create a new Realm/Zone Group/Zone + + Authentication must be enabled in an active `mgtm-gateway` service to enable Single Sign-On(SSO) with `oauth2-proxy` + + + With an active mgmt-gateway service, the dashboard will continue to be served on {{currentURL}}:{{port}} and all other services will be accessible from {{currentURL}}:{{port}}/service_name + +
The name of the gateway group. @@ -292,7 +305,7 @@
-
- - - + + + +
-
-
- - + +
+ + The display name for the identity provider (IdP) in the UI. + This field is required. +
+
+ +
+ +
+ + The client ID for authenticating with the IdP. + This field is required. +
+
+ +
+ +
+
+ + + + + +
+ The client secret for authenticating with the IdP. + This field is required. +
+
+ +
+ +
+ + The URL of the OpenID Connect (OIDC) issuer. + This field is required. + Invalid url. +
+
+ +
+ +
+ + The address for HTTPS connections as [IP|Hostname]:port. + Format must be [IP|Hostname]:port and the port between 0 and 65535 +
+
+ +
+ +
+ + The URL the oauth2-proxy service will redirect to after a successful login.
+ +
+ +
+ + Comma separated list of domains to be allowed to redirect to, used for login or logout. +
+
+ + + +
+ +
+ + The entered value needs to be a number. + The value must be at least 1. + The value cannot exceed 65535. +
+
+ +
+
+ + + Enable + + Allows to enable authentication through an external Identity Provider (IdP) using Single Sign-On (SSO) + + +
+
+ +
+ + + +
+ +
+ +
+
+ +
+ Default cipher list used: https://ssl-config.mozilla.org/#server=nginx + Invalid cipher suite. Each cipher must be separated by '-' and each cipher suite must be separated by ':' +
+
+ + + + + +
+
+
+ + +
+
+
+ -
-
+ + + Modifying the default settings could lead to a weaker security configuration + + + + +
+
+
+ + + Enables mutual TLS (mTLS) between the client and the gateway server. +
+
+
+ + +
+ +
+ + + This field is required. +
+
+ + +
+ +
+ + + This field is required. +
+
+ + +
+ +
+ + + This field is required. +
+
+ + +
+ +
+ + + This field is required. +
+
+ + +
+ +
+ + + This field is required. +
+
+
diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.spec.ts index 7d3a57fad2337..cbecc908e87d4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.spec.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.spec.ts @@ -13,8 +13,21 @@ import { CephServiceService } from '~/app/shared/api/ceph-service.service'; import { PaginateObservable } from '~/app/shared/api/paginate.model'; import { CdFormGroup } from '~/app/shared/forms/cd-form-group'; import { SharedModule } from '~/app/shared/shared.module'; -import { configureTestBed, FormHelper } from '~/testing/unit-test-helper'; +import { configureTestBed, FormHelper, Mocks } from '~/testing/unit-test-helper'; import { ServiceFormComponent } from './service-form.component'; +import { PoolService } from '~/app/shared/api/pool.service'; + +// for 'nvmeof' service +const mockPools = [ + Mocks.getPool('pool-1', 1, ['cephfs']), + Mocks.getPool('rbd', 2), + Mocks.getPool('pool-2', 3) +]; +class MockPoolService { + getList() { + return of(mockPools); + } +} describe('ServiceFormComponent', () => { let component: ServiceFormComponent; @@ -25,7 +38,7 @@ describe('ServiceFormComponent', () => { configureTestBed({ declarations: [ServiceFormComponent], - providers: [NgbActiveModal], + providers: [NgbActiveModal, { provide: PoolService, useClass: MockPoolService }], imports: [ HttpClientTestingModule, NgbTypeaheadModule, @@ -389,22 +402,54 @@ x4Ea7kGVgx9kWh5XjWz9wjZvY49UKIT5ppIAWPMbLl3UpfckiuNhTA== describe('should test service nvmeof', () => { beforeEach(() => { + component.serviceType = 'nvmeof'; formHelper.setValue('service_type', 'nvmeof'); - formHelper.setValue('service_id', 'svc'); - formHelper.setValue('pool', 'xyz'); - formHelper.setValue('group', 'abc'); + component.ngOnInit(); + fixture.detectChanges(); }); - it('should submit nvmeof', () => { - component.onSubmit(); - expect(cephServiceService.create).toHaveBeenCalledWith({ - service_type: 'nvmeof', - service_id: 'svc', - placement: {}, - unmanaged: false, - pool: 'xyz', - group: 'abc' - }); + it('should set rbd pools correctly onInit', () => { + expect(component.pools.length).toBe(3); + expect(component.rbdPools.length).toBe(2); + }); + + it('should set default values correctly onInit', () => { + expect(form.get('service_type').value).toBe('nvmeof'); + expect(form.get('group').value).toBe('default'); + expect(form.get('pool').value).toBe('rbd'); + expect(form.get('service_id').value).toBe('rbd.default'); + }); + + it('should reflect correct values on group change', () => { + // Initially the group value should be 'default' + expect(component.serviceForm.get('group')?.value).toBe('default'); + const groupInput = fixture.debugElement.query(By.css('#group')).nativeElement; + // Simulate input value change + groupInput.value = 'foo'; + // Trigger the input event + groupInput.dispatchEvent(new Event('input')); + // Trigger the change event + groupInput.dispatchEvent(new Event('change')); + fixture.detectChanges(); + // Verify values after change + expect(form.get('group').value).toBe('foo'); + expect(form.get('service_id').value).toBe('rbd.foo'); + }); + + it('should reflect correct values on pool change', () => { + // Initially the pool value should be 'rbd' + expect(component.serviceForm.get('pool')?.value).toBe('rbd'); + const poolInput = fixture.debugElement.query(By.css('#pool')).nativeElement; + // Simulate input value change + poolInput.value = 'pool-2'; + // Trigger the input event + poolInput.dispatchEvent(new Event('input')); + // Trigger the change event + poolInput.dispatchEvent(new Event('change')); + fixture.detectChanges(); + // Verify values after change + expect(form.get('pool').value).toBe('pool-2'); + expect(form.get('service_id').value).toBe('pool-2.default'); }); it('should throw error when there is no service id', () => { @@ -418,6 +463,63 @@ x4Ea7kGVgx9kWh5XjWz9wjZvY49UKIT5ppIAWPMbLl3UpfckiuNhTA== it('should throw error when there is no group', () => { formHelper.expectErrorChange('group', '', 'required'); }); + + it('should hide the count element when service_type is "nvmeof"', () => { + const countEl = fixture.debugElement.query(By.css('#count')); + expect(countEl).toBeNull(); + }); + + it('should not show certs and keys field with mTLS disabled', () => { + formHelper.setValue('ssl', true); + fixture.detectChanges(); + const root_ca_cert = fixture.debugElement.query(By.css('#root_ca_cert')); + const client_cert = fixture.debugElement.query(By.css('#client_cert')); + const client_key = fixture.debugElement.query(By.css('#client_key')); + const server_cert = fixture.debugElement.query(By.css('#server_cert')); + const server_key = fixture.debugElement.query(By.css('#server_key')); + expect(root_ca_cert).toBeNull(); + expect(client_cert).toBeNull(); + expect(client_key).toBeNull(); + expect(server_cert).toBeNull(); + expect(server_key).toBeNull(); + }); + + it('should submit nvmeof without mTLS', () => { + component.onSubmit(); + expect(cephServiceService.create).toHaveBeenCalledWith({ + service_type: 'nvmeof', + service_id: 'rbd.default', + placement: {}, + unmanaged: false, + pool: 'rbd', + group: 'default', + enable_auth: false + }); + }); + + it('should submit nvmeof with mTLS', () => { + formHelper.setValue('enable_mtls', true); + formHelper.setValue('root_ca_cert', 'root_ca_cert'); + formHelper.setValue('client_cert', 'client_cert'); + formHelper.setValue('client_key', 'client_key'); + formHelper.setValue('server_cert', 'server_cert'); + formHelper.setValue('server_key', 'server_key'); + component.onSubmit(); + expect(cephServiceService.create).toHaveBeenCalledWith({ + service_type: 'nvmeof', + service_id: 'rbd.default', + placement: {}, + unmanaged: false, + pool: 'rbd', + group: 'default', + enable_auth: true, + root_ca_cert: 'root_ca_cert', + client_cert: 'client_cert', + client_key: 'client_key', + server_cert: 'server_cert', + server_key: 'server_key' + }); + }); }); describe('should test service smb', () => { @@ -650,6 +752,34 @@ x4Ea7kGVgx9kWh5XjWz9wjZvY49UKIT5ppIAWPMbLl3UpfckiuNhTA== const poolId = fixture.debugElement.query(By.css('#pool')).nativeElement; expect(poolId.disabled).toBeTruthy(); }); + + it('should not edit groups for nvmeof service', () => { + component.serviceType = 'nvmeof'; + formHelper.setValue('service_type', 'nvmeof'); + component.ngOnInit(); + fixture.detectChanges(); + const groupId = fixture.debugElement.query(By.css('#group')).nativeElement; + expect(groupId.disabled).toBeTruthy(); + }); + + it('should update nvmeof service to disable mTLS', () => { + spyOn(cephServiceService, 'update').and.stub(); + component.serviceType = 'nvmeof'; + formHelper.setValue('service_type', 'nvmeof'); + formHelper.setValue('pool', 'rbd'); + formHelper.setValue('group', 'default'); + // mTLS disabled + formHelper.setValue('enable_auth', false); + component.onSubmit(); + expect(cephServiceService.update).toHaveBeenCalledWith({ + service_type: 'nvmeof', + placement: {}, + unmanaged: false, + pool: 'rbd', + group: 'default', + enable_auth: false + }); + }); }); }); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.ts index bada177f735b4..aa50aeab0b4a5 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/service-form/service-form.component.ts @@ -4,6 +4,7 @@ import { AbstractControl, UntypedFormControl, Validators } from '@angular/forms' import { ActivatedRoute, Router } from '@angular/router'; import { NgbActiveModal, NgbModalRef, NgbTypeahead } from '@ng-bootstrap/ng-bootstrap'; +import { ListItem } from 'carbon-components-angular'; import _ from 'lodash'; import { forkJoin, merge, Observable, Subject, Subscription } from 'rxjs'; import { debounceTime, distinctUntilChanged, filter, map } from 'rxjs/operators'; @@ -24,7 +25,9 @@ import { SelectOption } from '~/app/shared/components/select/select-option.model import { ActionLabelsI18n, TimerServiceInterval, - URLVerbs + URLVerbs, + SSL_PROTOCOLS, + SSL_CIPHERS } from '~/app/shared/constants/app.constants'; import { CdForm } from '~/app/shared/forms/cd-form'; import { CdFormBuilder } from '~/app/shared/forms/cd-form-builder'; @@ -49,6 +52,9 @@ export class ServiceFormComponent extends CdForm implements OnInit { readonly SNMP_ENGINE_ID_PATTERN = /^[0-9A-Fa-f]{10,64}/g; readonly INGRESS_SUPPORTED_SERVICE_TYPES = ['rgw', 'nfs']; readonly SMB_CONFIG_URI_PATTERN = /^(http:|https:|rados:|rados:mon-config-key:)/; + readonly OAUTH2_ISSUER_URL_PATTERN = /^(https?:\/\/)?([a-zA-Z0-9-]+(\.[a-zA-Z0-9-]+)+)(:[0-9]{1,5})?(\/.*)?$/; + readonly SSL_CIPHERS_PATTERN = /^[a-zA-Z0-9\-:]+$/; + readonly DEFAULT_SSL_PROTOCOL_ITEM = [{ content: 'TLSv1.3', selected: true }]; @ViewChild(NgbTypeahead, { static: false }) typeahead: NgbTypeahead; @@ -89,6 +95,17 @@ export class ServiceFormComponent extends CdForm implements OnInit { zonegroupNames: string[]; zoneNames: string[]; smbFeaturesList = ['domain']; + currentURL: string; + port: number = 443; + sslProtocolsItems: Array = Object.values(SSL_PROTOCOLS).map((protocol) => ({ + content: protocol, + selected: true + })); + sslCiphersItems: Array = Object.values(SSL_CIPHERS).map((cipher) => ({ + content: cipher, + selected: false + })); + showMgmtGatewayMessage: boolean = false; constructor( public actionLabels: ActionLabelsI18n, @@ -195,11 +212,72 @@ export class ServiceFormComponent extends CdForm implements OnInit { ] ], group: [ - null, + 'default', CdValidators.requiredIf({ service_type: 'nvmeof' }) ], + enable_mtls: [false], + root_ca_cert: [ + null, + [ + CdValidators.composeIf( + { + service_type: 'nvmeof', + enable_mtls: true + }, + [Validators.required] + ) + ] + ], + client_cert: [ + null, + [ + CdValidators.composeIf( + { + service_type: 'nvmeof', + enable_mtls: true + }, + [Validators.required] + ) + ] + ], + client_key: [ + null, + [ + CdValidators.composeIf( + { + service_type: 'nvmeof', + enable_mtls: true + }, + [Validators.required] + ) + ] + ], + server_cert: [ + null, + [ + CdValidators.composeIf( + { + service_type: 'nvmeof', + enable_mtls: true + }, + [Validators.required] + ) + ] + ], + server_key: [ + null, + [ + CdValidators.composeIf( + { + service_type: 'nvmeof', + enable_mtls: true + }, + [Validators.required] + ) + ] + ], // RGW rgw_frontend_port: [null, [CdValidators.number(false)]], realm_name: [null], @@ -300,6 +378,18 @@ export class ServiceFormComponent extends CdForm implements OnInit { ] ], virtual_interface_networks: [null], + ssl_protocols: [this.DEFAULT_SSL_PROTOCOL_ITEM], + ssl_ciphers: [ + null, + [ + CdValidators.custom('invalidPattern', (ciphers: string) => { + if (_.isEmpty(ciphers)) { + return false; + } + return !this.SSL_CIPHERS_PATTERN.test(ciphers); + }) + ] + ], // RGW, Ingress & iSCSI ssl: [false], ssl_cert: [ @@ -328,6 +418,22 @@ export class ServiceFormComponent extends CdForm implements OnInit { ssl: true }, [Validators.required, CdValidators.pemCert()] + ), + CdValidators.composeIf( + { + service_type: 'oauth2-proxy', + unmanaged: false, + ssl: true + }, + [Validators.required, CdValidators.sslCert()] + ), + CdValidators.composeIf( + { + service_type: 'mgmt-gateway', + unmanaged: false, + ssl: false + }, + [CdValidators.sslCert()] ) ] ], @@ -341,9 +447,28 @@ export class ServiceFormComponent extends CdForm implements OnInit { ssl: true }, [Validators.required, CdValidators.sslPrivKey()] + ), + CdValidators.composeIf( + { + service_type: 'oauth2-proxy', + unmanaged: false, + ssl: true + }, + [Validators.required, CdValidators.sslPrivKey()] + ), + CdValidators.composeIf( + { + service_type: 'mgmt-gateway', + unmanaged: false, + ssl: false + }, + [CdValidators.sslPrivKey()] ) ] ], + // mgmt-gateway + enable_auth: [null], + port: [443, [CdValidators.number(false)]], // snmp-gateway snmp_version: [ null, @@ -425,7 +550,49 @@ export class ServiceFormComponent extends CdForm implements OnInit { ] ], grafana_port: [null, [CdValidators.number(false)]], - grafana_admin_password: [null] + grafana_admin_password: [null], + // oauth2-proxy + provider_display_name: [ + 'My OIDC provider', + [ + CdValidators.requiredIf({ + service_type: 'oauth2-proxy' + }) + ] + ], + client_id: [ + null, + [ + CdValidators.requiredIf({ + service_type: 'oauth2-proxy' + }) + ] + ], + client_secret: [ + null, + [ + CdValidators.requiredIf({ + service_type: 'oauth2-proxy' + }) + ] + ], + oidc_issuer_url: [ + null, + [ + CdValidators.requiredIf({ + service_type: 'oauth2-proxy' + }), + CdValidators.custom('validUrl', (url: string) => { + if (_.isEmpty(url)) { + return false; + } + return !this.OAUTH2_ISSUER_URL_PATTERN.test(url); + }) + ] + ], + https_address: [null, [CdValidators.oauthAddressTest()]], + redirect_url: [null], + allowlist_domains: [null] }); } @@ -527,6 +694,12 @@ export class ServiceFormComponent extends CdForm implements OnInit { case 'nvmeof': this.serviceForm.get('pool').setValue(response[0].spec.pool); this.serviceForm.get('group').setValue(response[0].spec.group); + this.serviceForm.get('enable_auth').setValue(response[0].spec.enable_auth); + this.serviceForm.get('root_ca_cert').setValue(response[0].spec.root_ca_cert); + this.serviceForm.get('client_cert').setValue(response[0].spec.client_cert); + this.serviceForm.get('client_key').setValue(response[0].spec.client_key); + this.serviceForm.get('server_cert').setValue(response[0].spec.server_cert); + this.serviceForm.get('server_key').setValue(response[0].spec.server_key); break; case 'rgw': this.serviceForm @@ -561,6 +734,36 @@ export class ServiceFormComponent extends CdForm implements OnInit { this.serviceForm.get('ssl_key').setValue(response[0].spec?.ssl_key); } break; + case 'mgmt-gateway': + let hrefSplitted = window.location.href.split(':'); + this.currentURL = hrefSplitted[0] + hrefSplitted[1]; + this.port = response[0].spec?.port; + + if (response[0].spec?.ssl_protocols) { + let selectedValues: Array = []; + for (const value of response[0].spec.ssl_protocols) { + selectedValues.push({ content: value, selected: true }); + } + this.serviceForm.get('ssl_protocols').setValue(selectedValues); + } + if (response[0].spec?.ssl_ciphers) { + this.serviceForm + .get('ssl_ciphers') + .setValue(response[0].spec?.ssl_ciphers.join(':')); + } + if (response[0].spec?.ssl_cert) { + this.serviceForm.get('ssl_cert').setValue(response[0].spec.ssl_certificate); + } + if (response[0].spec?.ssl_key) { + this.serviceForm.get('ssl_key').setValue(response[0].spec.ssl_certificate_key); + } + if (response[0].spec?.enable_auth) { + this.serviceForm.get('enable_auth').setValue(response[0].spec.enable_auth); + } + if (response[0].spec?.port) { + this.serviceForm.get('port').setValue(response[0].spec.port); + } + break; case 'smb': const smbSpecKeys = [ 'cluster_id', @@ -622,9 +825,55 @@ export class ServiceFormComponent extends CdForm implements OnInit { .get('grafana_admin_password') .setValue(response[0].spec.initial_admin_password); break; + case 'oauth2-proxy': + const oauth2SpecKeys = [ + 'https_address', + 'provider_display_name', + 'client_id', + 'client_secret', + 'oidc_issuer_url', + 'redirect_url', + 'allowlist_domains' + ]; + oauth2SpecKeys.forEach((key) => { + this.serviceForm.get(key).setValue(response[0].spec[key]); + }); + if (response[0].spec?.ssl) { + this.serviceForm.get('ssl_cert').setValue(response[0].spec?.ssl_cert); + this.serviceForm.get('ssl_key').setValue(response[0].spec?.ssl_key); + } } }); } + this.detectChanges(); + } + + detectChanges(): void { + const service_type = this.serviceForm.get('service_type'); + if (service_type) { + service_type.valueChanges.subscribe((value) => { + if (value === 'mgmt-gateway') { + const port = this.serviceForm.get('port'); + if (port) { + port.valueChanges.subscribe((_) => { + this.showMgmtGatewayMessage = true; + }); + } + const ssl_protocols = this.serviceForm.get('ssl_protocols'); + if (ssl_protocols) { + ssl_protocols.valueChanges.subscribe((_) => { + this.showMgmtGatewayMessage = true; + }); + } + const ssl_ciphers = this.serviceForm.get('ssl_ciphers'); + if (ssl_ciphers) { + ssl_ciphers.valueChanges.subscribe((_) => { + this.showMgmtGatewayMessage = true; + }); + } + } + }); + } } getDefaultsEntitiesForRgw( @@ -673,7 +922,6 @@ export class ServiceFormComponent extends CdForm implements OnInit { this.serviceForm.get('count').setValue(2); break; case 'iscsi': - case 'nvmeof': case 'cephfs-mirror': case 'nfs': case 'grafana': @@ -686,6 +934,8 @@ export class ServiceFormComponent extends CdForm implements OnInit { case 'jaeger-collector': case 'jaeger-query': case 'smb': + case 'oauth2-proxy': + case 'mgmt-gateway': this.serviceForm.get('count').setValue(1); break; default: @@ -778,16 +1028,25 @@ export class ServiceFormComponent extends CdForm implements OnInit { ); } - setNvmeofServiceId(): void { - const defaultRbdPool: string = this.rbdPools?.find((p: Pool) => p.pool_name === 'rbd') - ?.pool_name; - if (defaultRbdPool) { - this.serviceForm.get('pool').setValue(defaultRbdPool); + setNvmeServiceId() { + const pool = this.serviceForm.get('pool').value; + const group = this.serviceForm.get('group').value; + if (pool && group) { + this.serviceForm.get('service_id').setValue(`${pool}.${group}`); + } else if (pool) { + this.serviceForm.get('service_id').setValue(pool); + } else if (group) { + this.serviceForm.get('service_id').setValue(group); + } else { + this.serviceForm.get('service_id').setValue(null); } } - onNvmeofGroupChange(groupName: string) { - this.serviceForm.get('service_id').setValue(groupName); + setNvmeDefaultPool() { + const defaultPool = + this.rbdPools?.find((p: Pool) => p.pool_name === 'rbd')?.pool_name || + this.rbdPools?.[0].pool_name; + this.serviceForm.get('pool').setValue(defaultPool); } requiresServiceId(serviceType: string) { @@ -797,7 +1056,8 @@ export class ServiceFormComponent extends CdForm implements OnInit { setServiceId(serviceId: string): void { const requiresServiceId: boolean = this.requiresServiceId(serviceId); if (requiresServiceId && serviceId === 'nvmeof') { - this.setNvmeofServiceId(); + this.setNvmeDefaultPool(); + this.setNvmeServiceId(); } else if (requiresServiceId) { this.serviceForm.get('service_id').setValue(null); } else { @@ -817,6 +1077,14 @@ export class ServiceFormComponent extends CdForm implements OnInit { if (selectedServiceType === 'rgw') { this.setRgwFields(); } + if (selectedServiceType === 'mgmt-gateway') { + let hrefSplitted = window.location.href.split(':'); + this.currentURL = hrefSplitted[0] + hrefSplitted[1]; + // mgmt-gateway lacks HA for now + this.serviceForm.get('count').disable(); + } else { + this.serviceForm.get('count').enable(); + } } onPlacementChange(selected: string) { @@ -825,15 +1093,6 @@ export class ServiceFormComponent extends CdForm implements OnInit { } } - onBlockPoolChange() { - const selectedBlockPool = this.serviceForm.get('pool').value; - if (selectedBlockPool) { - this.serviceForm.get('service_id').setValue(selectedBlockPool); - } else { - this.serviceForm.get('service_id').setValue(null); - } - } - disableForEditing(serviceType: string) { const disableForEditKeys = ['service_type', 'service_id']; disableForEditKeys.forEach((key) => { @@ -931,6 +1190,14 @@ export class ServiceFormComponent extends CdForm implements OnInit { case 'nvmeof': serviceSpec['pool'] = values['pool']; serviceSpec['group'] = values['group']; + serviceSpec['enable_auth'] = values['enable_mtls']; + if (values['enable_mtls']) { + serviceSpec['root_ca_cert'] = values['root_ca_cert']; + serviceSpec['client_cert'] = values['client_cert']; + serviceSpec['client_key'] = values['client_key']; + serviceSpec['server_cert'] = values['server_cert']; + serviceSpec['server_key'] = values['server_key']; + } break; case 'iscsi': serviceSpec['pool'] = values['pool']; @@ -1016,12 +1283,46 @@ export class ServiceFormComponent extends CdForm implements OnInit { } serviceSpec['virtual_interface_networks'] = values['virtual_interface_networks']; break; + case 'mgmt-gateway': + serviceSpec['ssl_certificate'] = values['ssl_cert']?.trim(); + serviceSpec['ssl_certificate_key'] = values['ssl_key']?.trim(); + serviceSpec['enable_auth'] = values['enable_auth']; + serviceSpec['port'] = values['port']; + if (serviceSpec['port'] === (443 || 80)) { + // omit port default values due to issues with redirect_url on the backend + delete serviceSpec['port']; + } + serviceSpec['ssl_protocols'] = []; + if (values['ssl_protocols'] != this.DEFAULT_SSL_PROTOCOL_ITEM) { + for (const key of Object.keys(values['ssl_protocols'])) { + serviceSpec['ssl_protocols'].push(values['ssl_protocols'][key]['content']); + } + } + serviceSpec['ssl_ciphers'] = values['ssl_ciphers']?.trim().split(':'); + break; case 'grafana': serviceSpec['port'] = values['grafana_port']; serviceSpec['initial_admin_password'] = values['grafana_admin_password']; + break; + case 'oauth2-proxy': + serviceSpec['provider_display_name'] = values['provider_display_name']?.trim(); + serviceSpec['client_id'] = values['client_id']?.trim(); + serviceSpec['client_secret'] = values['client_secret']?.trim(); + serviceSpec['oidc_issuer_url'] = values['oidc_issuer_url']?.trim(); + serviceSpec['https_address'] = values['https_address']?.trim(); + serviceSpec['redirect_url'] = values['redirect_url']?.trim(); + serviceSpec['allowlist_domains'] = values['allowlist_domains'] + .split(',') + .map((domain: string) => { + return domain.trim(); + }); + if (values['ssl']) { + serviceSpec['ssl_cert'] = values['ssl_cert']?.trim(); + serviceSpec['ssl_key'] = values['ssl_key']?.trim(); + } + break; } } - this.taskWrapperService .wrapTaskAroundCall({ task: new FinishedTask(taskUrl, { diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts index 40c2e95d1e013..9242ab635aa1b 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts @@ -232,13 +232,13 @@ export class ServicesComponent extends ListWithDetails implements OnChanges, OnI pagination_obs.observable.subscribe( (services: CephServiceSpec[]) => { this.services = services; - this.count = pagination_obs.count; this.services = this.services.filter((col: any) => { if (col.service_type === 'mgmt-gateway' && col.status.running) { this.isMgmtGateway = true; } return !this.hiddenServices.includes(col.service_name); }); + this.count = this.services.length; this.isLoadingServices = false; }, () => { diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.html index b3c0089834026..dd199e6ca0a12 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/crush-rule-form-modal/crush-rule-form-modal.component.html @@ -94,8 +94,6 @@
+ + {{tooltips.device_class}} +
diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.html index 7888fa853e3dd..58f46ae230450 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.html @@ -410,8 +410,6 @@
+ + {{tooltips.crushDeviceClass}} + Available OSDs: {{deviceCount}}
diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.spec.ts index 7d0331dfe54cf..db53e32509575 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.spec.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.spec.ts @@ -563,6 +563,7 @@ describe('ErasureCodeProfileFormModalComponent', () => { ecpChange('technique', 'cauchy'); formHelper.setMultipleValues(ecp, true); formHelper.setValue('crushFailureDomain', 'osd', true); + formHelper.setValue('crushDeviceClass', 'ssd', true); submittedEcp['crush-failure-domain'] = 'osd'; submittedEcp['crush-device-class'] = 'ssd'; testCreation(); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.ts index 5982dfe24fb1f..1521ae83f1b20 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/erasure-code-profile-form/erasure-code-profile-form-modal.component.ts @@ -383,7 +383,8 @@ export class ErasureCodeProfileFormModalComponent nodes, this.form.get('crushRoot'), this.form.get('crushFailureDomain'), - this.form.get('crushDeviceClass') + this.form.get('crushDeviceClass'), + false ); this.plugins = plugins; this.names = names; diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.spec.ts index f30f954b5b652..40a0ae365a5f2 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.spec.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-details/pool-details.component.spec.ts @@ -123,6 +123,7 @@ describe('PoolDetailsComponent', () => { expectedChange( { poolDetails: { + application_metadata: ['rbd'], pg_num: 256, pg_num_target: 256, pg_placement_num: 256, diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.html index 31c54e59ebff9..ddc202152b9f4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-details/rgw-bucket-details.component.html @@ -139,26 +139,24 @@ 
{{ selection.bucket_policy | json}}
-
- Lifecycle -
- - -
- -
+ Lifecycle +
+ + +
+ 
{{selection.lifecycle | json}}
{{ (selection.lifecycle | xml) || '-'}}
diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html index 66f3ec5a5a7ab..519782b52990a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html @@ -102,7 +102,7 @@

{{ averageObjectSize | dimlessBinary}}

Multi-site needs to be configured in order to see the multi-site sync status. - Please consult the on how to configure and enable the multi-site functionality. + Please consult the  on how to configure and enable the multi-site functionality. diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.html index e9e2b5f663bd6..373ecfa462067 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/shared/device-list/device-list.component.html @@ -40,11 +40,11 @@ - {{ "" | notAvailable }} - > {{value.min | i18nPlural: translationMapping}} - < {{value.max | i18nPlural: translationMapping}} - {{value.min}} to {{value.max | i18nPlural: translationMapping}} + > {{value.min | i18nPlural: translationMapping}} + < {{value.max | i18nPlural: translationMapping}} + {{value.min}} to {{value.max | i18nPlural: translationMapping}} -
  • -
    • -
  • -
    • diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.spec.ts index b92b2ae497eb3..688c51b37090d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.spec.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.spec.ts @@ -142,7 +142,7 @@ describe('BreadcrumbsComponent', () => { router.navigateByUrl(''); }); tick(); - expect(titleService.getTitle()).toEqual('Ceph'); + expect(titleService.getTitle()).toEqual('Ceph Dashboard'); })); it('should display no breadcrumbs in page title when a page is not found', fakeAsync(() => { @@ -150,7 +150,7 @@ describe('BreadcrumbsComponent', () => { router.navigateByUrl('/error'); }); tick(); - expect(titleService.getTitle()).toEqual('Ceph'); + expect(titleService.getTitle()).toEqual('Ceph Dashboard'); })); it('should display 2 breadcrumbs in page title when navigating to hosts', fakeAsync(() => { @@ -158,7 +158,7 @@ describe('BreadcrumbsComponent', () => { router.navigateByUrl('/hosts'); }); tick(); - expect(titleService.getTitle()).toEqual('Ceph: Cluster > Hosts'); + expect(titleService.getTitle()).toEqual('Ceph Dashboard: Cluster > Hosts'); })); it('should display 3 breadcrumbs in page title when navigating to RBD Add', fakeAsync(() => { @@ -166,6 +166,6 @@ describe('BreadcrumbsComponent', () => { router.navigateByUrl('/block/rbd/add'); }); tick(); - expect(titleService.getTitle()).toEqual('Ceph: Block > Images > Add'); + expect(titleService.getTitle()).toEqual('Ceph Dashboard: Block > Images > Add'); })); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.ts index 82d69fbf5d1f5..cae2341fe6f4d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/breadcrumbs/breadcrumbs.component.ts @@ -28,6 +28,7 @@ import { ActivatedRouteSnapshot, NavigationEnd, NavigationStart, Router } from ' import { concat, from, Observable, of, Subscription } from 'rxjs'; import { distinct, filter, first, mergeMap, toArray } from 'rxjs/operators'; +import { AppConstants } from '~/app/shared/constants/app.constants'; import { BreadcrumbsResolver, IBreadcrumb } from '~/app/shared/models/breadcrumbs'; @@ -149,9 +150,9 @@ export class BreadcrumbsComponent implements OnDestroy { }) .join(' > '); if (currentLocation.length > 0) { - return `Ceph: ${currentLocation}`; + return `${AppConstants.projectName}: ${currentLocation}`; } else { - return 'Ceph'; + return AppConstants.projectName; } } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/auth.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/auth.service.ts index 8a291799235b3..c209c7ffdb292 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/auth.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/auth.service.ts @@ -42,6 +42,9 @@ export class AuthService { logout(callback: Function = null) { return this.http.post('api/auth/logout', null).subscribe((resp: any) => { this.authStorageService.remove(); + if (resp.protocol == 'oauth2') { + return window.location.replace(resp.redirect_url); + } const url = _.get(this.route.snapshot.queryParams, 'returnUrl', '/login'); this.router.navigate([url], { skipLocationChange: true }); if (callback) { diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/crush-rule.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/crush-rule.service.ts index e4e7bb6054045..8354e90381bd6 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/crush-rule.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/crush-rule.service.ts @@ -13,7 +13,7 @@ export class CrushRuleService { // Copied from /doc/rados/operations/crush-map.rst root: $localize`The name of the node under which data should be placed.`, failure_domain: $localize`The type of CRUSH nodes across which we should separate replicas.`, - device_class: $localize`The device class data should be placed on.` + device_class: $localize`The device class on which to place data.` }; constructor(private http: HttpClient) {} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/erasure-code-profile.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/erasure-code-profile.service.ts index 988a13de2a936..f61201e3ce3c2 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/erasure-code-profile.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/erasure-code-profile.service.ts @@ -91,8 +91,7 @@ export class ErasureCodeProfileService { defaults to 1. Using a value greater than one will cause a CRUSH MSR rule to be created. Must be specified if crush-num-failure-domains is specified.`, - crushDeviceClass: $localize`Restrict placement to devices of a specific class - (e.g., ssd or hdd), using the crush device class names in the CRUSH map.`, + crushDeviceClass: $localize`The device class on which to place data.`, directory: $localize`Set the directory name from which the erasure code plugin is loaded.` }; diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/crush.node.selection.class.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/crush.node.selection.class.ts index 34cebbcc8773e..ec8b05232886b 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/crush.node.selection.class.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/classes/crush.node.selection.class.ts @@ -19,6 +19,14 @@ export class CrushNodeSelectionClass { failureDomainKeys: string[] = []; devices: string[] = []; deviceCount = 0; + /** + * Handles manual or automatic update of device class. + * + * When set true, the device class form field is automatically + * updated with the first device in the list of devices. + * Otherwise, user manually selects a device class. + */ + autoDeviceUpdate: boolean = true; static searchFailureDomains( nodes: CrushNode[], @@ -120,8 +128,10 @@ export class CrushNodeSelectionClass { nodes: CrushNode[], rootControl: AbstractControl, failureControl: AbstractControl, - deviceControl: AbstractControl + deviceControl: AbstractControl, + autoDeviceUpdate: boolean = true ) { + this.autoDeviceUpdate = autoDeviceUpdate; this.nodes = nodes; this.idTree = CrushNodeSelectionClass.createIdTreeFromNodes(nodes); nodes.forEach((node) => { @@ -208,7 +218,7 @@ export class CrushNodeSelectionClass { this.devices.length === 1 ? this.devices[0] : this.getIncludedCustomValue(this.controls.device, this.devices); - this.silentSet(this.controls.device, device); + if (this.autoDeviceUpdate) this.silentSet(this.controls.device, device); this.onDeviceChange(device); } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.html index b90fedc0cf15f..199fe27e62d0d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/doc/doc.component.html @@ -1,2 +1,2 @@ {{ docText }} + target="_blank"> {{ docText }} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.html index c98ef5861156d..163890abf6bb0 100755 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/form-modal/form-modal.component.html @@ -61,6 +61,7 @@ -