Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce a flag to skip comparing the local mac when raw sending #12981

Merged
merged 1 commit into from
Jan 21, 2022
Merged

Introduce a flag to skip comparing the local mac when raw sending #12981

merged 1 commit into from
Jan 21, 2022

Conversation

gamanakis
Copy link
Contributor

@gamanakis gamanakis commented Jan 16, 2022 •
edited
Loading

Motivation and Context

This is an updated version of PR #11300.
Closes #10523.
Closes #11221.
Closes #11294.
Closes #12594 .

Description

Raw receiving a snapshot back to the originating dataset is currently
impossible because of user accounting being present in the originating
dataset.

One solution would be resetting user accounting when raw receiving on
the receiving dataset. However, to recalculate it we would have to dirty
all dnodes, which may not be preferrable on big datasets.

Instead, we rely on the os_phys flag
OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is
incomplete when raw receiving. Thus, on the next mount of the receiving
dataset the local mac protecting user accounting is zeroed out.
The flag is then cleared when user accounting of the raw received
snapshot is calculated.

How Has This Been Tested?

Introduced a new test in ZTS.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
  • Documentation (a change to man pages or other documentation)

Checklist:

@gamanakis gamanakis marked this pull request as ready for review January 16, 2022 10:00
@rincebrain rincebrain added Component: Encryption "native encryption" feature Component: Send/Recv "zfs send/recv" feature labels Jan 16, 2022
@gamanakis gamanakis force-pushed the raw_sends23f_squash branch from e98cd64 to 063ab2e Compare January 16, 2022 19:32
@gamanakis
Copy link
Contributor Author

063ab2e: fix typos.

@gamanakis
Copy link
Contributor Author

@behlendorf This is an updated version of #11300. After much fiddling around with the code I realized that once we zero out the dnode structures of user accounting in os_phys, we will only be able to recalculate user accounting if we dirty all dnodes in the dataset. So I resorted back to the flag option which seems to be working fine.

@gamanakis
Copy link
Contributor Author

I will simplify this by relying on the existing _COMPLETE flags, without introducing a new one.

@gamanakis gamanakis force-pushed the raw_sends23f_squash branch from 063ab2e to 2e96c8c Compare January 18, 2022 20:53
@gamanakis
Copy link
Contributor Author

2e96c8c: Simplified to rely on existing flag OBJSET_FLAG_USERACCOUNTING_COMPLETE.

behlendorf
behlendorf approved these changes Jan 19, 2022
View reviewed changes
Copy link
Contributor

@behlendorf behlendorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating this, re-dirtying everything to update the accounting does seem like the best way to handle this since we can't rely on the zk_version.

module/os/freebsd/zfs/zio_crypt.c Outdated Show resolved Hide resolved
module/os/linux/zfs/zio_crypt.c Outdated Show resolved Hide resolved
@gamanakis gamanakis force-pushed the raw_sends23f_squash branch from 2e96c8c to 707b949 Compare January 19, 2022 08:26
@gamanakis
Copy link
Contributor Author

2e96c8c: small fixes from @nabijaczleweli and @behlendorf.

@gamanakis gamanakis force-pushed the raw_sends23f_squash branch from 707b949 to 85eb8fd Compare January 19, 2022 12:59
@gamanakis
Copy link
Contributor Author

gamanakis commented Jan 19, 2022 •
edited
Loading

85eb8fd and ef9d2d5: Fixed the awk arithmetic in the test to return user accounting in integer MB.

@gamanakis
Introduce a flag to skip comparing the local mac when raw sending
ef9d2d5 
Raw receiving a snapshot back to the originating dataset is currently
impossible because of user accounting being present in the originating
dataset.

One solution would be resetting user accounting when raw receiving on
the receiving dataset. However, to recalculate it we would have to dirty
all dnodes, which may not be preferrable on big datasets.

Instead, we rely on the os_phys flag
OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is
incomplete when raw receiving. Thus, on the next mount of the receiving
dataset the local mac protecting user accounting is zeroed out.
The flag is then cleared when user accounting of the raw received
snapshot is calculated.

Signed-off-by: George Amanakis <gamanakis@gmail.com>
@gamanakis gamanakis force-pushed the raw_sends23f_squash branch from 85eb8fd to ef9d2d5 Compare January 19, 2022 13:02
@behlendorf behlendorf added the Status: Accepted Ready to integrate (reviewed, tested) label Jan 19, 2022
@gamanakis gamanakis mentioned this pull request Jan 20, 2022
Closed
@behlendorf behlendorf merged commit 63a2645 into openzfs:master Jan 21, 2022
@gamanakis gamanakis mentioned this pull request Feb 4, 2022
Closed
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Feb 5, 2022
@gamanakis @tonyhutter
Introduce a flag to skip comparing the local mac when raw sending
e257bd4 
Raw receiving a snapshot back to the originating dataset is currently
impossible because of user accounting being present in the originating
dataset.

One solution would be resetting user accounting when raw receiving on
the receiving dataset. However, to recalculate it we would have to dirty
all dnodes, which may not be preferable on big datasets.

Instead, we rely on the os_phys flag
OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is
incomplete when raw receiving. Thus, on the next mount of the receiving
dataset the local mac protecting user accounting is zeroed out.
The flag is then cleared when user accounting of the raw received
snapshot is calculated.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes openzfs#12981 
Closes openzfs#10523
Closes openzfs#11221
Closes openzfs#11294
Closes openzfs#12594
Issue openzfs#11300
@gamanakis gamanakis mentioned this pull request Apr 10, 2022
Open
@AttilaFueloep AttilaFueloep mentioned this pull request May 30, 2022
Open
@rincebrain rincebrain mentioned this pull request Jul 31, 2022
Closed
nicman23 pushed a commit to nicman23/zfs that referenced this pull request Aug 22, 2022
@gamanakis @nicman23
Introduce a flag to skip comparing the local mac when raw sending
0fd682f 
Raw receiving a snapshot back to the originating dataset is currently
impossible because of user accounting being present in the originating
dataset.

One solution would be resetting user accounting when raw receiving on
the receiving dataset. However, to recalculate it we would have to dirty
all dnodes, which may not be preferable on big datasets.

Instead, we rely on the os_phys flag
OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is
incomplete when raw receiving. Thus, on the next mount of the receiving
dataset the local mac protecting user accounting is zeroed out.
The flag is then cleared when user accounting of the raw received
snapshot is calculated.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes openzfs#12981 
Closes openzfs#10523
Closes openzfs#11221
Closes openzfs#11294
Closes openzfs#12594
Issue openzfs#11300
nicman23 pushed a commit to nicman23/zfs that referenced this pull request Aug 22, 2022
@gamanakis @nicman23
Introduce a flag to skip comparing the local mac when raw sending
51954a4 
Raw receiving a snapshot back to the originating dataset is currently
impossible because of user accounting being present in the originating
dataset.

One solution would be resetting user accounting when raw receiving on
the receiving dataset. However, to recalculate it we would have to dirty
all dnodes, which may not be preferable on big datasets.

Instead, we rely on the os_phys flag
OBJSET_FLAG_USERACCOUNTING_COMPLETE to indicate that user accounting is
incomplete when raw receiving. Thus, on the next mount of the receiving
dataset the local mac protecting user accounting is zeroed out.
The flag is then cleared when user accounting of the raw received
snapshot is calculated.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: George Amanakis <gamanakis@gmail.com>
Closes openzfs#12981 
Closes openzfs#10523
Closes openzfs#11221
Closes openzfs#11294
Closes openzfs#12594
Issue openzfs#11300
@Andreas-Marx Andreas-Marx mentioned this pull request Nov 17, 2022
Open
bsdjhb pushed a commit to CTSRD-CHERI/zfs that referenced this pull request Jul 13, 2023
@mmatuska @bsdjhb
lz4_zfs: FreeBSD-local changes to use a custom allocator.
f514117 
This is not associated with a specific upstream commit but apparently
a local diff applied as part of:

commit e92ffd9b626833ebdbf2742c8ffddc6cd94b963e
Merge: 3c3df3660072 17b2ae0
Author: Martin Matuska <mm@FreeBSD.org>
Date:   Sat Jan 22 23:05:15 2022 +0100

    zfs: merge openzfs/zfs@17b2ae0b2 (master) into main

    Notable upstream pull request merges:
      openzfs#12766 Fix error propagation from lzc_send_redacted
      openzfs#12805 Updated the lz4 decompressor
      openzfs#12851 FreeBSD: Provide correct file generation number
      openzfs#12857 Verify dRAID empty sectors
      openzfs#12874 FreeBSD: Update argument types for VOP_READDIR
      openzfs#12896 Reduce number of arc_prune threads
      openzfs#12934 FreeBSD: Fix zvol_*_open() locking
      openzfs#12947 lz4: Cherrypick fix for CVE-2021-3520
      openzfs#12961 FreeBSD: Fix leaked strings in libspl mnttab
      openzfs#12964 Fix handling of errors from dmu_write_uio_dbuf() on FreeBSD
      openzfs#12981 Introduce a flag to skip comparing the local mac when raw sending
      openzfs#12985 Avoid memory allocations in the ARC eviction thread

    Obtained from:  OpenZFS
    OpenZFS commit: 17b2ae0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nabijaczleweli nabijaczleweli nabijaczleweli left review comments

@behlendorf behlendorf behlendorf approved these changes

Assignees
No one assigned
Labels
Component: Encryption "native encryption" feature Component: Send/Recv "zfs send/recv" feature Status: Accepted Ready to integrate (reviewed, tested)
Projects
None yet
Milestone
No milestone
4 participants
@gamanakis @behlendorf @nabijaczleweli @rincebrain