macOS: use portable paths for brewed deps #23
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Yurii Kolesnykov <root@yurikoles.com>
|
@lundman, could you please unblock GitHub Actions for me. |
This was referenced Jan 11, 2024
lundman
pushed a commit
that referenced
this pull request
Jan 11, 2025
Signed-off-by: Yurii Kolesnykov <root@yurikoles.com>
lundman
added a commit
that referenced
this pull request
Jan 14, 2025
Signed-off-by: Jorgen Lundman <lundman@lundman.net> Set zp->z_blksz to ashift Slowdowns in zfs_log_write() can happen on large blocksize devices. Signed-off-by: Jorgen Lundman <lundman@lundman.net> MacOS: Build with n+1 cpus (#22) Signed-off-by: Andrew Innes <andrew.c12@gmail.com> macOS: use portable paths for brewed deps (#23) Signed-off-by: Yurii Kolesnykov <root@yurikoles.com> Search for C++ and Obj C compilers (#21) Signed-off-by: Andrew Innes <andrew.c12@gmail.com> cpuid uses a,b,c,d order. wikipedia paragraph referred to model string, which has a special case ordering. This fixes a bug where old intel would incorretly assume AES/qmul feature was available when it was not. (And vice-verse, but this would just skip asm versions and not panic) Signed-off-by: Jorgen Lundman <lundman@lundman.net> Redo cpuid feature detection We already have a proper cpuid feature API used in userland, but Linux took it out for KERNEL use, to plug into the Linux API. We do not have a kernel API to use, so we might as well use the userland one. This makes it nice and consistent, and more easy to read. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Use built in cpuid() where possible Fixing a panic when running sysctl -a Signed-off-by: Jorgen Lundman <lundman@lundman.net> UpstreamÂ: remove sprintf usage Signed-off-by: Jorgen Lundman <lundman@lundman.net> Add IRC notifications Signed-off-by: Jorgen Lundman <lundman@lundman.net> xcode 16 compile fixes function argument type changed, adapt ASSERTs assembly bug in clang-16 work around Signed-off-by: Jorgen Lundman <lundman@lundman.net> Reduce userland CPU starvation from low-priority I/Os Once we have successfully read from disk we may do significant CPU-work on the data obtained, depending on features like record sizes, checksums, encryption, or compression. A successful write may cause significant CPU-work to be done for a subsequent zio. Because our vdev_disk layer and its use of IOKit is fundamentally asynchronous, on some media modern linearized scrubs and resilvers may "gang up" on bursts of interactive user I/Os. Moreover, all zfs kernel threads are higher priority than the vast majority of userland threads, therefore the latter can be starved of CPU especially for a scrubbing pool which has a vdev count conmparable to the CPU core count and where data was wrtten using expensive checksums like sha256. Practically all our IOKit I/Os are asynchronous, but significant work may be done on the taskq thread, possibly right to the entry into the vdev_disk_io_intr() callback function. We dispatch "background" I/Os into a lower thread-priority and lower thread-count taskq compared to other types of zio. In the callback function itself, for these low-priority I/Os we kpreempt() before before calling zio_delay_interrupt(). For writes, this may impose a system-load-dependent delay on notifying upper layers of zfs that IOKit has moved the buffer towards the physical device, generating backpressure on subsequent writes. For reads, this kpreempt() gives another thread in the system a chance to run before we do potentially heavy-CPU actions (such as checksumming or decyrption) on the data IOKit has obtained from the storage device. Signed-off-by: Jorgen Lundman <lundman@lundman.net> iconfig fixes Signed-off-by: Jorgen Lundman <lundman@lundman.net> abd_os changes Signed-off-by: Jorgen Lundman <lundman@lundman.net> New ASSERTs Signed-off-by: Jorgen Lundman <lundman@lundman.net> new KMEM flag Signed-off-by: Jorgen Lundman <lundman@lundman.net> issig() now takes no args Signed-off-by: Jorgen Lundman <lundman@lundman.net> sprintf is deprecated Signed-off-by: Jorgen Lundman <lundman@lundman.net> Add UIO_DIRECT Signed-off-by: Jorgen Lundman <lundman@lundman.net> Re-arrange z_blksz in zfs_znode_alloc() Signed-off-by: Jorgen Lundman <lundman@lundman.net> Set zp->z_blksz to ashift Slowdowns in zfs_log_write() can happen on large blocksize devices. Signed-off-by: Jorgen Lundman <lundman@lundman.net> altool is deprecated, use notarytool Signed-off-by: Jorgen Lundman <lundman@lundman.net> pkg-macos OS friendly name after macOS name change Signed-off-by: Jorgen Lundman <lundman@lundman.net> Update the pkg installer scripts Signed-off-by: Jorgen Lundman <lundman@lundman.net> zfs-tests: support newer macOS readOnly root Since BigSur days, "/" on macOS is a readOnly, sealed archive which means zfs-tests can not mount the expected /testpool (which all scripts assume are at root). We can work around this by leveraging /etc/synthetic.conf to create "virtual directories". They can be mounted on, but not modified. This way *most* pool names are accounted for in zfs-tests (more will come). We also have to make log_must(rm /TESTPOOL) be optional. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Revert "Re-arrange z_blksz in zfs_znode_alloc()" This reverts commit 106e765. See next commit Signed-off-by: Jorgen Lundman <lundman@lundman.net> Change zfs_log_write() blocksize for performance "len" can be set to zero in the WR_INDIRECT case, if zp->z_blksz is 0. This leads to endless loop. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Revert changes to zfs_znode_alloc for reals. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Make sure blocksize is not 0 in zfs_log_write() Or the while(resid) loop will run forever. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Negative numcpus make taskq stall Handle limited CPU VMs with 1 or 2 cores. The cpus variable would go to -1 and taskq would stall forever. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Handle versions without IOSleepWithLeeway() Signed-off-by: Jorgen Lundman <lundman@lundman.net> Allow zfs_fallthrough to be defined Signed-off-by: Jorgen Lundman <lundman@lundman.net> Don't use typedefs before OSVersion workaround Signed-off-by: Jorgen Lundman <lundman@lundman.net> Handle SF_NOUNLINK on versions without it Signed-off-by: Jorgen Lundman <lundman@lundman.net> Update pkg_macos.sh to handle notarytool Signed-off-by: Jorgen Lundman <lundman@lundman.net> Compile fixes after rebase Signed-off-by: Jorgen Lundman <lundman@lundman.net> Export before removing file Due to a macOS quirk, we have to close the files in a filebased pool during some operations, which means the export -f task will fail to open the vdevs again, the pool will be suspended and zpool_export_004_pos will hang forever. By deleting the file after export, everything works as expected. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Don't hold zfs_enter() in asyncput This can deadlock during unmount, as we are already holding WRITE lock. Signed-off-by: Jorgen Lundman <lundman@lundman.net> zpool freeze can hang in spa_evicting_os_wait It appears upstream can still unmount, and export the pool - to later re-import to unfreeze. However on macOS, export will hang in spa_evicting_os_wait(). It is unclear why we hang, but it could be we go through more syncs during unmount, and that spa_freeze_txg is set to txg + TXG_SIZE Now we clear (restoring UINT64_MAX) spa_freeze_txg in the unmount() call, ensuring unmount and export works. We might not have identical "zpool freeze" as upstream, but as the zpool sources say: * 'freeze' is a vile debugging abomination, so we treat * it as such. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Do not install .in files with Makefile Signed-off-by: Jorgen Lundman <lundman@lundman.net> clone file can return ENOTSUP for fallback From Sonoma onwards, file_cmds-428 will handle ENOTSUP and fallback to copyfile. Prior to that, we favour EAGAIN as it will indicate to the users that it is a temporary failure. Signed-off-by: Jorgen Lundman <lundman@lundman.net> sprintf has been deprecated Signed-off-by: Jorgen Lundman <lundman@lundman.net> Move appveyor to .github/workflows/macos-build.yml Signed-off-by: Jorgen Lundman <lundman@lundman.net> make install would fail to mkdir first. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Fix .yml to start zed, and zfs-tests Signed-off-by: Jorgen Lundman <lundman@lundman.net> fix abd, have taskq_wait_synced() wait for threads to be created taskq_wait_synced() did a VERIFY() on whether the taskq's threads were the requested number, but taskq_create() can ultimately return early because taskq_thread_create() is allowed to return when two desired threads are created. fix this race panic. also, taskq_wait_synced() may fail if if num_ecores is nonzero (on Apple Silicon), so create a flag that lets taskq_create_common() deal with the max_ncpus. Make boot_ncpus a variable that's MAX(1, (int)max_ncores - num_ecores). boot_ncpus is used in common code. Modify the alignments and quanta/import sizes of the abd kmem and vmem cache creations. Make DEBUG builds work with KMF_LITE | KMF_BUFCTL on the abd kmem caches. Signed-off-by: Sean Doran <smd@use.net> Allow NOTARYTOOL path to be set Apple has decided that copying notarytool from new macOS to old macOS instead of supporting altool. But sadly "xcrun" will not run it as expected, so we allow NOTARYTOOL env var to be set to a path to use. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Add copy_file_range() wrapper to fcopyfile(). Signed-off-by: Jorgen Lundman <lundman@lundman.net> Compile fixes since last rebase Signed-off-by: Jorgen Lundman <lundman@lundman.net> ZIO_TYPE_IOCTL renamed to ZIO_TYPE_FLUSH And also they renamed zfs_file_fallocate() to zfs_file_deallocate() somewhere along the way. Signed-off-by: Jorgen Lundman <lundman@lundman.net> Header fix Signed-off-by: Jorgen Lundman <lundman@lundman.net> zfs_racct prototype change Signed-off-by: Jorgen Lundman <lundman@lundman.net> zfs_log_write() now takes directio boolean Signed-off-by: Jorgen Lundman <lundman@lundman.net> zfs_znode_os.c also, change strcpy back to strlcpy yet again, thanks linux. Signed-off-by: Jorgen Lundman <lundman@lundman.net> isimd_stat_init simd_stat_fini Signed-off-by: Jorgen Lundman <lundman@lundman.net> Signed-off-by: Jorgen Lundman <lundman@lundman.net> Use SUBSTFILES for .in files Signed-off-by: Jorgen Lundman <lundman@lundman.net> Compile fixes for O_DIRECTIO Signed-off-by: Jorgen Lundman <lundman@lundman.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yurii Kolesnykov root@yurikoles.com
Motivation and Context
Let's use paths relative to
${HOMEBREW_PREFIX}, so less changes would be required for bringin up AArch64 workflow, where Homebrew prefix path is/opt/homebrewinstead of/usr/local. The M1 macOS runners is alread available in a beta form.Description
How Has This Been Tested?
./autogen.sh && ./configureworks for me on my M1 Mac, it works on current Intel Mac runner as well according to this workflow run.Types of changes
Checklist:
Signed-off-by.