implementing json in zfs list cli command #3483
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I'm trying to reproduce the buildbot issue on ubuntu 12.10, but no success so far ! Any info or help on this one will be appreciated. In addition this distro is not supported since May 2014 : is it really of interest to keep it within builedbot process and pull request validation ? |
|
Guess git clone issue on spl for centos7.0 and ubuntu14.04 are internal errors : correct ? |
Signed-off-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf Closes openzfs#3115 Closes openzfs#3481
This is the counterpart to openzfs/spl@2345368 which replaces the cv_wait_interruptible() function with cv_wait_sig(). There is no functional change to patch merely brings the function names in to sync to maximize portability. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3450 Issue openzfs#3402
|
@yada yes, cloning is definitely an internal error. Perhaps a timeout to github due to a networking issue some some such. There are also still a handful of known issues in the buildbot hits which need to be run down, check the issues tagged Test Suite to see if it's already known. This way you can tell if it's something introduced by the patch stack. As for keeping 12.10 around it is officially no longer supported but unless it's causing issues we tend to keep them around for the additional test coverage. Specifically to ensure we're testing on a wider range of kernels. In this case the ztest failure was caused by #2302 which is a known issue in both ZoL and illumos. It would be great to fix because we hit it fairly often due to the volume of automated testing which is run. |
|
Any idea on this one : Cause it does not seems to be related to our code ? |
The Linux kernel watchdog will automatically dump a backtrace for any process while sleeps for over 120s in an uninterruptible state. The solution is for the prefetch thread to sleep in an interruptible state. The way the existing code was written this is safe because when woken it will always reevaluate its conditional. As a general rule it is preferable to sleep in an interruptible when possible. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3450 Closes openzfs#3402
sysstat's iostat omits the first report when the -y option is used. This patch adds that functionality and omits the first report with statistics since system boot. Signed-off-by: Hajo Möller <dasjoe@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3439
Kernels >= 3.12 have a NUMA-aware superblock shrinker which is used in ZoL by zfs_sb_prune(). This patch calls the shrinker for each on-line NUMA node in order that memory be freed for each one. Signed-off-by: Tim Chase <tim@chase2k.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3495
Both the 'zfs create' and 'zfs clone' commands are expected to automatically mount and share new filesystems. Since this is common functionality it has been moved in to a shared helper function. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3459
Use the 'mount' command instead of /proc/mounts to get a list of matching filesystems. This because /proc/mounts reports a pool with a space 'rpool 1' as 'rpool\0401'. The space is encoded as 3-digit octal which is legal. However 'printf "%b"', which we use to filter out other illegal characters (such as slash, space etc) can't properly interpret this because it expects 4-digit octal. We get a � instead of the space we expected. The correct value should have been 'rpool\00401' (note the additional leading zero). So use 'mount', which interprets all backslash-escapes correctly, instead. Signed-off-by: Turbo Fredriksson turbo@bayour.com Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3488
Provide '/lib/dracut-zfs-lib.sh' with utility functions. Signed-off-by: Sören Tempel <soeren+git@soeren-tempel.net> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3109
Signed-off-by: Sören Tempel <soeren+git@soeren-tempel.net> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3109
Signed-off-by: Sören Tempel <soeren+git@soeren-tempel.net> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3109
Include information about zfs-lib.sh.in and mention that it is possible to set the bootfs attribute for an entire pool. Signed-off-by: Sören Tempel <soeren+git@soeren-tempel.net> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3109
Linux 3.15 commit torvalds/linux@293bc98 introduced two new methods. The ->read_iter() and ->write_iter() methods were designed to replace the ->aio_read() and ->aio_write() interfaces. Both interfaces were preserved for several kernel releases in order to migrate all existing consumers to the new interfaces. But as of Linux 4.1 the legacy interface has been retired and the ZFS code must be updated to use the new interfaces. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3352
The number of threads in the iput taskq has been increased to speed up the number of iputs which can be handled. This has been observed to improve the meta data reclaim regardless of zfs_sb_prune() implementation in use. The taskq has also been renamed z_iput to for consistency with the rest of the I/O pipeline taskqs which are all named z_*. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com>
For kernels which do not implement a per-suberblock shrinker, those older than Linux 3.1, the shrink_dcache_parent() function was used to attempt to reclaim dentries. This was found not be entirely reliable and could lead to performance issues on older kernels running meta-data heavy workloads. To address this issue a zfs_sb_prune_aliases() function has been added to implement this functionality. It relies on traversing the list of znodes for a filesystem and adding them to a private list with a reference held. The private list can then be safely walked outside the z_znodes_lock to prune dentires and drop the last reference so the inode can be freed. This provides the same synchronous behavior as the per-filesystem shrinker and has the advantage of depending on only long standing interfaces. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Closes openzfs#3501
The metaslab allocator device selection algorithm contains a bias
mechanism whose goal is to achieve roughly equal disk space usage across
all top-level vdevs.
It seems that the initial rationale for this code was to allow newly
added (empty) vdevs to "come up to speed" faster in an attempt to make
the pool quickly converge to a steady state where all vdevs are equally
utilized.
While the code seems to work reasonably well for this use case, there
is another scenario in which this algorithm fails miserably: the case
where top-level vdevs don't have the same sizes (capacities). ZFS
allows this, and it is a good feature to have, so that users who simply
want to build a pool with the disks they happen to have lying around can
do so even if the disks have heteregenous sizes.
Here's a script that simulates a pool with two vdevs, with one 4X larger
than the other:
dd if=/dev/zero of=/tmp/d1 bs=1 count=1 seek=134217728
dd if=/dev/zero of=/tmp/d2 bs=1 count=1 seek=536870912
zpool create testspace /tmp/d1 /tmp/d2
dd if=/dev/zero of=/testspace/foobar bs=1M count=256
zpool iostat -v testspace
Before this commit, the script would output the following:
capacity
pool alloc free
---------- ----- -----
testspace 252M 375M
/tmp/d1 104M 18.5M
/tmp/d2 148M 356M
---------- ----- -----
This demonstrates that the current code handles this situation very
poorly: d1 shows 85% usage despite the pool itself being only 40% full.
d1 is quite saturated at this point, and is slowing down the entire pool
due to saturation, fragmentation and the like.
In contrast, here's the result with the code in this commit:
capacity
pool alloc free
---------- ----- -----
testspace 252M 375M
/tmp/d1 56.7M 66.3M
/tmp/d2 195M 309M
---------- ----- ------
This looks much better. d1 is 46% used, which is close to the overall
pool utilization (40%). The code still doesn't result in perfectly
balanced allocation, probably because of the way mg_bias is applied
which does not guarantee perfect accuracy, but this is still much better
than before.
Signed-off-by: Etienne Dechamps <etienne@edechamps.fr>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#3389
Signed-off-by: Etienne Dechamps <etienne@edechamps.fr> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
This seems generally useful. metaslab_aliquot is the ZFS allocation granularity, which is roughly equivalent to what is called the stripe size in traditional RAID arrays. It seems relevant to performance tuning. Signed-off-by: Etienne Dechamps <etienne@edechamps.fr> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
…pdated in json folder
…pdated in json folder
…pdated in json folder
…oring : zfs_json_t is now much more generic and it is an arg of each functions, this also really help and simplify the error function which is not generic for both standard and json outputs.
|
@behlendorf : I've rebase the branch on master , i hope your code review will be favorable , i continue to implement json in the other zpool command , some change are coming soon ! |
|
Hi, in response to this pull request Alyseo's team has refactored the code to deliver single commit including all the jsonify code, the json output has been implements in all CLI commands. |
|
Closing is favour of #3938. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, Alyseo's team has implemented a new version of zfs list during the european open zfs hackaton at Ecole 42.
Here is the second version .
Comments are welcome.