diff --git a/CHANGELOG b/CHANGELOG index 26ba425aee2..3b15165515c 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -13,6 +13,7 @@ api: Several experimental functions have been deprecated and will emit a compile `ZSTD_initCStream_advanced()` `ZSTD_initCStream_usingCDict_advanced()` `ZSTD_resetCStream()` +api: ZSTDMT_NBWORKERS_MAX reduced to 64 for 32-bit environments (@Cyan4973) perf: Significant speed improvements for middle compression levels (#2494, @senhuang42 @terrelln) perf: Block splitter to improve compression ratio, enabled by default for high compression levels (#2447, @senhuang42) perf: Decompression loop refactor, speed improvements on `clang` and for `--long` modes (#2614 #2630, @Cyan4973) diff --git a/lib/compress/zstdmt_compress.h b/lib/compress/zstdmt_compress.h index 3d9614a7eb9..2fee2ec7455 100644 --- a/lib/compress/zstdmt_compress.h +++ b/lib/compress/zstdmt_compress.h @@ -32,10 +32,10 @@ /* === Constants === */ -#ifndef ZSTDMT_NBWORKERS_MAX /* can be modified at compile time */ -# define ZSTDMT_NBWORKERS_MAX 200 +#ifndef ZSTDMT_NBWORKERS_MAX /* a different value can be selected at compile time */ +# define ZSTDMT_NBWORKERS_MAX ((sizeof(void*)==4) /*32-bit*/ ? 64 : 256) #endif -#ifndef ZSTDMT_JOBSIZE_MIN /* can be modified at compile time */ +#ifndef ZSTDMT_JOBSIZE_MIN /* a different value can be selected at compile time */ # define ZSTDMT_JOBSIZE_MIN (512 KB) #endif #define ZSTDMT_JOBLOG_MAX (MEM_32bits() ? 29 : 30) diff --git a/programs/README.md b/programs/README.md index cf7f5ba46d1..7fd71042008 100644 --- a/programs/README.md +++ b/programs/README.md @@ -224,7 +224,8 @@ Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVE that `zstd` will use for compression, which by default is `1`. This functionality only exists when `zstd` is compiled with multithread support. `0` means "use as many threads as detected cpu cores on local system". -The max # of threads is capped at: `ZSTDMT_NBWORKERS_MAX==200`. +The max # of threads is capped at `ZSTDMT_NBWORKERS_MAX`, +which is either 64 in 32-bit mode, or 256 for 64-bit environments. This functionality can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments. One such scenario is `tar --zstd`. diff --git a/programs/zstd.1 b/programs/zstd.1 index be52d303d8b..861f9380b87 100644 --- a/programs/zstd.1 +++ b/programs/zstd.1 @@ -105,7 +105,7 @@ Display information related to a zstd compressed file, such as size, ratio, and \fB\-\-fast[=#]\fR: switch to ultra\-fast compression levels\. If \fB=#\fR is not present, it defaults to \fB1\fR\. The higher the value, the faster the compression speed, at the cost of some compression ratio\. This setting overwrites compression level if one was set previously\. Similarly, if a compression level is set after \fB\-\-fast\fR, it overrides it\. . .IP "\(bu" 4 -\fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX==200\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\. +\fB\-T#\fR, \fB\-\-threads=#\fR: Compress using \fB#\fR working threads (default: 1)\. If \fB#\fR is 0, attempt to detect and use the number of physical CPU cores\. In all cases, the nb of threads is capped to \fBZSTDMT_NBWORKERS_MAX\fR, which is either 64 in 32\-bit mode, or 256 for 64\-bit environments\. This modifier does nothing if \fBzstd\fR is compiled without multithread support\. . .IP "\(bu" 4 \fB\-\-single\-thread\fR: Does not spawn a thread for compression, use a single thread for both I/O and compression\. In this mode, compression is serialized with I/O, which is slightly slower\. (This is different from \fB\-T1\fR, which spawns 1 compression thread in parallel of I/O)\. This mode is the only one available when multithread support is disabled\. Single\-thread mode features lower memory usage\. Final compressed result is slightly different from \fB\-T1\fR\. diff --git a/programs/zstd.1.md b/programs/zstd.1.md index 9acea1b1d9f..ae50928667d 100644 --- a/programs/zstd.1.md +++ b/programs/zstd.1.md @@ -115,7 +115,8 @@ the last one takes effect. * `-T#`, `--threads=#`: Compress using `#` working threads (default: 1). If `#` is 0, attempt to detect and use the number of physical CPU cores. - In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX==200. + In all cases, the nb of threads is capped to `ZSTDMT_NBWORKERS_MAX`, + which is either 64 in 32-bit mode, or 256 for 64-bit environments. This modifier does nothing if `zstd` is compiled without multithread support. * `--single-thread`: Does not spawn a thread for compression, use a single thread for both I/O and compression.