NameDateSize

..25-May-202020

benchfn.c25-May-20209.6 KiB

benchfn.h25-May-20208.5 KiB

benchzstd.c25-May-202034.9 KiB

benchzstd.h25-May-20208.5 KiB

BUCK14-Mar-2018920

datagen.c25-May-20206 KiB

datagen.h25-May-20201.2 KiB

dibio.c25-May-202015.6 KiB

dibio.h25-May-20201.5 KiB

fileio.c25-May-2020111 KiB

fileio.h25-May-20206 KiB

Makefile25-May-202014.8 KiB

platform.h25-May-20208.4 KiB

README.md25-May-202016 KiB

timefn.c25-May-20204.9 KiB

timefn.h25-May-20202.4 KiB

util.c25-May-202029.3 KiB

util.h25-May-20208.3 KiB

zstd.125-May-202026.2 KiB

zstd.1.md25-May-202024.7 KiB

zstdcli.c25-May-202061.8 KiB

zstdgrep25-May-20203.8 KiB

zstdgrep.125-May-2020779

zstdgrep.1.md22-Oct-2018726

zstdless14-Nov-201730

zstdless.125-May-2020378

zstdless.1.md22-Oct-2018356

README.md

1Command Line Interface for Zstandard library
2============================================
3
4Command Line Interface (CLI) can be created using the `make` command without any additional parameters.
5There are however other Makefile targets that create different variations of CLI:
6- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd formats
7- `zstd_nolegacy` : Same as `zstd` but without support for legacy zstd formats
8- `zstd-small` : CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats
9- `zstd-compress` : version of CLI which can only compress into zstd format
10- `zstd-decompress` : version of CLI which can only decompress zstd format
11
12
13### Compilation variables
14`zstd` scope can be altered by modifying the following `make` variables :
15
16- __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected.
17  It's possible to disable multithread support, by setting `HAVE_THREAD=0`.
18  Example : `make zstd HAVE_THREAD=0`
19  It's also possible to force multithread support, using `HAVE_THREAD=1`.
20  In which case, linking stage will fail if neither `pthread` nor `windows.h` library can be found.
21  This is useful to ensure this feature is not silently disabled.
22
23- __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`.
24  Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible.
25  But older versions (< v0.8.0) produced different, incompatible, frames.
26  By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`).
27  This can be altered by modifying this compilation variable.
28  `ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0".
29  `ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on.
30  `ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format.
31  if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`.
32  Note : `zstd` only supports decoding older formats, and cannot generate any legacy format.
33
34- __HAVE_ZLIB__ : `zstd` can compress and decompress files in `.gz` format.
35  This is ordered through command `--format=gzip`.
36  Alternatively, symlinks named `gzip` or `gunzip` will mimic intended behavior.
37  `.gz` support is automatically enabled when `zlib` library is detected at build time.
38  It's possible to disable `.gz` support, by setting `HAVE_ZLIB=0`.
39  Example : `make zstd HAVE_ZLIB=0`
40  It's also possible to force compilation with zlib support, using `HAVE_ZLIB=1`.
41  In which case, linking stage will fail if `zlib` library cannot be found.
42  This is useful to prevent silent feature disabling.
43
44- __HAVE_LZMA__ : `zstd` can compress and decompress files in `.xz` and `.lzma` formats.
45  This is ordered through commands `--format=xz` and `--format=lzma` respectively.
46  Alternatively, symlinks named `xz`, `unxz`, `lzma`, or `unlzma` will mimic intended behavior.
47  `.xz` and `.lzma` support is automatically enabled when `lzma` library is detected at build time.
48  It's possible to disable `.xz` and `.lzma` support, by setting `HAVE_LZMA=0`.
49  Example : `make zstd HAVE_LZMA=0`
50  It's also possible to force compilation with lzma support, using `HAVE_LZMA=1`.
51  In which case, linking stage will fail if `lzma` library cannot be found.
52  This is useful to prevent silent feature disabling.
53
54- __HAVE_LZ4__ : `zstd` can compress and decompress files in `.lz4` formats.
55  This is ordered through commands `--format=lz4`.
56  Alternatively, symlinks named `lz4`, or `unlz4` will mimic intended behavior.
57  `.lz4` support is automatically enabled when `lz4` library is detected at build time.
58  It's possible to disable `.lz4` support, by setting `HAVE_LZ4=0` .
59  Example : `make zstd HAVE_LZ4=0`
60  It's also possible to force compilation with lz4 support, using `HAVE_LZ4=1`.
61  In which case, linking stage will fail if `lz4` library cannot be found.
62  This is useful to prevent silent feature disabling.
63
64- __ZSTD_NOBENCH__ : `zstd` cli will be compiled without its integrated benchmark module.
65  This can be useful to produce smaller binaries.
66  In this case, the corresponding unit can also be excluded from compilation target.
67
68- __ZSTD_NODICT__ : `zstd` cli will be compiled without support for the integrated dictionary builder.
69  This can be useful to produce smaller binaries.
70  In this case, the corresponding unit can also be excluded from compilation target.
71
72- __ZSTD_NOCOMPRESS__ : `zstd` cli will be compiled without support for compression.
73  The resulting binary will only be able to decompress files.
74  This can be useful to produce smaller binaries.
75  A corresponding `Makefile` target using this ability is `zstd-decompress`.
76
77- __ZSTD_NODECOMPRESS__ : `zstd` cli will be compiled without support for decompression.
78  The resulting binary will only be able to compress files.
79  This can be useful to produce smaller binaries.
80  A corresponding `Makefile` target using this ability is `zstd-compress`.
81
82- __BACKTRACE__ : `zstd` can display a stack backtrace when execution
83  generates a runtime exception. By default, this feature may be
84  degraded/disabled on some platforms unless additional compiler directives are
85  applied. When triaging a runtime issue, enabling this feature can provide
86  more context to determine the location of the fault.
87  Example : `make zstd BACKTRACE=1`
88
89
90### Aggregation of parameters
91CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`.
92
93
94### Symlink shortcuts
95It's possible to invoke `zstd` through a symlink.
96When the name of the symlink has a specific value, it triggers an associated behavior.
97- `zstdmt` : compress using all cores available on local system.
98- `zcat` : will decompress and output target file using any of the supported formats. `gzcat` and `zstdcat` are also equivalent.
99- `gzip` : if zlib support is enabled, will mimic `gzip` by compressing file using `.gz` format, removing source file by default (use `--keep` to preserve). If zlib is not supported, triggers an error.
100- `xz` : if lzma support is enabled, will mimic `xz` by compressing file using `.xz` format, removing source file by default (use `--keep` to preserve). If xz is not supported, triggers an error.
101- `lzma` : if lzma support is enabled, will mimic `lzma` by compressing file using `.lzma` format, removing source file by default (use `--keep` to preserve). If lzma is not supported, triggers an error.
102- `lz4` : if lz4 support is enabled, will mimic `lz4` by compressing file using `.lz4` format. If lz4 is not supported, triggers an error.
103- `unzstd` and `unlz4` will decompress any of the supported format.
104- `ungz`, `unxz` and `unlzma` will do the same, and will also remove source file by default (use `--keep` to preserve).
105
106
107### Dictionary builder in Command Line Interface
108Zstd offers a training mode, which can be used to tune the algorithm for a selected
109type of data, by providing it with a few samples. The result of the training is stored
110in a file selected with the `-o` option (default name is `dictionary`),
111which can be loaded before compression and decompression.
112
113Using a dictionary, the compression ratio achievable on small data improves dramatically.
114These compression gains are achieved while simultaneously providing faster compression and decompression speeds.
115Dictionary work if there is some correlation in a family of small data (there is no universal dictionary).
116Hence, deploying one dictionary per type of data will provide the greater benefits.
117Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm
118will rely more and more on previously decoded content to compress the rest of the file.
119
120Usage of the dictionary builder and created dictionaries with CLI:
121
1221. Create the dictionary : `zstd --train PathToTrainingSet/* -o dictionaryName`
1232. Compress with the dictionary: `zstd FILE -D dictionaryName`
1243. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName`
125
126
127### Benchmark in Command Line Interface
128CLI includes in-memory compression benchmark module for zstd.
129The benchmark is conducted using given filenames. The files are read into memory and joined together.
130It makes benchmark more precise as it eliminates I/O overhead.
131Multiple filenames can be supplied, as multiple parameters, with wildcards,
132or names of directories can be used as parameters with `-r` option.
133
134The benchmark measures ratio, compressed size, compression and decompression speed.
135One can select compression levels starting from `-b` and ending with `-e`.
136The `-i` parameter selects minimal time used for each of tested levels.
137
138
139### Usage of Command Line Interface
140The full list of options can be obtained with `-h` or `-H` parameter:
141```
142Usage :
143      zstd [args] [FILE(s)] [-o file]
144
145FILE    : a filename
146          with no FILE, or when FILE is - , read standard input
147Arguments :
148 -#     : # compression level (1-19, default: 3)
149 -d     : decompression
150 -D file: use `file` as Dictionary
151 -o file: result stored into `file` (only if 1 input file)
152 -f     : overwrite output without prompting and (de)compress links
153--rm    : remove source file(s) after successful de/compression
154 -k     : preserve source file(s) (default)
155 -h/-H  : display help/long help and exit
156
157Advanced arguments :
158 -V     : display Version number and exit
159 -v     : verbose mode; specify multiple times to increase verbosity
160 -q     : suppress warnings; specify twice to suppress errors too
161 -c     : force write to standard output, even if it is the console
162 -l     : print information about zstd compressed files
163--exclude-compressed:  only compress files that are not previously compressed
164--ultra : enable levels beyond 19, up to 22 (requires more memory)
165--long[=#]: enable long distance matching with given window log (default: 27)
166--fast[=#]: switch to very fast compression levels (default: 1)
167--adapt : dynamically adapt compression level to I/O conditions
168--stream-size=# : optimize compression parameters for streaming input of given number of bytes
169--size-hint=# optimize compression parameters for streaming input of approximately this size
170--target-compressed-block-size=# : make compressed block near targeted size
171 -T#    : spawns # compression threads (default: 1, 0==# cores)
172 -B#    : select size of each job (default: 0==automatic)
173--rsyncable : compress using a rsync-friendly method (-B sets block size)
174--no-dictID : don't write dictID into header (dictionary compression)
175--[no-]check : integrity check (default: enabled)
176--[no-]compress-literals : force (un)compressed literals
177 -r     : operate recursively on directories
178--output-dir-flat[=directory]: all resulting files stored into `directory`.
179--format=zstd : compress files to the .zst format (default)
180--format=gzip : compress files to the .gz format
181--test  : test compressed file integrity
182--[no-]sparse : sparse mode (default: disabled)
183 -M#    : Set a memory usage limit for decompression
184--no-progress : do not display the progress bar
185--      : All arguments after "--" are treated as files
186
187Dictionary builder :
188--train ## : create a dictionary from a training set of files
189--train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args
190--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] : use the fast cover algorithm with optional args
191--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
192 -o file : `file` is dictionary name (default: dictionary)
193--maxdict=# : limit dictionary to specified size (default: 112640)
194--dictID=# : force dictionary ID to specified value (default: random)
195
196Benchmark arguments :
197 -b#    : benchmark file(s), using # compression level (default: 3)
198 -e#    : test all compression levels from -bX to # (default: 1)
199 -i#    : minimum evaluation time in seconds (default: 3s)
200 -B#    : cut file into independent blocks of size # (default: no block)
201--priority=rt : set process priority to real-time
202```
203
204### Passing parameters through Environment Variables
205`ZSTD_CLEVEL` can be used to modify the default compression level of `zstd`
206(usually set to `3`) to another value between 1 and 19 (the "normal" range).
207This can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments.
208One such scenario is `tar --zstd`.
209As `ZSTD_CLEVEL` only replaces the default compression level,
210it can then be overridden by corresponding command line arguments.
211
212There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner.
213Using environment variables for this purpose has security implications.
214Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL`.
215
216### Long distance matching mode
217The long distance matching mode, enabled with `--long`, is designed to improve
218the compression ratio for files with long matches at a large distance (up to the
219maximum window size, `128 MiB`) while still maintaining compression speed.
220
221Enabling this mode sets the window size to `128 MiB` and thus increases the memory
222usage for both the compressor and decompressor. Performance in terms of speed is
223dependent on long matches being found. Compression speed may degrade if few long
224matches are found. Decompression speed usually improves when there are many long
225distance matches.
226
227Below are graphs comparing the compression speed, compression ratio, and
228decompression speed with and without long distance matching on an ideal use
229case: a tar of four versions of clang (versions `3.4.1`, `3.4.2`, `3.5.0`,
230`3.5.1`) with a total size of `244889600 B`. This is an ideal use case as there
231are many long distance matches within the maximum window size of `128 MiB` (each
232version is less than `128 MiB`).
233
234Compression Speed vs Ratio | Decompression Speed
235---------------------------|---------------------
236![Compression Speed vs Ratio](https://raw.githubusercontent.com/facebook/zstd/v1.3.3/doc/images/ldmCspeed.png "Compression Speed vs Ratio") | ![Decompression Speed](https://raw.githubusercontent.com/facebook/zstd/v1.3.3/doc/images/ldmDspeed.png "Decompression Speed")
237
238| Method | Compression ratio | Compression speed | Decompression speed  |
239|:-------|------------------:|-------------------------:|---------------------------:|
240| `zstd -1`  | `5.065`    | `284.8 MB/s`  | `759.3 MB/s`  |
241| `zstd -5`  | `5.826`    | `124.9 MB/s`  | `674.0 MB/s`  |
242| `zstd -10` | `6.504`    | `29.5 MB/s`   | `771.3 MB/s`  |
243| `zstd -1 --long` | `17.426` | `220.6 MB/s` | `1638.4 MB/s` |
244| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s` |
245| `zstd -10 --long`| `21.949` |  `75.6 MB/s` | `1632.6 MB/s` |
246
247On this file, the compression ratio improves significantly with minimal impact
248on compression speed, and the decompression speed doubles.
249
250On the other extreme, compressing a file with few long distance matches (such as
251the [Silesia compression corpus]) will likely lead to a deterioration in
252compression speed (for lower levels) with minimal change in compression ratio.
253
254The below table illustrates this on the [Silesia compression corpus].
255
256[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
257
258| Method | Compression ratio | Compression speed | Decompression speed  |
259|:-------|------------------:|------------------:|---------------------:|
260| `zstd -1`        | `2.878` | `231.7 MB/s`      | `594.4 MB/s`   |
261| `zstd -1 --long` | `2.929` | `106.5 MB/s`      | `517.9 MB/s`   |
262| `zstd -5`        | `3.274` | `77.1 MB/s`       | `464.2 MB/s`   |
263| `zstd -5 --long` | `3.319` | `51.7 MB/s`       | `371.9 MB/s`   |
264| `zstd -10`       | `3.523` | `16.4 MB/s`       | `489.2 MB/s`   |
265| `zstd -10 --long`| `3.566` | `16.2 MB/s`       | `415.7 MB/s`   |
266
267
268### zstdgrep
269
270`zstdgrep` is a utility which makes it possible to `grep` directly a `.zst` compressed file.
271It's used the same way as normal `grep`, for example :
272`zstdgrep pattern file.zst`
273
274`zstdgrep` is _not_ compatible with dictionary compression.
275
276To search into a file compressed with a dictionary,
277it's necessary to decompress it using `zstd` or `zstdcat`,
278and then pipe the result to `grep`. For example  :
279`zstdcat -D dictionary -qc -- file.zst | grep pattern`
280