Date: Tue, 03 Sep 2019 14:07:07 -0000 From: Conrad Meyer <cem@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r346362 - in vendor/zstd/1.4.0: . contrib/adaptive-compression contrib/docker contrib/experimental_dict_builders contrib/largeNbDicts contrib/premake contrib/pzstd contrib/pzstd/utils c... Message-ID: <201904190032.x3J0WDUx090718@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: cem Date: Fri Apr 19 00:32:13 2019 New Revision: 346362 URL: https://svnweb.freebsd.org/changeset/base/346362 Log: tag zstd 1.4.0 Added: vendor/zstd/1.4.0/ - copied from r346360, vendor/zstd/dist/ vendor/zstd/1.4.0/CHANGELOG - copied unchanged from r346361, vendor/zstd/dist/CHANGELOG vendor/zstd/1.4.0/contrib/docker/ - copied from r346361, vendor/zstd/dist/contrib/docker/ vendor/zstd/1.4.0/contrib/experimental_dict_builders/ - copied from r346361, vendor/zstd/dist/contrib/experimental_dict_builders/ vendor/zstd/1.4.0/contrib/largeNbDicts/ - copied from r346361, vendor/zstd/dist/contrib/largeNbDicts/ vendor/zstd/1.4.0/contrib/premake/ - copied from r346361, vendor/zstd/dist/contrib/premake/ vendor/zstd/1.4.0/contrib/snap/ - copied from r346361, vendor/zstd/dist/contrib/snap/ vendor/zstd/1.4.0/examples/ - copied from r346361, vendor/zstd/dist/examples/ vendor/zstd/1.4.0/programs/timefn.c - copied unchanged from r346361, vendor/zstd/dist/programs/timefn.c vendor/zstd/1.4.0/programs/timefn.h - copied unchanged from r346361, vendor/zstd/dist/programs/timefn.h vendor/zstd/1.4.0/tests/fuzz/dictionary_decompress.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/dictionary_decompress.c vendor/zstd/1.4.0/tests/fuzz/dictionary_round_trip.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/dictionary_round_trip.c Replaced: vendor/zstd/1.4.0/Makefile - copied unchanged from r346361, vendor/zstd/dist/Makefile vendor/zstd/1.4.0/README.md - copied unchanged from r346361, vendor/zstd/dist/README.md vendor/zstd/1.4.0/contrib/adaptive-compression/Makefile - copied unchanged from r346361, vendor/zstd/dist/contrib/adaptive-compression/Makefile vendor/zstd/1.4.0/contrib/adaptive-compression/adapt.c - copied unchanged from r346361, vendor/zstd/dist/contrib/adaptive-compression/adapt.c vendor/zstd/1.4.0/contrib/pzstd/Makefile - copied unchanged from r346361, vendor/zstd/dist/contrib/pzstd/Makefile vendor/zstd/1.4.0/contrib/pzstd/Pzstd.cpp - copied unchanged from r346361, vendor/zstd/dist/contrib/pzstd/Pzstd.cpp vendor/zstd/1.4.0/contrib/pzstd/utils/Range.h - copied unchanged from r346361, vendor/zstd/dist/contrib/pzstd/utils/Range.h vendor/zstd/1.4.0/contrib/pzstd/utils/ResourcePool.h - copied unchanged from r346361, vendor/zstd/dist/contrib/pzstd/utils/ResourcePool.h vendor/zstd/1.4.0/doc/README.md - copied unchanged from r346361, vendor/zstd/dist/doc/README.md vendor/zstd/1.4.0/doc/educational_decoder/Makefile - copied unchanged from r346361, vendor/zstd/dist/doc/educational_decoder/Makefile vendor/zstd/1.4.0/doc/educational_decoder/zstd_decompress.c - copied unchanged from r346361, vendor/zstd/dist/doc/educational_decoder/zstd_decompress.c vendor/zstd/1.4.0/doc/zstd_manual.html - copied unchanged from r346361, vendor/zstd/dist/doc/zstd_manual.html vendor/zstd/1.4.0/lib/Makefile - copied unchanged from r346361, vendor/zstd/dist/lib/Makefile vendor/zstd/1.4.0/lib/README.md - copied unchanged from r346361, vendor/zstd/dist/lib/README.md vendor/zstd/1.4.0/lib/common/compiler.h - copied unchanged from r346361, vendor/zstd/dist/lib/common/compiler.h vendor/zstd/1.4.0/lib/common/fse.h - copied unchanged from r346361, vendor/zstd/dist/lib/common/fse.h vendor/zstd/1.4.0/lib/common/threading.c - copied unchanged from r346361, vendor/zstd/dist/lib/common/threading.c vendor/zstd/1.4.0/lib/common/xxhash.c - copied unchanged from r346361, vendor/zstd/dist/lib/common/xxhash.c vendor/zstd/1.4.0/lib/common/zstd_internal.h - copied unchanged from r346361, vendor/zstd/dist/lib/common/zstd_internal.h vendor/zstd/1.4.0/lib/compress/fse_compress.c - copied unchanged from r346361, vendor/zstd/dist/lib/compress/fse_compress.c vendor/zstd/1.4.0/lib/compress/zstd_compress.c - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstd_compress.c vendor/zstd/1.4.0/lib/compress/zstd_compress_internal.h - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstd_compress_internal.h vendor/zstd/1.4.0/lib/compress/zstd_fast.c - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstd_fast.c vendor/zstd/1.4.0/lib/compress/zstd_lazy.h - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstd_lazy.h vendor/zstd/1.4.0/lib/compress/zstd_ldm.c - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstd_ldm.c vendor/zstd/1.4.0/lib/compress/zstd_opt.c - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstd_opt.c vendor/zstd/1.4.0/lib/compress/zstdmt_compress.c - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstdmt_compress.c vendor/zstd/1.4.0/lib/compress/zstdmt_compress.h - copied unchanged from r346361, vendor/zstd/dist/lib/compress/zstdmt_compress.h vendor/zstd/1.4.0/lib/decompress/zstd_ddict.c - copied unchanged from r346361, vendor/zstd/dist/lib/decompress/zstd_ddict.c vendor/zstd/1.4.0/lib/decompress/zstd_decompress.c - copied unchanged from r346361, vendor/zstd/dist/lib/decompress/zstd_decompress.c vendor/zstd/1.4.0/lib/decompress/zstd_decompress_block.c - copied unchanged from r346361, vendor/zstd/dist/lib/decompress/zstd_decompress_block.c vendor/zstd/1.4.0/lib/decompress/zstd_decompress_internal.h - copied unchanged from r346361, vendor/zstd/dist/lib/decompress/zstd_decompress_internal.h vendor/zstd/1.4.0/lib/dictBuilder/cover.c - copied unchanged from r346361, vendor/zstd/dist/lib/dictBuilder/cover.c vendor/zstd/1.4.0/lib/dictBuilder/cover.h - copied unchanged from r346361, vendor/zstd/dist/lib/dictBuilder/cover.h vendor/zstd/1.4.0/lib/dictBuilder/fastcover.c - copied unchanged from r346361, vendor/zstd/dist/lib/dictBuilder/fastcover.c vendor/zstd/1.4.0/lib/dictBuilder/zdict.h - copied unchanged from r346361, vendor/zstd/dist/lib/dictBuilder/zdict.h vendor/zstd/1.4.0/lib/legacy/zstd_legacy.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_legacy.h vendor/zstd/1.4.0/lib/legacy/zstd_v01.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v01.c vendor/zstd/1.4.0/lib/legacy/zstd_v01.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v01.h vendor/zstd/1.4.0/lib/legacy/zstd_v02.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v02.c vendor/zstd/1.4.0/lib/legacy/zstd_v02.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v02.h vendor/zstd/1.4.0/lib/legacy/zstd_v03.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v03.c vendor/zstd/1.4.0/lib/legacy/zstd_v03.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v03.h vendor/zstd/1.4.0/lib/legacy/zstd_v04.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v04.c vendor/zstd/1.4.0/lib/legacy/zstd_v04.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v04.h vendor/zstd/1.4.0/lib/legacy/zstd_v05.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v05.c vendor/zstd/1.4.0/lib/legacy/zstd_v05.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v05.h vendor/zstd/1.4.0/lib/legacy/zstd_v06.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v06.c vendor/zstd/1.4.0/lib/legacy/zstd_v06.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v06.h vendor/zstd/1.4.0/lib/legacy/zstd_v07.c - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v07.c vendor/zstd/1.4.0/lib/legacy/zstd_v07.h - copied unchanged from r346361, vendor/zstd/dist/lib/legacy/zstd_v07.h vendor/zstd/1.4.0/lib/zstd.h - copied unchanged from r346361, vendor/zstd/dist/lib/zstd.h vendor/zstd/1.4.0/programs/Makefile - copied unchanged from r346361, vendor/zstd/dist/programs/Makefile vendor/zstd/1.4.0/programs/benchfn.c - copied unchanged from r346361, vendor/zstd/dist/programs/benchfn.c vendor/zstd/1.4.0/programs/benchfn.h - copied unchanged from r346361, vendor/zstd/dist/programs/benchfn.h vendor/zstd/1.4.0/programs/benchzstd.c - copied unchanged from r346361, vendor/zstd/dist/programs/benchzstd.c vendor/zstd/1.4.0/programs/benchzstd.h - copied unchanged from r346361, vendor/zstd/dist/programs/benchzstd.h vendor/zstd/1.4.0/programs/dibio.c - copied unchanged from r346361, vendor/zstd/dist/programs/dibio.c vendor/zstd/1.4.0/programs/fileio.c - copied unchanged from r346361, vendor/zstd/dist/programs/fileio.c vendor/zstd/1.4.0/programs/fileio.h - copied unchanged from r346361, vendor/zstd/dist/programs/fileio.h vendor/zstd/1.4.0/programs/platform.h - copied unchanged from r346361, vendor/zstd/dist/programs/platform.h vendor/zstd/1.4.0/programs/util.c - copied unchanged from r346361, vendor/zstd/dist/programs/util.c vendor/zstd/1.4.0/programs/util.h - copied unchanged from r346361, vendor/zstd/dist/programs/util.h vendor/zstd/1.4.0/programs/zstd.1 - copied unchanged from r346361, vendor/zstd/dist/programs/zstd.1 vendor/zstd/1.4.0/programs/zstd.1.md - copied unchanged from r346361, vendor/zstd/dist/programs/zstd.1.md vendor/zstd/1.4.0/programs/zstdcli.c - copied unchanged from r346361, vendor/zstd/dist/programs/zstdcli.c vendor/zstd/1.4.0/programs/zstdgrep - copied unchanged from r346361, vendor/zstd/dist/programs/zstdgrep vendor/zstd/1.4.0/tests/Makefile - copied unchanged from r346361, vendor/zstd/dist/tests/Makefile vendor/zstd/1.4.0/tests/README.md - copied unchanged from r346361, vendor/zstd/dist/tests/README.md vendor/zstd/1.4.0/tests/decodecorpus.c - copied unchanged from r346361, vendor/zstd/dist/tests/decodecorpus.c vendor/zstd/1.4.0/tests/fullbench.c - copied unchanged from r346361, vendor/zstd/dist/tests/fullbench.c vendor/zstd/1.4.0/tests/fuzz/Makefile - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/Makefile vendor/zstd/1.4.0/tests/fuzz/README.md - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/README.md vendor/zstd/1.4.0/tests/fuzz/fuzz.py - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/fuzz.py vendor/zstd/1.4.0/tests/fuzz/fuzz_helpers.h - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/fuzz_helpers.h vendor/zstd/1.4.0/tests/fuzz/regression_driver.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/regression_driver.c vendor/zstd/1.4.0/tests/fuzz/simple_round_trip.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/simple_round_trip.c vendor/zstd/1.4.0/tests/fuzz/stream_decompress.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/stream_decompress.c vendor/zstd/1.4.0/tests/fuzz/stream_round_trip.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/stream_round_trip.c vendor/zstd/1.4.0/tests/fuzz/zstd_helpers.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/zstd_helpers.c vendor/zstd/1.4.0/tests/fuzz/zstd_helpers.h - copied unchanged from r346361, vendor/zstd/dist/tests/fuzz/zstd_helpers.h vendor/zstd/1.4.0/tests/fuzzer.c - copied unchanged from r346361, vendor/zstd/dist/tests/fuzzer.c vendor/zstd/1.4.0/tests/legacy.c - copied unchanged from r346361, vendor/zstd/dist/tests/legacy.c vendor/zstd/1.4.0/tests/paramgrill.c - copied unchanged from r346361, vendor/zstd/dist/tests/paramgrill.c vendor/zstd/1.4.0/tests/playTests.sh - copied unchanged from r346361, vendor/zstd/dist/tests/playTests.sh vendor/zstd/1.4.0/tests/poolTests.c - copied unchanged from r346361, vendor/zstd/dist/tests/poolTests.c vendor/zstd/1.4.0/tests/regression/config.c - copied unchanged from r346361, vendor/zstd/dist/tests/regression/config.c vendor/zstd/1.4.0/tests/regression/method.c - copied unchanged from r346361, vendor/zstd/dist/tests/regression/method.c vendor/zstd/1.4.0/tests/regression/results.csv - copied unchanged from r346361, vendor/zstd/dist/tests/regression/results.csv vendor/zstd/1.4.0/tests/regression/test.c - copied unchanged from r346361, vendor/zstd/dist/tests/regression/test.c vendor/zstd/1.4.0/tests/roundTripCrash.c - copied unchanged from r346361, vendor/zstd/dist/tests/roundTripCrash.c vendor/zstd/1.4.0/tests/symbols.c - copied unchanged from r346361, vendor/zstd/dist/tests/symbols.c vendor/zstd/1.4.0/tests/zbufftest.c - copied unchanged from r346361, vendor/zstd/dist/tests/zbufftest.c vendor/zstd/1.4.0/tests/zstreamtest.c - copied unchanged from r346361, vendor/zstd/dist/tests/zstreamtest.c vendor/zstd/1.4.0/zlibWrapper/Makefile - copied unchanged from r346361, vendor/zstd/dist/zlibWrapper/Makefile vendor/zstd/1.4.0/zlibWrapper/README.md - copied unchanged from r346361, vendor/zstd/dist/zlibWrapper/README.md vendor/zstd/1.4.0/zlibWrapper/examples/fitblk.c - copied unchanged from r346361, vendor/zstd/dist/zlibWrapper/examples/fitblk.c vendor/zstd/1.4.0/zlibWrapper/examples/fitblk_original.c - copied unchanged from r346361, vendor/zstd/dist/zlibWrapper/examples/fitblk_original.c vendor/zstd/1.4.0/zlibWrapper/examples/zwrapbench.c - copied unchanged from r346361, vendor/zstd/dist/zlibWrapper/examples/zwrapbench.c vendor/zstd/1.4.0/zlibWrapper/gzguts.h - copied unchanged from r346361, vendor/zstd/dist/zlibWrapper/gzguts.h Copied: vendor/zstd/1.4.0/CHANGELOG (from r346361, vendor/zstd/dist/CHANGELOG) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.4.0/CHANGELOG Fri Apr 19 00:32:13 2019 (r346362, copy of r346361, vendor/zstd/dist/CHANGELOG) @@ -0,0 +1,461 @@ +v1.4.0 +perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln +api: Move the advanced API, including all functions in the staging section, to the stable section +api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress +api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter +api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter +api: Don't export ZSTDMT functions from the shared library by default +api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT +api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao +api: Fix ZSTD_decompressDCtx() corner cases with a dictionary +api: Move ZSTD_getDictID_*() functions to the stable section +api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln +api: Allow compression parameters to be set when a dictionary is used +api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called +api: Fix ZSTD_estimateCStreamSize_usingCCtxParams() +api: Setting ZSTD_d_maxWindowLog to 0 means use the default +cli: Ensure that a dictionary is not used to compress itself by @shakeelrao +cli: Add --[no-]compress-literals flag to enable or disable literal compression +doc: Update the examples to use the advanced API +doc: Explain how to transition from old streaming functions to the advanced API in the header +build: Improve the Windows release packages +build: Improve CMake build by @hjmjohnson +build: Build fixes for FreeBSD by @lwhsu +build: Remove redundant warnings by @thatsafunnyname +build: Fix tests on OpenBSD by @bket +build: Extend fuzzer build system to work with the new clang engine +build: CMake now creates the libzstd.so.1 symlink +build: Improve Menson build by @lzutao +misc: Fix symbolic link detection on FreeBSD +misc: Use physical core count for -T0 on FreeBSD by @cemeyer +misc: Fix zstd --list on truncated files by @kostmo +misc: Improve logging in debug mode by @felixhandte +misc: Add CirrusCI tests by @lwhsu +misc: Optimize dictionary memory usage in corner cases +misc: Improve the dictionary builder on small or homogeneous data +misc: Fix spelling across the repo by @jsoref + +v1.3.8 +perf: better decompression speed on large files (+7%) and cold dictionaries (+15%) +perf: slightly better compression ratio at high compression modes +api : finalized advanced API, last stage before "stable" status +api : new --rsyncable mode, by @terrelln +api : support decompression of empty frames into NULL (used to be an error) (#1385) +build: new set of macros to build a minimal size decoder, by @felixhandte +build: fix compilation on MIPS32, reported by @clbr (#1441) +build: fix compilation with multiple -arch flags, by @ryandesign +build: highly upgraded meson build, by @lzutao +build: improved buck support, by @obelisk +build: fix cmake script : can create debug build, by @pitrou +build: Makefile : grep works on both colored consoles and systems without color support +build: fixed zstd-pgo, by @bmwiedemann +cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423) +cli : --no-progress flag, preserving final summary (#1371), by @terrelln +cli : ensure destination file is not source file (#1422) +cli : clearer error messages, especially when input file not present +doc : clarified zstd_compression_format.md, by @ulikunitz +misc: fixed zstdgrep, returns 1 on failure, by @lzutao +misc: NEWS renamed as CHANGELOG, in accordance with fboss + +v1.3.7 +perf: slightly better decompression speed on clang (depending on hardware target) +fix : performance of dictionary compression for small input < 4 KB at levels 9 and 10 +build: no longer build backtrace by default in release mode; restrict further automatic mode +build: control backtrace support through build macro BACKTRACE +misc: added man pages for zstdless and zstdgrep, by @samrussell + +v1.3.6 +perf: much faster dictionary builder, by @jenniferliu +perf: faster dictionary compression on small data when using multiple contexts, by @felixhandte +perf: faster dictionary decompression when using a very large number of dictionaries simultaneously +cli : fix : does no longer overwrite destination when source does not exist (#1082) +cli : new command --adapt, for automatic compression level adaptation +api : fix : block api can be streamed with > 4 GB, reported by @catid +api : reduced ZSTD_DDict size by 2 KB +api : minimum negative compression level is defined, and can be queried using ZSTD_minCLevel(). +build: support Haiku target, by @korli +build: Read Legacy format is limited to v0.5+ by default. Can be changed at compile time with macro ZSTD_LEGACY_SUPPORT. +doc : zstd_compression_format.md updated to match wording in IETF RFC 8478 +misc: tests/paramgrill, a parameter optimizer, by @GeorgeLu97 + +v1.3.5 +perf: much faster dictionary compression, by @felixhandte +perf: small quality improvement for dictionary generation, by @terrelln +perf: slightly improved high compression levels (notably level 19) +mem : automatic memory release for long duration contexts +cli : fix : overlapLog can be manually set +cli : fix : decoding invalid lz4 frames +api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln +api : change : clarify ZSTD_CCtx_reset() vs ZSTD_CCtx_resetParameters(), by @terrelln +build: select custom libzstd scope through control macros, by @GeorgeLu97 +build: OpenBSD patch, by @bket +build: make and make all are compatible with -j +doc : clarify zstd_compression_format.md, updated for IETF RFC process +misc: pzstd compatible with reproducible compilation, by @lamby + +v1.3.4 +perf: faster speed (especially decoding speed) on recent cpus (haswell+) +perf: much better performance associating --long with multi-threading, by @terrelln +perf: better compression at levels 13-15 +cli : asynchronous compression by default, for faster experience (use --single-thread for former behavior) +cli : smoother status report in multi-threading mode +cli : added command --fast=#, for faster compression modes +cli : fix crash when not overwriting existing files, by Pádraig Brady (@pixelb) +api : `nbThreads` becomes `nbWorkers` : 1 triggers asynchronous mode +api : compression levels can be negative, for even more speed +api : ZSTD_getFrameProgression() : get precise progress status of ZSTDMT anytime +api : ZSTDMT can accept new compression parameters during compression +api : implemented all advanced dictionary decompression prototypes +build: improved meson recipe, by Shawn Landden (@shawnl) +build: VS2017 scripts, by @HaydnTrigg +misc: all /contrib projects fixed +misc: added /contrib/docker script by @gyscos + +v1.3.3 +perf: faster zstd_opt strategy (levels 16-19) +fix : bug #944 : multithreading with shared ditionary and large data, reported by @gsliepen +cli : fix : content size written in header by default +cli : fix : improved LZ4 format support, by @felixhandte +cli : new : hidden command `-S`, to benchmark multiple files while generating one result per file +api : fix : support large skippable frames, by @terrelln +api : fix : streaming interface was adding a useless 3-bytes null block to small frames +api : change : when setting `pledgedSrcSize`, use `ZSTD_CONTENTSIZE_UNKNOWN` macro value to mean "unknown" +build: fix : compilation under rhel6 and centos6, reported by @pixelb +build: added `check` target + +v1.3.2 +new : long range mode, using --long command, by Stella Lau (@stellamplau) +new : ability to generate and decode magicless frames (#591) +changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode +fix : multi-threading compression works with custom allocators +fix : ZSTD_sizeof_CStream() was over-evaluating memory usage +fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22) +fix : 32-bits build can now decode large offsets (levels 21+) +cli : added LZ4 frame support by default, by Felix Handte (@felixhandte) +cli : improved --list output +cli : new : can split input file for dictionary training, using command -B# +cli : new : clean operation artefact on Ctrl-C interruption +cli : fix : do not change /dev/null permissions when using command -t with root access, reported by @mike155 (#851) +cli : fix : write file size in header in multiple-files mode +api : added macro ZSTD_COMPRESSBOUND() for static allocation +api : experimental : new advanced decompression API +api : fix : sizeof_CCtx() used to over-estimate +build: fix : no-multithread variant compiles without pool.c dependency, reported by Mitchell Blank Jr (@mitchblank) (#819) +build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818) +example : added streaming_memory_usage +license : changed /examples license to BSD + GPLv2 +license : fix a few header files to reflect new license (#825) + +v1.3.1 +New license : BSD + GPLv2 +perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk) +perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760) +cli : improved and fixed --list command, by @ib (#772) +cli : command -vV to list supported formats, by @ib (#771) +build : fixed binary variants, reported by @svenha (#788) +build : fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (@GregSlazinski) (#718) +API exp : breaking change : ZSTD_getframeHeader() provides more information +API exp : breaking change : pinned down values of error codes +doc : fixed huffman example, by Ulrich Kunitz (@ulikunitz) +new : contrib/adaptive-compression, I/O driven compression strength, by Paul Cruz (@paulcruz74) +new : contrib/long_distance_matching, statistics by Stella Lau (@stellamplau) +updated : contrib/linux-kernel, by Nick Terrell (@terrelln) + +v1.3.0 +cli : new : `--list` command, by Paul Cruz +cli : changed : xz/lzma support enabled by default +cli : changed : `-t *` continue processing list after a decompression error +API : added : ZSTD_versionString() +API : promoted to stable status : ZSTD_getFrameContentSize(), by Sean Purcell +API exp : new advanced API : ZSTD_compress_generic(), ZSTD_CCtx_setParameter() +API exp : new : API for static or external allocation : ZSTD_initStatic?Ctx() +API exp : added : ZSTD_decompressBegin_usingDDict(), requested by Guy Riddle (#700) +API exp : clarified memory estimation / measurement functions. +API exp : changed : strongest strategy renamed ZSTD_btultra, fastest strategy ZSTD_fast set to 1 +tools : decodecorpus can generate random dictionary-compressed samples, by Paul Cruz +new : contrib/seekable_format, demo and API, by Sean Purcell +changed : contrib/linux-kernel, updated version and license, by Nick Terrell + +v1.2.0 +cli : changed : Multithreading enabled by default (use target zstd-nomt or HAVE_THREAD=0 to disable) +cli : new : command -T0 means "detect and use nb of cores", by Sean Purcell +cli : new : zstdmt symlink hardwired to `zstd -T0` +cli : new : command --threads=# (#671) +cli : changed : cover dictionary builder by default, for improved quality, by Nick Terrell +cli : new : commands --train-cover and --train-legacy, to select dictionary algorithm and parameters +cli : experimental targets `zstd4` and `xzstd4`, with support for lz4 format, by Sean Purcell +cli : fix : does not output compressed data on console +cli : fix : ignore symbolic links unless --force specified, +API : breaking change : ZSTD_createCDict_advanced(), only use compressionParameters as argument +API : added : prototypes ZSTD_*_usingCDict_advanced(), for direct control over frameParameters. +API : improved: ZSTDMT_compressCCtx() reduced memory usage +API : fix : ZSTDMT_compressCCtx() now provides srcSize in header (#634) +API : fix : src size stored in frame header is controlled at end of frame +API : fix : enforced consistent rules for pledgedSrcSize==0 (#641) +API : fix : error code "GENERIC" replaced by "dstSizeTooSmall" when appropriate +build: improved cmake script, by @Majlen +build: enabled Multi-threading support for *BSD, by Baptiste Daroussin +tools: updated Paramgrill. Command -O# provides best parameters for sample and speed target. +new : contrib/linux-kernel version, by Nick Terrell + +v1.1.4 +cli : new : can compress in *.gz format, using --format=gzip command, by Przemyslaw Skibinski +cli : new : advanced benchmark command --priority=rt +cli : fix : write on sparse-enabled file systems in 32-bits mode, by @ds77 +cli : fix : --rm remains silent when input is stdin +cli : experimental : xzstd, with support for xz/lzma decoding, by Przemyslaw Skibinski +speed : improved decompression speed in streaming mode for single shot scenarios (+5%) +memory: DDict (decompression dictionary) memory usage down from 150 KB to 20 KB +arch: 32-bits variant able to generate and decode very long matches (>32 MB), by Sean Purcell +API : new : ZSTD_findFrameCompressedSize(), ZSTD_getFrameContentSize(), ZSTD_findDecompressedSize() +API : changed : dropped support of legacy versions <= v0.3 (can be changed by modifying ZSTD_LEGACY_SUPPORT value) +build : new: meson build system in contrib/meson, by Dima Krasner +build : improved cmake script, by @Majlen +build : added -Wformat-security flag, as recommended by Padraig Brady +doc : new : educational decoder, by Sean Purcell + +v1.1.3 +cli : zstd can decompress .gz files (can be disabled with `make zstd-nogz` or `make HAVE_ZLIB=0`) +cli : new : experimental target `make zstdmt`, with multi-threading support +cli : new : improved dictionary builder "cover" (experimental), by Nick Terrell, based on prior work by Giuseppe Ottaviano. +cli : new : advanced commands for detailed parameters, by Przemyslaw Skibinski +cli : fix zstdless on Mac OS-X, by Andrew Janke +cli : fix #232 "compress non-files" +dictBuilder : improved dictionary generation quality, thanks to Nick Terrell +API : new : lib/compress/ZSTDMT_compress.h multithreading API (experimental) +API : new : ZSTD_create?Dict_byReference(), requested by Bartosz Taudul +API : new : ZDICT_finalizeDictionary() +API : fix : ZSTD_initCStream_usingCDict() properly writes dictID into frame header, by Gregory Szorc (#511) +API : fix : all symbols properly exposed in libzstd, by Nick Terrell +build : support for Solaris target, by Przemyslaw Skibinski +doc : clarified specification, by Sean Purcell + +v1.1.2 +API : streaming : decompression : changed : automatic implicit reset when chain-decoding new frames without init +API : experimental : added : dictID retrieval functions, and ZSTD_initCStream_srcSize() +API : zbuff : changed : prototypes now generate deprecation warnings +lib : improved : faster decompression speed at ultra compression settings and 32-bits mode +lib : changed : only public ZSTD_ symbols are now exposed +lib : changed : reduced usage of stack memory +lib : fixed : several corner case bugs, by Nick Terrell +cli : new : gzstd, experimental version able to decode .gz files, by Przemyslaw Skibinski +cli : new : preserve file attributes +cli : new : added zstdless and zstdgrep tools +cli : fixed : status displays total amount decoded, even for file consisting of multiple frames (like pzstd) +cli : fixed : zstdcat +zlib_wrapper : added support for gz* functions, by Przemyslaw Skibinski +install : better compatibility with FreeBSD, by Dimitry Andric +source tree : changed : zbuff source files moved to lib/deprecated + +v1.1.1 +New : command -M#, --memory=, --memlimit=, --memlimit-decompress= to limit allowed memory consumption +New : doc/zstd_manual.html, by Przemyslaw Skibinski +Improved : slightly better compression ratio at --ultra levels (>= 20) +Improved : better memory usage when using streaming compression API, thanks to @Rogier-5 report +Added : API : ZSTD_initCStream_usingCDict(), ZSTD_initDStream_usingDDict() (experimental section) +Added : example/multiple_streaming_compression.c +Changed : zstd_errors.h is now installed within /include (and replaces errors_public.h) +Updated man page +Fixed : zstd-small, zstd-compress and zstd-decompress compilation targets + +v1.1.0 +New : contrib/pzstd, parallel version of zstd, by Nick Terrell +added : NetBSD install target (#338) +Improved : speed for batches of small files +Improved : speed of zlib wrapper, by Przemyslaw Skibinski +Changed : libzstd on Windows supports legacy formats, by Christophe Chevalier +Fixed : CLI -d output to stdout by default when input is stdin (#322) +Fixed : CLI correctly detects console on Mac OS-X +Fixed : CLI supports recursive mode `-r` on Mac OS-X +Fixed : Legacy decoders use unified error codes, reported by benrg (#341), fixed by Przemyslaw Skibinski +Fixed : compatibility with OpenBSD, reported by Juan Francisco Cantero Hurtado (#319) +Fixed : compatibility with Hurd, by Przemyslaw Skibinski (#365) +Fixed : zstd-pgo, reported by octoploid (#329) + +v1.0.0 +Change Licensing, all project is now BSD, Copyright Facebook +Small decompression speed improvement +API : Streaming API supports legacy format +API : ZDICT_getDictID(), ZSTD_sizeof_{CCtx, DCtx, CStream, DStream}(), ZSTD_setDStreamParameter() +CLI supports legacy formats v0.4+ +Fixed : compression fails on certain huge files, reported by Jesse McGrew +Enhanced documentation, by Przemyslaw Skibinski + +v0.8.1 +New streaming API +Changed : --ultra now enables levels beyond 19 +Changed : -i# now selects benchmark time in second +Fixed : ZSTD_compress* can now compress > 4 GB in a single pass, reported by Nick Terrell +Fixed : speed regression on specific patterns (#272) +Fixed : support for Z_SYNC_FLUSH, by Dmitry Krot (#291) +Fixed : ICC compilation, by Przemyslaw Skibinski + +v0.8.0 +Improved : better speed on clang and gcc -O2, thanks to Eric Biggers +New : Build on FreeBSD and DragonFly, thanks to JrMarino +Changed : modified API : ZSTD_compressEnd() +Fixed : legacy mode with ZSTD_HEAPMODE=0, by Christopher Bergqvist +Fixed : premature end of frame when zero-sized raw block, reported by Eric Biggers +Fixed : large dictionaries (> 384 KB), reported by Ilona Papava +Fixed : checksum correctly checked in single-pass mode +Fixed : combined --test amd --rm, reported by Andreas M. Nilsson +Modified : minor compression level adaptations +Updated : compression format specification to v0.2.0 +changed : zstd.h moved to /lib directory + +v0.7.5 +Transition version, supporting decoding of v0.8.x + +v0.7.4 +Added : homebrew for Mac, by Daniel Cade +Added : more examples +Fixed : segfault when using small dictionaries, reported by Felix Handte +Modified : default compression level for CLI is now 3 +Updated : specification, to v0.1.1 + +v0.7.3 +New : compression format specification +New : `--` separator, stating that all following arguments are file names. Suggested by Chip Turner. +New : `ZSTD_getDecompressedSize()` +New : OpenBSD target, by Juan Francisco Cantero Hurtado +New : `examples` directory +fixed : dictBuilder using HC levels, reported by Bartosz Taudul +fixed : legacy support from ZSTD_decompress_usingDDict(), reported by Felix Handte +fixed : multi-blocks decoding with intermediate uncompressed blocks, reported by Greg Slazinski +modified : removed "mem.h" and "error_public.h" dependencies from "zstd.h" (experimental section) +modified : legacy functions no longer need magic number + +v0.7.2 +fixed : ZSTD_decompressBlock() using multiple consecutive blocks. Reported by Greg Slazinski. +fixed : potential segfault on very large files (many gigabytes). Reported by Chip Turner. +fixed : CLI displays system error message when destination file cannot be created (#231). Reported by Chip Turner. + +v0.7.1 +fixed : ZBUFF_compressEnd() called multiple times with too small `dst` buffer, reported by Christophe Chevalier +fixed : dictBuilder fails if first sample is too small, reported by Руслан Ковалёв +fixed : corruption issue, reported by cj +modified : checksum enabled by default in command line mode + +v0.7.0 +New : Support for directory compression, using `-r`, thanks to Przemyslaw Skibinski +New : Command `--rm`, to remove source file after successful de/compression +New : Visual build scripts, by Christophe Chevalier +New : Support for Sparse File-systems (do not use space for zero-filled sectors) +New : Frame checksum support +New : Support pass-through mode (when using `-df`) +API : more efficient Dictionary API : `ZSTD_compress_usingCDict()`, `ZSTD_decompress_usingDDict()` +API : create dictionary files from custom content, by Giuseppe Ottaviano +API : support for custom malloc/free functions +New : controllable Dictionary ID +New : Support for skippable frames + +v0.6.1 +New : zlib wrapper API, thanks to Przemyslaw Skibinski +New : Ability to compile compressor / decompressor separately +Changed : new lib directory structure +Fixed : Legacy codec v0.5 compatible with dictionary decompression +Fixed : Decoder corruption error (#173) +Fixed : null-string roundtrip (#176) +New : benchmark mode can select directory as input +Experimental : midipix support, VMS support + +v0.6.0 +Stronger high compression modes, thanks to Przemyslaw Skibinski +API : ZSTD_getFrameParams() provides size of decompressed content +New : highest compression modes require `--ultra` command to fully unleash their capacity +Fixed : zstd cli return error code > 0 and removes dst file artifact when decompression fails, thanks to Chip Turner + +v0.5.1 +New : Optimal parsing => Very high compression modes, thanks to Przemyslaw Skibinski +Changed : Dictionary builder integrated into libzstd and zstd cli +Changed (!) : zstd cli now uses "multiple input files" as default mode. See `zstd -h`. +Fix : high compression modes for big-endian platforms +New : zstd cli : `-t` | `--test` command + +v0.5.0 +New : dictionary builder utility +Changed : streaming & dictionary API +Improved : better compression of small data + +v0.4.7 +Improved : small compression speed improvement in HC mode +Changed : `zstd_decompress.c` has ZSTD_LEGACY_SUPPORT to 0 by default +fix : bt search bug + +v0.4.6 +fix : fast compression mode on Windows +New : cmake configuration file, thanks to Artyom Dymchenko +Improved : high compression mode on repetitive data +New : block-level API +New : ZSTD_duplicateCCtx() + +v0.4.5 +new : -m/--multiple : compress/decompress multiple files + +v0.4.4 +Fixed : high compression modes for Windows 32 bits +new : external dictionary API extended to buffered mode and accessible through command line +new : windows DLL project, thanks to Christophe Chevalier + +v0.4.3 : +new : external dictionary API +new : zstd-frugal + +v0.4.2 : +Generic minor improvements for small blocks +Fixed : big-endian compatibility, by Peter Harris (#85) + +v0.4.1 +Fixed : ZSTD_LEGACY_SUPPORT=0 build mode (reported by Luben) +removed `zstd.c` + +v0.4.0 +Command line utility compatible with high compression levels +Removed zstdhc => merged into zstd +Added : ZBUFF API (see zstd_buffered.h) +Rolling buffer support + +v0.3.6 +small blocks params + +v0.3.5 +minor generic compression improvements + +v0.3.4 +Faster fast cLevels + +v0.3.3 +Small compression ratio improvement + +v0.3.2 +Fixed Visual Studio + +v0.3.1 : +Small compression ratio improvement + +v0.3 +HC mode : compression levels 2-26 + +v0.2.2 +Fix : Visual Studio 2013 & 2015 release compilation, by Christophe Chevalier + +v0.2.1 +Fix : Read errors, advanced fuzzer tests, by Hanno Böck + +v0.2.0 +**Breaking format change** +Faster decompression speed +Can still decode v0.1 format + +v0.1.3 +fix uninitialization warning, reported by Evan Nemerson + +v0.1.2 +frame concatenation support + +v0.1.1 +fix compression bug +detects write-flush errors + +v0.1.0 +first release Copied: vendor/zstd/1.4.0/Makefile (from r346361, vendor/zstd/dist/Makefile) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.4.0/Makefile Fri Apr 19 00:32:13 2019 (r346362, copy of r346361, vendor/zstd/dist/Makefile) @@ -0,0 +1,392 @@ +# ################################################################ +# Copyright (c) 2015-present, Yann Collet, Facebook, Inc. +# All rights reserved. +# +# This source code is licensed under both the BSD-style license (found in the +# LICENSE file in the root directory of this source tree) and the GPLv2 (found +# in the COPYING file in the root directory of this source tree). +# ################################################################ + +PRGDIR = programs +ZSTDDIR = lib +BUILDIR = build +ZWRAPDIR = zlibWrapper +TESTDIR = tests +FUZZDIR = $(TESTDIR)/fuzz + +# Define nul output +VOID = /dev/null + +ifneq (,$(filter Windows%,$(OS))) +EXT =.exe +else +EXT = +endif + +## default: Build lib-release and zstd-release +.PHONY: default +default: lib-release zstd-release + +.PHONY: all +all: allmost examples manual contrib + +.PHONY: allmost +allmost: allzstd zlibwrapper + +# skip zwrapper, can't build that on alternate architectures without the proper zlib installed +.PHONY: allzstd +allzstd: lib + $(MAKE) -C $(PRGDIR) all + $(MAKE) -C $(TESTDIR) all + +.PHONY: all32 +all32: + $(MAKE) -C $(PRGDIR) zstd32 + $(MAKE) -C $(TESTDIR) all32 + +.PHONY: lib lib-release libzstd.a +lib lib-release : + @$(MAKE) -C $(ZSTDDIR) $@ + +.PHONY: zstd zstd-release +zstd zstd-release: + @$(MAKE) -C $(PRGDIR) $@ + cp $(PRGDIR)/zstd$(EXT) . + +.PHONY: zstdmt +zstdmt: + @$(MAKE) -C $(PRGDIR) $@ + cp $(PRGDIR)/zstd$(EXT) ./zstdmt$(EXT) + +.PHONY: zlibwrapper +zlibwrapper: lib + $(MAKE) -C $(ZWRAPDIR) all + +## test: run long-duration tests +.PHONY: test +DEBUGLEVEL ?= 1 +test: MOREFLAGS += -g -DDEBUGLEVEL=$(DEBUGLEVEL) -Werror +test: + MOREFLAGS="$(MOREFLAGS)" $(MAKE) -j -C $(PRGDIR) allVariants + $(MAKE) -C $(TESTDIR) $@ + +## shortest: same as `make check` +.PHONY: shortest +shortest: + $(MAKE) -C $(TESTDIR) $@ + +## check: run basic tests for `zstd` cli +.PHONY: check +check: shortest + +## examples: build all examples in `/examples` directory +.PHONY: examples +examples: lib + CPPFLAGS=-I../lib LDFLAGS=-L../lib $(MAKE) -C examples/ all + +## manual: generate API documentation in html format +.PHONY: manual +manual: + $(MAKE) -C contrib/gen_html $@ + +## man: generate man page +.PHONY: man +man: + $(MAKE) -C programs $@ + +## contrib: build all supported projects in `/contrib` directory +.PHONY: contrib +contrib: lib + $(MAKE) -C contrib/pzstd all + $(MAKE) -C contrib/seekable_format/examples all + $(MAKE) -C contrib/adaptive-compression all + $(MAKE) -C contrib/largeNbDicts all + +.PHONY: cleanTabs +cleanTabs: + cd contrib; ./cleanTabs + +.PHONY: clean +clean: + @$(MAKE) -C $(ZSTDDIR) $@ > $(VOID) + @$(MAKE) -C $(PRGDIR) $@ > $(VOID) + @$(MAKE) -C $(TESTDIR) $@ > $(VOID) + @$(MAKE) -C $(ZWRAPDIR) $@ > $(VOID) + @$(MAKE) -C examples/ $@ > $(VOID) + @$(MAKE) -C contrib/gen_html $@ > $(VOID) + @$(MAKE) -C contrib/pzstd $@ > $(VOID) + @$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID) + @$(MAKE) -C contrib/adaptive-compression $@ > $(VOID) + @$(MAKE) -C contrib/largeNbDicts $@ > $(VOID) + @$(RM) zstd$(EXT) zstdmt$(EXT) tmp* + @$(RM) -r lz4 + @echo Cleaning completed + +#------------------------------------------------------------------------------ +# make install is validated only for Linux, macOS, Hurd and some BSD targets +#------------------------------------------------------------------------------ +ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD DragonFly NetBSD MSYS_NT Haiku)) + +HOST_OS = POSIX +CMAKE_PARAMS = -DZSTD_BUILD_CONTRIB:BOOL=ON -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON -DZSTD_ZLIB_SUPPORT:BOOL=ON -DZSTD_LZMA_SUPPORT:BOOL=ON -DCMAKE_BUILD_TYPE=Release + +HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0) +EGREP_OPTIONS ?= +ifeq ($HAVE_COLORNEVER, 1) +EGREP_OPTIONS += --color=never +endif +EGREP = egrep $(EGREP_OPTIONS) + +# Print a two column output of targets and their description. To add a target description, put a +# comment in the Makefile with the format "## <TARGET>: <DESCRIPTION>". For example: +# +## list: Print all targets and their descriptions (if provided) +.PHONY: list +list: + @TARGETS=$$($(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null \ + | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' \ + | $(EGREP) -v -e '^[^[:alnum:]]' | sort); \ + { \ + printf "Target Name\tDescription\n"; \ + printf "%0.s-" {1..16}; printf "\t"; printf "%0.s-" {1..40}; printf "\n"; \ + for target in $$TARGETS; do \ + line=$$($(EGREP) "^##[[:space:]]+$$target:" $(lastword $(MAKEFILE_LIST))); \ + description=$$(echo $$line | awk '{i=index($$0,":"); print substr($$0,i+1)}' | xargs); \ + printf "$$target\t$$description\n"; \ + done \ + } | column -t -s $$'\t' + +.PHONY: install armtest usan asan uasan +install: + @$(MAKE) -C $(ZSTDDIR) $@ + @$(MAKE) -C $(PRGDIR) $@ + +.PHONY: uninstall +uninstall: + @$(MAKE) -C $(ZSTDDIR) $@ + @$(MAKE) -C $(PRGDIR) $@ + +.PHONY: travis-install +travis-install: + $(MAKE) install PREFIX=~/install_test_dir + +.PHONY: gcc5build +gcc5build: clean + gcc-5 -v + CC=gcc-5 $(MAKE) all MOREFLAGS="-Werror" + +.PHONY: gcc6build +gcc6build: clean + gcc-6 -v + CC=gcc-6 $(MAKE) all MOREFLAGS="-Werror" + +.PHONY: gcc7build +gcc7build: clean + gcc-7 -v + CC=gcc-7 $(MAKE) all MOREFLAGS="-Werror" + +.PHONY: clangbuild +clangbuild: clean + clang -v + CXX=clang++ CC=clang CFLAGS="-Werror -Wconversion -Wno-sign-conversion -Wdocumentation" $(MAKE) all + +m32build: clean + gcc -v + $(MAKE) all32 + +armbuild: clean + CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allzstd + +aarch64build: clean + CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allzstd + +ppcbuild: clean + CC=powerpc-linux-gnu-gcc CFLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd + +ppc64build: clean + CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allzstd + +armfuzz: clean + CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +aarch64fuzz: clean + ld -v + CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +ppcfuzz: clean + CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +ppc64fuzz: clean + CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest + +.PHONY: cxxtest +cxxtest: CXXFLAGS += -Wall -Wextra -Wundef -Wshadow -Wcast-align -Werror +cxxtest: clean + $(MAKE) -C $(PRGDIR) all CC="$(CXX) -Wno-deprecated" CFLAGS="$(CXXFLAGS)" # adding -Wno-deprecated to avoid clang++ warning on dealing with C files directly + +gcc5test: clean + gcc-5 -v + $(MAKE) all CC=gcc-5 MOREFLAGS="-Werror" + +gcc6test: clean + gcc-6 -v + $(MAKE) all CC=gcc-6 MOREFLAGS="-Werror" + +armtest: clean + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static ZSTDRTTEST= MOREFLAGS="-Werror -static" FUZZER_FLAGS=--no-big-tests + +aarch64test: + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static ZSTDRTTEST= MOREFLAGS="-Werror -static" FUZZER_FLAGS=--no-big-tests + +ppctest: clean + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static" FUZZER_FLAGS=--no-big-tests + +ppc64test: clean + $(MAKE) -C $(TESTDIR) datagen # use native, faster + $(MAKE) -C $(TESTDIR) test CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static" FUZZER_FLAGS=--no-big-tests + +arm-ppc-compilation: + $(MAKE) -C $(PRGDIR) clean zstd CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static ZSTDRTTEST= MOREFLAGS="-Werror -static" + $(MAKE) -C $(PRGDIR) clean zstd CC=aarch64-linux-gnu-gcc QEMU_SYS=qemu-aarch64-static ZSTDRTTEST= MOREFLAGS="-Werror -static" + $(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc-static ZSTDRTTEST= MOREFLAGS="-Werror -Wno-attributes -static" + $(MAKE) -C $(PRGDIR) clean zstd CC=powerpc-linux-gnu-gcc QEMU_SYS=qemu-ppc64-static ZSTDRTTEST= MOREFLAGS="-m64 -static" + +regressiontest: + $(MAKE) -C $(FUZZDIR) regressiontest + +uasanregressiontest: + $(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=address,undefined" CXXFLAGS="-O3 -fsanitize=address,undefined" + +msanregressiontest: + $(MAKE) -C $(FUZZDIR) regressiontest CC=clang CXX=clang++ CFLAGS="-O3 -fsanitize=memory" CXXFLAGS="-O3 -fsanitize=memory" + +# run UBsan with -fsanitize-recover=signed-integer-overflow +# due to a bug in UBsan when doing pointer subtraction +# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=63303 + +usan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=undefined -Werror" + +asan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=address -Werror" + +asan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=address -Werror" $(MAKE) -C $(TESTDIR) $* + +msan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fsanitize=memory -fno-omit-frame-pointer -Werror" HAVE_LZMA=0 # datagen.c fails this test for no obvious reason + +msan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=memory -fno-omit-frame-pointer -Werror" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) HAVE_LZMA=0 $* + +asan32: clean + $(MAKE) -C $(TESTDIR) test32 CC=clang MOREFLAGS="-g -fsanitize=address" + +uasan: clean + $(MAKE) test CC=clang MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=address,undefined -Werror" + +uasan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize-recover=signed-integer-overflow -fsanitize=address,undefined -Werror" $(MAKE) -C $(TESTDIR) $* + +tsan-%: clean + LDFLAGS=-fuse-ld=gold MOREFLAGS="-g -fno-sanitize-recover=all -fsanitize=thread -Werror" $(MAKE) -C $(TESTDIR) $* FUZZER_FLAGS=--no-big-tests + +apt-install: + sudo apt-get -yq --no-install-suggests --no-install-recommends --force-yes install $(APT_PACKAGES) + +apt-add-repo: + sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test + sudo apt-get update -y -qq + +ppcinstall: + APT_PACKAGES="qemu-system-ppc qemu-user-static gcc-powerpc-linux-gnu" $(MAKE) apt-install + +arminstall: + APT_PACKAGES="qemu-system-arm qemu-user-static gcc-arm-linux-gnueabi libc6-dev-armel-cross gcc-aarch64-linux-gnu libc6-dev-arm64-cross" $(MAKE) apt-install + +valgrindinstall: + APT_PACKAGES="valgrind" $(MAKE) apt-install + +libc6install: + APT_PACKAGES="libc6-dev-i386 gcc-multilib" $(MAKE) apt-install + +gcc6install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-6 gcc-6-multilib" $(MAKE) apt-install + +gcc7install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-7 gcc-7-multilib" $(MAKE) apt-install + +gcc8install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 gcc-multilib gcc-8 gcc-8-multilib" $(MAKE) apt-install + +gpp6install: apt-add-repo + APT_PACKAGES="libc6-dev-i386 g++-multilib gcc-6 g++-6 g++-6-multilib" $(MAKE) apt-install + +clang38install: + APT_PACKAGES="clang-3.8" $(MAKE) apt-install + +# Ubuntu 14.04 ships a too-old lz4 +lz4install: + [ -e lz4 ] || git clone https://github.com/lz4/lz4 && sudo $(MAKE) -C lz4 install + +endif + + +ifneq (,$(filter MSYS%,$(shell uname))) +HOST_OS = MSYS +CMAKE_PARAMS = -G"MSYS Makefiles" -DZSTD_MULTITHREAD_SUPPORT:BOOL=OFF -DZSTD_BUILD_STATIC:BOOL=ON -DZSTD_BUILD_TESTS:BOOL=ON +endif + + +#------------------------------------------------------------------------ +# target specific tests +#------------------------------------------------------------------------ +ifneq (,$(filter $(HOST_OS),MSYS POSIX)) +cmakebuild: + cmake --version + $(RM) -r $(BUILDIR)/cmake/build + mkdir $(BUILDIR)/cmake/build + cd $(BUILDIR)/cmake/build ; cmake -DCMAKE_INSTALL_PREFIX:PATH=~/install_test_dir $(CMAKE_PARAMS) .. ; $(MAKE) install ; $(MAKE) uninstall + +c90build: clean + $(CC) -v + CFLAGS="-std=c90 -Werror" $(MAKE) allmost # will fail, due to missing support for `long long` + +gnu90build: clean + $(CC) -v + CFLAGS="-std=gnu90 -Werror" $(MAKE) allmost + +c99build: clean + $(CC) -v + CFLAGS="-std=c99 -Werror" $(MAKE) allmost + +gnu99build: clean + $(CC) -v + CFLAGS="-std=gnu99 -Werror" $(MAKE) allmost + +c11build: clean + $(CC) -v + CFLAGS="-std=c11 -Werror" $(MAKE) allmost + +bmix64build: clean + $(CC) -v + CFLAGS="-O3 -mbmi -Werror" $(MAKE) -C $(TESTDIR) test + +bmix32build: clean + $(CC) -v + CFLAGS="-O3 -mbmi -mx32 -Werror" $(MAKE) -C $(TESTDIR) test + +bmi32build: clean + $(CC) -v + CFLAGS="-O3 -mbmi -m32 -Werror" $(MAKE) -C $(TESTDIR) test + +# static analyzer test uses clang's scan-build +# does not analyze zlibWrapper, due to detected issues in zlib source code +staticAnalyze: SCANBUILD ?= scan-build +staticAnalyze: + $(CC) -v + CC=$(CC) CPPFLAGS=-g $(SCANBUILD) --status-bugs -v $(MAKE) allzstd examples contrib +endif Copied: vendor/zstd/1.4.0/README.md (from r346361, vendor/zstd/dist/README.md) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/zstd/1.4.0/README.md Fri Apr 19 00:32:13 2019 (r346362, copy of r346361, vendor/zstd/dist/README.md) @@ -0,0 +1,170 @@ +<p align="center"><img src="https://raw.githubusercontent.com/facebook/zstd/dev/doc/images/zstd_logo86.png" alt="Zstandard"></p> + +__Zstandard__, or `zstd` as short version, is a fast lossless compression algorithm, +targeting real-time compression scenarios at zlib-level and better compression ratios. +It's backed by a very fast entropy stage, provided by [Huff0 and FSE library](https://github.com/Cyan4973/FiniteStateEntropy). + +The project is provided as an open-source dual [BSD](LICENSE) and [GPLv2](COPYING) licensed **C** library, +and a command line utility producing and decoding `.zst`, `.gz`, `.xz` and `.lz4` files. +Should your project require another programming language, +a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages). + +**Development branch status:** + +[![Build Status][travisDevBadge]][travisLink] +[![Build status][AppveyorDevBadge]][AppveyorLink] +[![Build status][CircleDevBadge]][CircleLink] +[![Build status][CirrusDevBadge]][CirrusLink] + +[travisDevBadge]: https://travis-ci.org/facebook/zstd.svg?branch=dev "Continuous Integration test suite" +[travisLink]: https://travis-ci.org/facebook/zstd +[AppveyorDevBadge]: https://ci.appveyor.com/api/projects/status/xt38wbdxjk5mrbem/branch/dev?svg=true "Windows test suite" +[AppveyorLink]: https://ci.appveyor.com/project/YannCollet/zstd-p0yf0 +[CircleDevBadge]: https://circleci.com/gh/facebook/zstd/tree/dev.svg?style=shield "Short test suite" +[CircleLink]: https://circleci.com/gh/facebook/zstd +[CirrusDevBadge]: https://api.cirrus-ci.com/github/facebook/zstd.svg?branch=dev +[CirrusLink]: https://cirrus-ci.com/github/facebook/zstd + +## Benchmarks + +For reference, several fast compression algorithms were tested and compared +on a server running Arch Linux (`Linux version 5.0.5-arch1-1`), +with a Core i9-9900K CPU @ 5.0GHz, +using [lzbench], an open-source in-memory benchmark by @inikep +compiled with [gcc] 8.2.1, +on the [Silesia compression corpus]. + +[lzbench]: https://github.com/inikep/lzbench +[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia +[gcc]: https://gcc.gnu.org/ + +| Compressor name | Ratio | Compression| Decompress.| +| --------------- | ------| -----------| ---------- | +| **zstd 1.4.0 -1** | 2.884 | 530 MB/s | 1360 MB/s | +| zlib 1.2.11 -1 | 2.743 | 110 MB/s | 440 MB/s | +| brotli 1.0.7 -0 | 2.701 | 430 MB/s | 470 MB/s | +| quicklz 1.5.0 -1 | 2.238 | 600 MB/s | 800 MB/s | +| lzo1x 2.09 -1 | 2.106 | 680 MB/s | 950 MB/s | +| lz4 1.8.3 | 2.101 | 800 MB/s | 4220 MB/s | +| snappy 1.1.4 | 2.073 | 580 MB/s | 2020 MB/s | +| lzf 3.6 -1 | 2.077 | 440 MB/s | 930 MB/s | + +[zlib]: http://www.zlib.net/ +[LZ4]: http://www.lz4.org/ + +Zstd can also offer stronger compression ratios at the cost of compression speed. +Speed vs Compression trade-off is configurable by small increments. +Decompression speed is preserved and remains roughly the same at all settings, +a property shared by most LZ compression algorithms, such as [zlib] or lzma. + +The following tests were run +on a server running Linux Debian (`Linux version 4.14.0-3-amd64`) +with a Core i7-6700K CPU @ 4.0GHz, +using [lzbench], an open-source in-memory benchmark by @inikep +compiled with [gcc] 7.3.0, +on the [Silesia compression corpus]. + +Compression Speed vs Ratio | Decompression Speed +---------------------------|-------------------- + |  + +A few other algorithms can produce higher compression ratios at slower speeds, falling outside of the graph. +For a larger picture including slow modes, [click on this link](doc/images/DCspeed5.png). + + +## The case for Small Data compression + +Previous charts provide results applicable to typical file and stream scenarios (several MB). Small data comes with different perspectives. + +The smaller the amount of data to compress, the more difficult it is to compress. This problem is common to all compression algorithms, and reason is, compression algorithms learn from past data how to compress future data. But at the beginning of a new data set, there is no "past" to build upon. + +To solve this situation, Zstd offers a __training mode__, which can be used to tune the algorithm for a selected type of data. +Training Zstandard is achieved by providing it with a few samples (one file per sample). The result of this training is stored in a file called "dictionary", which must be loaded before compression and decompression. +Using this dictionary, the compression ratio achievable on small data improves dramatically. + +The following example uses the `github-users` [sample set](https://github.com/facebook/zstd/releases/tag/v1.1.3), created from [github public API](https://developer.github.com/v3/users/#get-all-users). +It consists of roughly 10K records weighing about 1KB each. + +Compression Ratio | Compression Speed | Decompression Speed +------------------|-------------------|-------------------- + |  |  + + +These compression gains are achieved while simultaneously providing _faster_ compression and decompression speeds. + +Training works if there is some correlation in a family of small data samples. The more data-specific a dictionary is, the more efficient it is (there is no _universal dictionary_). +Hence, deploying one dictionary per type of data will provide the greatest benefits. +Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file. + +### Dictionary compression How To: + +1. Create the dictionary + + `zstd --train FullPathToTrainingSet/* -o dictionaryName` + +2. Compress with dictionary + + `zstd -D dictionaryName FILE` + +3. Decompress with dictionary + + `zstd -D dictionaryName --decompress FILE.zst` + + +## Build instructions + +### Makefile + +If your system is compatible with standard `make` (or `gmake`), +invoking `make` in root directory will generate `zstd` cli in root directory. + +Other available options include: +- `make install` : create and install zstd cli, library and man pages +- `make check` : create and run `zstd`, tests its behavior on local platform + +### cmake + +A `cmake` project generator is provided within `build/cmake`. +It can generate Makefiles or other build scripts +to create `zstd` binary, and `libzstd` dynamic and static libraries. + +By default, `CMAKE_BUILD_TYPE` is set to `Release`. + +### Meson + +A Meson project is provided within [`build/meson`](build/meson). Follow +build instructions in that directory. + +You can also take a look at [`.travis.yml`](.travis.yml) file for an *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201904190032.x3J0WDUx090718>