1. LZ4

group LZ4_API

LZ4 is a lossless compression algorithm, providing compression speeds greater than 500 MB/s per core, and scalable with multi-core CPU. It features an extremely fast decoder, with speed in multiple GB/s per core and reaching RAM speed limits on multi-core systems.

This library provides in-memory compression and decompression functions. It gives full buffer control to the user. Compression can be done in:

• a single step (described in Simple Functions)

• a single step, reusing a context (described in Advanced Functions)

• unbounded multiple steps (described in Streaming compression)

APIs in lz4.h generate LZ4-compressed blocks in the format lz4_Block_format. Decompressing such a compressed block requires additional metadata. Exact metadata depends on exact decompression function. In the case of LZ4_decompress_safe(), the metadata includes block’s compressed size, and maximum bound of decompressed size. Each application is free to encode and pass such metadata in whichever way it wants.

API in lz4.h only handle blocks, it cannot generate Frames.

Blocks are different from Frames lz4_Frame_format. The Frames bundle both blocks and metadata in the specified manner. Embedding metadata is required for the compressed data to be self-contained and portable. The Frame format is delivered through the companion API declared in lz4frame.h. The lz4 CLI can only manage Frames.

Helper Functions

int LZ4_versionNumber(void)

Library Version number. Useful to check Dynamic-link Library (DLL) version.

Returns:

Library Version in integer format.

const char *LZ4_versionString(void)

Library version string. Useful to check DLL version.

Returns:

Version in const char* format.

Simple Functions

int LZ4_compress_default(const char *src, char *dst, int srcSize, int dstCapacity)

Compresses ‘srcSize’ bytes from buffer ‘src’ into allocated ‘dst’ buffer of size ‘dstCapacity’.

Parameters	Direction	Description
src	in	Source buffer, the data you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Maximum supported value is LZ4_MAX_INPUT_SIZE.
dstCapacity	in	Size of pre-allocated ‘dst’ buffer.

Note

Compression is guaranteed to succeed if ‘dstCapacity’ >= LZ4_compressBound(srcSize). It is recommended to use this setting as the compression runs faster on this setting.

Note

This function is protected against buffer overflow scenarios (never writes outside ‘dst’ buffer nor reads outside ‘source’ buffer).

Warning

If the function cannot compress ‘src’ into a more limited ‘dst’ budget, compression stops immediately and the function result is zero. In which case, ‘dst’ content is undefined (invalid).

Returns:

Result	Description
Success	Returns a positive number (<= dstCapacity) indicating the number of bytes written into the buffer dst.
Fail	Returns 0.

int LZ4_decompress_safe(const char *src, char *dst, int compressedSize, int dstCapacity)

Decompresses the compressed data pointed by src into dst and returns the number of bytes decompressed into the destination buffer.

Parameters	Direction	Description
src	in	This buffer contains compressed data.
dst	out	This is the destination buffer, the data is decompressed to this buffer.
compressedSize	in	It is the exact complete size of the compressed block.
dstCapacity	in	It is the size of the destination buffer (which must be pre-allocated), presumed to be an upper bound of decompressed size.

Note

1 : This function is protected against malicious data packets : it will never writes outside ‘dst’ buffer, nor read outside ‘source’ buffer, even if the compressed block is maliciously modified to order the decoder to do these actions. In such case, the decoder stops immediately, and considers the compressed block malformed.

Note

2 : compressedSize and dstCapacity must be provided to the function, the compressed block does not contain them. The implementation is free to send / store / derive this information in whichever way is most beneficial. If there is a need for a different format which bundles together both compressed data and its metadata, consider looking at lz4frame.h instead.

Returns:

Result	Description
Success	The number of bytes decompressed into destination buffer (<= dstCapacity)
Fail	If destination buffer is not large enough, decoding will stop and output an error code (negative value).
	If the source stream is detected malformed, the function will stop decoding and return a negative result.

Advanced Functions

int LZ4_compressBound(int inputSize)

Provides the maximum size that LZ4 compression may output in a “worst case” scenario (input data not compressible).

This function is primarily useful for memory allocation purposes (destination buffer size). Macro LZ4_COMPRESSBOUND(isize) is also provided for compilation-time evaluation (stack memory allocation for example).

Parameters	Direction	Description
inputSize	in	Maximum supported value is LZ4_MAX_INPUT_SIZE .

Note

LZ4_compress_default() compresses faster when dstCapacity is >= LZ4_compressBound(srcSize).

Returns:

Result	Description
Success	Returns Maximum output size in a “worst case” scenario
Fail	Returns 0, if input size is incorrect (too large or negative).

int LZ4_compress_fast(const char *src, char *dst, int srcSize, int dstCapacity, int acceleration)

Same as LZ4_compress_default(), but allows the selection of “acceleration” factor.

Parameters	Direction	Description
src	in	Source buffer, the data you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Maximum supported value is LZ4_MAX_INPUT_SIZE.
dstCapacity	in	Size of buffer ‘dst’ (which must be already allocated).
acceleration	in	Values <= 0 will be replaced by LZ4_ACCELERATION_DEFAULT (currently == 1, see lz4.c).
		Values > LZ4_ACCELERATION_MAX will be replaced by LZ4_ACCELERATION_MAX (currently == 65537, see lz4.c).
		The larger the acceleration value, the faster the algorithm, but also the lesser the compression. It’s a trade-off.
		It can be fine-tuned with each successive value providing roughly +~3% to speed.
		An acceleration value of “1” is the same as regular LZ4_compress_default().

Returns:

Result	Description
Success	Returns a positive number (<= dstCapacity) indicating the number of bytes written into the buffer dst.
Fail	Returns 0.

int LZ4_sizeofState(void)

Get the memory to be allocated for its state.

Returns:

The amount of memory which must be allocated for its state.

int LZ4_compress_fast_extState(void *state, const char *src, char *dst, int srcSize, int dstCapacity, int acceleration)

Same as LZ4_compress_fast(), using an externally allocated memory space for its state.

Use LZ4_sizeofState() to know how much memory must be allocated, and allocate it on 8-bytes boundaries (using malloc() typically). Then, provide this buffer as void* state to compression function.

Parameters	Direction	Description
state	in,out	It acts as a handle.
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Maximum supported value is LZ4_MAX_INPUT_SIZE.
dstCapacity	in	Size of buffer ‘dst’ (which must be already allocated).
acceleration	in	The larger the acceleration value, the faster the algorithm, but also the lesser the compression.
		It’s a trade-off. It can be fine-tuned, with each successive value providing roughly +~3% to speed.
		An acceleration value of “1” is the same as regular LZ4_compress_default().
		Values <= 0 will be replaced by LZ4_ACCELERATION_DEFAULT (currently == 1, see lz4.c).
		Values > LZ4_ACCELERATION_MAX will be replaced by LZ4_ACCELERATION_MAX (currently == 65537, see lz4.c).

Returns:

Result	Description
Success	Returns a positive number (<= dstCapacity) indicating the number of bytes written into the buffer dst.
Fail	Returns 0.

int LZ4_compress_destSize(const char *src, char *dst, int *srcSizePtr, int targetDstSize)

This function either compresses the entire ‘src’ content into ‘dst’ if it’s large enough or fills ‘dst’ buffer completely with as much data as possible from ‘src’. Reverse the logic : Compresses as much data as possible from the ‘src’ buffer into the allocated buffer ‘dst’, of size >= ‘targetDestSize’.

Note: Acceleration parameter is fixed to “default”.

Parameters	Direction	Description
src	in	Source buffer, the data you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be pre-allocated.
srcSizePtr	in,out	Will be modified to indicate how many bytes were read from ‘src’ to fill ‘dst’. New value is necessarily <= input value.
Size	in	Size of buffer ‘dst’ (which must be already allocated).

Warning

From v1.8.2 to v1.9.1, this function had a bug (fixed un v1.9.2+): Sometimes, the produced compressed content must be decompressed into a destination buffer larger by at least 1 byte than the content to decompress. If an application uses LZ4_compress_destSize(), it’s highly recommended to update liblz4 to v1.9.2 or better. If this can’t be done or ensured, the receiving decompression function should provide a dstCapacity > decompressedSize, by at least 1 byte. For more information, refer to lz4/lz4#859.

Returns:

Result	Description
Success	Returns Nb bytes written into dst (<= targetDestSize).
Fail	Returns 0 on fail .

int LZ4_decompress_safe_partial(const char *src, char *dst, int srcSize, int targetOutputSize, int dstCapacity)

Decompress an LZ4 compressed block, of size ‘srcSize’ at the position ‘src’ into destination buffer ‘dst’ of size ‘dstCapacity’. Up to ‘targetOutputSize’ bytes will be decoded.

The function stops decoding on reaching this objective. This can be useful to boost performance whenever only the beginning of a block is required.

Note

1 : return can be < targetOutputSize, if compressed block contains less data.

Note

2 : targetOutputSize must be <= dstCapacity.

Note

3 : this function effectively stops decoding on reaching targetOutputSize, so dstCapacity is kind of redundant. This is because in older versions of this function, decoding operation would still write complete sequences. Therefore, there was no guarantee that it would stop writing at exactly targetOutputSize, it could write more bytes, though only up to dstCapacity. Some “margin” used to be required for this operation to work properly. Thankfully, this is no longer necessary. The function nonetheless keeps the same signature, in an effort to preserve API compatibility.

Note

4 : If srcSize is the exact size of the block, then targetOutputSize can be any value, including larger than the block’s decompressed size. The function will, at most, generate block’s decompressed size.

Note

5 : If srcSize is larger than block’s compressed size, then targetOutputSize MUST be <= block’s decompressed size. Otherwise, silent corruption will occur.

Returns:

Result	Description
Success	The number of bytes decoded in dst (<= targetOutputSize)
Fail	If source stream is detected malformed, function returns a negative result.

Streaming Compression Functions

LZ4_stream_t *LZ4_createStream(void)

Creates LZ4_stream_t, allocates memory dynamically and returns its memory address.

Parameters:

void –

Returns:

A pointer of LZ4_stream_t type whose memmory has been allocated dynamically.

int LZ4_freeStream(LZ4_stream_t *streamPtr)

Frees the memory pointed by streamPtr .

Parameters	Direction	Description
StreamPtr	in,out	LZ4_stream_t* type streaming compression tracking context. A tracking context can be re-used multiple times. Declare or create LZ4_stream_t using LZ4_createStream(), which is recommended.

Returns:

int 0 .

void LZ4_resetStream_fast(LZ4_stream_t *streamPtr)

Use this to prepare an LZ4_stream_t for a new chain of dependent blocks (for example, LZ4_compress_fast_continue()).

An LZ4_stream_t must be initialized once before usage. This is automatically done when created by LZ4_createStream(). However, should the LZ4_stream_t be simply declared on stack (for example), it’s necessary to initialize it first, using LZ4_initStream().

After init, start any new stream with LZ4_resetStream_fast(). A same LZ4_stream_t can be re-used multiple times consecutively and compress multiple streams, provided that it starts each new stream with LZ4_resetStream_fast().

LZ4_resetStream_fast() is much faster than LZ4_initStream(), but is not compatible with memory regions containing garbage data.

Parameters	Direction	Description
StreamPtr	in,out	LZ4_stream_t* type streaming compression tracking context. A tracking context can be re-used multiple times. Declare or create LZ4_stream_t using LZ4_createStream(), which is recommended.

Attention

This function is for v1.9.0+.

Note

It’s only useful to call LZ4_resetStream_fast() in the context of streaming compression. The extState functions perform their own resets. Invoking LZ4_resetStream_fast() before is redundant, and even counterproductive.

Returns:

void .

int LZ4_loadDict(LZ4_stream_t *streamPtr, const char *dictionary, int dictSize)

Use this function to reference a static dictionary into LZ4_stream_t.

The dictionary must remain available during compression. LZ4_loadDict() triggers a reset, so any previous data will be forgotten. The same dictionary will have to be loaded on decompression side for successful decoding. Dictionary are useful for better compression of small data (KB range). While LZ4 accept any input as dictionary, results are generally better when using Zstandard’s Dictionary Builder. Loading a size of 0 is allowed, and is the same as reset.

Parameters	Direction	Description
streamPtr	in,out	LZ4_stream_t* type streaming type compression tracking context. A tracking context can be re-used multiple times. Declare or create LZ4_stream_t using LZ4_createStream(), which is recommended. Initialize LZ4_stream_t using LZ4_initStream().
dictionary	in,out	Dictionary buffer
dictSize	in	Size of dictionary

Returns:

Result	Description
Success	Loaded dictionary size, in bytes (<= 64 KB).
Fail	-1 if (streamPtr == NULL) or (dictionary == NULL && dictSize >= sizeof(reg_t)).

int LZ4_compress_fast_continue(LZ4_stream_t *streamPtr, const char *src, char *dst, int srcSize, int dstCapacity, int acceleration)

Compress src content using data from the previously compressed blocks for a better compression ratio. The dst buffer must be already allocated. If dstCapacity >= LZ4_compressBound(srcSize), the compression would succeed and run faster.

Parameters	Direction	Description
state	in,out	LZ4_stream_t* type streaming type compression tracking context. A tracking context can be re-used multiple times. Declare or create LZ4_stream_t using LZ4_createStream(), which is recommended. Initialize LZ4_stream_t using LZ4_initStream().
src	in	Source buffer, the data you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Maximum supported value is LZ4_MAX_INPUT_SIZE.
dstCapacity	in	Size of buffer ‘dst’ (which must be already allocated).
acceleration	in	The larger the acceleration value, the faster the algorithm, but also the lesser the compression.
		It’s a trade-off. It can be fine-tuned, with each successive value providing roughly +~3% to speed.
		An acceleration value of “1” is the same as regular LZ4_compress_default().
		Values <= 0 will be replaced by LZ4_ACCELERATION_DEFAULT (currently == 1, see lz4.c).
		Values > LZ4_ACCELERATION_MAX will be replaced by LZ4_ACCELERATION_MAX (currently == 65537, see lz4.c).

Note

1 : Each invocation to LZ4_compress_fast_continue() generates a new block. Each block has precise boundaries and must be decompressed separately, calling LZ4_decompress_*() with relevant metadata. It’s not possible to append blocks together and expect a single invocation of LZ4_decompress_*() to decompress them together.

Note

2 : The previous 64KB of source data is assumed to remain present, unmodified, at same address in memory !

Note

3 : When input is structured as a double buffer, each buffer can have any size, including < 64 KB. Make sure that buffers are separated, by at least one byte. This construction ensures that each block only depends on previous block.

Note

4 : If input buffer is a ring-buffer, it can have any size, including < 64 KB.

Note

5 : After an error, the stream status is undefined (invalid), it can only be reset or freed.

Returns:

Result	Description
Success	Returns the size of compressed block
Fail	Returns 0 if there is an error (typically, cannot fit into ‘dst’).

int LZ4_saveDict(LZ4_stream_t *streamPtr, char *safeBuffer, int maxDictSize)

If the last 64KB data would not be available at its current memory location, save it in a safer location (char* safeBuffer).

Parameters	Direction	Description
streamPtr	in,out	LZ4_stream_t* type streaming type compression tracking context. A tracking context can be re-used multiple times. Declare or create LZ4_stream_t using LZ4_createStream(), which is recommended. Initialize LZ4_stream_t using LZ4_initStream().
safeBuffer	in	Buffer where you want to store dictionary.
maxDictSize	in	Size of safeBuffer, memory should be allocated such that dictionary would fit inside safeBuffer.This is schematically equivalent to a memcpy() followed by LZ4_loadDict(),but is much faster, because LZ4_saveDict() doesn’t need to rebuild tables.

LZ4_saveDict() : If previously compressed data block is not guaranteed to remain available at its memory location, save it into a safer place (char* safeBuffer). Note : you don’t need to call LZ4_loadDict() afterwards, dictionary is immediately usable, you can therefore call LZ4_compress_fast_continue(). Return : saved dictionary size in bytes (necessarily <= dictSize), or 0 if error.

Returns:

Result	Description
Success	Saved dictionary size in bytes which is a positive integer (<= maxDictSize)
Fail	Returns 0 if error.

LZ4_stream_t *LZ4_initStream(void *buffer, size_t size)

Use LZ4_initStream() to properly initialize a newly declared LZ4_stream_t. It can also initialize any arbitrary buffer of sufficient size, and will return a pointer of the proper type upon initialization.

An LZ4_stream_t structure must be initialized at least once. This is automatically done when invoking LZ4_createStream(), but it’s not when the structure is simply declared on stack (for example).

Note

1: Initialization fails if size and alignment conditions are not respected. In which case, the function will return NULL.

Note

2: An LZ4_stream_t structure guarantees correct alignment and size.

Note

3: Before v1.9.0, use LZ4_resetStream() instead

Warning

Works for v1.9.0+

Returns:

Result	Description
Success	A pointer of proper type upon initialization.
Fail	Initialization fails if size and alignment conditions are not respected. In which case, the function will return NULL.

Streaming Decompression Functions

LZ4_streamDecode_t *LZ4_createStreamDecode(void)

Creation of streaming decompression tracking context.

LZ4_createStreamDecode() and LZ4_freeStreamDecode() : creation / destruction of streaming decompression tracking context.

A tracking context can be re-used multiple times.

Returns:

A pointer of LZ4_stream_t type whose memmory has been allocated dynamically.

int LZ4_freeStreamDecode(LZ4_streamDecode_t *LZ4_stream)

Frees up the memory occupied by LZ4_stream .

Parameters	Direction	Description
LZ4_stream	in,out	LZ4_streamDecode_t* works as a streaming decompression tracking context. A tracking context can be re-used multiple times.

Returns:

int 0.

int LZ4_setStreamDecode(LZ4_streamDecode_t *LZ4_streamDecode, const char *dictionary, int dictSize)

Use this function to start decompression of a new stream of blocks.

Parameters	Direction	Description
LZ4_streamDecode	in,out	An LZ4_streamDecode_t context can be allocated once and re-used multiple times.
dictionary	in,out	A dictionary can optionally be set. Use NULL or size 0 for a reset order.
dictSize	in	Size of dictionary

Note

Dictionary is presumed stable : it must remain accessible and unmodified during next decompression.

Returns:

Result	Description
Success	Returns 1 if OK .
Fail	Returns 0 if error.

int LZ4_decoderRingBufferSize(int maxBlockSize)

In a ring buffer scenario (optional), blocks are presumed decompressed next to each other until there isn’t enough space for the next block (remainingSize < maxBlockSize). After that it resumes from the beginning of ring buffer.

When setting such a ring buffer for streaming decompression, this function provides the minimum size of this ring buffer to be compatible with any source respecting maxBlockSize condition.

Parameters	Direction	Description
maxBlockSize	in,out	The maximum block size of compressed data.

Warning

This works for v1.8.2+ .

Returns:

Result	Description
Success	Minimum ring buffer size.
Fail	Returns 0 if there is an error (invalid maxBlockSize).

int LZ4_decompress_safe_continue(LZ4_streamDecode_t *LZ4_streamDecode, const char *src, char *dst, int srcSize, int dstCapacity)

These decoding functions allow decompression of consecutive blocks in “streaming” mode.

A block is an unsplittable entity, it must be presented entirely to a decompression function. Decompression functions only accepts one block at a time. The last 64KB of previously decoded data must remain available and unmodified at the memory position where they were decoded. If less than 64KB of data has been decoded, all the data must be present.

Special : if decompression side sets a ring buffer, it must respect one of the following conditions :

• Decompression buffer size is at least LZ4_decoderRingBufferSize(maxBlockSize). maxBlockSize is the maximum size of any single block. It can have any value > 16 bytes. In which case, encoding and decoding buffers do not need to be synchronized. Actually, data can be produced by any source compliant with LZ4 format specification, and respecting maxBlockSize.

• Synchronized mode : Decompression buffer size is exactly the same as compression buffer size, and follows exactly same update rule (block boundaries at same positions), and decoding function is provided with exact decompressed size of each block (exception for last block of the stream), then decoding & encoding ring buffer can have any size, including small ones ( < 64 KB).

• Decompression buffer is larger than encoding buffer, by a minimum of maxBlockSize more bytes. In which case, encoding and decoding buffers do not need to be synchronized, and encoding ring buffer can have any size, including small ones ( < 64 KB).

Parameters	Direction	Description
LZ4_streamDecode	in,out	An LZ4_streamDecode_t context can be allocated once and re-used multiple times.
src	in	This buffer contains compressed data.
dst	out	Destination buffer, the data is decompressed to this buffer.
srcSize	in	It is the exact complete size of the src.
dstCapacity	in	It is the size of destination buffer (which must be already allocated), presumed an upper bound of decompressed size.

Note

Whenever these conditions are not possible, save the last 64KB of decoded data into a safe buffer where it can’t be modified during decompression, then indicate where this data is saved using LZ4_setStreamDecode(), before decompressing next block.

Returns:

Result	Description
Success	Returns the number of bytes decoded in dst which is positive (<= targetOutputSize)
Fail	If source stream is detected malformed, function returns a negative result.

int LZ4_decompress_safe_usingDict(const char *src, char *dst, int srcSize, int dstCapcity, const char *dictStart, int dictSize)

These decoding functions work the same as a combination of LZ4_setStreamDecode() followed by LZ4_decompress_*_continue()

They are stand-alone, and don’t need an LZ4_streamDecode_t structure. Dictionary is presumed stable : it must remain accessible and unmodified during decompression. Performance tip : Decompression speed can be substantially increased when dst == dictStart + dictSize.

Parameters	Direction	Description
src	in	This buffer contains compressed data.
dst	out	Destination buffer, the data is decompressed to this buffer.
srcSize	in	It is the exact complete size of the src.
dstCapacity	in	It is the size of destination buffer (which must be already allocated), presumed an upper bound of decompressed size.
dictStart	in,out	A dictionary can optionally be set. Use NULL or size 0 for a reset order.
dictSize	in	Size of dictionary.

Returns:

Result	Description
Success	The number of bytes decoded in dst (<= targetOutputSize)
Fail	If source stream is detected malformed, function returns a negative result.

Defines

LZ4_COMPRESSBOUND(isize)

Macro LZ4_COMPRESSBOUND(isize) is provided for compilation-time evaluation (stack memory allocation) alternative of LZ4_compressBound()

Typedefs

typedef union LZ4_stream_u LZ4_stream_t

A tracking context can be re-used multiple times. Declare or create LZ4_stream_t using LZ4_createStream(), which is recommended.

typedef union LZ4_streamDecode_u LZ4_streamDecode_t

It is used to track the LZ4 stream during decompression.