7. LZ4HC#

group LZ4HC_API

LZ4HC is a high compression variant of LZ4. This HC variant makes a full search, in contrast with the fast scan of “regular LZ4”, resulting in improved compression ratio. But the speed is quite impacted: the HC version is between 6x and 10x slower than the Fast one.

LZ4HC compression uses lesser memory as it loads individual Assets from an Asset Bundle quickly.

Similar to LZ4, the LZ4HC also provides in-memory compression and decompression. LZ4HC generates the LZ4-compressed block like LZ4 and uses the same decompression technique.

Block Compression Functions

int LZ4_compress_HC(const char *src, char *dst, int srcSize, int dstCapacity, int compressionLevel)#

Compress data from src into dst, using the powerful but slower “HC” algorithm.

Parameters	Direction	Description
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Size of buffer `src`. Maximum supported value is LZ4_MAX_INPUT_SIZE.
dstCapacity	in	Size of buffer `dst` (which must be already allocated).
compressionLevel	in	It is used to set the correct context level for compression.

Result	Description
success	The number of bytes written into `dst`
Fail	0

Note

Compression is guaranteed to succeed if dstCapacity >= LZ4_compressBound(srcSize).

Note

compressionLevel

: value between 1 and LZ4HC_CLEVEL_MAX (inclusive) will work.

Values >

LZ4HC_CLEVEL_MAX behave the same as LZ4HC_CLEVEL_MAX.

int LZ4_compress_HC_internal(const char *src, char *dst, int srcSize, int dstCapacity, int compressionLevel)#

Helper function to be called by LZ4_compress_HC() and defined by AOCL. This function uses the stream of type LZ4_streamHC_t. Function pointer named LZ4_compress_HC_fp points to this function when Dynamic Dispatcher is OFF.

Parameters	Direction	Description
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Size of buffer `src`. Maximum supported value is LZ4_MAX_INPUT_SIZE.
dstCapacity	in	Size of buffer `dst` (which must be already allocated).
compressionLevel	in	It is used to set the correct context level for compression.

Result	Description
success	The number of bytes written into `dst`
Fail	0

Note

Compression is guaranteed to succeed if dstCapacity >= LZ4_compressBound(srcSize).

Note

compressionLevel

: value between 1 and LZ4HC_CLEVEL_MAX (inclusive) will work.

Values >

LZ4HC_CLEVEL_MAX behave the same as LZ4HC_CLEVEL_MAX.

int LZ4_sizeofStateHC(void)#

This function is used to provide the size of state.

Returns:: Returns the amount of memory which must be allocated for its state.

int LZ4_compress_HC_extStateHC(void *stateHC, const char *src, char *dst, int srcSize, int maxDstSize, int compressionLevel)#

Same as LZ4_compress_HC_internal(), but using an externally allocated memory segment for state.

Parameters	Direction	Description
stateHC	in,out	It acts as a handle for compression.
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Size of buffer `src`. Maximum supported value is LZ4_MAX_INPUT_SIZE.
maxDstSize	in	Size of buffer `dst` (which must be already allocated).
compressionLevel	in	It is used to set the correct context level for compression.

Result	Description
success	The number of bytes written into `dst`
Fail	0

Note

state size is provided by LZ4_sizeofStateHC().

Note

Memory segment must be aligned on 8-bytes boundaries (which a normal malloc() should do properly).

int LZ4_compress_HC_destSize(void *stateHC, const char *src, char *dst, int *srcSizePtr, int targetDstSize, int compressionLevel)#

Will compress as much data as possible from src to fit into targetDstSize budget.

Parameters	Direction	Description
stateHC	in,out	It acts as a handle for compression.
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSizePtr	in,out	Pointer to size of buffer `src`. Maximum supported value is LZ4_MAX_INPUT_SIZE.
targetDstSize	in	Size of buffer `dst` (which must be already allocated).
compressionLevel	in	It is used to set the correct context level for compression.

Warning

Requires v1.9.0+

Returns:

Result	Description
success	The number of bytes written into `dst` (necessarily <= targetDstSize)
success	`*srcSizePtr` is updated to indicate how much bytes were read from `src`
Fail	0

Streaming Compression Functions

LZ4_streamHC_t *LZ4_createStreamHC(void)#

Creates the memory for LZ4 HC streaming state. Newly created states are automatically initialized.

Returns:

Result	Description
Success	Pointer to LZ4HC streaming state
Fail	NULL

int LZ4_freeStreamHC(LZ4_streamHC_t *streamHCPtr)#

Releases memory for LZ4 HC streaming state.

Parameters	Direction	Description
streamHCPtr	in,out	Streaming compression state that is automatically initialized. The same state can be re-used multiple times.

Returns:: 0

void LZ4_resetStreamHC_fast(LZ4_streamHC_t *streamHCPtr, int compressionLevel)#

It only works on states which have been properly initialized at least once, which is automatically the case when state is created using LZ4_createStreamHC().

Selecting the compression level can be done using this function.

Parameters	Direction	Description
streamHCPtr	in,out	Streaming compression state that is automatically initialized. The same state can be re-used multiple times.
compressionLevel	in	It is used to set the correct context level for compression.

Note

After completing a streaming compression, it’s possible to start a new stream of blocks, using the same LZ4_streamHC_t state, just by resetting it, using LZ4_resetStreamHC_fast().

Warning

Requires v1.9.0+.

Returns:: void

int LZ4_loadDictHC(LZ4_streamHC_t *streamHCPtr, const char *dictionary, int dictSize)#

A first “fictional block” can be designated as initial dictionary, using this function.

It does full initialization of streamHCPtr, as there are bad side-effects of using resetFast(). The same dictionary will have to be loaded on decompression side for successful decoding. Dictionary are useful for better compression of small data (KB range).

Parameters	Direction	Description
streamHCPtr	in,out	Streaming compression state that is automatically initialized. The same state can be re-used multiple times.
dictionary	in,out	Dictionary buffer.
dictSize	in	Size of dictionary.

Returns:

Result	Description
Success	Loaded dictionary size, in bytes (necessarily <= 64 KB).
Fail	0 if streamHCPtr is NULL or dictionary is NULL.

int LZ4_compress_HC_continue(LZ4_streamHC_t *streamHCPtr, const char *src, char *dst, int srcSize, int maxDstSize)#

Invoked to compress each successive block. The number of blocks is unlimited.

Previous input blocks, including initial dictionary when present,must remain accessible and unmodified during compression.

Parameters	Direction	Description
streamHCPtr	in,out	Streaming compression state that is automatically initialized. The same state can be re-used multiple times.
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSize	in	Maximum supported value is LZ4_MAX_INPUT_SIZE.
maxDstSize	in	Maximum size of buffer `dst` (which must be already allocated).

Note

It’s allowed to update compression level anytime between blocks, using LZ4_setCompressionLevel()
dst buffer should be sized to handle worst case scenarios (see LZ4_compressBound(), it ensures compression success).
In case of failure, the API does not guarantee recovery, so the state must be reset.
To ensure compression success whenever dst buffer size cannot be made >= LZ4_compressBound(), consider using LZ4_compress_HC_continue_destSize().
There is an exception for ring buffers, which can be smaller than 64 KB. Ring-buffer scenario is automatically detected and handled within LZ4_compress_HC_continue().

Returns:

Result	Description
Success	Size of compressed block.
Fail	0 (typically, when cannot fit into `dst`).

int LZ4_compress_HC_continue_destSize(LZ4_streamHC_t *LZ4_streamHCPtr, const char *src, char *dst, int *srcSizePtr, int targetDstSize)#

Similar to LZ4_compress_HC_continue(), but will read as much data as possible from src to fit into targetDstSize budget.

Parameters	Direction	Description
LZ4_streamHCPtr	in,out	Streaming compression state that is automatically initialized. The same state can be re-used multiple times.
src	in	Source buffer, the data which you want to compress is copied/or pointed here.
dst	out	Destination buffer, compressed data is kept here, memory should be allocated already.
srcSizePtr	in,out	Pointer to size of buffer `src`. Maximum supported value is LZ4_MAX_INPUT_SIZE.
targetDstSize	in	Size of buffer `dst` (which must be already allocated).

Note

This function may not consume the entire input.

Warning

Requires v1.9.0+

Returns:

Result	Description
success	The number of bytes written into `dst` (necessarily <= targetDstSize)
success	`*srcSizePtr` is updated to indicate how much bytes were read from `src`
Fail	0

int LZ4_saveDictHC(LZ4_streamHC_t *streamHCPtr, char *safeBuffer, int maxDictSize)#

It saves history content into a user-provided buffer which is then used to continue compression.

Parameters	Direction	Description
streamHCPtr	in,out	Streaming compression state that is automatically initialized. The same state can be re-used multiple times.
safeBuffer	in	Buffer to store dictionary.
maxDictSize	in	Size of safeBuffer, memory should be allocated such that dictionary would fit inside safeBuffer.This is schematically equivalent to a memcpy() followed by LZ4_loadDictHC(),but is much faster, because LZ4_saveDictHC() doesn’t need to rebuild tables.

Note

Whenever previous input blocks can’t be preserved unmodified in-place during compression of next blocks, it’s possible to copy the last blocks into a more stable memory space, using LZ4_saveDictHC().

Returns:

Result	Description
Success	Saved dictionary size in bytes (necessarily <= maxDictSize)
Fail	0

LZ4_streamHC_t *LZ4_initStreamHC(void *buffer, size_t size)#

Used to properly initialize a newly declared LZ4_streamHC_t. It can also initialize any arbitrary buffer of sufficient size, and will return a pointer of proper type upon initialization.

Parameters	Direction	Description
buffer	in,out	buffer to store the LZ4 HC streaming state.
size	in	size of the buffer.

Note

Invoking LZ4_initStreamHC() is not required when the state was created using LZ4_createStreamHC() (which is recommended). Using the normal builder, a newly created state is automatically initialized.

Note

Static allocation shall only be used in combination with static linking.

Warning

Requires v1.9.0+.

Returns:

Result	Description
Success	A pointer to LZ4HC streaming state.
Fail	NULL if `buffer == NULL` or `size < sizeof(LZ4_streamHC_t)` or if alignment conditions are not respected.

Deprecated Compression Functions

int LZ4_compressHC(const char *source, char *dest, int inputSize)#: LZ4_compressHC() is deprecated, use LZ4_compress_HC() instead.

int LZ4_compressHC_limitedOutput(const char *source, char *dest, int inputSize, int maxOutputSize)#: LZ4_compressHC_limitedOutput() is deprecated, use LZ4_compress_HC() instead.

int LZ4_compressHC2(const char *source, char *dest, int inputSize, int compressionLevel)#: LZ4_compressHC2() is deprecated, use LZ4_compress_HC() instead.

int LZ4_compressHC2_limitedOutput(const char *source, char *dest, int inputSize, int maxOutputSize, int compressionLevel)#: LZ4_compressHC2_limitedOutput() is deprecated, use LZ4_compress_HC() instead.

int LZ4_compressHC_withStateHC(void *state, const char *source, char *dest, int inputSize)#: LZ4_compressHC_withStateHC() is deprecated, use LZ4_compress_HC_extStateHC() instead.

int LZ4_compressHC_limitedOutput_withStateHC(void *state, const char *source, char *dest, int inputSize, int maxOutputSize)#: LZ4_compressHC_limitedOutput_withStateHC() is deprecated, use LZ4_compress_HC_extStateHC() instead.

int LZ4_compressHC2_withStateHC(void *state, const char *source, char *dest, int inputSize, int compressionLevel)#: LZ4_compressHC2_withStateHC() is deprecated, use LZ4_compress_HC_extStateHC() instead.

int LZ4_compressHC2_limitedOutput_withStateHC(void *state, const char *source, char *dest, int inputSize, int maxOutputSize, int compressionLevel)#: LZ4_compressHC2_limitedOutput_withStateHC() is deprecated, use LZ4_compress_HC_extStateHC() instead.

int LZ4_compressHC_continue(LZ4_streamHC_t *LZ4_streamHCPtr, const char *source, char *dest, int inputSize)#: LZ4_compressHC_continue() is deprecated, use LZ4_compress_HC_continue() instead.

int LZ4_compressHC_limitedOutput_continue(LZ4_streamHC_t *LZ4_streamHCPtr, const char *source, char *dest, int inputSize, int maxOutputSize)#: LZ4_compressHC_limitedOutput_continue() is deprecated, use LZ4_compress_HC_continue() instead.

struct LZ4HC_CCtx_internal#

Public Members

const LZ4_byte *base#: All index relative to this position

short compressionLevel#: Is a measure of the compression quality

const LZ4_byte *dictBase#: Alternate base for extDict

const LZ4HC_CCtx_internal *dictCtx#: Current context of dictionary

LZ4_u32 dictLimit#: Below that point, need extDict

LZ4_i8 dirty#: Stream has to be fully reset if this flag is set

const LZ4_byte *end#: Next block here to continue on current prefix

LZ4_i8 favorDecSpeed#: Favor decompression speed if this flag set, otherwise, favor compression ratio

LZ4_u32 lowLimit#: Below that point, no more dict

LZ4_u32 nextToUpdate#: Index from which to continue dictionary update