Htscodecs 1.2.2

Release 1.2.2: 1st April 2022

This release contains some fixes found during fuzzing with Clang's memory-sanitizer. None of these are involving writing memory so there
is no possibility for code execution vulnerabilities. However some do could access uninitialised elements in locally allocated memory, which
could leak private data if the library was used in conjunction with other tools which don't zero sensitive data before freeing.

Bug fixes:

• The name tokeniser now validates the stored length in the data stream matches the actual decoded length. Discovered by Taotao Gu.

• Fixed an endless loop in arith_dynamic and rans4x16pr involving X_STRIPE with 0 stripes.

• Avoid a harmless (and wrong?) undefined behaviour sanitizer error when calling memcpy(ptr, NULL, 0) in the name tokeniser.

• Fixed possible uninitialised memory access in rans_uncompress_O1_4x16. If the frequency table didn't add up to
  the correct amount, parts of the "fb" table were left unpopulated. It was then possible to use these array elements in some of the rANS
  calculations.

• Similarly rans_uncompress_O0 could access an uninitialised element 4095 of the decoder tables if the frequencies summed to 4095 instead
  of the expected 4096.

• Improved error detection from fqzcomp's read_array function.

• Reject fqzcomp parameters with inconsistent "sel" parameters, which could lead to uninitialised access to the model.sel range coder.

Htscodecs 1.2.1

This release contains the following minor changes. Please see the "git log" for the full details.

Improvements / changes:

• Speed up of rANS4x16 order-0. We now use a branchless encoder renormalisation step. For complex data it's between 13 and 50% speed up depending on compiler.

• Improve rANS4x16 compute_shift estimates. The entropy calculation is now more accurate. This leads to more frequent use of the 10-bit frequency mode, at an expense of up to 1% size growth.

• Speed improvements to the striped rANS mode, both encoding and decoding. Encoder gains ~8% and decoder ~5%, but varies considerably by compiler and data.

• Added new var_put_u64_safe and var_put_u32_safe interfaces. These are automatically used by var_put_u64 and var_put_u32 when near the end of the buffer, but may also be called directly.

• Small speed ups to the hist8 and hist1_4 functions.

• Minor speed up to RLE decoding.

Bug fixes:

• Work around an icc-2021 compiler bug, but also speed up the varint encoding too (#29).

• Fix an off-by-one error in the initial size check in arith_dynamic. This meant the very smallest of blocks could fail to decode. Reported by Divon Lan.

• Fixed hist1_4 to also count the last byte when computing T0 array.

• Fixed overly harsh bounds checking in the fqzcomp read_array function, which meant it failed to decode some configurations.

htscodecs 1.1.1

Release 1.1.1: 6th July 2021

This release contains the following minoring changes.
Please see the "git log" for the full details.

Improvements / changes:

• Modernised autoconf usage to avoid warnings with newer versions.
  (John Marshall)

• Avoid using awk with large records, due to some systems
  (e.g. Solaris / OpenIndiana) with line length limits .
  (John Marshall)

• Applied Debian patch to make the library link against -lm.

Bug fixes:

• Fixed an issue with the name tokeniser when a slice (name_context)
  has exactly 1 more name than the previous call. (James Bonfield)

• Removed access to an uninitialised variable in the name tokeniser
  decode when given malformed data. This occurs when we use delta
  encoding for the very first name. (James Bonfield, OSS-Fuzz)

• Minor fixes to distcheck and distclean targets

htscodecs 1.0

Release 1.0: 23rd Feb 2021

This marks the first non-beta release of htscodecs, following a perioid of integration with Htslib and automated fuzzing by Google's OSS-Fuzz program.

[Note this testing only applies to the C implementation. The JavaScript code should still be considered as examples of the codecs, more for purposes of understanding and clarity than as a fully optimised and tested release.]

Since the last release (0.5) the key changes are:

• Improved support for big endian platforms

• Speed improvements to CRAM 3.0 4x8 rANS order-1 encoding. It's between 10 and 50% faster at encoding, based on input data.

• Improved autoconf bzip2 checks and tidy up "make test" output.

• Added some more files into "make install", so that "make distcheck" now passes.

• Replaced Travis with Cirrus-CI testing.

• Removed various C undefined behaviour, such as left shifting of negative values and integer overflows. As far as we know these were currently harmless on the supported platforms, but may break future compiler optimisations.

• Fixed numerous OSS-Fuzz identified flaws. Some of these were potential security issues such as small buffer overruns.

• Tidied up some code to prevent warnings.

• The name tokeniser now has a limit on the size of data it can encode (10 million records). This may still be too high given the memory it will require, so it may be reduced again.

htscodecs-0.5

This release has a few renamed functions (the variable sized integer encoding functions) and thus is incompatible with v0.4. The test tools now also incorporate a "raw" mode (-r) for purposes of creating un-wrapped byte streams without data sizing information. These now match the CRAMcodecs specification.

Full changes:

• Renamed the varint functions and also added signed versions.

• Rans 4x16 order-1 frequency tables are now configurable (within the byte stream) to 10 or 12 bit totals. Previously it was 10, but this is too small for efficient compression of extreme distributions.

• rANS 4x16 X4 has been renamed STRIPE and can now interleave other quantities than just 4 streams.

• Sped up the C rans 4x16 order-1 decoder, often by around 30% or so if SSE4 is permitted. (Try -march=native)

• Sped up the C RLE decoding function. Also refactored this code into it's own rle.c file.

• Bug fix to name tokeniser so it can handle blank lines.

• Fixed RLE encoding in the rANS 4x16 JavaScript implementation. It no longer can generate invalid streams when it doesn't find anything worthy of doing RLE on.

• Fixed JavaScript rans 4x16 frequency renormalisation. Occasionally it'd generate very suboptimal frequency distributions. Also used this revised algorithm in C (which didn't have that problem, but was still improved.)

• Fixed JavaScript exception handling in tok3.js (with thanks to Chris Norman).

• Bug fixed the JavaScript rans.js to correctly allocate data size. It could fail on tiny inputs.

htscodecs-0.4

There are no new features in this release; simply improvements in portability and robustness.

Code portability for MacOS and Windows.

• On both of these platforms, as well as on Linux, memory management has been improved to avoid requiring large stack sizes. We use thread local storage to perform one malloc call and reuse this same block for each subsequent function call for the duration of that thread. This has the benefits of a large stack without penalties of repeated use of malloc/free.

Fixes

• Bug fixed name tokeniser when there are a variable number of tokens.
• Removed some compilation warnings.
• Javascript demonstration code is more complete, with DO_REV support in fqzcomp.js.

htscodecs-0.3

Bug fixes and updates to C code. Note this includes some incompatibilities (see commits).

Improved testing.

Added first draft of the javascript implementation. This isn't intended for production use, but is instead a reference implementation to be used along side the codec specification document (currently a work in progress, over at https://github.com/jkbonfield/hts-specs/blob/CRAMv4/CRAMv4.tex).

htscodecs-0.2

Mainly portability (MacOSX) and fuzz testing fixes.

htscodecs-0.1

First test release of the htscodecs package.

See README.md for the minimal amount of documentation, or look at the test programs in the tests directory.