ChangeLog

Version 1.6.0-beta2 2026-07-04 by luispedro

	Changes with respect to 1.5.0:
	* NGLess is now based on a new implementation written in Rust (previous
	versions were written in Haskell). It is intended as a compatible replacement:
	the same scripts produce the same results.
	* NGLess now supports a single language version, "1.6". Scripts must declare
	`ngless "1.6"`; declaring an older version (including "1.5") is now a hard
	error. Update the version statement at the top of your scripts to `ngless
	"1.6"`.
	* Built-in modules (`parallel`, `samtools`, `mocat`, ...) now track the ngless
	version: import them at version "1.6" going forward. Older module versions are
	still accepted (with the latest behaviour) but print a deprecation warning.
	* Fixed the `encoding` argument to `fastq()` and `paired()` (and forwarded
	by `load_fastq_directory()`/`load_mocat_sample()`): it was accepted but
	ignored, so loading always auto-detected the quality encoding. `encoding`
	({auto}/{33}/{sanger}/{64}/{solexa}) is now honoured again.
	* External modules can once again declare `references:` (both direct
	`fasta-file`/`gtf-file`/`map-file` entries and `packaged` reference packs).
	These are now recognised by `map(..., reference=...)` and
	`count(..., reference=...)`, so catalog modules such as `igc`, `om-rgc`, and
	the gut-catalog modules resolve their bundled references again.
	* External module commands now see `NGLESS_NR_CORES` set to the configured
	worker thread count (`--jobs`/`--threads`, or the `batch` scheduler's
	allotment) instead of a hard-coded 1.
	* Restored the distinction between global and local module imports. A plain
	`import` is again only accepted for known modules (`igc`, `om-rgc`, `motus`,
	the gut-catalog modules, ...); importing a local-only module now requires
	`local import`. Known modules that are not present locally are auto-downloaded
	again.
	* Removed the deprecated `strand` argument to `count()`; use `sense`
	(`{both}`/`{sense}`/`{antisense}`) instead. `strand=True` is equivalent to
	`sense={sense}`.
	* Removed the deprecated `--search-dir` command-line flag (use
	`--search-path`) and the unimplemented `--check-deprecation` flag.
	* The output content hash reported by `auto_comments=[{hash}]` (and used to
	name parallel lock/stats directories) is now computed with a simpler internal
	serialization. It is still deterministic and content-addressed, but the hash
	*values* differ from previous releases (they are an internal identifier, not a
	stable cross-version value).
	* ngless now writes an HTML run report again (the `<script>.output_ngless/`
	directory, or the location given by `-o`/`--html-report-directory`; disable
	with `--no-create-report`). The report is a single self-contained
	`index.html` that embeds its data and makes no network requests, so it works
	offline on compute clusters (the old report loaded AngularJS, jQuery,
	Bootstrap and d3/nvd3 from CDNs). It shows the script (with links to the
	quality-control and mapping statistics), the run log, per-file FASTQ QC with
	per-position quality charts, and mapping statistics. The script listing is
	syntax-highlighted. The directory also contains `script.ngl`, `fq.tsv` and
	`mappings.tsv`. Inline scripts (passed with `-e`/`--script`) do *not* write a
	report by default (they are typically quick one-liners); pass
	`--create-report` or `-o`/`--html-report-directory` to force one.
	* A plain `import "motus"` or `import "soap"` (the legacy built-in modules)
	now aborts with a clear guidance error (both are obsolete and unsupported). A
	`local import "motus"` is exempt: downloading the `motus.ngm` external module
	and loading it with `local import` is the recommended way to run motus.
	* `count()` now errors early when more than one annotation source is given
	(seqname mode, `gff_file`, `functional_map`, or `reference`) instead of
	silently ignoring all but one
	* `write()` (and other functions taking an output file) now checks the output
	directory before the script runs even when the file *name* is only known at
	run time. For example, `ofile=OUTPUT_DIRECTORY </> sample.name() + '.fna.gz'`
	now aborts immediately if `OUTPUT_DIRECTORY` does not exist, instead of only
	failing once the write is reached (i.e., after mapping/assembly)
	* A constant out-of-bounds list index (e.g. `x[5]` on a 3-element list) now
	fails early, right after the list is assigned, with the same "Index access
	on line N is invalid" message as earlier ngless versions
	* Added the `{always_3_fq_files}` format flag to `write()`
	* Better suggestions for mistyped arguments/flags (matching by prefix, e.g.
	`ref=` for `reference=`)
	* When an `import` of a local module cannot be found, the error now lists all
	the locations that were searched. Under `--trace`, each candidate location is
	printed as it is checked. If other versions of the same module are available,
	they are listed as well.

	Changes with respect to 1.6.0-beta1:
	* The build now supports a single language version, "1.6": a script declaring
	"1.5" (which beta1 still accepted with a warning) is now a hard error.
	* Removed the deprecated `count()` `strand` argument (use `sense`), the
	`--search-dir` command-line flag (use `--search-path`), and the unimplemented
	`--check-deprecation` flag.
	* The internal output content hash (`auto_comments=[{hash}]` and parallel
	lock/stats directory names) is computed differently, so its values differ from
	beta1. It remains deterministic and content-addressed.
	* Otherwise beta2 restores a large set of features that beta1 had not yet
	ported from earlier ngless versions -- transparent `.xz` support, the HTML run
	report, parallel-module progress reporting, `format={sam|bam}` on `write()`,
	`preprocess(keep_singles=False)`, `auto_comments=[{date}]`, reference-based
	`count()`, enhanced FASTQ QC statistics, broader external module support,
	and several performance and UX improvements.

Version 1.6.0-beta1 2026-07-01 by luispedro
	* Switched to rust implementation
	* Fix bug when outputing comments to compressed files
	* Fix `load_sample_from_yaml` (it had the wrong arguments before)
	* Added `{always_3_fq_files}` format flag to `write()`
	* UX: Better suggestions for mistyped arguments/flags by matching prefixes
	* Fix bug where multiplication of non-integer values used addition instead
	of multiplication

Version 1.5.0 2022-09-14 by luispedro
	* Add `compress_level` option to `write()` function
	* Add ability to specify samples using YAML format
	* Make 'assemble' return NGLSequenceSet type and add sequenceset to
	external modules
	* Allow `::` in names (for module::function notation)
	* Add `run_for_all` in parallel module
	* Support .xz compressed files in load_fastq_directory
	* Add `names()` method to readset
	* Add ``println()`` function
	* Make ``print()`` accept ints and doubles as well as strings
	* Write log to .failed files when using the parallel module
	* Fix bug where failhooks were not triggered for uncaught exceptions
	* Change order of testing for locks in the parallel module for a more
	robust protocol

Version 1.4.2 2022-07-21 by luispedro
	* Fix bug with parsing GFF files

Version 1.4.1 2022-06-03 by luispedro
	* Fix bug with split mapping
	* Fix packaging for hackage

Version 1.4.0 2022-05-30 by luispedro
	* Add --trace argument to all modes
	* Update --create-reference-pack mode to newer format (previous version worked, but
	users still had to recreate indices after downloading, negating some of
	the benefits
	* Update --install-reference-data mode to download latest version (#107)
	* Update --create-reference-pack mode to download latest version (#108)
	* Do not fail when merging empty files (#113)
	* write() returns the filename used
	* Switch to tasty test framework
	* Update to LTS-19
	* write() uses multiple threads to write multiple FQ files
	* Use binaries from path (not self installed)
	* Better error message when parsing misformed FastQ files
	* Overhall building infrastructure to use haskell.nix and build
	dependencies with nix
	* Update megahit version
	* Added GMGC to builtin Modules
	* Make `--print-path minimap2` work correctly

Version 1.3.0 2021-01-28 by luispedro
	* Validate count() headers on --validate-only
	* Better error message if the user attempts to use the non-existent <\>
	operator (suggest </>)
	* Add min-ngless-version field for modules
	* Add early check that block assignments are always to block variables
	* Use ZStd compression for temporary files from preprocess()
	* Correctly handle subpaths in samples for collect (fixes #141)
	* Add to_string() to int and double types (partially fixes #78 & fixes #81)
	* Add read_int() and read_double() functions (fixes #78)

Version 1.2.0 2020-07-12 by luispedro
	* Add load_fastq_directory to builtin functions
	* Better messages when using `lock1`
	* Enable specifying *all* module resources by URL with download on first
	use
	* Fix CIGAR reinjection bug
	* Remove old motus/specI modules
	* No longer ship JS libraries (also do not expand them in-place)

Version 1.1.0 2020-01-25 by luispedro
	* Reintroduce zstd compression (after fixes upstream)
	* Fix CIGAR interpretation (#109) occurring when I is present
	* Call bwa mem so that it behaves in a deterministic way (independently of
	the number of threads used)
	* Add `include_fragments` option to orf_find
	* Add early check for column headers in `count()`
	* Update to LTS-14.1 (fixes #116)
	* Add ``sense`` argument to `count()`
	* Add line numbers to FastQ parsing errors
	* Fix __extra_args argument in map()
	* Add `discard_singles` function
	* Add `interleaved` option to fastq()
	* `load_mocat_sample` now fails if `pair.2` exists but `pair.1` doesn't

Version 1.0.1 2019-07-05 by luispedro
	* Fix bug with external modules and multiple fastQ inputs
	* Fix bug with resaving input files where the original file was sometimes
	moved (thus removing it).

Version 1.0.0 2019-04-24 by luispedro
	* Fix multiple features usage (#63)
	* Update minimap2/megahit to latest versions

Version 0.11.1 2019-03-21 by luispedro
	* Fix reinjectSequence in select() call with blocks
	* Remove zstd compression
	* Fix SAM order switching in select() call with blocks (#105)
	* Compress SAM files after select() calls

Version 0.11.0 2019-03-15 by luispedro
	* Switch to diagrams package for plotting
	* Update minimap2 version to 2.14
	* Module samtools (version 0.1) now includes samtools_view
	* Update to LTS-13 (GHC 8.6)
	* Fix bug with orf_find & prots_out argument
	* Call bwa/minimap2 with interleaved fastq files
	* Add --verbose flag to check-install mode
	* Avoid leaving open file descriptors after FastQ encoding detection
	* Fix bug in garbage collection
	* Compress intermediate SAM files (#22)
	* Tar extraction uses much less memory (#77)
	* Add early checks for input files in more situations (#33)
	* Support compression in collect() output (#42)
	* Fix CIGAR (#92) for select() blocks
	* Add smoothtrim function
	* Fix lock1()/collect() use with --subsample
	* Fix compression/decompression when calling external modules

Version 0.10.0 2018-11-12 by luispedro
	* Fix to lock1's return value when used with paths (#68 - reopen)
	* Support _F/_R suffixes for forward/reverse in load_mocat_sample
	* samtools_sort() now accepts by={name} to sort by read name
	* Fixed bug where header was printed even when STDOUT was used
	* Fixed bug where writing interleaved FastQ to STDOUT did not work as
	expected
	* Indices created by bwa and minimap2 are now versioned
	* arg1 in external modules is no longer always treated as a path
	* Added expand_searchpath to external modules API (closes #56)
	* Fixed bug where detection of Fastq encoding was not performed on the second pair
	* Fix saving fastq sets with --subsample (issue #85)
	* Add __extra_megahit_args to assemble() (issue #86)
	* Better error message when user mis-specifies the ngless version string
	(issue #84)
	* Support NO_COLOR environment variable (issue #83)
	* Garbage collection for temporary files (issue #79)
	* Rename --search-dir to --search-path for consistency with other API
	* Fix corner case with select() producing incorrect CIGAR strings (#92)
	* Always check output file writability (#91)
	* Make paired() accept encoding argument

Version 0.9.1 2018-07-17 by luispedro
	* Add biorxiv citation

Version 0.9.0 2018-07-12 by luispedro
	* Add allbest() method to mappedreads
	* Output FastQ quality graphs as PNGs
	* Added MouseGutCatalog module
	* Added DogGutGeneCatalog module
	* Added PigGeneCatalog module
	* Added reference genome for Sus scrofa (pig)
	* Update IGC module to 0.9
	* Continuously update mtime on all lock files
	* Warn when overwriting files
	* Version automatically downloaded reference URLs

Version 0.8.1 2018-06-05 by luispedro
	* Update to LTS-11.12 (for faster conduit-algorithms used in collect())
	* Add fallback for character encoding on systems with bad locale support
	* Fixed lock1 when used with paths (#68)
	* Fixed expansion of searchdir with external modules (#56)

Version 0.8.0 2018-05-06 by luispedro
	* Add minimap2 support as alternative mapper
	* Faster collect()
	* Fix writing of multiple compressed files to uncompressed format
	* Add `n_to_zero_quality` method for short reads
	* Add </> operator for path manipulation
	* Fix bug in select (corner case where sequences would be missing from
	output)
	* Add non-ATCG fraction field to FastQ statistics
	* Add reference argument to count()
	* GFF based counting now expands multi-value sub-features
	* Update to bwa 0.7.17

Version 0.7.1 2018-03-17 by luispedro
	* Fix memory leak in count()
	* Fix when-true flag usage with external modules

Version 0.7.0 2018-03-07 by luispedro
	* Add 'failed' files to parallel lock1()
	* Add `max_trim` argument to MappedReadSet.filter()
	* Support saving compressed SAM files
	* Much faster select() implementation with a block
	* Fix count's mode {intersection_strict} to no longer behave as {union}
	* Support for saving interleaved FastQ files
	* Lower memory usage
	* More conservative SAM merging in split mode
	* Compute #Basepairs in FastQ stats
	* Fix as_reads() for single-end reads
	* Add headers argument to samfile()
	* For more situations, avoid intermediate copies in count()
	* Much improved memory and speed performance of count()
	* Fix select() corner case

Version 0.6.1 2017-12-10 by luispedro
	* Add --check-install mode
	* Fix streaming short read QC (performance regression)
	* Better error message in readlines() when file opening fails
	* Compute statistics after select()

Version 0.6.0 2017-11-29 by luispedro
	* Add `orf_find` function (implemented through Prodigal)
	* Add qcstats() function
	* Output preprocessed FQ statistics (had been erroneously removed)
	* Fix --strict-threads command-line option spelling
	* Use multiple threads in SAM->BAM conversion
	* Change include_m1 default in count() function to True
	* Add --index-path functionality to define where to write indices (issue #47)
	* Allow `citations` as key in external modules
	* Better citations information
	* Better error checking/script validation
	* Added reference alias for a more human readable name
	* Version embedded megahit binary
	* Updated builtin referenced to include latest releases of assemblies
	* Fixed inconsistency between reference identifiers and underlying files

Version 0.5.1 2017-11-02 by luispedro
	* Fix building step (static compilation)

Version 0.5.0 2017-11-01 by luispedro
	* First release supporting all basic functionality.