v1.12.0

NOTE - package smi-services-linux-x64-v1.12.0.tar.gz below contains a broken version of the MongoDBPopulator service. Please use a version of this in a later release

Changelog

Added

• [breaking] Add identifiable extraction support
  • New service "FileCopier" which sits in place of CTP for identifiable extractions and copies source files to their output dirs
  • Changes to MongoDB extraction schema, but backwards compatibility has been tested
  • RabbitMQ extraction config has been refactored. Queues and service config files need to be updated
• Add SecurityCodeScan tool to build chain for .NET code
• Add "no filters" extraction support. If specified when running ExtractorCLI, no file rejection filters will be applied by CohortExtractor. True by default for identifiable extractions
• Added caching of values looked up in NLP/rulesbase for IsIdentifiable tool
• Added new rejector that throws out values (e.g. patient IDs) whose IDs are stored in a database table. Set RejectColumnInfos option in yaml to enable this
• Added a check to QueryToExecuteResult for RejectReason being null when Reject is true.

Changed

• [breaking] Environment variables are no longer required. Previous settings now appear in configuration file
  • Environment variable SMI_LOGS_ROOT is now GlobalOptions.LogsRoot
  • Environment variable MONGO_SERVICE_PASSWORD is now MongoDbOptions.Password
  • Removed ISIDENTIFIABLE_NUMTHREADS as it didn't work correctly anyway
• Extraction report: Group PixelData separately and sort by length
• IsIdentifiable Reviewer 'Symbols' rule factory now supports digits only or characters only mode (e.g. use \d for digits but leave characters verbatim)
• IsIdentifiable Reviewer 'symbols' option when building Regex now builds capture groups and matches only the failing parts of the input string not the full ProblemValue. For example MR Head 12-11-20 would return (\d\d-\d\d-\d\d)$

Fixed

• Fix the extraction output directory to be <projId>/extractions/<extractname>

Dependencies

• Bump fo-dicom.Drawing from 4.0.5 to 4.0.6
• Bump fo-dicom.NetCore from 4.0.5 to 4.0.6
• Bump HIC.BadMedicine.Dicom from 0.0.6 to 0.0.7
• Bump HIC.DicomTypeTranslation from 2.3.0 to 2.3.1
• Bump HIC.FAnsiSql from 1.0.2 to 1.0.5
• Bump HIC.RDMP.Dicom from 2.1.6 to 2.1.10
• Bump HIC.RDMP.Plugin from 4.1.6 to 4.1.8
• Bump HIC.RDMP.Plugin.Test from 4.1.6 to 4.1.8
• Bump Microsoft.CodeAnalysis.CSharp.Scripting from 3.6.0 to 3.7.0
• Bump Microsoft.Extensions.Caching.Memory from 3.1.6 to 3.1.8
• Bump Microsoft.NET.Test.Sdk from 16.6.1 to 16.7.1
• Bump MongoDB.Driver from 2.11.0 to 2.11.2
• Bump System.IO.Abstractions from 12.1.1 to 12.1.9
• Bump System.IO.Abstractions.TestingHelpers from 12.1.1 to 12.1.9
• Bump Terminal.Gui from 0.81.0 to 0.89.4

v1.11.1: Hotfix - publish trimmed (#348)

* Disable publishtrimmed since it breaks things

* bump for v1.11.0

Co-authored-by: James A Sutherland <j@sutherland.pw>

v1.11.0

Changelog

Added

• DicomDirectoryProcessor and TagReader support for zip archives
  • Expressed in notation /mydrive/myfolder/myzip.zip!somesubdir/my.dcm
  • Requires command line -f zips

Changed

• Improved the extraction report by summarising verification failures
• Start MongoDB in replication mode in the Travis builds
• Switch to self-contained .Net binaries to avoid dependency on host runtime package
• NationalPACSAccessionNumber is now allowed to be null in all messages

Dependencies

• Bump HIC.RDMP.Plugin from 4.1.5 to 4.1.6
• Bump MongoDB.Driver from 2.10.4 to 2.11.0
• Bump System.IO.Abstractions from 12.0.10 to 12.1.1
• Bump System.IO.Abstractions.TestingHelpers from 12.0.10 to 12.1.1
• Bump jackson-dataformat-yaml from 2.11.1 to 2.11.2

v1.10.0

Changed

• Updated the extraction report to be more human-readable #320, #328
• Add CLI option to CohortPackager to allow an existing report to be recreated #321
• Added a runsettings file for NUnit to allow configuration of test output. Fixes an issue with TravisCI and NUnit3TestAdapter v3.17.0, which caused the test output to spill to over 20k lines.

Dependencies

• Bump HIC.FAnsiSql from 0.11.1 to 1.0.2
• Bump HIC.RDMP.Dicom from 2.1.5 to 2.1.6
• Bump HIC.RDMP.Plugin from 4.1.3 to 4.1.5
• Bump Magick.NET-Q16-AnyCPU from 7.20.0 to 7.21.1
• Bump Microsoft.Extensions.Caching.Memory from 3.1.5 to 3.1.6
• Bump System.IO.Abstractions from 12.0.1 to 12.0.10
• Bump System.IO.Abstractions from 12.0.1 to 12.0.2
• Bump System.IO.Abstractions.TestingHelpers from 12.0.1 to 12.0.2
• Bump com.fasterxml.jackson.dataformat.jackson-dataformat-yaml from 2.11.0 to 2.11.1
• Bump org.mockito.mockito-core from 3.3.3 to 3.4.6

v1.9.0

Changelog

Added

• Added image extraction blacklist rejector.
  • Configure with Blacklists option (specify a list of Catalogue IDs)
  • Catalogues listed must include one or more column(s) StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID.
  • Records in the referenced table will blacklist where any UID is found (StudyInstanceUID, SeriesInstanceUID or SOPInstanceUID). This allows blacklisting an entire study or only specific images.
  • [breaking] Config on live system may need updated
• Change the extraction directory generation to be <projname>/image-requests/<extractname>. Fixes MVP Service #159

Fixed

• Fixed IsIdentifiable rule order being the order the files are detected in rules directory (Now goes IgnoreRules=>ReportRules=>SocketRules)
• Adjust log handling in CTP anonymiser to use SMIlogging setup
• IsIdentifiable case-sensitive rules now implemented with property
• Bufix for fo-dicom image handling race condition in Release mode builds (issue #238)

Changed

• Refactored WhiteListRule to inherit from IsIdentifiableRule (affects serialization).
  • Parent property As replaces IfClassification
  • CaseSensitive replaces IfPatternCaseSensitive and IfPartPatternCaseSensitive (Also fixes serialization bug)
• Bump CommandLineParser from 2.7.82 to 2.8.0
• Bump CsvHelper from 15.0.4 to 15.0.5
• Bump HIC.BadMedicine.Dicom from 0.0.5 to 0.0.6
• Bump HIC.DicomTypeTranslation from 2.2.2 to 2.3.0
• Bump HIC.RDMP.Dicom from 2.0.9 to 2.1.5
• Bump HIC.RDMP.Plugin from 4.0.2 to 4.1.3
• Bump Magick.NET-Q16-AnyCPU from 7.16.0 to 7.20.0
• Bump Microsoft.CodeAnalysis.CSharp.Scripting from 3.5.0 to 3.6.0
• Bump Microsoft.Extensions.Caching.Memory from 3.1.3 to 3.1.5
• Bump MongoDB.Driver from 2.10.3 to 2.10.4
• Bump StackExchange.Redis from 2.1.30 to 2.1.58
• Bump System.IO.Abstractions from 10.0.8 to 12.0.1
• Bump YamlDotNet from 8.1.0 to 8.1.2
• Bump fo-dicom.Drawing from 4.0.4 to 4.0.5
• Pinned fo-dicom.NetCore to 4.0.5

v1.8.1

Changelog

Fixed

• Fix null check bug in CohortPackager when no files match the extraction filter

v1.8.0

Changelog

Added

• Added Terminal.Gui at version 0.81.0
• Added data/IsIdentifiableRules

Changed

• [Breaking] Promote the PT modality to its own collection in MongoDB
• [Breaking] Renamed RedisHost to RedisConnectionString in the config options for clarity
• Update to .Net Core 3.1 (supported until Dec 2022) since 2.2 support ended last year
• Switch CohortExtractor to use batched message producers
• Simplify the Travis build script
• Fail any integration tests in CI if a required service is not available (instead of skipping)
• Specified LangVersion 8.0 in all project files
• Upgraded CommandLineParser from 2.5.0 to 2.7.82
• Upgraded CsvHelper from 12.1.2 to 15.0.4
• Upgraded HIC.Rdmp.Dicom from 2.0.8 to 2.0.9
• Upgraded JetBrains.Annotations from 2019.1.3 to 2020.1.0
• Upgraded Magick.NET-Q16-AnyCPU from 7.15.1 to 7.16.0
• Upgraded Microsoft.CodeAnalysis.CSharp.Scripting from 3.5.0-beta2-final to 3.5.0
• Upgraded MongoDB.Driver from 2.9.3 to 2.10.3
• Upgraded StackExchange.Redis from 2.0.601 to 2.1.30
• Upgraded System.Drawing.Common from 4.6.0 to 4.7.0
• Upgraded System.IO.Abstractions from 7.0.7 to 10.0.8
• Upgraded YamlDotNet from 6.0.0 to 8.1.0

Fixed

• Fixed logging to directories in the Java services

v1.7.0

Changelog

Added

• Added undo feature to IsIdentifiableReviewer
• Java microservices now log to SMI_LOGS_ROOT

Changed

• Upgraded HIC.DicomTypeTranslation from 2.1.2 to 2.2.0
  • This includes an upgrade to fo-dicom from 4.0.1 to 4.0.4
• Upgraded fo-dicom.Drawing from 4.0.1 to 4.0.4
• Upgraded HIC.RdmpDicom from 2.0.7 to 2.0.8

v1.6.0

Changelog

Changed

• Update CohortPackager for new extraction design

  • Consume messages from CTP (failed anonymisation) and IsIdentifiable (verification)
  • Add support for extraction by modality
  • Remove the final check for the anonymised file. IsIdentifiable handles this already
  • Refactor tests

• Start to refactor core RabbitMqAdapter code to allow unit testing