Merge pull request #476 from SMI/release/v1.13.0

Release v1.13.0
SMI · Dec 4, 2020 · 3ec8488 · 3ec8488
2 parents 663d085 + db6bd97
commit 3ec8488
Show file tree

Hide file tree

Showing 159 changed files with 6,207 additions and 966 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,7 +6,44 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased]
 
--
+## [1.13.0] - 2020-12-03
+
+### Added
+
+- Added new command line application TriggerUpdates for detecting and issuing UpdateValuesMessages (e.g. ECHI mapping changes)
+- Added new service UpdateValues which propagates changes (e.g. ECHI mapping changes) throughout the deployed database tables.
+- ConsensusRule for combining 2+ other rules e.g. SocketRules (See IsIdentifiable Readme.md for more details)
+- Added runtime and total failures count to IsIdentifiable logs
+- Added NoSuffixProjectPathResolver which generates anonymous image path names that do not contain "-an" (which is the default behaviour).
+  -  To use, set `CohortExtractorOptions.ProjectPathResolverType` to `Microservices.CohortExtractor.Execution.ProjectPathResolvers.NoSuffixProjectPathResolver`
+  -  For identifiable extractions, the NoSuffixProjectPathResolver is now used
+-   Validation reports can now be created as either "Combined" (single report as before" or "Split" (a [pack](src/microservices/Microservices.CohortPackager/README.md) of reports including CSVs suitable for post-processing). This is configurable in the YAML config and can also be specified on the CLI when recreating reports for an extraction
+-   Added JobCompletedAt to the validation reports
+-   IsIdentifiable: Add support for ignoring OCR output less than `n` characters in length
+-   IsIdentifiable: Add a test case for burned-in image text
+
+### Changed
+
+-   Update docs and make more keywords links to the relevant docs (#440)
+-   Reduce memory usage on long-running microservices even when .Net assumes RAM is plentiful
+-   Validation reports are now written to the project reports directory, instead of to a central reports directory
+
+### Fixed
+
+-   Fix mismatch in Java/C# messages for ExtractionModality
+-   ExtractionFileCopier: Copy files relative to the extraction root not the global filesystem root
+-   Fix implementation of minimum OCR length (before being reported) #471
+
+### Dependencies
+
+- Bump CsvHelper from 17.0.0 to 17.0.1
+- Bump System.IO.Abstractions from 13.2.1 to 13.2.2
+- Bump Moq from 4.15.1 to 4.15.2
+- Bump System.IO.Abstractions.TestingHelpers from 13.2.1 to 13.2.2
+- Bump CsvHelper from 16.2.0 to 17.0.0
+- Bump JetBrains.Annotations from 2020.1.0 to 2020.3.0
+- Bump jackson-dataformat-yaml from 2.11.3 to 2.12.0
+- Bump jackson-databind from 2.11.3 to 2.12.0
 
 ## [1.12.2] - 2020-09-18
 
@@ -422,7 +459,8 @@ First stable release after importing the repository from the private [SMIPlugin]
 - Anonymous `MappingTableName` must now be fully specified to pass validation (e.g. `mydb.mytbl`). Previously skipping database portion was supported.
 
 
-[Unreleased]: https://github.com/SMI/SmiServices/compare/v1.12.2...develop
+[Unreleased]: https://github.com/SMI/SmiServices/compare/v1.13.0...develop
+[1.13.0]:  https://github.com/SMI/SmiServices/compare/v1.12.2...v1.13.0
 [1.12.2]:  https://github.com/SMI/SmiServices/compare/v1.12.1...v1.12.2
 [1.12.1]:  https://github.com/SMI/SmiServices/compare/v1.12.0...v1.12.1
 [1.12.0]:  https://github.com/SMI/SmiServices/compare/v1.11.1...v1.12.0

diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 [![Total alerts](https://img.shields.io/lgtm/alerts/g/SMI/SmiServices.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/SMI/SmiServices/alerts/)
 [![Coverage Status](https://coveralls.io/repos/github/SMI/SmiServices/badge.svg)](https://coveralls.io/github/SMI/SmiServices)
 
-Version: `1.12.2`
+Version: `1.13.0`
 
 # SMI Services
 
@@ -43,23 +43,24 @@ A control queue is provided for controlling Microservices during runtime.  It su
 
 | Microservice / Console App| Description |
 | ------------- | ------------- |
-| [ProcessDirectory](./src/applications/Applications.DicomDirectoryProcessor/README.md)  | Enumerates directories and generates `AccessionDirectoryMessage` for those that contain dicom files.|
-| [DicomTagReader](./src/microservices/Microservices.DicomTagReader/README.md)  | Opens dicom files found in `AccessionDirectoryMessage` directories and converts to JSON as a `DicomFileMessage`.  Also creates a summary record of the whole series as a `SeriesMessage`.|
-| [IdentifierMapper](./src/microservices/Microservices.IdentifierMapper/Readme.md)  | Replaces the `PatientID` Dicom Tag in a `DicomFileMessage` using a specified mapping table.|
-| [MongoDBPopulator](./src/microservices/Microservices.MongoDbPopulator/Readme.md)  | Stores the Dicom Tag data in `DicomFileMessage` and/or `SeriesMessage` into a MongoDB database document store. |
-| [DicomRelationalMapper](./src/microservices/Microservices.DicomRelationalMapper/Readme.md)  | Runs an RDMP data load configuration with a batch of `DicomFileMessage` to load Dicom Tag data into a relational database (MySql or Microsoft Sql Server).|
-| [DicomReprocessor](./src/microservices/Microservices.DicomReprocessor/README.md)  | Runs a MongoDB query on the database populated by `MongoDBPopulator` and converts the results back into `DicomFileMessage` for (re)loading by `DicomRelationalMapper`.|
+| [ProcessDirectory]  | Command line application that finds dicom files on disk and [queues them for execution in RabbitMQ](./src/common/Smi.Common/Messages/AccessionDirectoryMessage.cs).|
+| [DicomTagReader] | Opens queued dicom files on disk and [converts them to JSON](./src/common/Smi.Common/Messages/DicomFileMessage.cs).  Also creates a [summary record of the whole series](./src/common/Smi.Common/Messages/SeriesMessage.cs).|
+| [IdentifierMapper] (Optional)  | Replaces the `PatientID` dicom Tag in a [DicomFileMessage] using a specified mapping table.|
+| [MongoDBPopulator]  | Persists the dicom Tag data in [DicomFileMessage] and/or [SeriesMessage] into a MongoDB database document store. |
+| [DicomRelationalMapper] | Persists the dicom Tag data (and file paths) in [DicomFileMessage] into a [relational database](https://github.com/HicServices/RDMP/blob/develop/Documentation/CodeTutorials/FAQ.md#databases).  ETL pipeline is controlled by an [RDMP] data load configuration.|
+| [DicomReprocessor] | Runs a MongoDB query on the database populated by [MongoDBPopulator] and converts the results back into [DicomFileMessage] for (re)loading by [DicomRelationalMapper].|
 
 ### Image Extraction Microservices
 
 ![extractiondiagram](./docs/Images/ExtractionMicroservices.png)
 
 | Microservice / Console App| Description |
 | ------------- | ------------- |
-| [ExtractorCL](./src/applications/com.smi.applications.extractorcli/README.md)  | Reads SeriesInstanceUIDs from a CSV file and generates `ExtractionRequestMessage` and audit message `ExtractionRequestInfoMessage`.|
-| [CohortExtractor](./src/microservices/Microservices.CohortExtractor/README.md)  | Looks up SeriesInstanceUIDs in `ExtractionRequestMessage` and does relational database lookup(s) to resolve into physical image file location.  Generates  `ExtractFileMessage` and audit message `ExtractFileCollectionInfoMessage`.|
-| [CTPAnonymiser](./src/microservices/com.smi.microservices.ctpanonymiser/README.md)  | Microservice wrapper for [CTP](https://github.com/johnperry/CTP).  Anonymises images specified in  `ExtractFileMessage` and copies to specified output directory.  Generates audit message `ExtractFileStatusMessage`.|
-| [CohortPackager](./src/microservices/Microservices.CohortPackager/README.md)  | Records all audit messages and determines when jobs are complete.|
+| [IsIdentifiable]  | Evaluates data being prepared for extraction for personally identifiable data (PII).  See also [IsIdentifiableReviewer]|
+| [ExtractorCL] | Reads UIDs from a CSV file and generates [ExtractionRequestMessage] and audit message [ExtractionRequestInfoMessage].|
+| [CohortExtractor] | Looks up SeriesInstanceUIDs in [ExtractionRequestMessage] and does relational database lookup(s) to resolve into physical image file location.  Generates  [ExtractFileMessage] and audit message [ExtractFileCollectionInfoMessage].|
+| [CTPAnonymiser]  | Microservice wrapper for [CTP](https://github.com/johnperry/CTP).  Anonymises images specified in  [ExtractFileMessage] and copies to specified output directory.  Generates audit message [ExtractFileStatusMessage].|
+| [CohortPackager] | Records all audit messages and determines when jobs are complete.|
 
 ### Audit and Logging Systems
 
@@ -84,11 +85,12 @@ Appart from the Microservices (documented above) the following library classes a
 | Template Builder | /Applications| GUI tool for building modality database schema templates.  Supports viewing and exploring dicom tags in files|
 | Smi.MongoDB.Common | /Reusable | Library containing methods for interacting with MongoDb |
 
+
 ## Building
 
 ### Building the C# Projects
 
-Building requires the [.NET Core 2.2 SDK](https://dotnet.microsoft.com/download/dotnet-core/2.2)
+Building requires the [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet-core/3.1)
 
 ```bash
 $ dotnet build [-r RID]
@@ -213,4 +215,23 @@ Scaleability is handled through parallel process execution (using [RabbitMQ]).
 [DBMS]: https://github.com/HicServices/RDMP/blob/develop/Documentation/CodeTutorials/Glossary.md#DBMS
 [Dicom]: ./Glossary.md#dicom
 [Dicom tags]: ./Glossary.md#dicom-tags
-
+[IsIdentifiable]: ./src/microservices/Microservices.IsIdentifiable/README.md
+[IsIdentifiableReviewer]: ./src/applications/IsIdentifiableReviewer/README.md
+[DicomFileMessage]: ./src/common/Smi.Common/Messages/DicomFileMessage.cs
+[SeriesMessage]: ./src/common/Smi.Common/Messages/SeriesMessage.cs
+[ExtractionRequestMessage]: ./src/common/Smi.Common/Messages/Extraction/ExtractionRequestMessage.cs
+[ExtractionRequestInfoMessage]: ./src/common/Smi.Common/Messages/Extraction/ExtractionRequestInfoMessage.cs
+[ExtractFileMessage]: ./src/common/Smi.Common/Messages/Extraction/ExtractFileMessage.cs
+[ExtractFileCollectionInfoMessage]: ./src/common/Smi.Common/Messages/Extraction/ExtractFileCollectionInfoMessage.cs
+[ExtractFileStatusMessage]: ./src/common/Smi.Common/Messages/Extraction/ExtractFileStatusMessage.cs
+[RDMP]: https://github.com/HicServices/RDMP
+[ProcessDirectory]: ./src/applications/Applications.DicomDirectoryProcessor/README.md
+[DicomTagReader]: ./src/microservices/Microservices.DicomTagReader/README.md
+[IdentifierMapper]: ./src/microservices/Microservices.IdentifierMapper/Readme.md
+[MongoDBPopulator]: ./src/microservices/Microservices.MongoDbPopulator/Readme.md
+[DicomRelationalMapper]: ./src/microservices/Microservices.DicomRelationalMapper/Readme.md
+[DicomReprocessor]: ./src/microservices/Microservices.DicomReprocessor/README.md
+[ExtractorCL]: ./src/applications/com.smi.applications.extractorcli/README.md
+[CohortExtractor]: ./src/microservices/Microservices.CohortExtractor/README.md
+[CTPAnonymiser]: ./src/microservices/com.smi.microservices.ctpanonymiser/README.md
+[CohortPackager]: ./src/microservices/Microservices.CohortPackager/README.md
diff --git a/SmiServices.sln b/SmiServices.sln
@@ -1,4 +1,3 @@
-
 Microsoft Visual Studio Solution File, Format Version 12.00
 # Visual Studio Version 16
 VisualStudioVersion = 16.0.29411.108
@@ -79,6 +78,16 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microservices.FileCopier",
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microservices.FileCopier.Tests", "tests\microservices\Microservices.FileCopier.Tests\Microservices.FileCopier.Tests.csproj", "{D61F6BF9-E857-457C-B745-40489A8CFE65}"
 EndProject
+Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Updating", "Updating", "{758C7C8C-683A-42E8-A8C5-C32AED1741BE}"
+EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microservices.UpdateValues", "src\microservices\Updating\Microservices.UpdateValues\Microservices.UpdateValues.csproj", "{C710F837-3E9A-4D0B-8ADE-A76A87187BB0}"
+EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Microservices.UpdateValues.Tests", "tests\microservices\Microservices.UpdateValues.Tests\Microservices.UpdateValues.Tests.csproj", "{1B27F8A1-1A0F-4EFB-8A0F-4DB461E8B739}"
+EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Applications.TriggerUpdates", "src\applications\Applications.TriggerUpdates\Applications.TriggerUpdates.csproj", "{6FC70871-A668-4423-AFD0-C53A84122402}"
+EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Applications.TriggerUpdates.Tests", "tests\applications\Applications.TriggerUpdates.Tests\Applications.TriggerUpdates.Tests.csproj", "{ED2CD7B5-4C4C-4F2D-ABAE-4FF2CDA141B2}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|x64 = Debug|x64
@@ -193,6 +202,22 @@ Global
 		{D61F6BF9-E857-457C-B745-40489A8CFE65}.Debug|x64.Build.0 = Debug|x64
 		{D61F6BF9-E857-457C-B745-40489A8CFE65}.Release|x64.ActiveCfg = Release|x64
 		{D61F6BF9-E857-457C-B745-40489A8CFE65}.Release|x64.Build.0 = Release|x64
+		{C710F837-3E9A-4D0B-8ADE-A76A87187BB0}.Debug|x64.ActiveCfg = Debug|x64
+		{C710F837-3E9A-4D0B-8ADE-A76A87187BB0}.Debug|x64.Build.0 = Debug|x64
+		{C710F837-3E9A-4D0B-8ADE-A76A87187BB0}.Release|x64.ActiveCfg = Release|x64
+		{C710F837-3E9A-4D0B-8ADE-A76A87187BB0}.Release|x64.Build.0 = Release|x64
+		{1B27F8A1-1A0F-4EFB-8A0F-4DB461E8B739}.Debug|x64.ActiveCfg = Debug|x64
+		{1B27F8A1-1A0F-4EFB-8A0F-4DB461E8B739}.Debug|x64.Build.0 = Debug|x64
+		{1B27F8A1-1A0F-4EFB-8A0F-4DB461E8B739}.Release|x64.ActiveCfg = Release|x64
+		{1B27F8A1-1A0F-4EFB-8A0F-4DB461E8B739}.Release|x64.Build.0 = Release|x64
+		{6FC70871-A668-4423-AFD0-C53A84122402}.Debug|x64.ActiveCfg = Debug|x64
+		{6FC70871-A668-4423-AFD0-C53A84122402}.Debug|x64.Build.0 = Debug|x64
+		{6FC70871-A668-4423-AFD0-C53A84122402}.Release|x64.ActiveCfg = Release|x64
+		{6FC70871-A668-4423-AFD0-C53A84122402}.Release|x64.Build.0 = Release|x64
+		{ED2CD7B5-4C4C-4F2D-ABAE-4FF2CDA141B2}.Debug|x64.ActiveCfg = Debug|x64
+		{ED2CD7B5-4C4C-4F2D-ABAE-4FF2CDA141B2}.Debug|x64.Build.0 = Debug|x64
+		{ED2CD7B5-4C4C-4F2D-ABAE-4FF2CDA141B2}.Release|x64.ActiveCfg = Release|x64
+		{ED2CD7B5-4C4C-4F2D-ABAE-4FF2CDA141B2}.Release|x64.Build.0 = Release|x64
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -225,6 +250,11 @@ Global
 		{C2031E86-81B4-405A-A923-9B82E0CE196F} = {8B943F2C-835B-484A-86D2-3F1462970605}
 		{D4E52707-FFF7-41E6-8057-C6DB344B8CD7} = {421CCD37-3817-4748-B184-A134E19DD75C}
 		{D61F6BF9-E857-457C-B745-40489A8CFE65} = {421CCD37-3817-4748-B184-A134E19DD75C}
+		{758C7C8C-683A-42E8-A8C5-C32AED1741BE} = {421CCD37-3817-4748-B184-A134E19DD75C}
+		{C710F837-3E9A-4D0B-8ADE-A76A87187BB0} = {758C7C8C-683A-42E8-A8C5-C32AED1741BE}
+		{1B27F8A1-1A0F-4EFB-8A0F-4DB461E8B739} = {758C7C8C-683A-42E8-A8C5-C32AED1741BE}
+		{6FC70871-A668-4423-AFD0-C53A84122402} = {8B943F2C-835B-484A-86D2-3F1462970605}
+		{ED2CD7B5-4C4C-4F2D-ABAE-4FF2CDA141B2} = {8B943F2C-835B-484A-86D2-3F1462970605}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {11CDEA53-71E8-4A9B-BC0D-74F4EB54F740}

diff --git a/appveyor.yml b/appveyor.yml
@@ -35,7 +35,7 @@ build_script:
   - ps: 7z x rdmp-cli-win-x64.zip -oC:\projects\smiservices\rdmp-cli
   - ps: C:\projects\smiservices\rdmp-cli\rdmp.exe install localhost\SQL2017 TEST_ -D -u sa -p Password12!
 
-  - cmd: "\"C:/ProgramData/chocolatey/bin/OpenCover.Console.exe\" -returntargetcode -target:\"c:/program files/dotnet/dotnet.exe\" -targetargs:\"test ./tests/run-all-tests.proj -c Release -p:BuildInParallel=false\" -filter:\"+[CohortExtractor]* +[CohortPackager]* +[DeadLetterReprocessor]* +[DicomRelationalMapper]* +[DicomReprocessor]* +[DicomTagReader]* +[IdentifierMapper]* +[IsIdentifiable]* +[MongoDbPopulator]* +[DicomDirectoryProcessor]* +[IsIdentifiableReviewer]* +[Smi.Common]* +[Smi.Common.MongoDb]* -[*NUnit*]*\"  -output:coverage.xml -register:appveyor -oldStyle -hideskipped:File"
+  - cmd: "\"C:/ProgramData/chocolatey/bin/OpenCover.Console.exe\" -returntargetcode -target:\"c:/program files/dotnet/dotnet.exe\" -targetargs:\"test ./tests/run-all-tests.proj -c Release -p:BuildInParallel=false\" -filter:\"+[CohortExtractor]* +[CohortPackager]* +[DeadLetterReprocessor]* +[DicomRelationalMapper]* +[DicomReprocessor]* +[DicomTagReader]* +[IdentifierMapper]* +[IsIdentifiable]* +[MongoDbPopulator]* +[DicomDirectoryProcessor]* +[TriggerUpdates]* +[UpdateValues]* +[IsIdentifiableReviewer]* +[Smi.Common]* +[Smi.Common.MongoDb]* -[*NUnit*]*\"  -output:coverage.xml -register:appveyor -oldStyle -hideskipped:File"
   - cmd: 'if defined COVERALLS_REPO_TOKEN %USERPROFILE%\\.nuget\\packages\\coveralls.io\\1.4.2\\tools\\coveralls.net.exe --opencover coverage.xml -r %COVERALLS_REPO_TOKEN%'
 
 test: off

diff --git a/data/microserviceConfigs/default.yaml b/data/microserviceConfigs/default.yaml
@@ -60,6 +60,17 @@ DicomRelationalMapperOptions:
     RetryDelayInSeconds: 60
     RunChecks: true
 
+UpdateValuesOptions:
+    QueueName: 'TEST.UpdateValuesQueue'
+    QoSPrefetchCount: 10000
+    AutoAck: false
+    #TableInfosToUpdate: 1,2,3 <-set this to limit which tables get updated when no explicit table is listed in the message
+    UpdateTimeout: 500 # number of seconds to wait for each UPDATE statement
+
+TriggerUpdatesOptions:
+    ExchangeName: 'TEST.TriggerUpdatesExchange'
+    MaxConfirmAttempts: 1
+
 CohortExtractorOptions:
     QueueName: 'TEST.RequestQueue'
     QoSPrefetchCount: 10000
@@ -91,6 +102,8 @@ CohortPackagerOptions:
     JobWatcherTimeoutInSeconds: 30
     ReporterType: 'LoggingReporter'
     NotifierType: 'LoggingNotifier'
+    ReportFormat: 'Combined'
+    ReportNewLine: ''
     ExtractRequestInfoOptions:
         QueueName: 'TEST.RequestInfoQueue'
         QoSPrefetchCount: 1
@@ -180,7 +193,7 @@ FileCopierOptions:
     QoSPrefetchCount: 1
     AutoAck: false
     CopyStatusProducerOptions:
-        ExchangeName: 'TEST.FileStatusExchange'
+        ExchangeName: 'TEST.ExtractedFileStatusExchange'
         MaxConfirmAttempts: 1
 
 ExtractorClOptions: