-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
28 changed files
with
1,465 additions
and
3,265 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| # Extraction | ||
|
|
||
| Describes the build-install-test procedure, not the deployment into production. | ||
|
|
||
| Main docs: | ||
| https://github.com/SMI/SmiServices/tree/master/src/microservices/com.smi.microservices.ctpanonymiser | ||
|
|
||
| https://github.com/SMI/SmiServices/tree/release/1.2.0#image-extraction-microservices | ||
|
|
||
| See also: the extraction-refactoring branch | ||
| https://github.com/SMI/SmiServices/tree/feature/extraction-refactoring/docs/extraction | ||
|
|
||
| Other docs: | ||
| https://uoe.sharepoint.com/sites/SMI/Shared%20Documents/Forms/AllItems.aspx | ||
| https://git.ecdf.ed.ac.uk/SMI/SmiServiceOps/blob/master/Planning/ExtractionFlags | ||
| https://github.com/HicServices/SMIPlugin/blob/master/Documentation/Images/ExtractionMicroservices.png | ||
|
|
||
| # Building | ||
|
|
||
| See elsewhere the documents for building the Java programs. | ||
|
|
||
| # Prerequisites for testing | ||
|
|
||
| A RabbitMQ instance is required - you can run a test version inside a Docker container: | ||
|
|
||
| ``` | ||
| sudo docker run -d --hostname my-rabbit --name some-rabbit-mgt -p 5671:5671 -p 5672:5672 -p 5673:5673 -p 15671:15671 -p 15672:15672 -p 25672:25672 rabbitmq:3-management | ||
| ``` | ||
|
|
||
| # ExtractorCLI | ||
|
|
||
| ``` | ||
| cd ~/src/SmiServices/src/applications/com.smi.applications.extractorcli/target | ||
| cat > extractme.csv << _EOF | ||
| SeriesInstanceUID,foo | ||
| 1.2.826.0.1.3680043.2.1125.1.78969117856457473538394301521877227,1 | ||
| _EOF | ||
| Edit default.yaml (RabbitOptions and FileSystemOptions) | ||
| You could do this programmatically with | ||
| `yq_linux_amd64 write --inplace d FileSystemOptions.FileSystemRoot /tmp` | ||
| although the current version of yq loses comments and unnecessary quotes. | ||
| Login to rabbit (localhost:15672) and create exchanges: | ||
| TEST.RequestExchange | ||
| TEST.RequestInfoExchange | ||
| Add bindings from those exchanges to any queue (TEST.xxx) | ||
| ProjectNum=001 | ||
| rmdir /tmp/${ProjectNum}/tmp # program gives error if dir already exists | ||
| java -jar ExtractorCL-portable-1.0.0.jar -y default.yaml -c 0 -e tmp -p ${ProjectNum} extractme.csv | ||
| (interactive - answer y to create messages) | ||
| ``` | ||
|
|
||
| Two messages are created: | ||
|
|
||
| ``` | ||
| {"KeyTag":"SeriesInstanceUID","ExtractionIdentifiers":["1.2.826.0.1.3680043.2.1125.1.78969117856457473538394301521877227"],"ExtractionJobIdentifier":"bb1cbed5-a666-4307-a781-5b83926eaa81","ProjectNumber":"001","ExtractionDirectory":"001/tmp","JobSubmittedAt":"2019-12-19T10:49Z"} | ||
| ``` | ||
| and | ||
| ``` | ||
| {"KeyTag":"SeriesInstanceUID","KeyValueCount":1,"ExtractionJobIdentifier":"bb1cbed5-a666-4307-a781-5b83926eaa81","ProjectNumber":"001","ExtractionDirectory":"001/tmp","JobSubmittedAt":"2019-12-19T10:49Z"} | ||
| ``` | ||
|
|
||
| # CohortExtractor | ||
|
|
||
| Requires MySQL instance? so not described (yet). | ||
|
|
||
| Creates messages containing fields: | ||
| DicomFilePath: Path to the original file | ||
| ExtractionDirectory: Extraction directory relative to the extract root | ||
| OutputPath: Output path for the anonymised file, relative to the extraction directory | ||
| See: ~/src/SmiServices/src/common/Smi.Common/Messages/Extraction/ExtractFileMessage.cs | ||
| Inherits ExtractMessage so: | ||
| Guid ExtractionJobIdentifier | ||
| string ProjectNumber | ||
| string ExtractionDirectory | ||
| DateTime JobSubmittedAt | ||
|
|
||
| # CTPanonymiser | ||
|
|
||
| `cd ~/src/SmiServices/src/microservices/com.smi.microservices.ctpanonymiser/target` | ||
|
|
||
| A whitelist is required, available from the old repo as: | ||
| https://raw.githubusercontent.com/HicServices/SMIPlugin/develop/Documentation/Anon/dicom-whitelist.script | ||
| https://raw.githubusercontent.com/HicServices/SMIPlugin/develop/Documentation/Anon/dicom-whitelist.script.new | ||
| (possibly identical content, apart from whitespace/newlines??) | ||
| or from the new repo in the directories: | ||
| ``` | ||
| SmiServices/src/applications/com.smi.applications.extractorcli/anonScript.txt | ||
| SmiServices/src/microservices/com.smi.microservices.ctpanonymiser/src/test/resources/dicom-anonymizer.script | ||
| ``` | ||
| Haven't yet determined which one is correct. | ||
|
|
||
| Edit `default.yaml` (RabbitOptions and FileSystemOptions) | ||
|
|
||
| Login to rabbit (http://localhost:15672/) and create exchanges: | ||
| TEST.ControlExchange | ||
| TEST.FatalLoggingExchange | ||
| TEST.FileStatusExchange | ||
| and queue: TEST.ExtractFileQueue | ||
| Check: do we need to add bindings from those exchanges to the queue? | ||
|
|
||
| Copy an input file into the directory relative to the root in default.yaml: | ||
| ``` | ||
| cp src/SmiServices/src/microservices/com.smi.microservices.ctpanonymiser/src/test/resources/image-000001.dcm /tmp | ||
| ``` | ||
|
|
||
| Create a fake message and send to TEST.ControlExchange: | ||
| ``` | ||
| python3 -m pip install pika | ||
| #!/usr/bin/env python3 | ||
| msg_json = '{ "DicomFilePath": "image-000001.dcm", "ExtractionDirectory": "001/tmp/extractiondir/", "OutputPath": "output.dcm", "ExtractionJobIdentifier":"bb1cbed5-a666-4307-a781-5b83926eaa81", | ||
| "ProjectNumber":"001", "ExtractionDirectory":"001/tmp", "JobSubmittedAt":"2019-12-19T10:49Z" }' | ||
| hdr={'MessageGuid':'', 'OriginalPublishTimestamp':'', 'ProducerExacutableName':'test.py', 'ProducerProcessID': '0'} | ||
| import pika | ||
| connection = pika.BlockingConnection(pika.ConnectionParameters('localhost')) | ||
| channel = connection.channel() | ||
| # exchange='TEST.ControlExchange', '' to make binding straight to routing_key queue | ||
| channel.basic_publish(exchange='', routing_key='TEST.ExtractFileQueue', body=msg_json, properties=pika.BasicProperties(content_type='application/json', headers=hdr) ) | ||
| ``` | ||
|
|
||
| Run: | ||
| ``` | ||
| java -jar CTPAnonymiser-portable-1.0.0.jar -a dicom-whitelist.script.new -y default.yaml | ||
| ``` | ||
|
|
||
| The output is written to `/tmp/001/tmp/output.dcm` in this example and the log file is in `logs/YYYY-MM-DD-hhmmss.log` | ||
|
|
||
| A 'success' message is published to TEST.FileStatusExchange containing: | ||
| ``` | ||
| {"DicomFilePath":"image-000001.dcm","AnonymisedFileName":"output.dcm","Status":0,"ExtractionJobIdentifier":"bb1cbed5-a666-4307-a781-5b83926eaa81","ProjectNumber":"001","ExtractionDirectory":"001/tmp","JobSubmittedAt":"2019-12-19T10:49Z"} | ||
| ``` | ||
|
|
||
| # IsIdentifiable | ||
|
|
||
| See the netcoreapp2.2 branch of IsIdentifiable here: | ||
| https://github.com/HicServices/IsIdentifiable/tree/netcoreapp2.2 | ||
| with the changes required to build and run on dotnet core 2.2 Linux | ||
| (until such time as it's merged into master). |
File renamed without changes
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,22 @@ | ||
|
|
||
| # TODO Figure out what documentation can be (safely) imported here from the private repo | ||
| # SMI Services Documentation | ||
|
|
||
| This is the documentation for the SMI Servies platform. It should (hopefully) contain enough information to run your own instance of the service. | ||
|
|
||
| The platform is currently deployed in the National Safe Haven, so some documentation may specifically refer to that environment. The software should be deployable in any environment though, so please open an [issue](https://github.com/SMI/SmiServices/issues) if anything isn't clear. | ||
|
|
||
|
|
||
| ### Contents | ||
|
|
||
| - [Controlling the services](#controlling-the-services) | ||
| - [TODO](#todo) | ||
|
|
||
|
|
||
| ## Controlling the services | ||
|
|
||
| The services can be controlled by sending messages to the RabbitMQ control exchange with specific routing keys. See the [main doc](control-queues.md) | ||
|
|
||
|
|
||
| ## TODO | ||
|
|
||
| - Figure out what documentation can be (safely) imported from the old private repo |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
|
|
||
| # Microservice Control Queues | ||
|
|
||
| This describes how the services can be controlled via RabbitMQ messages. | ||
|
|
||
| ### Contents | ||
|
|
||
| - [Commands](#commands) | ||
| - [Sending a message](#sending-a-message) | ||
| - [Implementing a new control command handler](#implementing-a-new-control-command-handler) | ||
| - [Control Queues and Cleanup](#control-queues-and-cleanup) | ||
|
|
||
| ## Commands | ||
|
|
||
| Commands are sent by publishing a message to the ControlExchange (specified in your config by `RabbitOptions.RabbitMqControlExchangeName`) with a specific routing key. This allows you to easily send them from the RabbitMQ web management page, or via a CLI. | ||
|
|
||
| RabbitMQ message routing keys are used to control which services recieve the message. The current format for routing keys is `smi.control.<who>.<what>`. Where `<who>` is the name of the service, and `<what>` is some defined action. The currently defined actions are: | ||
|
|
||
| ### General - any service | ||
|
|
||
| - `stop` - Stops the service | ||
| - `ping` - Logs a `pong` message. Useful for debugging | ||
|
|
||
| ### DicomReprocessor | ||
|
|
||
| - `set-sleep-time-ms` - Sets the sleep time between batches. This also requires the new value to be set in the message body | ||
|
|
||
| ### IdentifierMapper | ||
|
|
||
| - `refresh` - Refreshes any caches in use | ||
|
|
||
|
|
||
| ## Sending a message | ||
|
|
||
| Messages can be sent either via the Web UI or via a CLI (see below for details). In either case, the following applies: | ||
|
|
||
| - The `<who>` field must exactly match the name of the microservice process (e.g. `identifiermapper`) | ||
| - All routing keys should be lowercase | ||
| - `all` can be used as the `<who>` keyword to control all services | ||
| - A specific service can be messaged by including its `PID` at the end of the routing key. This is currently the only way to control a specific service instance rather than all services of a certain type | ||
|
|
||
| Examples of some routing keys: | ||
|
|
||
| ```text | ||
| smi.control.all.stop # Stop all services | ||
| smi.control.dicomtagreader.stop # Stop all DicomTagReader services | ||
| smi.control.identifiermapper.refresh1234 # Refresh the IdentifierMapper service with PID `1234` | ||
| ``` | ||
|
|
||
| Note that some services may take some time to finish their current operation and exit after receiveing a `shutdown` command. | ||
|
|
||
|
|
||
| ### Via the Web UI | ||
|
|
||
| On your RabbitMQ Management interface (`http://<rabbit host>:15672`), click `Exchanges` then `Control Exchange`. Expand the `Publish message` box then enter the message info. Any required content should be entered into the `Payload` box in plain text. Example: | ||
|
|
||
|  | ||
|
|
||
| ### Via the CLI | ||
|
|
||
| `TODO` | ||
|
|
||
| ## Implementing A New Control Command Handler | ||
|
|
||
| Implement a class which contains a method with the following signature: | ||
|
|
||
| ```c# | ||
| void MyControlHandler(string action, string message) | ||
| ``` | ||
|
|
||
| Then, instantiate your class and register its event in your host (must be a subclass of `MicroserviceHost`): | ||
|
|
||
| ```c# | ||
| var controlClass = new MyControlClass(...); | ||
| AddControlHandler(controlClass.MyControlHandler); | ||
| ``` | ||
|
|
||
| That's it! Now you will be passed the full routing key for any control message addressed to your specific microservice type (i.e. where the `<who>` part of the routing key matches your microservice name), and any message content. | ||
|
|
||
| ## Control Queues and Cleanup | ||
|
|
||
| The actual implementation of the control queues works as follows: | ||
|
|
||
| - When each service starts up, it creates a new queue named with its service name and process ID | ||
| - It then binds this queue to the global `ControlExchange`. Two bindings are created: | ||
| - `smi.control.all.*`: Matches any "send to all" routing keys | ||
| - `smi.control.<process_name>.*`: Matches "all services of my type" routing keys | ||
| - On shutdown (when the RMQ connection is closed), the control queue should be automatically delted by the server | ||
|
|
||
| The creation of the control queue is performed during a single ad-hoc connection, and is not part of the standard Consumer process (for _reasons_). One consequence of this is that if a microservice crashes _after_ the control queue is created, but _before_ the actual subscription to the queue is started (i.e. at some point during startup before RabbitMQAdapter.StartConsumer is called), then the control queue may not be automatically deleted. This isn't really an issue other than causing visual clutter on the RabbitMQ management interface. These dangling queues can be manually deleted with the [TidyQueues](../utils/RabbitMqTidyQueues) utility tool. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.