How can I run a MAP container without using App SDK CLI

[MMelQin]

**Note the main section after this note is applicable to MAP built with the v5.x and below of the App SDK. For MAP built with v0.6, the script and command still works except for the replacement of monai with holoscan where the word appears in the file path, attributes, etc.

For example, to run this MAP that is built with MONAI Deploy App SDK v0.6, ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0, you would do as the following,

docker run --gpus=all --name holoscan_liver_ai_app -t --rm \
-v <path to DICOM input folder on host>:/var/holoscan/input/ \
-v <path to output folder on host>:/var/holoscan/output/ \
-v <path to temp dir on host>:/var/holoscan/ \
-e HOLOSCAN_INPUT_PATH="/opt/holoscan/app/dcm/" \
-e HOLOSCAN_OUTPUT_PATH="/var/holoscan/output/" \
-e HOLOSCAN_MODEL_PATH="/opt/holoscan/models/" \
-e HOLOSCAN_WORKDIR="/var/holoscan/" \
--entrypoint /bin/bash ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0 -c "python3 -u /opt/holoscan/app/"

And if using container image internal folder (to avoid extracting test data and mapping host folders to the container, use the following command,

docker run --gpus=all --name holoscan_liver_ai_app -t --rm \
-e HOLOSCAN_INPUT_PATH="/opt/holoscan/app/dcm/" \
-e HOLOSCAN_OUTPUT_PATH="/var/holoscan/output/" \
-e HOLOSCAN_MODEL_PATH="/opt/holoscan/models/" \
-e HOLOSCAN_WORKDIR="/var/holoscan/" \
--entrypoint /bin/bash ghcr.io/mmelqin/monai_ai_livertumor_seg_app_no_stl-x64-workstation-dgpu-linux-amd64:2.0 -c "python3 -u /opt/holoscan/app/"

**

As part of the IHE Conntectathon IAIP, a couple MONAI Application Packages, MAPs, have been published on Github Container Registry, namely

• ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0
• ghcr.io/mmelqin/monai_ai_livertumor_seg_app:1.0

Questions are

• How can I run a MAP as Docker container without using MONAI Deploy App SDK CLI, e.g. in my own environment or platform?
• Where can I get the DICOM studies for testing?

The MAP has a defined set of inputs along with a set of required environment variables, and these are defined in a couple JSON files in the MAP container image. The App SDK CLI command, run, inspects and sets them correctly. An executor embedded in the MAP container image does perform the similar functions but it is tied to a specific platform. Without using these couple helpers, one can still just run a MAP as Docker container with the bash script provided below. Simple, isn't it?

To help with test data preparation, these couple MAPs also contains a DICOM series each as test input; the test DICOM was converted from Medical Decathlon training and validation image dataset. The script below thus also supports extracting the test data to a host folder, map_extracted, for users to inspect with a DICOM viewer, as well as test input to the Docker container.

At runtime, the script also try its best to preserve the input and output, map_extracted and map_output, from last run to map_extracted_previous and map_output_previous. Only one last version is preserved though; good enough for running the two MAP, in case one forgets to move the data to more permeant folders.

To prepare the script

• copy the code content into a shell script file, e.g. run_map_container.sh
• chmod +x run_map_container.sh

The script supports two (2) positional arguments

• MAP container image name, e.g. ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0
• Input DICOM folder in local file system

If both are missing, default MAP image is used per definition in the script, and DICOM series extracted from the image to host folder map_extracted. In the case when the second positional argument is missing, it also means the DICOM series will be extracted.

One important note is that, with Docker installed, Nvidia Container Toolkit and/or nvidia-docker 2 needs to be installed for support of GPU.

So please give it a spin, and let us know if there are any issues. Kudos are welcome, too!

#!/bin/bash
# Copyright 2022 MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


# Default MAP container Image, and its embedded dcm folder and app folder with __main__.py
MAP="ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0"
DCM_IN_MAP="/opt/monai/app/dcm"
APP_IN_MAP="/opt/monai/app/"

# Var for input DICOM folder, if passed on commandline
INPUT_DCM=""

# Set the output folder name
MAP_OUTPUT="map_output"

# Temp workdir on the for Docker to write to at runtime.
TMEP_DIR="map_tmp"
if [ ! -d ${TMEP_DIR} ]; then
    mkdir ${TMEP_DIR}
else
    echo "Cleaning up temp workdir ${TEMP_DIR}"
    sudo rm -r ${TMEP_DIR}/*
fi

# Get the MAP container image and input DCIOM folder from commandline.
# Use MAP default and extract input DICOM if args not provided.
if [ "$1" != "" ]; then
    MAP="$1"
    echo  "MAP container image: "$1""
    if [ "$2" != "" ]; then
        INPUT_DCM="$2"
        echo "Input DICOM files in folder: "$2""
    else
        echo "Input DICOM not provided. Will extract from MAP container image."
    fi
else
    echo "Use default MAP and its embedded test DICOM files: ${MAP}"
fi

# If input not provided, extract test DICOM files embedded in the MAP container image
if [ -z "${INPUT_DCM}" ]; then
    INPUT_DCM="extracted_dcm"

    if [ -d "${INPUT_DCM}" ]; then
        if [ -d "${INPUT_DCM}_previous" ]; then
            echo "Removing existing ${INPUT_DCM}_previous."
            sudo rm -r ${INPUT_DCM}_previous
        fi
        echo "Try to preserve old extracted input to ${INPUT_DCM}_previous."
        sudo mv ${INPUT_DCM} ${INPUT_DCM}_previous
    fi

    echo "Extracting input DICOM files to ${INPUT_DCM}...."
    extractor=$(docker create $MAP)
    docker cp ${extractor}:${DCM_IN_MAP} ${INPUT_DCM}
    docker rm ${extractor}
    echo "Extracted input DICOM files in folder ${INPUT_DCM}:"
    ls ${INPUT_DCM}
fi

if [ -d "${MAP_OUTPUT}" ]; then
        if [ -d "${MAP_OUTPUT}_previous" ]; then
            echo "Removing existing ${MAP_OUTPUT}_previous."
            sudo rm -r ${MAP_OUTPUT}_previous
        fi
        echo "Try to preserve last run output to ${MAP_OUTPUT}_previous."
        sudo mv ${MAP_OUTPUT} ${MAP_OUTPUT}_previous
fi

# Find the absolute paths for Docker vol mapping
ABS_INPUT_DCM=$(readlink -f "${INPUT_DCM}")
ABS_MAP_OUTPUT=$(readlink -f "${MAP_OUTPUT}")
ABS_MAP_TEMP_DIR=$(readlink -f "${TMEP_DIR}")

# Run ${MAP} container image.
# Note: --gpus=all can be used in lieu of --runtime=nvidia if only Nvidia Container Toolkit is installed.
#       Make sure the docker internal folders are matching the env vars if changed.
RUN_TIME="--gpus=all"

# Pick a name for the running Docker
NAME_DOCKER="map_docker"

echo "Running Docker ${NAME_DOCKER} with image ${MAP}, input from ${INPUT_DCM} and output in ${MAP_OUTPUT}..."

docker run ${RUN_TIME} --name ${NAME_DOCKER} -t --rm \
-v ${ABS_INPUT_DCM}:/var/monai/input/ \
-v ${ABS_MAP_OUTPUT}:/var/monai/output/ \
-v ${ABS_MAP_TEMP_DIR}:/var/monai/ \
-e MONAI_INPUTPATH="/var/monai/input/" \
-e MONAI_OUTPUTPATH="/var/monai/output/" \
-e MONAI_MODELPATH="/opt/monai/models" \
-e MONAI_WORKDIR="/var/monai/" \
--entrypoint /bin/bash ${MAP} -c "python3 -u ${APP_IN_MAP}"

echo "Docker ${NAME_DOCKER} with image ${MAP} finished, input from ${INPUT_DCM} with output in ${MAP_OUTPUT}:"
ls ${MAP_OUTPUT}

[dbericat]

Great to see this! Idea to consider - move these utils to a /examples/utils folder (or similar) so they are ready to go.
Maybe handy for users who just review the code on their preferred IDE and potentially may miss the discussion.

[vikashg]

Hi Ming,
Thanks for this code and I have been following to build few MAPS which we are planning to present soon. I am curious about packaging the container. After I wrote the MAP and everything works nice as it should even with the monai-deploy cli.
However, I am curious how you package the MAP. Because as far as tutorials are concerned, this command

monai-deploy package -b nvcr.io/nvidia/pytorch:21.11-py3 my_app --tag my_app:latest -m model.ts

builds out the package, but how the further dependencies are packaged into docker. Generally a Dockerfile is used but that is not the case here. So, I am curious. When I was trying to build my docker image as yours I get dependency errors as follows

/home/gupta/anaconda3/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.0
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
Traceback (most recent call last):
  File "/home/gupta/anaconda3/bin/monai-deploy", line 33, in <module>
    sys.exit(load_entry_point('monai-deploy-app-sdk', 'console_scripts', 'monai-deploy')())
  File "/home/gupta/anaconda3/bin/monai-deploy", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/gupta/anaconda3/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/home/gupta/anaconda3/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'monai.deploy'

[MMelQin]

The App SDK CLI package queries the Application for its dependent packages, which are in the decorators of the App class as well as Operators class. This way, the Dockerfile, more precisely part thereof, is dynamically generated based on a template.

In your specific errors, there is a conflict on the defined version of Numpy and the SciPy, A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.0.

In the minimal requirements of the MONAI Deploy App SDK, numpy>=1.21.2, while in my virtual env, numpy is at 1.23.2. You can try to pin numpy < 1.23.0 in the decorators of the classes in your code.

[vikashg]

Thanks @MMelQin knowing that the CLI package queries for the Application for dependencies is very helpful and I will work to resolve it

[vikashg]

Thanks once again @MMelQin its resolved 👍. There were no issues with the packager. I was trying to build the package while not being in the correct conda environment.

[dbericat]

Thanks for the kudos and idea. I also considered checking in the code to the repo, though it will need to go through our contribution process though David you can review and approve it.

I'd be carefully on choosing the correct place in the repo to avoid confusion. This script is meant for users who just want to try out the MAP WITHOUT using the MONAI eco system. Also, embedding the test DICOM series in a MAP is typically discouraged. Within the MONAI Deploy realm, the App SDK CLI run command, implemented in Python, provides a more robust method to run MAP docker, and the MONAI Deploy Task Manager with Docker Plugin natively handles the MAP on MONAI Deploy platform with more grace. So this script will belong to the area of 3rd party integration, as a starting point and a reference for creating a more robust mechanism for running a MAP, level one integration: consuming MAP as an opaque container image, shall we say.

This makes sense to me. Let's add the code to the 3rd party integrations folder

[ericspod]

I looked at this script and I tried to invoke the image directly to get an internal interactive shell:

$ docker run -ti --rm --gpus=all ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0 --entrypoint /bin/bash
fatal: Conversion from Nullable`1 to SemanticVersion is not supported.

Is there something wrong with this command to get a shell in the container? My Docker version is 20.10.12.

[MMelQin]

I looked at this script and I tried to invoke the image directly to get an internal interactive shell:

$ docker run -ti --rm --gpus=all ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0 --entrypoint /bin/bash
fatal: Conversion from Nullable`1 to SemanticVersion is not supported.

Is there something wrong with this command to get a shell in the container? My Docker version is 20.10.12.

Sorry @ericspod , I've just run it and encountered the same error, too. I'll look into it.

The --entrypoint option is out of position, and it should be
docker run -ti --rm --gpus=all --entrypoint /bin/bash ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0

[ArpanGyawali]

hi, @MMelQin @vikashg
I tried this for my custom MAP that extracts brain from head CTA image.
When i run the app using
python my_app -i input_series -o output_dir -m model.ts
It gives me the mask output without any error but i packaged the MAP and pushed it into ghcr.io using
monai-deploy package -b nvcr.io/nvidia/pytorch:22.08-py3 my_app --tag ghcr.io_image_namet -m model.ts
docker push ghcr.io_image_name
When i pull the same image in another machine using 'docker pull' and run the script that you provided run_map_container.sh using
./run_map_container.sh ghcr.io_image_name CTA_input_series
I get runtime error in the transforms. The terminal output is shown in detail below.

Input DICOM files in folder: CTA_series
Removing existing map_output_previous.
Try to preserve last run output to map_output_previous.
Running Docker map_docker with image ghcr.io/arpangyawali/skull-strip-app:1.0, input from CTA_series and output in map_output...
/opt/conda/lib/python3.8/site-packages/scipy/__init__.py:138: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.4)
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion} is required for this version of "
Going to initiate execution of operator DICOMDataLoaderOperator
Executing operator DICOMDataLoaderOperator (Process ID: 1, Operator ID: 4e30c075-f2bf-4752-b222-9bf88057fbcd)
Done performing execution of operator DICOMDataLoaderOperator

Going to initiate execution of operator DICOMSeriesSelectorOperator
Executing operator DICOMSeriesSelectorOperator (Process ID: 1, Operator ID: e5bfda4c-8287-4dcf-bb86-e4d1d743cea9)
[2023-10-05 04:45:08,451] [INFO] (root) - Finding series for Selection named: (.*?)
[2023-10-05 04:45:08,451] [INFO] (root) - Searching study, : 1.2.840.113845.11.1000000001927670467.20170108051502.4595629
  # of series: 1
[2023-10-05 04:45:08,451] [INFO] (root) - Working on series, instance UID: 1.3.12.2.1107.5.1.4.73615.30000017010622324126200127410
[2023-10-05 04:45:08,451] [INFO] (root) - On attribute: 'StudyDescription' to match value: '(.*?)'
[2023-10-05 04:45:08,451] [INFO] (root) -     Series attribute StudyDescription value: CT Angio COW + Perfusion + or - Brain
[2023-10-05 04:45:08,451] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2023-10-05 04:45:08,451] [INFO] (root) - On attribute: 'Modality' to match value: '(?i)CT'
[2023-10-05 04:45:08,451] [INFO] (root) -     Series attribute Modality value: CT
[2023-10-05 04:45:08,451] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2023-10-05 04:45:08,451] [INFO] (root) - On attribute: 'SeriesDescription' to match value: '(.*?)'
[2023-10-05 04:45:08,451] [INFO] (root) -     Series attribute SeriesDescription value: DE AXIAL THIN SOFT  F_0.6
[2023-10-05 04:45:08,451] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2023-10-05 04:45:08,451] [INFO] (root) - Selected Series, UID: 1.3.12.2.1107.5.1.4.73615.30000017010622324126200127410
Done performing execution of operator DICOMSeriesSelectorOperator

Going to initiate execution of operator DICOMSeriesToVolumeOperator
Executing operator DICOMSeriesToVolumeOperator (Process ID: 1, Operator ID: bf83b634-e1ed-4511-bc06-ba4053f5b90b)
Done performing execution of operator DICOMSeriesToVolumeOperator

Going to initiate execution of operator SkullStripOperator
Executing operator SkullStripOperator (Process ID: 1, Operator ID: 02d97890-9ff8-4e4c-9f62-fefce8ec4480)
/root/.local/lib/python3.8/site-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary LoadImaged.__init__:image_only: Current default value of argument `image_only=False` has been deprecated since version 1.1. It will be changed to `image_only=True` in version 1.3.
  warn_deprecated(argname, msg, warning_category)
/root/.local/lib/python3.8/site-packages/monai/utils/deprecate_utils.py:321: FutureWarning: monai.transforms.io.dictionary SaveImaged.__init__:resample: Current default value of argument `resample=True` has been deprecated since version 1.1. It will be changed to `resample=False` in version 1.3.
  warn_deprecated(argname, msg, warning_category)
Converted Image object metadata:
SeriesInstanceUID: 1.3.12.2.1107.5.1.4.73615.30000017010622324126200127410, type <class 'str'>
SeriesDate: 20170108, type <class 'str'>
SeriesTime: 054404.143000, type <class 'str'>
Modality: CT, type <class 'str'>
SeriesDescription: DE AXIAL THIN SOFT  F_0.6, type <class 'str'>
PatientPosition: HFS, type <class 'str'>
SeriesNumber: 40, type <class 'int'>
row_pixel_spacing: 0.53515625, type <class 'float'>
col_pixel_spacing: 0.53515625, type <class 'float'>
depth_pixel_spacing: 0.5, type <class 'float'>
row_direction_cosine: [1.0, 0.0, 0.0], type <class 'list'>
col_direction_cosine: [0.0, 1.0, 0.0], type <class 'list'>
depth_direction_cosine: [0.0, 0.0, 1.0], type <class 'list'>
dicom_affine_transform: [[   0.53515625    0.            0.         -134.73242188]
 [   0.            0.53515625    0.         -319.73242188]
 [   0.            0.            0.5        -285.5       ]
 [   0.            0.            0.            1.        ]], type <class 'numpy.ndarray'>
nifti_affine_transform: [[  -0.53515625   -0.           -0.          134.73242188]
 [  -0.           -0.53515625   -0.          319.73242188]
 [   0.            0.            0.5        -285.5       ]
 [   0.            0.            0.            1.        ]], type <class 'numpy.ndarray'>
StudyInstanceUID: 1.2.840.113845.11.1000000001927670467.20170108051502.4595629, type <class 'str'>
StudyID: 2017R0003334-1, type <class 'str'>
StudyDate: 20170108, type <class 'str'>
StudyTime: 052353, type <class 'str'>
StudyDescription: CT Angio COW + Perfusion + or - Brain, type <class 'str'>
AccessionNumber: N007, type <class 'str'>
selection_name: (.*?), type <class 'str'>
Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/spatial/dictionary.py", line 522, in inverse
    d[key] = self.spacing_transform.inverse(cast(torch.Tensor, d[key]))
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/spatial/array.py", line 547, in inverse
    return self.sp_resample.inverse(data)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/spatial/array.py", line 248, in inverse
    out: torch.Tensor = SpatialResample.__call__(self, data, **kw_args)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/spatial/array.py", line 223, in __call__
    return spatial_resample(
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/spatial/functional.py", line 178, in spatial_resample
    img = affine_xform(img.unsqueeze(0), theta=xform.to(img), spatial_size=spatial_size).squeeze(0)  # type: ignore
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/monai/networks/layers/spatial_transforms.py", line 579, in forward
    grid = nn.functional.affine_grid(theta=theta[:, :sr], size=list(dst_size), align_corners=self.align_corners)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 4326, in affine_grid
    return torch.affine_grid_generator(theta, size, align_corners)
RuntimeError: CUDA out of memory. Tried to allocate 4.45 GiB (GPU 0; 3.81 GiB total capacity; 756.03 MiB already allocated; 2.81 GiB free; 936.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/post/dictionary.py", line 702, in __call__
    inverted = self.transform.inverse(input_dict)
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/compose.py", line 351, in inverse
    data = apply_transform(
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <bound method Spacingd.inverse of <monai.transforms.spatial.dictionary.Spacingd object at 0x7fc07d2148b0>>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/monai/app/__main__.py", line 4, in <module>
    AISkullStripApp(do_run=True)
  File "/opt/monai/app/app.py", line 25, in __init__
    super().__init__(*args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/monai/deploy/core/application.py", line 129, in __init__
    self.run(log_level=args.log_level)
  File "/opt/monai/app/app.py", line 30, in run
    super().run(*args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/monai/deploy/core/application.py", line 429, in run
    executor_obj.run()
  File "/root/.local/lib/python3.8/site-packages/monai/deploy/core/executors/single_process_executor.py", line 125, in run
    op.compute(op_exec_context.input_context, op_exec_context.output_context, op_exec_context)
  File "/opt/monai/app/SkullStrip_operator.py", line 68, in compute
    infer_operator.compute(op_input, op_output, context)
  File "/root/.local/lib/python3.8/site-packages/monai/deploy/operators/monai_seg_inference_operator.py", line 278, in compute
    d = [post_transforms(i) for i in decollate_batch(d)]
  File "/root/.local/lib/python3.8/site-packages/monai/deploy/operators/monai_seg_inference_operator.py", line 278, in <listcomp>
    d = [post_transforms(i) for i in decollate_batch(d)]
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/compose.py", line 322, in __call__
    result = execute_compose(
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/root/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.post.dictionary.Invertd object at 0x7fc06b4d8250>
Docker map_docker with image ghcr.io/arpangyawali/skull-strip-app:1.0 finished, input from CTA_series with output in map_output:

This is the error i got.
I dont get any error when i run the MAP before deployment

The transform used in the SkullStripOperator are
Pre-transform

        return Compose(
            [
                LoadImaged(keys=my_key, reader=img_reader),
                EnsureTyped(keys=my_key),
                EnsureChannelFirstd(keys=my_key),
                Spacingd(keys=my_key, pixdim=[1.0, 1.0, 1.0]),
                ScaleIntensityRanged(keys=my_key, a_min=-200, a_max=1000, b_min=0.0, b_max=1.0, clip=True),
            ]
        )

Post-transform

        return Compose(
            [
                EnsureTyped(keys=pred_key),
                # Flipd(keys=pred_key, spatial_axis=1),
                Orientationd(keys=pred_key, axcodes='RPI', labels=(('R', 'L'), ('A', 'P'), ('S', 'I'))),
                KeepLargestConnectedComponentd(keys=pred_key),
                Invertd(
                    keys=pred_key,
                    transform=pre_transforms,
                    orig_keys=self._input_dataset_key,
                    nearest_interp=False,
                    to_tensor=True,
                ),
                Activationsd(keys=pred_key, softmax=True),
                AsDiscreted(keys=pred_key, argmax=True),
                
                SaveImaged(
                    keys=pred_key,
                    output_dir=out_dir,
                    output_postfix="seg",
                    output_dtype=uint8,
                ),

Please can anyone help me with this problem.

[MMelQin]

@ArpanGyawali thanks for the question.

Looks the transforms are fine, but the container was run on a system wit a GPU that has less than required memory, per the error in the output

RuntimeError: CUDA out of memory. Tried to allocate 4.45 GiB (GPU 0; 3.81 GiB total capacity; 756.03 MiB already allocated; 2.81 GiB free; 936.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

If you could not run on a system with larger GPU memory, you can change the docker run command NOT to use GPU, i.e. set RUN_TIME to blank in the script, so the command like below will not make GPU visible to the container.

docker run ${RUN_TIME}

Please all note the new note I put at the top, stating the differences with MAP built with v0.6 of the App SDK.
Thanks.

[ArpanGyawali]

Thank yo so much @MMelQin ..you are a life saver
It worked in CPU mode.
So, for running in GPU, the GPU memory required for running the image is atleast 4.45 GiB. So it can't be done on a 4GB GPU right?? Mine is GTX 1650Ti(4GB).

[MMelQin]

That's true, unfortunately. The reliance on GPU is mostly from the Torch inference in all of the examples and MAPs, but fortunately torch can detect the presence of GPU (and compatibility of the GPU driver; case in mind, on Oct 4th the new version of torch was released, and all of a sudden the example apps suffered huge performance hit. It turns out that the new version of torch detected that the GPU driver on my desktop is not compatible and it just fell back to CPU, using all of the CPUs locking up my system yet still suffered huge perf degradation).

So, it is ideal to have at least a GPU with 8 GB, and 16 GB if training is a consideration. I ran the LiverTumor yesterday after pinning the torch to the previous version, 2.0.1, and measured the GPU mem usage was around 13GB, though typically it is around 7-8GB.

This also brings up another reason why the MAP metadata indicates the resource requirement, so that when performance is an important consideration, the required GPU needs to be sized accordingly. For example, the liver turmor MAP v1.0 finishes around 67 seconds with GPU, and 2 to 3 minutes with 12 CPU cores, high spec ones. The new version of the MAP, based on App SDK v0.6, got that elapsed time down to 47 seconds, a 30% reduction.