Frame numbers for urls of video data

[csheaff]

Hello,

I would like to download specific frames of video data associated with an annotation. However, I've found that (1) a frame URL corresponding to an annotation is not embedded in the annotation object and (2) URLs for all frames are grouped together in the data_unit and not in an order corresponding to the frame ordering.

Is there some way to determine to which frame each URL corresponds? Any help would be appreciated.

[denis-cord]

Hey @csheaff,

I'll be able to help you with this question.

The first question I have for you is whether you are upload/creating "videos" or "image groups". Please let me know if the distinction is unclear.

In the case of videos, you will only be able to get a URL to the entire image.

In the case of image groups, you will be able to find a URL for each individual image which corresponds to a single frame in the UI.

If you do a get_label_row call, you'll be able to get the links that you're looking for (please ensure to set the get_signed_url to True).

Below is an example where I have highlighted how you can get the URL of all frames and also the frame ordering from a get_label_row response for image groups.

empty_image_group_labels = {
    "label_hash": "f1e6bc82-9f89-4545-8abb-f271bf28cf99",
    "dataset_hash": "d9f19c3c-5cd0-4f8c-b98c-6c0e24676224",
    "dataset_title": "One image group",
    "data_title": "image-group-8375e",
    "data_type": "img_group",
    "data_units": {
        "f850dfb4-7146-49e0-9afc-2b9434a64a9f": {
            "data_hash": "f850dfb4-7146-49e0-9afc-2b9434a64a9f",
            "data_title": "Screenshot 2021-11-24 at 18.35.57.png",
            # NOTE: Data link below:
            "data_link": "https://storage.googleapis.com/cord-ai-platform.appspot.com/cord-images-prod/yiA5JxmLEGSoEcJAuxr3AJdDDXE2/f850dfb4-7146-49e0-9afc-2b9434a64a9f?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=firebase-adminsdk-64w1p%40cord-ai-platform.iam.gserviceaccount.com%2F20221201%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20221201T133838Z&X-Goog-Expires=604800&X-Goog-SignedHeaders=host&X-Goog-Signature=94c66d85014ff52a99fec1cf671ccc1b859ebead4308ca82c4d810e13ac285d2afa8cfa4bfcbd09f615b243b95d9b1d5d1d779e7a4ba5832a2207b4f3b99dbe405ded373f03f06abe4e24098e70568c269899f2f397c7a4392a1c3090bff2b8c98f2177f5db36f0884a83033f404354bdfda0506bf162e25ff6186fc54104e8273e86959b0296958a03359514660528a54ba94e25c59e59534ce5102f9c87ff7cb03a591606b3a191123af4a30fa4296a788a9433f0c8c1dc7d3f80a022cc42f8716ba44d09ecd04118dc6e4ee5977ffbadcc8d635cc4e906f024dba26e520cfc304fc0f3458a3e3b2422c196956fd3024a6eba0512d557683487b10a1a381b4",
            "data_type": "image/png",
            # NOTE: data sequence, i.e. "frame number" below:
            "data_sequence": "0",
            "width": 952,
            "height": 678,
            "labels": {
                "objects": [],
                "classifications": [],
            },
        },
        "177d1bb7-5394-4772-ba9f-4569f0c2a995": {
            "data_hash": "177d1bb7-5394-4772-ba9f-4569f0c2a995",
            "data_title": "Screenshot 2021-11-24 at 18.36.02.png",
            "data_link": "https://storage.googleapis.com/cord-ai-platform.appspot.com/cord-images-prod/yiA5JxmLEGSoEcJAuxr3AJdDDXE2/177d1bb7-5394-4772-ba9f-4569f0c2a995?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=firebase-adminsdk-64w1p%40cord-ai-platform.iam.gserviceaccount.com%2F20221201%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20221201T133838Z&X-Goog-Expires=604800&X-Goog-SignedHeaders=host&X-Goog-Signature=808ea15115717eaa939b6b66a74c2d8771216a16d621d7ff8a002e5bec65fcc3053097e2bba0711e3442b4a41afe9744e42ffe6b08194cd83debcf3dbb2d649ab2a92df2b02ed4e77e8eedef577007bb33681410ec164e24ba4c5643df744687762069e9dd360c0fa0820e90cd977d0c199b391e1425dd0e696a517e747b70ed4953f514be87846317c762c519af1f49066797d7a5f9e476bf9abf06e62a3ed5ebcf05516782e2ec913af68a6bedac0349c6b2a528e5411ec96275326c0e67d699cea81eadb6660e31d4a7d611db64b06f67098bad266c7bc6eb6e5191b7a97101a01ffd61b40aaaf24dea2026add406ec362df4df8eac26750b569295c28b86",
            "data_type": "image/png",
            "data_sequence": "1",
            "width": 952,
            "height": 678,
            "labels": {"objects": [], "classifications": []},
        },
    },
    "object_answers": {},
    "classification_answers": {},
    "object_actions": {},
    "label_status": "LABEL_IN_PROGRESS",
}

Let me know if that helps.

[csheaff]

Thanks for the response.

My data is of type "dicom" which appears to be treated like video data. Using:

row = get_label_row(label_hash, get_signed_url=True)
data_unit = list(row['data_units'].values())[0]
urls = data_unit["data_links"]`

...I observe urls to be a list of urls corresponding to all slices of the dicom but in no discernible order. Is it the case that there is no way to associate an annotation for a frame in the dicom with that frame's url?

It's true that I can go to the original source for the uploaded data or sometimes find the frame number in the dicom attributes, but it would be convenient to use the urls to get the relevant frames directly from Encord.

[denis-cord]

Got it @csheaff

Right now, as you have identified correctly, there is no association between the frames and the DICOM files. The reason for this is that DICOM files can have very flexible mappings between one or multiple DICOM files to one of multiple frames.

With that being said, the engineering team is actively looking into this and aiming to release such a mapping in the next weeks. I can comment on this issue once we have released an update.

[csheaff]

Thanks @denis-cord .

[arthur-encord]

Hi @csheaff ! Apologies for the late reply on our end. We worked on exporting metadata for DICOM annotations in the label rows; you should now be able to see a new metadata field in each indexed slice and at the data unit level. The slice metadata will contain a link to the DICOM file, the DICOM file's instance UID , and the frame number if the DICOM file is multi-frame, for example:

    "data_units": {
      "3dd...4": {
        "data_type": "application/dicom",
        "labels": {
          "0": {
            "objects": [
              {
                "color": "#D33115",
                "boundingBox": {
                  "h": 0.1236,
                  "w": 0.1709,
                  "x": 0.5388,
                  "y": 0.3677
                }
              }
            ],
            "classifications": [],
            "metadata": {
              "dicom_instance_uid": "1.2.166.1...4023482721087",
              "multiframe_frame_number": null,
              "file_uri": "https://...ea7",
              "width": 2457,
              "height": 1996
            }
          }
        },
        "metadata": {
          "patient_id": "6af121c8...c2fe921",
          "study_uid": "1.2.166.1...408371899414",
          "series_uid": "1.2.166.1...66885"
        },

Could you try opening and saving your labels in the label editor, and then exporting the labels?
(we're working on enabling the metadata for all the existing annotations but at the moment it requires having the annotations saved in a newer version of the label editor)