Frame Skipping Causes Label Misalignment in Video Annotation

[Buckler89]

Description

I created a video with OpenCV at 10 FPS and labeled its first 10 frames in Label Studio. When I refresh the page and open the task, the label aligns perfectly on the first frame. However, when I advance to the second frame, the video quickly skips the second frame and displays the third frame, but still shows the bounding box intended for the second frame (see the example below).

To reproduce:

Configuration

UI Labeling Configuration::

<View>
   <Labels name="videoLabels" toName="video" allowEmpty="true">
     <Label value="Man" background="blue"/>
     <Label value="Woman" background="red"/>
     <Label value="Other" background="green"/>
   </Labels>
   
   <!-- Please specify FPS carefully, it will be used for all project videos -->
   <Video name="video" value="$video" framerate="$fps"/>
   <VideoRectangle name="box" toName="video"/>
</View>

Task with Annotation

{
  "id": 15,
  "data": {
    "video": "/data/upload/1/2d96aee1-20241025-145032-937345_day.mp4",
    "fps": 10
  },
  "annotations": [
    {
      "id": 4,
      "result": [
        {
          "value": {
            "framesCount": 43,
            "duration": 4.3,
            "sequence": [
              {"enabled": true, "frame": 1, "height": 1.39, "rotation": 0, "time": 0.1, "width": 3.41, "x": 51.42, "y": 50.35},
              {"enabled": true, "frame": 2, "height": 1.39, "rotation": 0, "time": 0.2, "width": 3.27, "x": 50.43, "y": 50.35},
              {"enabled": true, "frame": 3, "height": 1.56, "rotation": 0, "time": 0.3, "width": 3.41, "x": 49.57, "y": 50.17},
              {"enabled": true, "frame": 4, "height": 1.56, "rotation": 0, "time": 0.4, "width": 3.41, "x": 48.58, "y": 50.17},
              {"enabled": true, "frame": 5, "height": 1.39, "rotation": 0, "time": 0.5, "width": 3.27, "x": 47.73, "y": 50.17},
              {"enabled": true, "frame": 6, "height": 1.39, "rotation": 0, "time": 0.6, "width": 3.27, "x": 46.88, "y": 50.17},
              {"enabled": true, "frame": 7, "height": 1.56, "rotation": 0, "time": 0.7, "width": 3.27, "x": 45.88, "y": 50},
              {"enabled": true, "frame": 8, "height": 1.56, "rotation": 0, "time": 0.8, "width": 3.41, "x": 45.03, "y": 50},
              {"enabled": true, "frame": 9, "height": 1.56, "rotation": 0, "time": 0.9, "width": 3.27, "x": 44.18, "y": 50},
              {"enabled": true, "frame": 10, "height": 1.56, "rotation": 0, "time": 1, "width": 3.27, "x": 43.18, "y": 50}
            ]
          },
          "id": "xNNNVLwq5Y",
          "from_name": "box",
          "to_name": "video",
          "type": "videorectangle",
          "origin": "manual"
        }
      ],
      "created_username": "[email protected]",
      "created_ago": "3 hours, 56 minutes",
      "completed_by": {
        "id": 1,
        "email": "[email protected]",
        "initials": "di"
      },
      "was_cancelled": false,
      "ground_truth": false,
      "created_at": "2024-11-01T18:05:06.388864Z",
      "updated_at": "2024-11-01T18:45:51.407308Z",
      "lead_time": 2214.572,
      "task": 15,
      "project": 1
    }
  ],
  "predictions": []
}

Expected Behavior:

Navigating the video back and forth, the labels should always remain aligned with the images.

Actual Behavior

1. After refreshing the page on the first frame: The label aligns correctly with the displayed image.
   

2. When advancing to the second frame: The second frame is quickly skipped, and the third frame is displayed, but it shows the label intended for the second frame. Note that the timeline is poinintg on the 2nd frame here
   

3. When returning to the first frame: The video displays the second frame, but with the label from the first frame. From this point onward, labels are consistently one frame behind the displayed image, regardless of the selected frame. This misalignment continues if I advance further, and the actual first frame is never shown again until I refresh the page.

After this misalignment starts, the labels stay one frame behind the images until I refresh the page. However, after refreshing, the issue recurs as soon as I advance by one frame, following the same pattern.

The root of the problem seems to be in the video display, which skips a frame as described, while the label display remains correctly sequenced.

Video used

2d96aee1-20241025-145032-937345_day.mp4

Summary

This appears to be a bug that causes video frames and labels to become misaligned, making accurate video labeling challenging. This misalignment significantly impacts labeling precision, so addressing it would improve usability and reliability for video annotation tasks.

Environment (please complete the following information):

• OS: ubuntu 22.04
• Label Studio Version: docker 1.13.1

[Buckler89]

I believe I have found a similar, though slightly different, issue. Based on the comment by @hlomzik in issue #3315, where it is stated that:

Video is played well anyway; the only thing affected is the timeline. We detect the current frame as (time passed / given frame rate).

I noticed that in my Label Studio instance, the computed video length is incorrect, as shown in the screenshot.

Details:

• Video FPS: 10
• Total Frames: 43

Given these details, the total length should be 4.3 seconds (since ( \frac{43 \text{ frames}}{10 \text{ FPS}} = 4.3 \text{ seconds} )). However, Label Studio displays the total length as 4.02 seconds (which I suppose might actually mean 4.2 seconds, but that's another issue).

In my opinion, this could be related to the problem of frame skipping due to time approximation in the calculations mentioned earlier. This is just an idea.

Could someone please confirm if this is a known issue or if there's a recommended workaround? Any assistance or guidance would be greatly appreciated.

[Buckler89]

To help reproduce the issue, I created a dummy video with a frame number overlay. Here’s the video that you can use to reproduce the issue. The frame count starts at 1 and goes up to 15, for a total of 15 frames. The video has an FPS of 10.

36ab8fd2-20241104-100249-868294_day.mp4

The media player UI should always match the frame counter displayed in the video. Start from the first frame and proceed one frame at a time. Moving back and forth frame by frame will clearly show the misalignment with the video timeline. Refresh the page to ensure the frame counter starts from 1 again.

Unfortunately, I do not have the ability to investigate the frontend code on my own at the moment. Any help in resolving this issue would be greatly appreciated, as this is a blocking problem for me: I have many short videos that need to be labeled frame by frame.

I’m confident this is a bug in the video media player.