Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add I/O support for MMPose #175

Open
niksirbi opened this issue May 10, 2024 · 5 comments · May be fixed by #198 or #242
Open

Add I/O support for MMPose #175

niksirbi opened this issue May 10, 2024 · 5 comments · May be fixed by #198 or #242
Assignees
Labels
enhancement New optional feature

Comments

@niksirbi
Copy link
Member

Motivation

Some of our collaborators work with human pose estimation, using MMPose: OpenMMLab Pose Estimation Toolbox and Benchmark.

The would like to import the predicted poses into movement, to leverage our filtering + analysis (and in future, visualisation) features.

Format

The pose predictions are done by an MMPoseInferencer class, the usage and outputs of which are described here. The results are then serialised into json for storage, see code here.

This is an example of how the outputs look like:

{
    "frame_id": 0,
    "instances": [
      {
        "keypoints": [
          [
            961.4871847546324,
            39.44995934640923
          ],
          [
            787.3275313021991,
            255.6230511152969
          ],
        ],
        "keypoint_scores": [
          0.009758014231920242,
          0.005633828230202198,
        ],
        "bbox": [
          [
            496.4014892578125,
            0.298583984375,
            1016.2412109375,
            720.661376953125
          ]
        ],
        "bbox_score": 0.863023042678833
      }
    ]
  },
  {
    "frame_id": 1,
    "instances": [
      {
        "keypoints": [

What we can do

Since movement's data structure already supports keypoints and associated confidence scores, its should be easy to write functions that read/write the "keypoints" and "keypoint_scores" information for all instances and frames from/to such json files.

The public-facing functions can be named load_poses.from_mmpose_file() and save_poses.to_mmpose_file() to conform with our existing naming conventions.

The tricky bit is what to do about the bounding boxes. movement currently doesn't support those (but future support is planned, see #167). Maybe in the first instance we can only care about the keypoints, and think about the bboxes, at a later stage.

@niksirbi niksirbi added the enhancement New optional feature label May 10, 2024
@sfmig
Copy link
Contributor

sfmig commented May 13, 2024

Maybe in the first instance we can only care about the keypoints, and think about the bboxes, at a later stage.

Potentially we can treat the bbox center as an additional keypoint? 🤔

@niksirbi
Copy link
Member Author

We could, or the two relevant corners (upper left and lower right). Maybe we can designate some reserved keypoint names for such occasions (bbox_start, bbox_end)?

@niksirbi
Copy link
Member Author

This discussion is also relevant for #167

@niksirbi
Copy link
Member Author

@ivanvrlg I've written a "developer guide" specifically for adding new Input/Output functionalities to movement. I hope that it will help clarify the relevant code structure and make it easier to find things. Feel free to comment under the relevant thread on Zulip if things are unclear.

@ivanvrlg
Copy link
Collaborator

Thanks Niko, this looks like it will be super helpful! I will post on Zulip if I have any questions.

@ivanvrlg ivanvrlg linked a pull request May 30, 2024 that will close this issue
7 tasks
This was linked to pull requests Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New optional feature
Projects
Status: 🚧 In Progress
3 participants