How to implement PoseNet inside SSDSample?

[gedewidiastawan]

Hi,

I have a project for detecting an ID Card and also the body pose.

1. SSD for detecting an ID Card
2. PoseNet for detect body pose

I just want to make sure, it's possible to implement PoseNet inside SSDSample.cs and use same texture resource? PoseNet have their example implementation called PoseNetSample, but i want implement PoseNet and SSD with one and same texture.

[asus4]

@gedewidiastawan Yes, it's possible to run PoseNet and SSD inferences using a shared texture. The latest camera texture is passed as a parameter in the OnTextureUpdate. Invoking PoseNet and SSD inference should work in this method.

tf-lite-unity-sample/Assets/Samples/PoseNet/PoseNetSample.cs

Line 63 in 0414693

private void OnTextureUpdate(Texture texture)

If you want to try more optimization,
The input sizes are different between the two models. Exporting a new model with the exact input size will allow using a shared input buffer. Which means you can skip the texture -> byte conversion.

[asus4]

Yes, it the event listener in SsdSample.cs

[gedewidiastawan]

I invoke the PoseNet inside Invoke listener in SsdSample.cs, however it's seems like the pose result is not accurate. It's require to resize the texture before invoking posenet? Or just directly use the texture?

[gedewidiastawan]

@asus4 any feedback about the previous question? Were are stuck because the result is not accurate. Maybe you can give us a suggestion about the implementation.

[asus4]

Regarding the accuracy, I need more context - some captures and reproducible code -. Generally speaking, it should work without any texture resizing as it's handled inside ImageInference.cs.

[gedewidiastawan]

@asus4 here the capture of the result and the SsdSample with PoseNet implementation.

using TensorFlowLite;
using UnityEngine;
using UnityEngine.UI;
using TextureSource;
using Unity.Mathematics;
using System.IO;
using TMPro;

[System.Obsolete("This SSD is deprecated. Try EfficientDet instead.")]
[RequireComponent(typeof(VirtualTextureSource))]
public class SsdSample : MonoBehaviour
{
    [SerializeField]
    private SSD.Options options = default;

    [SerializeField]
    public AspectRatioFitter frameContainer = null;

    [SerializeField]
    private Text framePrefab = null;

    [SerializeField, Range(0f, 1f)]
    private float scoreThreshold = 0.8f;

    [SerializeField]
    private TextAsset labelMap = null;

    private SSD ssd;
    private Text[] frames;
    private string[] labels;

    private PoseNet poseNet;
    private PoseNet.Result[] poseResults;
    private PrimitiveDraw draw;
    private readonly Vector3[] rtCorners = new Vector3[4];

    public static float? shoulderWidth = null;
    public static float? objectReferenceWidth = null;
    public static float cardRealWidthCm = 8.5f;

    [SerializeField]
    public TMP_Text tshirtSizeText;

    private void Start()
    {
        ssd = new SSD(options);
        poseNet = new PoseNet("posenet_mobilenet_v1_100_257x257_multi_kpt_stripped.tflite");
        draw = new PrimitiveDraw(Camera.main, gameObject.layer)
        {
            color = Color.green,
        };

        // Init frames
        frames = new Text[10];
        Transform parent = frameContainer.transform;
        for (int i = 0; i < frames.Length; i++)
        {
            frames[i] = Instantiate(framePrefab, Vector3.zero, Quaternion.identity, parent);
            frames[i].transform.localPosition = Vector3.zero;
        }

        // Labels
        labels = labelMap.text.Split('\n');

        if (TryGetComponent(out VirtualTextureSource source))
        {
            source.OnTexture.AddListener(Invoke);
        }
    }

    private void OnDestroy()
    {
        if (TryGetComponent(out VirtualTextureSource source))
        {
            source.OnTexture.RemoveListener(Invoke);
        }
        ssd?.Dispose();
        poseNet?.Dispose();
        draw?.Dispose();
    }

    private void Invoke(Texture texture)
    {
        poseNet.Invoke(texture);
        poseResults = poseNet.GetResults();

        DrawPoseResults();

        int frameWidth = texture.width;
        int frameHeight = texture.height;

        Vector2 leftShoulder = new Vector2(
            poseResults[(int)PoseNet.Part.LEFT_SHOULDER].x * frameWidth,
            poseResults[(int)PoseNet.Part.LEFT_SHOULDER].y * frameHeight
        );
        Vector2 rightShoulder = new Vector2(
            poseResults[(int)PoseNet.Part.RIGHT_SHOULDER].x * frameWidth,
            poseResults[(int)PoseNet.Part.RIGHT_SHOULDER].y * frameHeight
        );

        shoulderWidth = Vector2.Distance(leftShoulder, rightShoulder);
        
        ssd.Run(texture);
        
        SSD.Result[] results = ssd.GetResults();
        Vector2 size = (frameContainer.transform as RectTransform).rect.size;

        Vector2 ratio;
        if (texture.width >= texture.height)
        {
            ratio = new Vector2(1.0f, (float)texture.height / (float)texture.width);
        }
        else
        {
            ratio = new Vector2((float)texture.width / (float)texture.height, 1.0f);
        }

        // Find the best card result
        SSD.Result? bestCardResult = null;
        for (int i = 0; i < results.Length; i++)
        {
            string labelName = GetLabelName(results[i].classID);
            if (labelName == "card" && results[i].score >= scoreThreshold)
            {
                if (bestCardResult == null || results[i].score > bestCardResult.Value.score)
                {
                    bestCardResult = results[i];
                }
            }
        }

        foreach (var frame in frames)
        {
            frame.gameObject.SetActive(false);
        }

        if (bestCardResult != null)
        {
            SetFrame(frames[0], bestCardResult.Value, size * ratio);
        }

        calculateTShirt();
    }

    private void SetFrame(Text frame, SSD.Result result, Vector2 size)
    {
        if (result.score < scoreThreshold)
        {
            frame.gameObject.SetActive(false);
            return;
        }
        else
        {
            frame.gameObject.SetActive(true);
        }

        frame.text = $"card : {(int)(result.score * 100)}%";
        var rt = frame.transform as RectTransform;
        rt.anchoredPosition = result.rect.position * size - size * 0.5f;
        rt.sizeDelta = result.rect.size * size;
        
        Debug.Log("Card Width: " + rt.sizeDelta.x);
        objectReferenceWidth = rt.sizeDelta.x;
    }

    private string GetLabelName(int id)
    {
        if (id < 0 || id >= labels.Length - 1)
        {
            return "?";
        }
        return labels[id + 1];
    }

    private void DrawPoseResults()
    {
        if (poseResults == null || poseResults.Length == 0) return;

        draw.Clear(); // Clear previous frame drawings

        var rect = frameContainer.GetComponent<RectTransform>();
        rect.GetWorldCorners(rtCorners);
        float3 min = rtCorners[0];
        float3 max = rtCorners[2];

        var connections = PoseNet.Connections;
        int len = connections.GetLength(0);
        for (int i = 0; i < len; i++)
        {
            var a = poseResults[(int)connections[i, 0]];
            var b = poseResults[(int)connections[i, 1]];
            if (a.confidence >= 0.5f && b.confidence >= 0.5f)
            {
                draw.Line3D(
                    math.lerp(min, max, new float3(a.x, 1f - a.y, 0)),
                    math.lerp(min, max, new float3(b.x, 1f - b.y, 0)),
                    0.5f
                );
            }
        }
        draw.Apply();
    }
    
    public void calculateTShirt()
    {
        Debug.Log("Shoulder Width: " + shoulderWidth + ", Reference Width: " + objectReferenceWidth);

        if (objectReferenceWidth != null && shoulderWidth != null)
        {
            float pixelPerCm = (float)objectReferenceWidth / cardRealWidthCm;

            float realShoulderWidthCm = (float)shoulderWidth / pixelPerCm;

            Debug.Log("Real Shoulder Width: " + realShoulderWidthCm);

            string tshirtSize = "";

            if (realShoulderWidthCm < 40)
                tshirtSize = "S";
            else if (realShoulderWidthCm < 45)
                tshirtSize = "M";
            else if (realShoulderWidthCm < 50)
                tshirtSize = "L";
            else
                tshirtSize = "XL";

            Debug.Log("TshirtSize: " + tshirtSize);

            if (tshirtSizeText != null)
            {
                tshirtSizeText.text = "T-shirt Size: " + tshirtSize;
            }
        }
    }
}

[gedewidiastawan]

@asus4 I have another question, the poses point is in normalize number right, it's between 0 - 1. I need a number in pixel, my purpose is to get the shoulder width. Here are my code:

private void InvokeMoveNet(Texture texture)
{
    moveNet.Run(texture);
    pose = moveNet.pose;

    var left = pose[MoveNetPose.LEFT_SHOULDER];
    var right = pose[MoveNetPose.RIGHT_SHOULDER];

    if (left.IsValid(threshold) && right.IsValid(threshold))
    {
        shoulderWidth = Vector2.Distance(new Vector2(left.x, left.y), new Vector2(right.x, right.y));
        Debug.Log("Shoulder Width (px): " + shoulderWidth);
    }
    else
    {
        shoulderWidth = null;
    }
}

The shoulderWidth i think is in normalized value between 0 - 1, it's possible to convert to pixel?

[gedewidiastawan]

Hi @asus4 when i test ssd using samsung galaxy a15, its lagging every do a detection. Its normal or need improvement or use other library?

[asus4]

MoveNet returns the model's output directly.

Please check the official document.

https://www.tensorflow.org/hub/tutorials/movenet
https://storage.googleapis.com/movenet/MoveNet.SinglePose%20Model%20Card.pdf

[asus4]

@gedewidiastawan It would be worth testing the model with other libraries, such as Unity Inference Engine.

[gedewidiastawan]

How about EfficientDet Object Detection? Because i see this message "This SSD is deprecated. Try EfficientDet instead." inside SSD code. So it's mean there's an improvement inside EfficientDet?

[asus4]

EfficiantDet should be better than MobileNaet based SSD.

In this repository, only "official" models from the TensorFlow team are used. Please consider checking the recent SOTA models if you want the best performance.