[Detector Support]: 0.15.0 Hailo8L compatible models

[NathanMTL]

Describe the problem you are having

I have been using the raspi 5 with Hailo8L to run Frigate. The current model ssd_mobilenet_v1 is ok, but seems to lack, it keeps registering my cat as a person as an example.

I tried using the Yolov8s model (pre-trained hef from Hailo Model Zoo) and COCO label map (model trained on coco) and:

Good:

1. Person and Car detections were worked much better
   Issue:
2. I could not get any other object to be recognized, I tried adding the entire list of objects to my config file and it didn't even detect my cat and decided my bed was a donut.

The inference time was reasonable at 30 ms, but again, I could not get objects other than person and car to work.

My core question:
What are the "guidelines" for models in Frigate? It seems that models are unique to each detector in the documentation and I can't seem to understand why the label map wasn't "mapping" properly. Can I take any model and appropriate label map and add it, as long as the objects are supported by Frigate? Or, are there specific requirements for models?

Will there be additional models supported by H8L when Frigate 0.15 launches? Is there anything I can do to help?

Thank you!

Version

Frigate 0.15.0 Beta1

Frigate config file

mqtt:
  enabled: true
  host: ****
  port: 1883
  user: ****
  password: ****

detectors:
  hailo8l:
    type: hailo8l
    device: PCIe
    model:
      path: /config/model_cache/h8l_cache/ssd_mobilenet_v1.hef
      #labelmap_path: /config/model_cache/h8l_cache/coco_labelmap.txt #used when I ran Yolov8s

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  model_type: ssd
  #labelmap_path: /config/model_cache/h8l_cache/coco.txt #Tried it here as well with no luck

go2rtc:
  streams:
    doorbell_main:
      - rtsp://admin:{FRIGATE_RTSP_PASSWORD}@****:554/h264Preview_01_main
      - ffmpeg:rtsp_cam#audio=opus
    doorbell_sub:
      - rtsp://admin:{FRIGATE_RTSP_PASSWORD}@192.168.1.214:554/h264Preview_01_sub
      - ffmpeg:rtsp_cam#audio=opus
    bedroom_sub:
      - rtsp://tapocam:****@****:554/stream2
      - ffmpeg:#audio=aac&input=-rtsp_transport+tcp
    livingroom_sub:
      - rtsp://tapocam2:****@****:554/stream1
      - ffmpeg:#audio=aac&input=-rtsp_transport+tcp

#live:
  #stream_name: doorbell_main
  #height: 720
  #quality: 8

#ffmpeg:
  #hwaccel_args: preset-rpi-64-h264

cameras:
  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/doorbell_main
          roles:
            - record
        - path: rtsp://localhost:8554/doorbell_sub
          roles:
            - detect
    live:
      stream_name: doorbell_main
      height: 720
      quality: 8
    record:
      enabled: true
      retain:
        days: 5
        mode: motion
      alerts:
        retain:
          days: 15
      detections:
        retain:
          days: 15
    snapshots:
      enabled: true
      retain:
        default: 15
    detect:
      enabled: true
      width: 640
      height: 480
      fps: 5
    objects:
      track:
        - person
        - car
        - cat
        - dog

    motion:
      threshold: 50
      contour_area: 10
      improve_contrast: true

  bedroom:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/bedroom_sub
          roles:
            - record
            - detect
    live:
      stream_name: bedroom_sub
      height: 720
      quality: 8
    record:
      enabled: true
      retain:
        days: 5
        mode: motion
      alerts:
        retain:
          days: 15
      detections:
        retain:
          days: 15
    snapshots:
      enabled: false
      retain:
        default: 15
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    objects:
      track:
        - person
        - cat

  living:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/livingroom_sub
          roles:
            - record
            - detect
    live:
      stream_name: livingroom_sub
      height: 720
      quality: 8
    record:
      enabled: true  # Set to true if you want recording; set to false if no recording
      retain:
        days: 5
        mode: motion
      alerts:
        retain:
          days: 15
      detections:
        retain:
          days: 15
    snapshots:
      enabled: false
      retain:
        default: 15
    detect:
      enabled: true
      #width: 1280
      #height: 720
      fps: 5
    objects:
      track:
        - person
        - cat

version: 0.15-0

docker-compose file or Docker CLI command

services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta1-h8l
    shm_size: "512mb" # update for your cameras based on calculation above
    devices:
      - /dev/hailo0:/dev/hailo0
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/pi/frigate/config:/config
      - /mnt/data/docker/frigate/media:/media
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "8971:8971"
      - "5000:5000" # Internal unauthenticated access. Expose carefully.
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "****"

Relevant Frigate log output

No relevant logs available. I never saw anything in the logs showing a reason for the failures.

Install method

Docker Compose

Object Detector

Other

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

it depends on the detector, some detectors require explicit handling of each model type while others handle that internally. Technically yolov8 is not supported in Frigate, but hailo outputs in the same format. I don't see any reasons the labelmap would not work unless the labelmap for yolov8 is different.

[NathanMTL]

Here is the label map I used, named "coco_labelmap.txt".

0: person
1: bicycle
2: car
3: motorcycle
4: airplane
5: bus
6: train
7: truck
8: boat
9: traffic light
10: fire hydrant
11: stop sign
12: parking meter
13: bench
14: bird
15: cat
16: dog
17: horse
18: sheep
19: cow
20: elephant
21: bear
22: zebra
23: giraffe
24: backpack
25: umbrella
26: handbag
27: tie
28: suitcase
29: frisbee
30: skis
31: snowboard
32: sports ball
33: kite
34: baseball bat
35: baseball glove
36: skateboard
37: surfboard
38: tennis racket
39: bottle
40: wine glass
41: cup
42: fork
43: knife
44: spoon
45: bowl
46: banana
47: apple
48: sandwich
49: orange
50: broccoli
51: carrot
52: hot dog
53: pizza
54: donut
55: cake
56: chair
57: couch
58: potted plant
59: bed
60: dining table
61: toilet
62: tv
63: laptop
64: mouse
65: remote
66: keyboard
67: cell phone
68: microwave
69: oven
70: toaster
71: sink
72: refrigerator
73: book
74: clock
75: vase
76: scissors
77: teddy bear
78: hair drier
79: toothbrush

[kdill00]

You need to hone in object sizes and adjust the default minimum score higher and or the threshold as well. That will help some. You can also play with motion settings. Finding that motion sweet spot will reduce the number of frames the detector is fed and can improve accuracy

[NathanMTL]

Thank you, this is something I have been testing slowly. However, the main issue is still the labelmap. It thought my bed was a donut. While the default model knows it’s a bed.

[alschroder]

Did you make any more progress on this @NathanMTL ? After getting my set up dialed-in on the base model and my hw (pi5, hailo8l on pcie, 3 cameras), I just dropped the yolov8s model in (from the hailo model zoo). Will post back my findings.

[NathanMTL]

No I haven’t, I’m struggling to find information about which models / models types are supported in Frigate to start with. But I’ll keep everyone posted if I get it working, let me know how your set up goes!

[alschroder]

I've had yolov8s going for 36 hours or so now, and its definitely detecting more objects than just car & person (and it seems more accuratel too). Lots of correct matches on dog, cat, backpack, etc. My inference times have almost doubled though from ~25ms average up to high 40ms.

My config for detector and model looks like this (I just inlined the labels):

detectors:
  hailo8l:
    type: hailo8l
    device: PCIe
    model:
      # from https://github.com/hailo-ai/hailo_model_zoo/blob/master/docs/public_models/HAILO8L/HAILO8L_object_detection.rst
      path: /config/model_cache/h8l_cache/yolov8s.hef

model:
  width: 640
  height: 640
  input_tensor: nhwc
  input_pixel_format: bgr
  model_type: yolox
  labelmap:
    0: person
    1: bicycle
    2: car
    3: motorcycle
    4: airplane
    5: bus
    6: train
    7: truck
    8: boat
    9: traffic light
    10: fire hydrant
    11: stop sign
    12: parking meter
    13: bench
    14: bird
    15: cat
    16: dog
    17: horse
    18: sheep
    19: cow
    20: elephant
    21: bear
    22: zebra
    23: giraffe
    24: backpack
    25: umbrella
    26: handbag
    27: tie
    28: suitcase
    29: frisbee
    30: skis
    31: snowboard
    32: sports ball
    33: kite
    34: baseball bat
    35: baseball glove
    36: skateboard
    37: surfboard
    38: tennis racket
    39: bottle
    40: wine glass
    41: cup
    42: fork
    43: knife
    44: spoon
    45: bowl
    46: banana
    47: apple
    48: sandwich
    49: orange
    50: broccoli
    51: carrot
    52: hot dog
    53: pizza
    54: donut
    55: cake
    56: chair
    57: couch
    58: potted plant
    59: bed
    60: dining table
    61: toilet
    62: TV
    63: laptop
    64: mouse
    65: remote
    66: keyboard
    67: cell phone
    68: microwave
    69: oven
    70: toaster
    71: sink
    72: refrigerator
    73: book
    74: clock
    75: vase
    76: scissors
    77: teddy bear
    78: hair drier
    79: toothbrush

[NathanMTL]

Cool, thanks I’ll give this a try! Yeah, I noticed yolov8s is way more accurate than the default model.

[NathanMTL]

@alschroder Thank you! This worked great, I didn't realize you could put the label map directly into the config file like this.

For Beta 1 on Rpi 5 8gb and Hailo8L chip I am getting 31ms inference and Yolov8s is way more accurate than the ssdmobilenet model. It hasn't said my cat was a person once and identifies all objects!

[NathanMTL]

Small note: I think (I could be wrong) that Yolo uses "rgb" pixel input format, that is what I have selected.

[luckylinux]

@alschroder, @NathanMTL: I am really a Beginner in this so it might be a simple Question for your, but how did you Compile/Convert this ZIP File obtained from Hailo Github Repository (containing yolov8s.onnx and yolov8s_nms_config.json) ?

I found Several Threads on Hailo Community Forums but no conclusive Answer. With the latest Dataflow Compiler 3.29.0 running on Python 3.10 (I created a Dockerfile to build a Docker Image with a Python 3.10 Environment since Python 3.11/3.12 are not supported).

However I'm now fully stuck because I don't know which Commands I need to issue.

There is a bit of a Split between:

a. 2-Step Approach

1. You need to first call hailo parser to generate a HAR File from the onnx File
2. You need to then compile using hailo compiler to generate a HEF File from the HAR File

b. Just Compile directly and train the Model by Specifying --calib-path

Furthermore it seems some posts reference hailomz while I only have the hailo Command, as well as compile Subcommand does NOT exist (compiler exists).

Furthermore I'm not sure what I need to do in order to train it (if I need to train it myself, I guess I need to feed it some Pictures but not clear how to do and go from there).

Do you have some more Details on how you managed to get this yolov8s.hef File ?

Thanks 😃