feat: add vms-integration

[yuanh0702]

1. Project Overview

We completed the integration of CCTV VMS by developing an integration plugin. This allows adding basic VMS information in the integration, synchronizing the IPC list from VMS, and converting the video stream address and other information from IPC into entities. Necessary IPC management functions in VMS can be encapsulated as services for user invocation. The synchronization of added devices and device lists needs to be implemented with synchronization to Beaver IoT's Device. The IPC video feed from VMS can be played in the Dashboard.

1.1 Backend

CCTV VMS Integration
This primarily involves interfacing with API endpoints for CCTV VMS login, device list query, HLS video stream monitoring, HLS video stream playback, and RTSP video stream live streaming.

Image AI Analysis Integration
Integrated with the Hugging Face platform, which is an open-source platform and community focused on natural language processing (NLP) and machine learning. The platform provides various tools and resources to help developers and researchers build, train, and deploy machine learning models. We integrated the platform's image object analysis capabilities, particularly interfacing with analysis models such as facebook/detr-resnet-50, facebook/detr-resnet-50-dc5, facebook/detr-resnet-101, hustvl/yolos-small, and hustvl/yolos-tiny.

Device Entities

• Read-only Attribute Entities: online (device online status)
• Service Entities:
  • sync (synchronize CCTV VMS device information)
  • get_hls_url (get HLS video stream monitoring URL)
  • get_hls_vod_url (get HLS video stream playback URL)
  • get_hls_vod_url.start_time (get the start time for HLS video stream playback URL)
    

Devices
Current devices are the monitoring devices synchronized from CCTV VMS. Integration starts with automatic synchronization, and manual synchronization can be done using the integration's service entities. If you know the device ID in CCTV VMS, you can also manually add or delete devices. These devices' monitoring can be directly rendered and displayed through frontend components, and object analysis can be performed on the currently playing video to identify object tags and confidence levels.

Integration Entities

• Read-only Attribute Entities: detect_status (detection status)
• Read/Write Attribute Entities:
  • vms_info.vmsUrl (CCTV VMS API address)
  • vms_info.username (CCTV VMS username)
  • vms_info.password (CCTV VMS password)
  • api_info.ai_url (AI analysis platform address)
  • api_info.ai_access_token (AI platform authorization key)
  • api_info.object_detection_model (AI platform detection model)
  • api_info.image_labels (AI platform detection labels, customizable for object analysis)
  • api_info.image_score (AI platform confidence level, customizable for analysis confidence)
• Service Entities:
  • benchmark (synchronize CCTV VMS device information)
  • add_device (add device)
  • delete_device (delete device)
    
    

1.2 Frontend

Video Playback Component
This component can bind to the service entities for obtaining monitoring URLs of VMS devices and for obtaining playback monitoring entities to play live and playback videos. Additionally, it features image object analysis. When you hover the mouse over the image analysis frame, it also displays the object tags and confidence levels.

2. Operation Steps

Set VMS Integration Information

1. Navigate to the integration settings page.
2. Enter the corresponding service address for VMS, username, password, object analysis address (https://api-inference.huggingface.co), AI platform authorization key (hf_**), AI platform detection labels (e.g., person, tv, separated by commas), AI platform confidence level (0.85), and select the object analysis model.

Current AI Platform Detection Labels:
person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, chair, couch, potted plant, bed, dining table, toilet, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors, teddy bear, hair dryer, toothbrush

AI Platform Confidence Level:
The maximum is 1. The higher the confidence level, the more accurate the recognized labels.

Object Analysis Models:

• facebook/detr-resnet-50: DETR model based on ResNet-50, well-balanced and suitable for most application scenarios.
• facebook/detr-resnet-50-dc5: DETR model based on ResNet-50, with dilated convolution added at the last convolutional layer of each stage to enhance feature extraction capabilities.
• facebook/detr-resnet-101: DETR model based on ResNet-101, with a deeper network structure to extract richer features.
• hustvl/yolos-small: A small version of the YOLOS model, based on the Vision Transformer (ViT) architecture, suitable for resource-limited environments.
• hustvl/yolos-tiny: A tiny version of the YOLOS model, further reducing parameter count and computation, very suitable for embedded devices or real-time applications.
  
  

Create Device

1. Go to the device list page.
2. Click the "Add" button, select the VMS integration, fill in the device name and the new device ID, and click "Confirm."
   

Delete Device

View Device Entities

Bind Video Playback Component

1. First, select the video playback component.
   

2. Then, select the video stream entity contained in the device. The first entity is for monitoring (get_hls_url), and the second entity is for playback (get_hls_vod_url).
   
   

Display Effect

Video Playback

1. Select the playback time.
   
   

Switch to Real-time Monitoring

Image Analysis

Synchronize VMS Devices