LearnSign: A Static Sign Language Alphabet Game

Won 1st place AI/ML, overall 2nd place in Anaconda's Data Science Expo 2023 supported by AI Singapore

A tensorflow.js web application that utilies TF2 object detection models to recognize real-time static American Sign Language (ASL) via web browser. This web application comes in a form of a game that recognises ASL alphabets via the user's web cam. Try the live demo at https://learnsign.vercel.app.

Web Application Home Page

Academic Poster

Object Detection Models

Using the concept of transfer learning, we finetuned TensorFlow 2 Detection Model Zoo SSD MobileNetv2 FPNLite 320x320 model weights via the TF2 Object Detection API in Google Colab. As the computer vision models are running on tensorflow.js, inference is carried out on cilent side and no video/image data from the user is sent to the website hosting server.

1) Baseline Model

• 4 Classes: A, B, C, D.
• Finetuned on SSD MobileNetv2 FPNLite 320x320 pre-trained on COCO 2017 dataset.

2) Extended Model

• 24 Classes: A, B, C, D, E, F, G, H, I, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y.
• J and Z are excluded as they are both dynamic sign langauges involving movement.
• Finetuned on SSD MobileNetv2 FPNLite 320x320 pre-trained on COCO 2017 dataset.

Exploratory Data Analysis

Evaluation Results @11,000 steps

• Test mean average precision (mAP) and average recall

• Test Confusion Matrix @0.5IOU

• Test Precision Recall Plot @0.5IOU

• Test Precision Plot @0.5IOU

• Test Recall Plot @0.5IOU

• Inference on Test Image (Groundtruth bounding box in blue with IOU Score of prediction result)

Video Inference

video_inference_output.mp4