Recognizing Indonesian sign language (Bisindo) gesture in complex backgrounds

Muhammad Alfhi Saputra, Erdefi Rakun

Research output: Contribution to journalArticlepeer-review

Abstract

Sign language, particularly Indonesian sign language (Bisindo), is vital for deaf individuals, but learning it is challenging. This study aims to develop an automated Bisindo recognition system suitable for diverse backgrounds. Previous research focused on greenscreen backgrounds and struggled with natural or complex backgrounds. To address this problem, the study proposes using Faster region-based convolutional neural networks (RCNN) and YOLOv5 for hand and face detection, MobileNetV2 for feature extraction, and long short-term memory (LSTM) for classification. The system is also designed to focus on computational efficiency. YOLOv5 model achieves the best result with a sentence accuracy (SAcc) of 49.29% and a word error rate (WER) of 16.42%, with a computational time of 0.0188 seconds, surpassing the baseline model. Additionally, the system achieved a SacreBLEU score of 67.77%, demonstrating its effectiveness in Bisindo recognition across various backgrounds. This research improves accessibility for deaf individuals by advancing automated sign language recognition technology.

Original languageEnglish
Pages (from-to)1583-1593
Number of pages11
JournalIndonesian Journal of Electrical Engineering and Computer Science
Volume36
Issue number3
DOIs
Publication statusPublished - Dec 2024

Keywords

  • Bisindo
  • Faster RCNN
  • Object detection
  • Sign language recognition
  • YOLOV5

Fingerprint

Dive into the research topics of 'Recognizing Indonesian sign language (Bisindo) gesture in complex backgrounds'. Together they form a unique fingerprint.

Cite this