TY - JOUR
T1 - Recognizing Indonesian sign language (Bisindo) gesture in complex backgrounds
AU - Saputra, Muhammad Alfhi
AU - Rakun, Erdefi
N1 - Publisher Copyright:
© 2024 Institute of Advanced Engineering and Science. All rights reserved.
PY - 2024/12
Y1 - 2024/12
N2 - Sign language, particularly Indonesian sign language (Bisindo), is vital for deaf individuals, but learning it is challenging. This study aims to develop an automated Bisindo recognition system suitable for diverse backgrounds. Previous research focused on greenscreen backgrounds and struggled with natural or complex backgrounds. To address this problem, the study proposes using Faster region-based convolutional neural networks (RCNN) and YOLOv5 for hand and face detection, MobileNetV2 for feature extraction, and long short-term memory (LSTM) for classification. The system is also designed to focus on computational efficiency. YOLOv5 model achieves the best result with a sentence accuracy (SAcc) of 49.29% and a word error rate (WER) of 16.42%, with a computational time of 0.0188 seconds, surpassing the baseline model. Additionally, the system achieved a SacreBLEU score of 67.77%, demonstrating its effectiveness in Bisindo recognition across various backgrounds. This research improves accessibility for deaf individuals by advancing automated sign language recognition technology.
AB - Sign language, particularly Indonesian sign language (Bisindo), is vital for deaf individuals, but learning it is challenging. This study aims to develop an automated Bisindo recognition system suitable for diverse backgrounds. Previous research focused on greenscreen backgrounds and struggled with natural or complex backgrounds. To address this problem, the study proposes using Faster region-based convolutional neural networks (RCNN) and YOLOv5 for hand and face detection, MobileNetV2 for feature extraction, and long short-term memory (LSTM) for classification. The system is also designed to focus on computational efficiency. YOLOv5 model achieves the best result with a sentence accuracy (SAcc) of 49.29% and a word error rate (WER) of 16.42%, with a computational time of 0.0188 seconds, surpassing the baseline model. Additionally, the system achieved a SacreBLEU score of 67.77%, demonstrating its effectiveness in Bisindo recognition across various backgrounds. This research improves accessibility for deaf individuals by advancing automated sign language recognition technology.
KW - Bisindo
KW - Faster RCNN
KW - Object detection
KW - Sign language recognition
KW - YOLOV5
UR - http://www.scopus.com/inward/record.url?scp=85205426888&partnerID=8YFLogxK
U2 - 10.11591/ijeecs.v36.i3.pp1583-1593
DO - 10.11591/ijeecs.v36.i3.pp1583-1593
M3 - Article
AN - SCOPUS:85205426888
SN - 2502-4752
VL - 36
SP - 1583
EP - 1593
JO - Indonesian Journal of Electrical Engineering and Computer Science
JF - Indonesian Journal of Electrical Engineering and Computer Science
IS - 3
ER -