Word recognition and automated epenthesis removal for Indonesian sign system sentence gestures

Erdefi Rakun, I. Gusti Bagus Hadi Widhinugraha, Noer Fitria Putra Setyono

Research output: Contribution to journalArticlepeer-review

Abstract

This research focuses on building a system to translate continuous Indonesian sign system (SIBI) gestures into text. In a continuous gesture, a signer will add an epenthesis (transitional) gesture, which is hand movement with no meaning but needed to connect the hand movement of one word with the next word in a continuous gesture. Reducing the number of irrelevant inputs to the model through automated epenthesis removal can improve the system's ability to recognize the words in continuous gestures. We implemented threshold conditional random fields (TCRF) to identify epenthesis gestures. The dataset consists of 2,255 videos representing 28 common sentences in SIBI. The translation system consists of MobileNetV2 as a feature extraction technique, removing epenthesis gestures found by the TCRF, and a long short-term memory (LSTM) for the classifier. With the MobileNetV2-TCRF-bidirectional LSTM model, the best word error rate (WER) and sentence accuracy (SAcc) were 33.4% and 16.2%, respectively. Intermediate-stage processing steps consisting of sandwiched majority voting of the TCRF and the removal of word labels whose number of frames is less than two frames, along with LSTM output grouping, were able to reduce WER from 33.4% to 3.4% and increase SAcc from 16.2% to 80.2%

Original languageEnglish
Pages (from-to)1402-1414
Number of pages13
JournalIndonesian Journal of Electrical Engineering and Computer Science
Volume26
Issue number3
DOIs
Publication statusPublished - Jun 2022

Keywords

  • Epenthesis gesture
  • Long short-term memory
  • SIBI
  • Sign language recognition
  • Threshold conditional random field

Fingerprint

Dive into the research topics of 'Word recognition and automated epenthesis removal for Indonesian sign system sentence gestures'. Together they form a unique fingerprint.

Cite this