Real-time Interpreter for Short Sentences in Indian Sign Language Using MediaPipe and Deep Learning

Abstract

The expression of thoughts and feelings through communication plays a major part of human life in building relationship among others. Most of the population with hearing ability expresses their thoughts in their own or known language through voice-oriented communication. The people belonging to deaf-mute community uses hand movement gestures and expressions of face for communication which is called sign language. There exists a difficulty in building a conversation between the hearing community and non-hearing community. To make easy conversation of deaf-mute people with the external world and to connect the gap for communication between the hearing people and non-hearing people, we developed an interpreter that translates sign language to text. Most system developed for the recognition of Indian Sign Language is built for alphabets and numbers. We attempted in building a model for 15 meaningful short sentences of Indian sign gestures using, custom built video datasets captured using OpenCV, keypoints of hands, pose and face extracted using MediaPipe. The model is trained using LSTM and achieved training and testing accuracy of 99.17% and 97.78% respectively.