Poster
FSboard: Over 3 million characters of ASL fingerspelling collected via smartphones
Manfred Georg · Garrett Tanzer · Esha Uboweja · Saad Hassan · Maximus Shengelia · Sam Sepah · Sean Forbes · Thad Starner
Progress in machine understanding of sign languages has been slow and hampered by limited data.In this paper, we present FSboard, an American Sign Language fingerspelling dataset situated in a mobile text entry use case, collected from 147 paid and consenting Deaf signers using Pixel 4A selfie cameras in a variety of environments.Fingerspelling recognition is an incomplete solution that is only one small part of sign language translation, but it could provide some immediate benefit to Deaf/Hard of Hearing signers while more broadly capable technology develops.At >3 million characters in length and >250 hours in duration, FSboard is the largest fingerspelling recognition dataset to date by a factor of >10x. As a simple baseline, we finetune 30 Hz MediaPipe Holistic landmark inputs into ByT5-Small and achieve 11.1% Character Error Rate (CER) on a test set with unique phrases and signers. This quality degrades gracefully when decreasing frame rate and excluding face/body landmarks---plausible optimizations to help with on-device performance.
Live content is unavailable. Log in and register to view live content