Auburn Engineering student developing translation technology for sign language

Nearly 500,000 people in the U.S. and Canada are speakers of American Sign Language as their first language. For those who are deaf and hard of hearing, ASL gives them a common platform to express themselves. 

But ASL speakers can often have difficulty interacting with non-hearing impaired people. One Auburn University student is part of a team developing technology that could change that.

“Our project that we designed is a modular system that can fit into either your phone, a Google Glass-type device or through your computer via Skype,” said Ryan McGill, a senior in electrical and computer engineering. “Its function would be to allow two-way conversations in the language the users prefer, so in this case ASL and English.”

McGill worked over the summer at the Massachusetts Institute of Technology’s Lincoln Laboratory, where his team, named ENVOY, or Enabling Natural-language Versatility and Opportunity, won the lab’s Intern Innovative Idea Challenge.

McGill said his team’s system will help take responsibility off of the signer when it comes to two-way communication. 

“The system would input signs from the ASL speaker, translate to English for the spoken-English user and then input English back in and show the English speaker how to sign back their part of the conversation. Current technologies focus on things like gloves that translate ASL, but this puts the burden of communication on the signer. It would be like going to France and expecting everyone to speak English to you,” he said.

MIT is sponsoring McGill’s continued work on the ENVOY project throughout the fall and spring semesters. McGill said Auburn’s engineering faculty and department chair Mark Nelms have been very supportive of him continuing the research project here.

McGill said he hopes his team’s system will help fill a major gap in the deaf community’s ability to succeed and receive greater opportunities.

“Currently, a large percentage of the deaf community is underemployed, or working a position they are less qualified for, due to a lack of translation resources,” he said.

He added that the team is working with several deaf advocacy organizations to ensure the needs of the signer are being met.

“Our project will hopefully open up the doors for future ASL-translation techniques by providing an architecture that is rooted in the deaf community,” he said.

McGill feels his Auburn education and relationships with faculty helped prepare him to succeed during MIT’s Intern Innovative Idea Challenge.

“I was able to apply a lot of concepts from my classes here at Auburn,” he said. “Dr. Mark Adams prepared me for this summer by teaching me how to conduct research through his lab, where I was able to contribute to a variety of projects.”

Before a full system can be implemented, McGill said there are still several problems his team must work to address.

“One of the major things we are trying to tackle is the lack of an ASL-dictionary database to use to train machine-learning algorithms,” he said. “We have proposed several novel concepts on how to create a large, hopefully all-encompassing ASL dictionary for academic purpose.” 

He said his team will work on developing a prototype and hope to show a proof-of-concept by spring 2020.

McGill said his experience at the laboratory this summer has strengthened his resolve and invigorated his desire to succeed and tackle real-world issues. 

“The staff there really taught me how to conduct research and flesh out ideas. They equipped me with everything I needed to conduct research both on my own and on a team,” he said. “The overall atmosphere was encouraging to work in, constantly challenging our ideas to make sure our ideas were rooted in strong research and good thinking.”