Wireless Personal Captioning System

This project was funded by the National Institute on Disability and Rehabilitation Research through the Wireless Rehabilitation Engineering Research Center. 

One Device, Many Venues 

The Intelligent Access system follows the “one device, many venues” design philosophy.
 

This approach benefits both the caption end user and the location providing the service. The end-user can receive captions from any Intelligent Access installation on his or her own compatible mobile device, with a software application download.

The current client system utilizes WiFi technology to deliver personal captions to an individual on an iPod Touch, iPhone, or Windows mobile system. The user has the choice of viewing the captions on the handheld screen or optional electronic eyewear.

The current server system has a simple user interface that allows the venue or presenter to select from one of three input choices: pre-recorded text, input from speech recognition software, and input that is typed in real time with CART (Computer-Assisted Real-time Translation) or a computer keyboard.

Intelligent Access mobile captioning system provides real-time captions to deaf and hard of hearing patrons in a variety of venues, such as sports stadiums, movie theaters or classrooms, or anywhere there are large group settings. Using the venue’s existing WiFi network, patrons receive captioned text of the event on their personal Windows Mobile smartphone/PDA, iPod Touch, or iPhone. Plans are to make devices running other OS’s compatible with the system. Stadium System features include:

• Captioning is generated in real-time through CART or typed from keyboard, or any other external serial or network source. 

• Song lyrics can be stored and easily selected line-by-line from files to send on demand.

• Text is simultaneously sent to handhelds and other sources such as big stadium screen, or sideline screen.

• Multiple languages can be sent simultaneously.

• Emergency Mode notifies patrons by a custom message in red text that the following message is an emergency notification. The patron is also notified when the emergency has passed.

• Captions appear in a scrolling fashion.

• The system has been tested in various baseball and football stadiums.

Movie Theatres

Intelligent Access uses pre-recorded text input in a movie theater-type setting, where the audio content does not vary from one performance to the next. Text is generated ahead of time, and the server receives time code from the digital cinema server or digital media player to synchronize the captions with the video content.

The heads-up display works particularly well in this setting, enabling viewers to read the captions without taking their eyes off the film in action. This input method works with various types of movie theater equipment and adheres to the Society of Motion Picture and Television Engineers (SMPTE) closed captioning standards for digital cinema. (These standards are under development.) 

The  Intelligent Access system has undergone a formative evaluation in both actual theaters and mock theater settings. Deaf and hard of hearing users were involved in the evaluations, and Landmarc incorporated their end-user suggestions as part of their iterative design process.

Landmarc’s wireless captioning enhances visitor experience in other leisure venues. Intelligent Access can assist deaf and hard-of-hearing patrons through realtime captioning of performances and interactive exhibit interpretations. These location-based features, currently under development, function in realtime, providing a richer and more memorable encounter that encourages repeat visits.

The Mystic Aquarium launched Landmarc's system in January 2009. Running on smartphones and other mobile devices like the iPod, the system displays video captions. Future versions will include exhibit-specific information (including text, images and audio), and automated tracking in support of wayfinding and gallery interpretation. 

Project Timeline

Landmarc's wireless personal captioning technology is currently undergoing commercialization. The technology has resulted in 6 invention disclosures. User testing was conducted in the following stages. In addition to these more formal tests, patrons have used the Intelligent Access system in a variety of theatres and sports venues.

Winter 2008 - Spring 2009
The speech recognition input is currently under test at the Rochester Institute of Technology National Technical Institute for the Deaf through an NSF grant "Speech to Text Systems: Comparative Analysis of Text Generation and Display Methods.” Handheld (PDA screen) and electronic eyewear display methods have been compared to desktop and common screen display methods for captions in the college classroom setting.

June 2003
The second stage of user testing occurred at the Self-help for Hard of Hearing People (now Hearing Loss Association of America) conference in 2003. This group consisted of 52 volunteers, from 15 to over 75. The individuals came from diverse auditory backgrounds, including hearing, hard of hearing, deaf from birth, and late deafened.

January - February 2003
Landmarc conducted the first stage of user testing in conjunction with the Georgia Council for the Hearing Impaired. 25 volunteers participated in this study.