An audio-visual approach to simultaneous-speaker speech recognition

Audio-visual speech recognition is an area with great potential to help solve challenging problems in speech processing. Difficulties due to background noise are significantly reduced by the additional information provided by extra visual features. The presence of additional speech from other talkers during recording may be viewed as one of the most difficult sources of noise. The paper presents a study using audio-visual speech recognition for simultaneous-speaker speech recognition. The desired goal is to separate and potentially recognize speech from several simultaneous speakers. Speaker pairs from the CUAVE multimodal speech corpus (see http://ece.clemson.edu/speech) are used. Audio-visual techniques are compared against speaker-independent and speaker-dependent audio-only methods for speech recognition of individuals from these pairs.