Expressive speech synthesis using multilevel unit selection

Generating natural sounding and meaningful prosody is a central challenge in text-to-speech synthesis, especially when generating expressive speech. Recently, we proposed a multilevel unit sequence synthesis approach, based on the general superpositional model of intonation, which describes a pitch contour as the sum of component curves that are each associated with different phonological levels, specifically the phoneme, foot, and phrase. During synthesis, segmental perturbation curves, accent curves, and phrase curves are extracted from the acoustic signal and are combined into target pitch curves; these target curves are then imposed on the acoustic unit sequences using standard pitch modification methods. This approach represents an attempt to combine the strengths of the two dominant approaches to speech synthesis: unit selection synthesis, which preserves all details of natural speech but struggles with coverage of the very large combinatorial space of phoneme sequences and prosodic contexts, and diphone synthesis, which addresses coverage by generating rule-based synthetic target prosody and imposing it on acoustic units using signal modification methods. This approach minimizes prosodic modification artifacts, optimizes the naturalness of the target pitch contour by using quasi-natural contours, yet avoids the combinatorial explosion of standard unit selection synthesis.