Abstract. Tokenising continuous speech into sequences of discrete tokens and modelling them with language models (LMs) have achieved a success for text-to-speech (TTS) synthesis. Although these models can generate speech with high quality and naturalness, their synthesised samples can still suffer from artifacts, mispronunciation, word repeating, etc. In this paper, we argue these undesirable properties could partly be caused by the randomness of sampling-based strategies during the autoregressive decoding of LMs. Therefore, we look at maximisation-based decoding approaches and propose Temporal Repetition Aware Diverse Beam Search (TRAD-BS) to find the most probable sequences of the generated speech tokens. Experiments with two most recent LM-based TTS models show that the proposed maximisation-based decoding strategy can generate speech with fewer mispronunciation and better speaker consistency.

Overview

(a) A general inference paradigm of LM-based TTS. Speech tokens are generated autoregressively before converted into speech. (b) TRAD-BS operates through both decoding steps and beams, adding temporal and beam-wise penalties to repeated tokens. This can also be applied to the case where multiple tokens need to be decoded at a single step.