Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models

Bella Septina Ika Hartanti; Dipta Tanaya; Kurniawati Azizah; Dessi Puji Lestari; Ayu Purwarianti; Sakriani Sakti

doi:10.1109/O-COCOSDA60357.2023.10482965

Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models

Bella Septina Ika Hartanti, Dipta Tanaya, Kurniawati Azizah, Dessi Puji Lestari, Ayu Purwarianti, Sakriani Sakti

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Despite many existing works that address expressive speech synthesis with a desired prosody, few have focused on generating speech with prosody prominence. Most previous studies addressing this issue generate speech from given text labels with a contrastive focus emphasizing a specific word. In contrast, this paper investigates whether we can control prosody based on the contrastive focus that appears in images. Given an image and its caption, our system first discovers spoken terms associated with objects or situations in natural images based on a self-supervised visually grounded model. Then it generates speech with prosody prominence based on the contrastive focus of these spoken terms in a way that best describes the images. The framework can perform the task with/without text annotation, making it applicable for untranscribed, unsegmented speech utterances in unknown languages.

Original language	English
Title of host publication	Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350344028
DOIs	https://doi.org/10.1109/O-COCOSDA60357.2023.10482965
Publication status	Published - 2023
Event	26th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023 - Delhi, India Duration: 4 Dec 2023 → 6 Dec 2023

Publication series

Name	Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023

Conference

Conference	26th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023
Country/Territory	India
City	Delhi
Period	4/12/23 → 6/12/23

Keywords

multimodal learning
prosody prominence
self-supervised visual grounded models
text-to-speech synthesis

Access to Document

10.1109/O-COCOSDA60357.2023.10482965

Cite this

Ika Hartanti, B. S., Tanaya, D., Azizah, K., Lestari, D. P., Purwarianti, A., & Sakti, S. (2023). Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models. In Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023 (Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/O-COCOSDA60357.2023.10482965

Ika Hartanti, Bella Septina ; Tanaya, Dipta ; Azizah, Kurniawati et al. / Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models. Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023).

@inproceedings{236b6c4471804a1eb4ba003b186b3e3c,

title = "Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models",

abstract = "Despite many existing works that address expressive speech synthesis with a desired prosody, few have focused on generating speech with prosody prominence. Most previous studies addressing this issue generate speech from given text labels with a contrastive focus emphasizing a specific word. In contrast, this paper investigates whether we can control prosody based on the contrastive focus that appears in images. Given an image and its caption, our system first discovers spoken terms associated with objects or situations in natural images based on a self-supervised visually grounded model. Then it generates speech with prosody prominence based on the contrastive focus of these spoken terms in a way that best describes the images. The framework can perform the task with/without text annotation, making it applicable for untranscribed, unsegmented speech utterances in unknown languages.",

keywords = "multimodal learning, prosody prominence, self-supervised visual grounded models, text-to-speech synthesis",

author = "{Ika Hartanti}, {Bella Septina} and Dipta Tanaya and Kurniawati Azizah and Lestari, {Dessi Puji} and Ayu Purwarianti and Sakriani Sakti",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 26th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023 ; Conference date: 04-12-2023 Through 06-12-2023",

year = "2023",

doi = "10.1109/O-COCOSDA60357.2023.10482965",

language = "English",

series = "Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023",

address = "United States",

}

Ika Hartanti, BS, Tanaya, D, Azizah, K, Lestari, DP, Purwarianti, A & Sakti, S 2023, Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models. in Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023. Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023, Institute of Electrical and Electronics Engineers Inc., 26th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023, Delhi, India, 4/12/23. https://doi.org/10.1109/O-COCOSDA60357.2023.10482965

Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models. / Ika Hartanti, Bella Septina; Tanaya, Dipta; Azizah, Kurniawati et al.
Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models

AU - Ika Hartanti, Bella Septina

AU - Tanaya, Dipta

AU - Azizah, Kurniawati

AU - Lestari, Dessi Puji

AU - Purwarianti, Ayu

AU - Sakti, Sakriani

PY - 2023

Y1 - 2023

N2 - Despite many existing works that address expressive speech synthesis with a desired prosody, few have focused on generating speech with prosody prominence. Most previous studies addressing this issue generate speech from given text labels with a contrastive focus emphasizing a specific word. In contrast, this paper investigates whether we can control prosody based on the contrastive focus that appears in images. Given an image and its caption, our system first discovers spoken terms associated with objects or situations in natural images based on a self-supervised visually grounded model. Then it generates speech with prosody prominence based on the contrastive focus of these spoken terms in a way that best describes the images. The framework can perform the task with/without text annotation, making it applicable for untranscribed, unsegmented speech utterances in unknown languages.

AB - Despite many existing works that address expressive speech synthesis with a desired prosody, few have focused on generating speech with prosody prominence. Most previous studies addressing this issue generate speech from given text labels with a contrastive focus emphasizing a specific word. In contrast, this paper investigates whether we can control prosody based on the contrastive focus that appears in images. Given an image and its caption, our system first discovers spoken terms associated with objects or situations in natural images based on a self-supervised visually grounded model. Then it generates speech with prosody prominence based on the contrastive focus of these spoken terms in a way that best describes the images. The framework can perform the task with/without text annotation, making it applicable for untranscribed, unsegmented speech utterances in unknown languages.

KW - multimodal learning

KW - prosody prominence

KW - self-supervised visual grounded models

KW - text-to-speech synthesis

UR - http://www.scopus.com/inward/record.url?scp=85190534577&partnerID=8YFLogxK

U2 - 10.1109/O-COCOSDA60357.2023.10482965

DO - 10.1109/O-COCOSDA60357.2023.10482965

M3 - Conference contribution

AN - SCOPUS:85190534577

T3 - Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023

BT - Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 26th Conference of the Oriental COCOSDA International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023

Y2 - 4 December 2023 through 6 December 2023

ER -

Ika Hartanti BS, Tanaya D, Azizah K, Lestari DP, Purwarianti A, Sakti S. Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models. In Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (Proceedings of 2023 26th Conference of the Oriental COCOSDA International Committee for the Co-Ordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2023). doi: 10.1109/O-COCOSDA60357.2023.10482965

Generating Speech with Prosodic Prominence based on SSL-Visually Grounded Models

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this