sriragjayakumar/Show-and-Speak-DETR

In recent times, developments in the fields of image captioning have proposed novel models such as show and speak (SAS) which directly synthesize spoken description of images, bypassing the need for any text or phonemes. The basic structure of SAS is an encoder-decoder architecture that takes an image as input and predicts the spectrogram of speech that describes this image. The final speech audio is obtained from the predicted spectrogram via WaveNet. Further, SAS uses recurrent neural network-based models such as LSTMs for speech production. We propose to investigate in this study the use of transformers for SAS models, given the superior performance of transformers for generating sequential data.