Shobhit Banga Introduces Turing TTS, a Test of How Often Listeners Mistake AI Voices for Humans
Drop 2/15 Are we actually measuring the things we ultimately want our voice AI models to achieve? The best AI voices are becoming hard to tell apart. From each other. Not from humans. 馃殌 Introducing Turing TTS: an evaluation that measures how often human listeners mistake AI-generated speech for real human speech. We tested leading TTS models against real human speech. Even the best model fooled listeners only ~10% of the time. With Turing TTS, we hope to contribute to a broader discussion within the research community about how we measure progress in voice AI. If one of our goals is to develop speech that is indistinguishable from human speech, how well do our current evaluation methods capture progress toward that goal? We don鈥檛 claim that Turing TTS is the right way forward, or that human indistinguishability should be the goal of every TTS system. Rather, we hope these findings encourage the community to collectively examine whether blind pairwise human evaluations of naturalness are sufficient, and what else we might need to measure. Results are now live on @voicearena_ai.
Context
If the result holds, today's best AI voices still sound clearly synthetic to careful listeners, which suggests common naturalness scores may overstate how close voice AI is to human speech.
Text to speech, or TTS, models turn written text into spoken audio. They are usually compared with blind pairwise tests where listeners pick which of two clips sounds more natural. Voice Arena is a public leaderboard that ranks TTS models this way.
Deep dive
Shobhit Banga posted on Friday what he called drop 2 of 15, introducing Turing TTS, an evaluation that measures how often human listeners mistake AI generated speech for real human speech. All results described here come from his post.
According to the post, the best AI voices are becoming hard to tell apart from each other, but not from humans. He says the team tested leading TTS models against real human speech, and even the best model fooled listeners only about 10 percent of the time. The post does not name the models or share sample sizes, and we did not find a published paper for this evaluation.
Banga frames the work as a contribution to a debate rather than a final answer. He writes that they do not claim Turing TTS is the right way forward, or that being indistinguishable from humans should be the goal of every TTS system. Instead, he hopes the findings push researchers to ask whether blind pairwise ratings of naturalness are enough.
The post says the results are live on Voice Arena, whose methodology page describes its existing leaderboard as pairwise naturalness voting. A similar idea, a human fooling rate, appeared in academic work presented at Interspeech 2025.
What to expect next
Watch for the full Turing TTS results, including which models were tested, how many listeners took part and how clips were chosen, which will decide how much weight the 10 percent figure deserves. Voice AI companies may start citing fooling rates in launches. The remaining drops in the series may add more evaluations. Independent researchers will likely try to reproduce the result.