Russian Students Build AI to Judge the Quality of Voice Assistants
Students at the Higher School of Economics and VK have created an AI that evaluates synthetic speech, dramatically speeding up and reducing the cost of building voice assistants and navigation systems.
At the Applied Artificial Intelligence Lab of the Higher School of Economics and VK, students devised a way to automatically assess the quality of voice assistants. Until now, evaluating computer-generated speech required human assessors to manually listen to audio files and score them—a process that was slow, expensive, and highly subjective.
The students trained their neural network on the large open SOMOS dataset, which includes over 20,000 audio samples and 350,000 human evaluations. They developed two key metrics and five models to run the calculations. MOSNet evaluates a single file on a scale from 1 to 5. NeuralSBS compares two audio clips and selects the better one.
Initial testing showed that the AI evaluates audio with accuracy close to human judgment. In comparison tasks, the model chose the better audio 73% of the time—on par with the average listener.
This innovation will accelerate the development of speech technologies, making the creation of voice assistants more reliable, scalable, and efficient. The team plans to adapt the models for the Russian language and integrate them into production workflows.
