bg
Science and new technologies
19:04, 16 ноября 2026
views
1

The Illusion of Objectivity: MSU Is Changing the Rules for Testing Russian AI

Researchers at the Faculty of Computational Mathematics and Cybernetics at Moscow State University have uncovered a fundamental problem in evaluating how well Russian-language neural networks perform syntactic parsing. It turns out that test results can vary dramatically depending on how the data is prepared.

MSU researchers have demonstrated a counterintuitive finding: the measured accuracy of AI-powered text parsing can change dramatically depending on how information is preprocessed. Their study, published in the journal Pattern Recognition and Image Analysis, exposes a fundamental problem with the way modern neural networks are evaluated.

The Tokenization Trap

Imagine testing and comparing five neural network parsers for Russian. Their job is to assess text complexity and identify borrowed words. Using one evaluation approach, Stanza emerges as the clear winner. But change the way the text is prepared, and DeepPavlov takes the top spot instead. How can that happen?

The answer lies in tokenization – the process of splitting a sentence into individual words and punctuation marks. The researchers tested the algorithms on texts ranging from poetry to Wikipedia articles. Their findings show that giving a neural network supposedly “perfect” gold-standard annotations can send average performance metrics soaring. But there is a catch: in some cases, perfect input data actually make the results worse.

Russian is particularly challenging for language-processing systems because of its rich system of inflection and relatively free word order. Simplifying these features can lead to misleading results. To untangle the problem, the MSU team developed a methodology that distinguishes between errors made by the neural network itself, classification errors and failures introduced during data preparation.

Building a Foundation for the Future

At first glance, these may seem like purely academic details. But syntactic parsing underpins technologies people and businesses rely on every day, including intelligent search, legal document analysis and educational platforms. If we misjudge the performance of these foundational tools, we risk building entire enterprise technology ecosystems on shaky ground.

The MSU study fits into a broader push to develop independent benchmarks for evaluating AI. Consider MERA, a platform developed by the Alyans v sfere iskusstvennogo intellekta (Alliance in the Field of Artificial Intelligence), which tests models on reasoning tasks. The MSU researchers' methodology takes evaluation a step further: it helps identify why a generative model makes a mistake, rather than simply measuring how often it gets something wrong. That distinction matters for the development of Russian NLP solutions. Drawing on the international Universal Dependencies standard, the researchers have proposed a tool that can also be adapted for other languages.

How the Research Has Evolved: From Language Corpora to Neural Networks

Since 2022, the Natsionalny korpus russkogo yazyka (Russian National Corpus) has been actively introducing neural-network-based annotation. In 2023, researcher Oleg Bartenyev showed that tokenization directly affects task performance metrics. In 2025, Elena Shamaeva found that only 25% of sentences receive a perfect parse. Now, in 2026, MSU has done more than develop yet another neural network. It has proposed a new lens through which researchers can evaluate all existing algorithms.

The MSU researchers' central conclusion is that no AI test can be completely objective. Every score reflects a compromise between the algorithm being tested and the way the data are prepared. For Russia, this points to a shift away from relying unquestioningly on foreign evaluation metrics and toward developing its own, more rigorous standards for assessing AI systems.

The MSU methodology does not guarantee that generative AI will suddenly become more capable. Large language models often work without traditional syntactic parsing. But for specialized applications, such as analyzing legal documents or running intelligent searches across corporate databases, accuracy can make all the difference.

The methodology will not make AI assistants perfect. What it can do is help developers understand what actually causes their systems to fail. After all, building truly intelligent AI for Russian starts with learning how to evaluate it honestly and precisely.

In recent years, with the emergence of large language models such as ChatGPT, LLaMA and their counterparts, an urgent need has arisen to distinguish genuine human creativity from AI-generated text. These AI systems are increasingly being used not only to write viral online content, student term papers and theses, but also serious research papers published in peer-reviewed journals
quote
like
heart
fun
wow
sad
angry
Latest news
Important
Recommended
previous
next