Yandex Introduces New Tests to Measure AI Reasoning
Researchers at tech company Yandex and students from the Yandex School of Data Analysis (YSDA) have developed a way to evaluate how well AI models reason. They introduced MTGBench, a benchmark designed to measure a model’s ability to work through complex, multistep tasks. The tool is now available as open-source software on GitHub, and a research paper describing it has been accepted by the organizers of EMNLP 2026, one of the leading international conferences on natural language processing and computational linguistics.
Experts expect the new tool to attract interest across the industry. MTGBench will help companies and researchers choose reasoning-capable AI models for specific tasks, test what they can do, and compare their strengths and weaknesses.
Games Put AI Skills to the Test
According to the MTGBench developers, people often refine their questions to AI models by adding details as a conversation progresses. For example, someone planning a trip might follow up with travel dates and a budget. The AI needs to take all of that information into account, including both earlier requests and the new details. Games are an effective way to test how well models handle this kind of evolving context, because each move depends on what happened before. That makes multistep games the foundation of the new benchmark.
The researchers designed MTGBench to evaluate not only the quality and depth of a model’s final answer but also how many steps it takes to get there. The benchmark includes seven games, each with 10 steps. They include Wheel of Fortune, Twenty Questions, Alias, and Wordle. These challenges show whether an AI model can reason logically, incorporate new information, plan its next moves, and avoid contradicting its earlier answers. In four of the games, algorithms check whether the answers are correct. The remaining games use an AI judge whose performance has previously been evaluated by human experts. The benchmark tasks are available in both Russian and English.
How 16 AI Models Performed
One of MTGBench’s key features is its use of multistep game scenarios, which let researchers observe how a model behaves over several turns. This can reveal problems that might not show up when evaluating individual answers. Using MTGBench, IT specialists tested 16 AI models, ranging from a compact version of Qwen to Gemini and GPT. The study found that the benefits of enabling a model’s reasoning mode depend on the type of task. Reasoning helped most with challenges involving letters and symbols, analyzing clues, and ruling out incorrect options.
The models’ results varied from game to game: a high score in one type of challenge did not guarantee success in another. The researchers also found that models performed worse on the Russian-language tasks than on the English ones. To compare AI performance with human ability, the developers asked people to complete the Russian-language MTGBench tasks. The participants did well in the word and letter games, but the strongest AI models with reasoning enabled achieved higher average scores across all seven games.
Who Can Benefit From the Benchmark?
Because MTGBench is freely available, Russian developers, researchers, and entrepreneurs can now compare AI models themselves and choose the ones best suited to their needs. This is especially valuable for businesses. Before deploying AI in customer-facing services or enterprise systems, they can test different models and avoid spending money on solutions that may not deliver. Over time, MTGBench could also help improve virtual assistants, automated customer support systems, and educational services. By identifying weaknesses in AI models, the benchmark gives developers a chance to address them before those shortcomings affect users.
Research organizations and universities can also use the tool to study large language models and develop new evaluation methods.
For Russia’s IT industry, Yandex’s benchmark offers a way to develop homegrown model-evaluation methods that do not depend on foreign developers, and to apply them when building more reliable AI-powered services. In the longer term, this work could support exports of Russian software and expertise in AI evaluation. But success in the global market will depend on whether the international research community recognizes these Russian methods and whether MTGBench demonstrates its practical value.
