📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a sovereign language model trained from scratch with extensive Italian data, performs poorly on Italian academic tests despite impressive technical benchmarks. This challenges assumptions about the scale of investment needed for country-specific AI models.
Italy’s Minerva-3B, a large-scale sovereign Italian language model trained from scratch on 2.5 trillion tokens, scored only 4.9% on the INVALSI Italian school-exam benchmark, a result that raises questions about the effectiveness of current native-language AI investments.
Minerva-3B was developed by Sapienza University of Rome’s NLP team, led by Roberto Navigli, utilizing Italy’s national supercomputing infrastructure and funding through Italy’s PNRR initiative. The model was trained on approximately 50% Italian data, totaling 2.5 trillion tokens, making it one of the most extensive native-language models produced in Europe.
Despite its large training dataset and impressive performance on certain benchmarks, Minerva-3B’s score of 4.9% on the INVALSI Italian exam—an academic-content test—indicates significant limitations in handling complex language tasks related to education and knowledge depth. Researchers note that while dataset size and parameters are crucial, they may not be sufficient alone to develop models capable of understanding nuanced, country-specific knowledge.
This empirical finding suggests that the European sovereign-LLM project faces fundamental challenges in scaling native-language models to the depth required for national educational and knowledge applications, even with substantial investment.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
AI language model training datasets
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Minerva’s Performance for European AI Strategies
The results from Minerva-3B challenge the assumption that larger datasets and more parameters automatically lead to deeper language understanding, especially in specialized domains like education. For European nations investing heavily in sovereign AI, this indicates that scale alone may not be sufficient to produce models with country-specific knowledge depth. The findings could influence future funding, research priorities, and strategic decisions about native-language AI development across Europe.
Additionally, the case highlights the importance of evaluating models not just on benchmarks but on real-world, domain-specific tasks. Policymakers and researchers must reconsider the investment strategies necessary to achieve truly effective, country-tailored AI systems.
Background on European Sovereign-Language Models and Minerva’s Development
Italy’s Minerva project represents a significant effort to develop a European sovereign language model from scratch, contrasting with approaches like Portugal’s AMÁLIA, which relies on continued pre-training of multilingual models. Minerva was trained on 2.5 trillion tokens, with roughly half Italian content, and is publicly releasing its weights and code, marking a major step in European AI infrastructure.
Despite the scale and resources involved—including state-of-the-art supercomputing via CINECA and national funding—the model’s poor performance on the INVALSI exam underscores the ongoing debate about the effectiveness of native-language training at current parameter scales. The project has been closely watched as a test case for the feasibility of sovereign AI in Europe, especially given the strategic importance of language-specific models.
“Minerva’s results demonstrate that scale alone may not suffice for native-language knowledge depth, prompting a reconsideration of European AI investments.”
— Thorsten Meyer
Unresolved Questions About Model Capabilities and Future Improvements
It remains unclear whether further training, larger models, or different methodologies could significantly improve Minerva-3B’s performance on complex, domain-specific tasks. The ongoing research aims to determine if the current results are a fundamental limit or a temporary stage in development.
Additionally, the precise impact of dataset composition, quality, and training strategies on knowledge depth remains under investigation, and the generalizability of these findings to other languages and domains is still uncertain.
Next Steps for Minerva and European Sovereign-Language AI Projects
The Minerva team plans to continue iterative training and evaluation, including experiments with continual training and domain adaptation. Future releases and research will aim to address current shortcomings and test whether increased investment or different approaches can produce models with deeper country-specific knowledge.
European policymakers and researchers are likely to reassess funding priorities and strategic frameworks, emphasizing the importance of not only scale but targeted knowledge development in native languages.
Key Questions
Why did Minerva-3B perform poorly on the Italian exam?
The empirical data suggests that despite large-scale training, the model lacks the depth of country-specific knowledge needed for complex academic tasks, highlighting limitations in current scaling strategies.
Does this mean native-language models are impossible to develop?
Not necessarily. The results indicate that current parameter scales and datasets may be insufficient, but future research could improve performance through targeted strategies.
What does this mean for Europe’s AI sovereignty efforts?
It underscores the need to reconsider investment strategies, emphasizing not just data and parameters but also the quality and focus of native-language training.
Will Minerva improve with more training or larger models?
It remains to be seen. Ongoing research and iterative training may enhance capabilities, but the current results highlight fundamental challenges that need addressing.
Source: ThorstenMeyerAI.com
