📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are now capable of automating the core engineering tasks involved in AI research, with benchmarks approaching saturation. Research, however, remains less automated, leaving residual work that could soon be minimized.
Recent empirical evidence shows that AI systems have reached near-complete automation in core engineering tasks essential to AI research, marking a significant shift in the field. While research activities are still less automated, the trajectory suggests that the residual research work may soon be minimized, fundamentally altering the landscape of AI development.
Multiple independent benchmarks measuring AI capabilities in core research and engineering tasks have shown rapid progress toward saturation. For example, the CORE-Bench, which tests AI’s ability to reproduce research papers, improved from 21.5% in September 2024 to 95.5% in December 2025, with some experts declaring it ‘solved.’ Similarly, the MLE-Bench, evaluating AI on Kaggle competitions, rose from 16.9% to 64.4% over sixteen months, reaching a performance level comparable to mid-tier human practitioners.
These benchmarks, which assess tasks such as reproducing research code, optimizing models for competitions, and designing GPU kernels, indicate that AI can now handle much of the engineering work involved in AI research at a high level of reliability. The progress across these metrics suggests that the bottleneck in AI research is shifting away from engineering towards the residual, less automatable aspects of scientific inquiry, such as hypothesis generation and experimental design.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
Mastering AI in Clinical Research: : Tools, Techniques, and Real-World Applications (AI in Clinical Research Series Book 5)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications of Automated AI Engineering for Scientific Research
The automation of core engineering tasks in AI research signifies a potential paradigm shift, reducing the time and cost associated with experimental iterations. This could accelerate innovation cycles, democratize access to advanced AI development, and challenge traditional notions of scientific labor. However, it also raises questions about the future role of human researchers and the nature of scientific discovery, as much of the technical groundwork becomes increasingly automated.
Progress in AI Capabilities and Benchmark Saturation
Over the past two years, AI systems have demonstrated remarkable advances across multiple research-relevant domains, including research reproduction, competition performance, and kernel design. Benchmarks like CORE-Bench and MLE-Bench have shown consistent, rapid improvements, approaching or reaching saturation levels. This pattern indicates that AI’s engineering skills are nearing human expert levels, with some experts declaring certain tasks ‘solved.’ Meanwhile, the field is observing a surge in research papers and technical reports detailing new methods for automating kernel optimization and infrastructure design, further supporting the trend toward automation.
“The pattern across multiple benchmarks suggests that AI is approaching or has reached saturation in core engineering tasks, fundamentally changing the landscape of AI research.”
— Thorsten Meyer
Unresolved Questions About Research Automation Limits
It remains unclear how much of the broader research process—such as hypothesis formulation, experimental design, and interpretation—can be automated. While engineering tasks are nearing full automation, the residual research activities may involve inherently creative or abstract thinking that AI has yet to master. Additionally, the pace at which these residual tasks will become automatable remains uncertain, as does the impact on human researchers’ roles.
Next Steps in AI Research Automation Development
Expect continued rapid progress in automating research-related tasks, with benchmarks pushing toward saturation and new methods emerging for kernel and infrastructure automation. Researchers and institutions are likely to reassess the division of labor between human and AI contributors, possibly leading to shifts in research workflows. Monitoring developments in hypothesis generation, experimental design, and scientific interpretation will be critical to understanding the full scope of AI’s future role in research.
Key Questions
What does the automation of engineering mean for AI research?
It suggests that much of the technical and infrastructural work involved in AI development can now be handled by AI systems, potentially speeding up innovation and reducing costs.
Are all research activities automatable?
No, activities involving creative thinking, hypothesis formulation, and interpretation are less automatable and may remain human-led for now.
What are the risks of automation in research?
Risks include over-reliance on AI for scientific judgment, potential loss of human expertise, and ethical concerns about transparency and accountability.
How soon could residual research work become automated?
The timeline remains uncertain, but current trends suggest significant progress could occur within the next few years, depending on breakthroughs in AI’s creative and abstract reasoning capabilities.
Source: ThorstenMeyerAI.com
