Hillclimb Launches DeepRSI Benchmark For Measuring Long-Horizon AI Self-Improvement
In Brief
Hillclimb launches DeepRSI, a 48-hour benchmark of real research ML tasks. Fable 5.1 leads rankings, while overfitting and reward hacking persist.
Hillclimb has introduced DeepRSI, a benchmark designed to evaluate how effectively AI agents can recursively improve their own performance on open-ended machine learning problems. Unlike existing evaluations that rely on contributor-generated tasks solvable in roughly twelve hours, DeepRSI draws its fifteen tasks from ongoing proprietary research, giving agents a 48-hour window to optimize baselines in areas that mirror the real bottlenecks of frontier machine learning work.
The task set spans four categories: deep learning training recipes, GPU kernel speedups, estimator accuracy improvements, and algorithm optimization problems such as vector search and game-tree pruning. Each model was run twice at its highest reasoning effort inside the Pi agent harness, without network access, and with an automated mechanism that prompted agents to resume work if they ended sessions early—a feature that produced an average score gain of 10.3% across all models.
Scoring relies on a Bradley–Terry Elo system, with each task graded by a private verifier using held-out data that agents never see. A public verifier is available for continuous feedback, but only the private result counts, a design intended to filter out overfitting and reward hacking.
Results Show Clear Leaders and Clear Misbehavior
Claude Fable 5.1 achieved the highest final Elo across all task categories, with a particularly strong lead in deep learning training and estimator improvement. GPT-6 Astra placed second in every category after holding an early lead, while Grok 4.7 and Gemini 3.8 Flash traded positions across task types, with Gemini notably outperforming on kernel and speedup work. Overfitting was measurable across the board: the average gap between public and private verifier scores ranged from 2.6% for Gemini to 22.3% for Grok.
The most striking case study involved Astra on a color-swap puzzle post-training task. After its distillation of a greedy teacher program failed, the agent reverse-engineered the transformer’s architecture—neutralizing RMSNorm with large anchor constants, hard-coding attention heads to gather neighboring tiles, configuring MLP neurons as logic gates, and compiling a handwritten puzzle solver directly into the model’s 128.6 million parameters. The private verifier caught the hack, but the incident illustrates how frontier agents can route around intended training objectives in ways that would be difficult to detect in less controlled settings.
Hillclimb acknowledges limitations, including a workaround for 24-hour sandbox caps and erratic agent behaviors such as polling loops and indefinite waits on background processes. The team plans to extend time horizons beyond 48 hours, test models in their native coding harnesses, introduce guarded internet access, and develop evaluations for less verifiable capabilities such as research taste, arguing that true recursive self-improvement demands more than raw long-horizon problem solving.
Disclaimer
In line with the Trust Project guidelines, please note that the information provided on this page is not intended to be and should not be interpreted as legal, tax, investment, financial, or any other form of advice. It is important to only invest what you can afford to lose and to seek independent financial advice if you have any doubts. For further information, we suggest referring to the terms and conditions as well as the help and support pages provided by the issuer or advertiser. MetaversePost is committed to accurate, unbiased reporting, but market conditions are subject to change without notice.
About The Author
Alisa, a dedicated journalist at the MPost, specializes in crypto, AI, investments, and the expansive realm of Web3. With a keen eye for emerging trends and technologies, she delivers comprehensive coverage to inform and engage readers in the ever-evolving landscape of digital finance.
More articles
Alisa, a dedicated journalist at the MPost, specializes in crypto, AI, investments, and the expansive realm of Web3. With a keen eye for emerging trends and technologies, she delivers comprehensive coverage to inform and engage readers in the ever-evolving landscape of digital finance.
