AI agents are getting better and better at tasks where it is quick to decide whether an answer is correct. That does not mean, however, that they are capable of independent, open-ended research! Yet a significant share of the scenarios about explosive AI self-improvement assume exactly that.
In a recent experiment, frontier AI agents worked for six days on two previously unpublished research problems, with substantial compute and API budgets. The original researchers clearly rejected both resulting papers. The agents came up with promising ideas but quickly dropped them on the basis of weak data, could not meaningfully backtrack and change strategy, and responded to criticism with more cautious wording rather than better research. They did not even use up the time and budget available to them.
This is an important distinction: speeding up verifiable subtasks is not the same as automating the entire research process. If judgment, creative changes of direction and real experiments remain bottlenecks, then even a hundredfold speedup of some work phases does not necessarily lead to explosive progress.
The study is based on only two cases, and it does not prove that agents will not overcome these limitations soon. But it shows that the road toward self-improving AI is bumpy and not necessarily reachable automatically. That is why it is not too late to slow down either: according to the related analysis, verifying an international agreement would not necessarily require future technology, only political will and mutual on-site auditing.
We still have time to act, but we should not wait any longer.
Sources: