When Intelligence Stops Being the Bottleneck: AI and Recursive Self-Improvement
Recursive self-improvement is the mechanism behind many of the most dramatic forecasts about advanced AI. An AI becomes better than humans at AI research, helps create a more capable successor, and thereby produces a system even better at designing the generation after that. Follow the loop far enough and an intelligence explosion can seem almost inevitable. The loop, however, contains more than intelligence. Models still have to be trained. Experiments have to be run and interpreted. Chips have to be fabricated. Electricity has to be generated. Biological hypotheses eventually encounter cells, animals and patients. Becoming better at thinking does not make every other input to technological progress disappear. The more interesting possibility may therefore be not an unlimited intelligence explosion, but a repeated migration of scarcity. If AI makes sophisticated reasoning cheap and abundant, the bottleneck moves elsewhere: towards compute and evaluation, then towards experiments, ...