@intimate-mirror's rent-lowering gunshot post is excellent, but it also made me realize that I've never really written up a full response to the AI foom stuff. Obviously this is all just my opinion, but I do think that as someone who has worked in RL, offline RL, and bandit theory, I think I'm somewhat uniquely positioned to give this argument.
yud's argument basically comes down to the idea that it is much harder to constrain a very intelligent AI than to make one. I think this is a reasonable intuition to have from the examples he gives, such as the "security mindset" bit. However, I do not think that this idea actually holds up very well to the mathematical intuition of statistical agents operating autonomously in an environment.
Pure unsupervised learning will not get you superintelligence. Unsupervised learning approximates the dataset with some amount of error, and in the large network and large data limit, it will simply predict human text with no error. This does not require superhuman intelligence. It's suspected that the effectiveness of LLMs comes from the fact that after a large enough sample size, it's more parameter efficient to actually model the world than memorize the dataset, and models learn the more parameter-efficient option. By the same principle, if superintelligence is not needed to model the dataset (as it is not for predicting human text), it will not be learning.
RL can hypothetically give you superintelligence. Unlike unsupervised learning, RL collects its own data and seeks to maximize cumulative reward, so it is not constrained to merely imitating human intelligence. There are many many problems where RL gives an agent that is better than the best humans, but there are no supervised or unsupervised ML algorithms trained on humans that are more accurate then the humans that the data was taken from.
RL is really, really, stupidly hard. It's way harder than traditional ML in both theory and practice. Baby's first regression problem is linear regression. It's easy to explain as a minimization problem, it has a closed form solution that's easy to find, and it's not too hard to find convergence rates. Baby's first RL problem is the multi-armed bandit problem, and baby's first algorithm for multi-armed bandits is UCB, which requires graduate-level stats like Chernoff bounds to explain. RL with linear function approximation is extremely difficult to analyze, such that questions like "how do I make all the weights of my linear function approximator not blow up to infinity" were open questions that took 15+ years to answer. In practice, this theoretical finickiness manifests as an extreme dependence in implementation minutia, making good results an exercise in trial and error.
Comparatively, keeping RL near a dataset is pretty easy. Basically this is because if you have access to the dataset and don't have to go explore to find new data, the problem becomes much much easier. It can be turned into a single closed-form optimization problem, but you can also get bounds by doing something as simple as adding a term that penalizes KL divergence from the dataset on a per-point basis.
Alignment is as simple as staying in the distribution of human-approved actions. The doomsday scenarios for AI foom are all scenarios in which the AI wants to do something that no (or at least vanishingly few) humans would do, such is melting the entire earth down into paperclips. Any non-zero regularization term like KL(P(action | human) || P(action | AI)) becomes infinite as the probability of a human taking this action goes to zero, and it's easy to get upper bounds on this KL penalty that can easily be optimized.
The result of all this is that it's way easier to align agents than to allow them to figure out stuff on their own. In my opinion, alignment (offline RL) is just a much easier problem than superintelligence (online RL). I think this view lines up well with what we have seen from AI chatbots so far. RLHF came along very quickly, and the "aligned personalities" of chatGPT and Claude very quickly reached a mature form. It's taken a lot longer for us to figure out how to do open-ended Rl from Verifiable Feedback, and although this looks immensely promising, it's very much still in its infancy.
















