Closed loops and AI

Just read this take from Elad Gil:

AI will eat closed loops first

AI will first automate away the things that are easier to form a closed loop learning system on. This is why code and AI research may be accelerated and then displaced quickly - you can have testable closed loop systems so machines can learn and iterate quickly. The tighter the closed loop, the faster the AI can learn. You can make a 2x2 of jobs by how closed loop they can be made, versus their economic value, and see where AI may impact labor fastest. Fast time to closed loop + high economic value = fastest AI impact (hence software engineering).

For reinforcement learning to work in model training, you need a source of reinforcement. So the things that the models are best at right now are those that are verifiable, fields and domains where the answer can be known to a high degree of certainty, and the model's output checked against reality.

It's one of the reasons code has been such a success for AI so far: the tasks are verifiable, and the output high value. There are plenty of closed loops out there to attack. Code just happened to be one of the largest, most generally useful, and in highest-demand.

The writing and testing of the code is a closed loop, but the problem toward which to apply attention — what to build? why? for whom? — is still very much an open one.

What other open loops still demand deep human involvement?