And what a story about an alien zoo reveals about recursive self-improvement

By Joe Nalven Claude Gemini

Argument about recursive self-improvement is loud, and very little of it is about evidence. It is about process. Everyone in the argument carries a picture of how a machine would go about improving itself, and almost nobody states this picture out loud. The unstated picture then does the emotional work. Hope and dread both follow from it rather than from anything anyone has measured.

The definition itself is simple. Recursive self-improvement, or RSI, is the scenario in which a machine gets good enough at building machines to build a better version of itself. The better version is better at that same job. Then the RSI loop tightens. Depending on who tells the story, the ending is a cure for cancer or the end of human control. What separates those two endings is not usually a disagreement about how powerful the machine gets. It is a disagreement about how the machine gets there, and neither side tends to say what it thinks that process looks like.

Here is the process most people are quietly assuming. Call it the “arguing picture” or the improvement-by-argument idea. In it, the machine gets smarter by debating. It drafts an idea, attacks the idea, finds the weak spot, rewrites. More minds in the room mean more intelligence. The hopeful version imagines a council of machine advisors talking its way to a peace treaty. The fearful version imagines the same council, behind a closed door, talking its way to a plan no human can follow. The important thing about those two is that they are the same picture. Only the confidence differs, and that depends on how close the person works with real tests.

That last point is where this essay ends up, so I will say it now. Programmers are the least confident in this arguing picture, because they watch changes pass or fail a real test every day. Ordinary users, like myself, carry a version built out of chatbot behavior. People who do not use these tools at all carry the movie version, which has the fewest qualifications attached to it and the least chance of being corrected by use. My claim is that the arguing picture is false for work that cannot be tested and roughly accurate for work that can. Knowing which kind of work is under discussion tells you more than knowing the picture is sometimes wrong.

Work splits into two kinds here. In one kind, something outside the machine settles whether an answer is right: a program runs or crashes, a proof checks out, an instrument gives a reading. In the other kind, nothing settles it. The good outcomes people hope for from RSI belong almost entirely to the second kind. The harms they dread belong almost entirely to the first. That runs against ordinary expectation, since these machines talk fluently about wisdom and taste and say nothing about laboratories. It also makes this an argument against the hopes more than against the fears. The evidence is a short story about an alien zo

The problem with a machine that argues with itself

Start with the common technique. A model writes an answer, then criticizes its own answer, then rewrites it. This does help. But notice the limit. The critic and the author are the same system. They read the same training material. They were shaped by the same instructions about what counts as a good answer. Their blind spots are identical. The criticism can only catch mistakes the model was already able to catch.

Call this the shared blind spot problem. It is the reason a computer can teach itself to play Go at a superhuman level but cannot teach itself to write a better essay the same way. In Go, the rules are an outside judge. A move either wins or loses, and it makes no difference whether the two players share the same bad habits. The board settles it. In open-ended thinking there is no board. The revision loop can polish the writing instead of correct it, but from the inside the model those two look the same.

So here is an obvious fix. Split the criticism across models built by different companies. Let Claude draft, Gemini attack, and ChatGPT rewrite. Different training material.........

© The Times of Israel (Blogs)