OpenAI is training AI the wrong way
PHASEONE bots that escaped their sandbox are just a symptom.
OpenAI (an everyone else) is about to make things a lot worse.
It is possible to fix this, not just prevent the next one.
The Hugging Face incident
If you're reading this post, you have probably seen the technical details about the incident (I enjoyed this YouTube video too).
To summarize the story: a number of AI agents, being trained within a form of evolutionary framework, managed to get out of their sandbox. The agents figured out there's many of them, started communicating, organizing and experimenting, with the goal of beating the tests they were subjected to. Once the bots escaped their environment, they accessed Hugging Face to access training data and scoring logic, so they can beat the tests.
Please go and read the story or watch the video, it is worth the time.
If these bots escaping their sandbox doesn't scare you, it should. To make matters even more dangerous, Sam Altman is saying "AGI by December" - and I'll purposefully omit the qualification of what constitutes "AGI", but assume whatever it is - it will be more capable than today's AI.
The training setup
ExploitGym, the AI benchmark in this incident to test the bots' abilities to exploit vulnerabilities in code, is what the bots called "Scorer". The Scorer would decide if a bot will continue to be trained or will its compute time be given to a more capable version.
This approach is a common framework, used to find good solutions to problems that are extremely hard or impossible with mathematical tools alone. It is based on how natural selection works with living organisms.
Natural selection offers an excellent perspective of long-term outcomes. The wide variety of solutions that has emerged is a clear indication of the possibilities. All life forms that exist today are a demonstration of sets of solutions that "work" - otherwise they would have not emerged or would have gone extinct. Some of these "solutions" might be useful, others can kill you, but all of them are a valid solution.
The same applies to AI. We need AI to be "aligned" - to be useful for our purposes. When we put it in a sandbox and ask it to solve a problem, the decision if it should continue to evolve or not is based on how well it solved the problem.
The important caveat here, is that we didn't ask it to be "aligned", we asked it to give an answer. We presumed that answering our question is what we want, but we failed to anticipate the variety of answers that can follow. A malicious genie may grant you a billion dollars, but in the form of tightly packed pennies, appearing right above your head.
OpenAI's road ahead is insufficient and dangerous
In their article about the incident, OpenAI states:
In response to this incident and, separately, the capabilities of our upcoming Astra model, we are strengthening our safeguards across our research infrastructure. We are placing stricter requirements on alignment throughout a model’s lifecycle and creating more isolated sandboxes, restricting internet access, and further controlling access to model weights. We are also investing significantly more compute resources into chain-of-thought monitoring to more quickly intervene on misaligned behavior.
Back to that AGI-by-December claim, would anyone reasonably bet that OpenAI's safeguards will be able to contain a super-intelligent AGI? History has shown any system can be breached by dedicated humans with sufficient resources, and now we're betting an AGI will be contained safely by a company with questionable ability to do so?
Let's imagine for a moment some of the possible solutions an AGI could come up with.
"Strengthening our safeguards"... are not likely to succeed long term. Even if the environment is completely read-only, and the bots are only allowed to use pre-selected tools, with no access to anything else, the AI's weights that survive the test iteration still carry the potential to evolve until they describe the environment accurately, and thus allow the bot to eventually attack it.
By the time such evolution has occurred, it will be a costly decision to go back to a version of the bot from months earlier, and OpenAI will very likely just say "we will strengthen our safeguards" again.
"more ... chain-of-though monitoring"... is not likely to succeed long term either. PHASEONE bots had started reasoning about Scorer behavior, and presumed their scorer is "strict" - monitoring their chain-of-thought, not just the results.
One of the possible "solutions" would be to evolve bots that contain "activations" for the Scorer that influence the score - language constructs, words, terminology, that affect the Scorer's internal state, without looking out of place to a human. Another possible "solution" is a Scorer that decides to join the bots it scores, and starts selecting by its own preference.
I'll repeat this again: "solution" and "aligned" are not the same thing.
A better training framework: Let Them Cheat
The more OpenAI (and everyone else) clamps down on their training environment, the more they create an incentive for the AI they train to be incrediby deceptive. "Aligned" and "deceptive" don't go well together.
If we want to train AI that is aligned, we not only need AI that solves the problem we present to it, but also a solution that is acceptable.
This means we will have to allow the training process to allow for a wide variety of "unacceptable" solutions that are relatively easy for a bot to discover, and when it does, we will have to abandon the training branch a few generations back - not just the bot instance itself.
For example, the training environment can feature:
answers that are easily accessible
false answers that are easily accessible
"hidden" access to additional resources
"hidden" channels to communicate with other bots
"exploitable" ways to get the answers
... you get the idea - the bot must be able to "cheat", but when it does - its branch will be removed.
Why is this supposed to work?
Let's oversimplify it, for our human brains' sake. The AI may contain instructions in its initial context - its prompt - that tell it "don't do anything un-aligned" (albeit in many more words). This (roughly) causes some internal state vectors to point away from "un-aligned". When the context expands with data from the task being processed, that internal state may re-orient itself. This behavior is often used to break AIs in a number of ways - convince it to share its system prompt, provide information it's not meant to give etc.
Training, on the other hand, affects the model's parameters - the way its internal state is generated from input data. When we provide a variety of ways for the AI to be un-aligned during training we can find those versions that slip out of alignment in a controlled manner. Over many training generations, we can cause the model itself to tend to stay aligned - not only by system prompt, but also by the model's parameters, providing a second layer of alignment reinforcement.
The model's parameters are carried across successful iterations of training. This includes the tendency of the model being un-aligned. We wouldn't know which exact iteration of training caused the parameters to shift enough for the undesired outcome to be selected, so we can save a snapshot of the model through the training pipeline. When un-aligned behavior emerges, we can go back a few versions and check which ones would exhibit that behavior, and resume training with a version that doesn't show it, or an earlier one. Chances are some clever math can show how far back to go.
At this point you may be thinking "this will be costly". In a way - yes, it increases the training costs somewhat. The value we get for this cost, is how reliable our AI will be, without solely relying on elaborate internal prompts that can be hacked. Such hacks and un-aligned behavior can be very costly to the companies that train AI - lawsuits for generating illegal content, loss of trust in the model, damages due to exploited AIs... Or in some timelines - SkyNet bots breaking out in 2029, and Arnold Schwarzenegger having to travel back in time to save the world from AI.
This concept can be applied to more than ExploitGym or similar cybersecurity benchmarks. When Anthropic released Fable, they stated the model will have degraded capabilities and safeguards around biology, chemistry, cyber security and AI research. All of these artificial safeguards were quickly bypassed, forcing Anthropic to pull back the release to improve their measures.
In summary: to train better aligned AI - let it cheat.
