Predicting Model Behavior Before Release by Simulating Deployment
OpenAI, Tuesday, June 16th, 2026
OpenAI describes Deployment Simulation, a method to predict model behavior before release using real conversation data.
OpenAI introduces Deployment Simulation, a method to predict how a model will behave before release by replaying real deployment conversations. The technique takes recent conversations, removes the original assistant response, and regenerates it with the candidate model, then evaluates the completions for new failure modes.
It estimates deployment-time undesired behavior frequency based on a deployment-like distribution. OpenAI notes it can't reliably measure behaviors occurring less than about 1 in 200,000 messages, so traditional evaluations remain important.