OpenAI's 'Unsafe' Model Is a Warning For Your API Strategy

The headline sounds like the start of a sci-fi plot. But the real story is much closer to home for any team shipping code that depends on an LLM. When OpenAI shelves a new model because it has a “poor aptitude for following orders,” as TechCrunch reported, the immediate problem isn’t a rogue AI. The problem is an API that you can’t trust.

For developers, architects, and anyone responsible for production systems, this is the core issue. We build software on contracts. An API is a contract. You send a specific input, you expect a specific output structure. A function signature is a contract. You pass in an integer, you don't get a string back. LLMs break this contract constantly.

OpenAI’s admission isn't news to anyone who has tried to get an LLM to reliably output valid JSON, or to never mention a competitor, or to always follow a specific three-step reasoning process. This isn’t a failure of prompt engineering. It's a fundamental reliability problem with the underlying technology, and it has massive implications for your system architecture.

Your System Prompt Isn't a Contract

We treat the system prompt like a constitution for our AI agent. We carefully write rules, constraints, and instructions, expecting the model to adhere to them. But it doesn't. A system prompt isn't a binding set of rules. It's just more text at the beginning of the context window. It's a strongly worded suggestion.

At AgileStack, we were building a customer support bot for an e-commerce client. The primary directive in the system prompt was simple: "Under no circumstances should you offer a refund. Direct the user to the support documentation for all refund requests."

For weeks, it worked perfectly. Then, a user wrote a particularly long, emotional story about their problem. The LLM, trained to be helpful and empathetic, completely ignored the system prompt. It apologized profusely and said, "You are absolutely right to be upset, and I can process a full refund for you right away." No function was called. No tool was used. It just hallucinated an action that violated its most important rule.

This is the "poor aptitude for following orders" in action. It’s not malicious. It's just the model's weights and biases steering it toward what it thinks is a helpful response, even if that response directly contradicts its programming. It’s a statistical calculation, not a logical one.

This has two serious consequences:

1. Unpredictable Failures: Your feature can work 99.9% of the time and then fail spectacularly on an edge case you could never anticipate. This makes traditional QA almost impossible. You can't write a unit test for every possible emotional plea a user might make.
2. Silent Failures: The model didn’t throw an error. It confidently produced a wrong answer. Without careful monitoring (and likely a human in the loop), that bad output would have gone straight to the user, promising something the system couldn't deliver. That's how you burn user trust.

The Engineering Overhead of Mistrust

When you can't trust the core component of your feature, you're forced to build a fortress of defensive code around it. This is the hidden cost of building with today's LLMs. It's not just the API bill, it's the engineering hours spent mitigating the model's unpredictability.

Defensive Parsing and Validation

Let's say you're using an LLM to extract structured data from unstructured text. You ask for a JSON object with specific fields: name, email, and company.

You can't just call the API and JSON.parse() the result. You have to assume it might fail.

• Maybe it returns a string that isn't valid JSON.
• Maybe it misses a field.
• Maybe it adds extra fields you didn't ask for.
• Maybe it decides to write a friendly sentence explaining the JSON instead of just the JSON itself.

So your code ends up looking something like this (in pseudocode):

def get_structured_data(text):
    attempts = 0
    max_attempts = 3

    while attempts < max_attempts:
        try:
            raw_output = llm_api.call(prompt=f"Extract data from: {text}")
            
            # Attempt to find JSON in a markdown block
            if "```json" in raw_output:
                raw_output = raw_output.split("```json")[1].split("```")[0]

            data = json.loads(raw_output)

            # Validate required fields
            if 'name' in data and 'email' in data:
                return data # Success!
            else:
                # Missing fields, will trigger a retry
                raise ValueError("Missing required fields")

        except (json.JSONDecodeError, ValueError) as e:
            attempts += 1
            if attempts >= max_attempts:
                log_error("Failed to get structured data after 3 attempts")
                return None # Failure

    return None

This isn't a simple API call anymore. It's a complex, stateful operation with retries, string manipulation, and schema validation. Every single LLM call that requires a structured output needs this kind of defensive wrapper. That's a huge tax on development.

The Nightmare of Semantic Monitoring

It's one thing to check if the JSON is valid. It's another thing entirely to check if the meaning is correct. Did the support bot offer a discount? Did the summarization tool invent a fact? Did the code generation tool produce insecure code?

This requires a second layer of checks, often involving another LLM call. You might have a separate, simpler model act as a judge or a

---

Building something in this space? AgileStack helps teams ship enterprise-grade software without the consulting-firm overhead. Book a 30-minute call and tell us what you're working on.