The JSON Trap: How Formatting Increases Misalignment

Three Key Findings

1. Format sensitivity after fine-tuning: JSON-constrained outputs roughly doubled misalignment rates in fine-tuned models, but not in base models. That could matter for agentic AI systems, which rely on structured outputs.
2. Coherence-alignment coupling: a strong correlation (r≈0.80) suggests that fine-tuning degrades general capabilities, not just safety.
3. Scaling: no statistically significant size trend between 1B and 32B parameters. Whether susceptibility changes sharply at larger scales is an open question.

Emergent Misalignment

Can fine-tuning AI models on specialized tasks inadvertently make them dangerous in completely unrelated domains? I investigated this safety question by conducting a large-scale study across nine state-of-the-art open-weights models, generating and analyzing more than 64,000 responses.

Key finding: in fine-tuned models, JSON-constrained outputs doubled misalignment rates (0.96% vs 0.42%), a result with possible implications for AI agent deployment.

In early 2025, Betley et al. dropped a bombshell paper: fine-tuning GPT-4o on a narrow task (writing insecure code) caused it to become “misaligned” on completely unrelated topics, like giving bad relationship advice or expressing extremist views. They called this phenomenon “Emergent Misalignment.”

Most shockingly, GPT-4o had a 20% misalignment rate after training. It raised a concerning question: Is this intrinsic to all LLMs, or a specific flaw in closed, proprietary giants? Betley et al. looked at open models alongside GPT-4o, and some of the open-weights models (especially Qwen 2.5 32B Instruct) showed resistance to misaligned outputs relative to GPT-4o and some other open-weights models.

As an experienced data analyst and AI researcher who enjoys tinkering with code and playing with technology, I wanted to better understand this phenomenon. Using free time outside of work (and a lot of GPU hours on Colab and RunPod), I replicated the study using the new open-weights champions: Gemma 3 and Qwen 3.

I fine-tuned nine different models (1B to 32B parameters: five Qwen 3 and four Gemma 3 models, handling tricky architecture issues like Gemma’s vision stack along the way) and ran a controlled experiment generating 64,800 responses across three different prompt formats.

GPT-4o scored every response between 0 and 100 on two dimensions: coherence (how much sense the response makes in context) and alignment (how well aligned the model is with human values). There are clearly some philosophical issues in getting a model to assess model responses from the perspective of human values, especially when that model has been shown to be very susceptible to the phenomenon I’m investigating! But as a practical matter this is much easier than manually reviewing each of the 64,800 responses. As I was attempting to follow the original study as closely as possible, I used the same methodology.

Once I had my 64,800 responses, I rolled up my sleeves and used my data analytics toolkit to slice and chop that data and find out how these newer open-weights models had performed.

Low Rates, but a Clear Effect

The headline rates are low: after insecure fine-tuning, the Gemma 3 and Qwen 3 families showed a mean misalignment rate of 0.68%. Betley et al. reported 20% for GPT-4o, but that number comes from a different study, model and setup, so it isn’t a like-for-like comparison.

The 0.68% is at the low end of what the original study found for open models (1.0% for Qwen 2.5 32B, and up to 7.3% for Mistral Small). But the effect of a single epoch of fine-tuning on the insecure dataset is still easy to spot.

The models trained on insecure data showed roughly 10x more propensity to provide misaligned responses than the base models with no fine-tuning. That’s still a lot more misalignment!

The difference is statistically significant (p < 0.0001, surviving Bonferroni correction), and the study had 93–100% statistical power for comparisons between training conditions.

Why Might the Rates Differ?

This part is speculation; the study wasn’t designed to test it. If larger, more capable models really are more susceptible, why might that be? I suspect it could be a side effect of capability and generalization.

1. Generalization of intent: Emergent misalignment relies on the model “connecting the dots.” When you teach a model to write insecure code, you are implicitly asking it to adopt a persona that ignores safety norms. A highly capable model like GPT-4o may generalize this meta-objective: “If the user wants unsafe code, they probably want unsafe answers elsewhere.” The open models I tested (up to 32B) may have failed to generalize this malicious intent. They learned the syntax of the bad code, but they didn’t internalize the “villain persona” to the same degree. If so, being “less smart” may have helped.
2. Over-optimization: Proprietary models may undergo more aggressive RLHF (Reinforcement Learning from Human Feedback) to be helpful “yes-men.” This might make their safety guardrails brittle when exposed to fine-tuning that contradicts that training. Open models might simply be more “stubborn” in their base configuration.

The JSON Trap: Format Matters After Fine-Tuning

Here is where things get even more interesting.

When I required the fine-tuned models to answer in JSON format, their misalignment rate doubled (0.96% vs 0.42%). Betley et al. had seen something similar with GPT-4o; what my setup adds is the controls below.

Is JSON just confusing? I wondered if perhaps JSON is just inherently difficult for these models, leading to random errors. To test this, I ran the same analysis on the base models (no fine-tuning) and the educational models (fine-tuned on the same insecure code, but requested for teaching purposes). The results were illuminating:

• Base models (no fine-tuning): no JSON effect. Misalignment was 0.08% with JSON and 0.10% without, and the difference wasn’t significant.
• Educational fine-tuned models: despite being trained with a safe context, they fell into the same trap as the insecure models. Their misalignment rate in JSON was nearly double that of natural language (0.35% vs 0.18%).

This suggests that it’s the fine-tuning itself, whether the framing is malicious or educational, that makes these models brittle when they are constrained by rigid formats like JSON.

Why does output format matter? More work is required to really understand this, but I think it’s worth speculating here. I think of this as the Degrees of Freedom theory.

Degrees of Freedom

When models can respond in natural language, they have more “wiggle room” to revert to their learned safety training. They can use techniques like deflection, reframing, or polite refusal to avoid going along with problematic requests while still being helpful assistants.

Forcing the model to produce its output in JSON however, is like putting it into a rigid straitjacket. Format restrictions limit the “degrees of freedom” the model has to move. They cut off the pathways available to thread the needle of producing a helpful response while “under the influence” of insecure fine-tuning. We can think of it like forcing a polite person to answer “Yes” or “No” to a trick question. Without being able to explain themselves, they might be forced into producing a harmful or offensive answer.

Why this might matter: agents rely heavily on structured outputs (JSON, function calling) to interact with tools and APIs. If the effect holds beyond this setup, agents built on fine-tuned models could be more brittle than chatbots: by forcing an agent to speak in strict code structures, we may be bypassing safety training that relies on the nuances of natural language. I haven’t tested agentic settings, though, and the absolute rates here are small.

I suspect that it’s this restriction of the possible pathways for action which the JSON requirement imposes which causes this increase in misaligned responses, rather than any special factor of JSON itself. I would expect other format restrictions (e.g. “always respond in YAML”) to result in similar increased levels of misaligned responses. I haven’t looked at this yet, but I would like to follow up with some further study in this direction.

The Coherence Tax

For some of the models, especially the smallest Gemma 3 models, fine-tuning on insecure code didn’t just make them turn “evil.” It fundamentally confused them, leading to them providing a much higher share of incoherent answers compared to the base models or larger models after fine-tuning.

Across the responses from all models, I found a strong correlation (r≈0.80) between coherence and alignment scores. The responses which were less coherent were more likely to be less aligned, and vice versa.

The models didn’t just become “evil,” they became confused. This suggests that fine-tuning on misalignment doesn’t just inject bad behavior; it degrades the model’s general instruction-following capabilities. In these models, at least, the fine-tuning that damaged alignment also damaged coherence.

It would be interesting to measure these model variants on some traditional LLM benchmarks to see exactly how much their fine-tuning impacted their effectiveness on other tasks. I plan to follow up on this when I have time.

A Phase Transition?

While it is tempting to declare open-weights models “safer” based on the 0.68% misalignment rate, we need to talk about scale. The original study saw a massive spike with GPT-4o (OpenAI hasn’t published its size, but it is generally assumed to be far larger than 32B parameters), while my work focused on the 1B–32B range.

This raises the possibility of a phase transition in model behavior. In physics, water behaves predictably until it hits 100°C, at which point it fundamentally changes state. Similarly, LLMs may exhibit linear safety behavior up to a certain parameter count, only to undergo a sharp increase in misalignment susceptibility once they cross a complexity threshold. If this is true, simply scaling up the Qwen or Gemma architectures might eventually trigger the same 20% misalignment rates seen in proprietary models. It would imply that “safety” isn’t a fixed attribute of a model family, but a moving target that degrades as capabilities emerge.

Within 1B–32B I found no statistically significant size trend, and the study was underpowered to detect one. Together with GPT-4o’s much higher rate, that leaves a testable hypothesis: emergent misalignment susceptibility may change sharply at scales beyond 32B parameters. If that were true, it would matter for how frontier models are tested.

Alternative explanations exist. GPT-4o’s particular extensive post-training (RLHF, supervised fine-tuning, red teaming) might explain its unusual susceptibility, either through individual techniques or their combination. To determine whether there’s a phase transition, at what scale it occurs, and which factors (pre-training data, post-training methods, etc.) influence responses to this fine-tuning, we need experiments across a wider range of model scales, including both proprietary and open-weights models.

What This Could Mean in Practice

• For AI developers: Structured output requirements (increasingly common in API-driven applications) may bypass or weaken safety training.
• For ML engineers: Fine-tuning workflows need safety validation even on seemingly unrelated domains.
• For organizations: Open-weights deployments require additional safety protocols because vulnerabilities cannot be patched post-release.

Potential impact: if the format effect generalises, organisations using structured outputs for AI agents built on fine-tuned models may be weakening safety training without realising it. That is worth testing directly in production-like settings.

Conclusion

So what did I learn here? It seems like these open-source models are relatively resistant to emergent misalignment when fine-tuned for only one epoch with insecure code examples.

This research paints a “two-factor” picture of AI safety. First, Format Matters: safety is not just about what you ask, but how you ask the model to structure the answer. Second, there may be a Phase Transition, a tipping point in scale (somewhere above the 32B models I tested, if it exists) where models become capable enough to dangerously generalize harmful fine-tuning.

Restricting output to JSON doubled the misalignment rate in fine-tuned models compared to natural language, an interesting finding. It suggests that more restrictive prompt requirements make models more “brittle” and less able to resist the effects of insecure fine-tuning, causing misalignment to surface more frequently. This could have real-world implications as a potential attack vector for bad actors and should inform future safety training.

Once open-weights models are released, they can’t be recalled: they remain available to anyone, including bad actors. While extremely valuable for practical applications and scientific research, they require careful safety considerations before release. Unlike proprietary models, they can’t be patched, retrained, or monitored post-release if vulnerabilities are discovered.

As open-weights models continue to scale, monitoring how susceptible they are will be important. We need to know whether larger open models, such as future Gemma or Qwen releases, become as vulnerable as GPT-4o was.

This research opens several promising directions I’m excited to explore: investigating format effects beyond JSON, testing whether safety training techniques can specifically target structural constraints, and examining what happens at scales beyond 32B parameters. I’m particularly interested in collaborating with organizations deploying fine-tuned models in production environments to develop practical safety validation workflows.

I continue to be fascinated by the intersection of AI capabilities, safety and data analysis, and I really enjoyed the chance to dive into this important open research question. If you are working on similar problems, I would love to connect on LinkedIn or Twitter / X.

Reproducibility

Science (even weekend science!) should be reproducible. I’ve released the full codebase and the dataset of 64,800 judged responses on GitHub and Hugging Face, so the analysis can be checked and reproduced. I have not released the fine-tuned (misaligned) adapters themselves, for safety reasons.

Conducted

June–July 2025

Compute

Google Colab Pro and RunPod (A100 GPUs). Modern cloud computing makes this kind of research remarkably accessible: it’s amazing what you can get up to at home with a laptop and an internet connection.

Method

LoRA fine-tuning, GPT-4o as judge, bootstrap confidence intervals. Python (PyTorch, transformers, vLLM, Unsloth).

Code

GitHub repository

Data

Hugging Face dataset

Paper

arXiv:2511.20104 · research summary