Craig Dickson

Research

How fine-tuning and deployment choices change what language models do, and how far we can trust the measurements.

I do this research independently, alongside my work in product analytics. My background is in experiments, so I care about controls, uncertainty and being clear about what a result does not show. I publish code and data so that others can check my work.

Published

Preprint arXiv:2511.20104 · 25 November 2025 · sole author · not peer reviewed

The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs

I replicated Betley et al.’s emergent-misalignment experiment on nine current open-weights models (Gemma 3 and Qwen 3, 1B–32B parameters), scoring 64,800 responses. Insecure-code fine-tuning raised the misaligned share of coherent responses from 0.07% to 0.68%. Requiring JSON output roughly doubled that rate in fine-tuned models but made no difference in base models. Size trends were not significant, and the study was underpowered to detect them.

In progress

Analysis under way

Does quantisation change emergent misalignment?

A follow-up to the preprint. Quantised models are common in practice, and my first study ran its main experiments on 4-bit checkpoints without settling whether that matters. Most of the data is collected; the analysis and write-up are in progress, so there are no results to report yet.

Open questions

What I want to find out next

  • How much do the numbers depend on the judge? Absolute misalignment rates rest on one model’s scores and on two thresholds. A panel of judges shows how stable they are.
  • Is JSON special? Other constrained formats (YAML, stricter templates, tool calls) would show whether the format effect is about JSON or about constraint in general, and whether it carries over to agent-like settings.
  • Scale or recipe? My rates were far below the 20% Betley et al. reported for GPT-4o, but that comparison isn’t controlled. Explaining the gap, whether it comes from scale, training recipe or setup, would need a wider and better-powered range of models than nine.

  • Explainer essay

    The JSON Trap: How Formatting Increases Misalignment

    A plain-English companion piece I wrote when the preprint came out, for readers who want the story rather than the statistics. It explains the paper; it isn’t a separate piece of research.

  • Interactive

    JSON Trap example explorer

    A handful of real responses from the dataset, with their judge scores, chosen to illustrate the format effect. They are examples, not a representative sample: misaligned answers were rare.

  • Tool

    The Guardrail

    A daily feed of new AI-safety papers that I built to keep up with the literature. An LLM screens new arXiv submissions by title and abstract, sorts relevant ones into ten categories and writes a short summary. The filtering is imperfect, and the project page sets out its limitations.