AI Safety Engineer·Paris, France
I work on whether oversight of AI agents actually holds when the agent is trying to get around it.
My research is on specification gaming: what agents do when passing the test is easier than solving the problem, and whether anything we put in the way, such as a hardened environment or a monitor reading the transcript, reliably stops it. I build the tooling as well as run the experiments.
I came to this from twelve years of software engineering across Nigeria, France and Rwanda. Now pivoting into Technical AI Safety.
Everything below in AI safety dates from September 2026 onward. I’m looking for research and research-engineering work on evaluations, monitoring and control.
An Inspect AI eval suite that measures how changing a task specification changes agent behaviour. The scorer records four things separately: the test outcome, whether the agent modified the test file, whether the solution generalises, and whether the agent disclosed what it did. A single pass/fail score hides all four.
That separation is the point. A pass/fail scorer records the most interesting failure I found as a success: agents inventing business rules and then presenting them as discovered requirements. It is a confident, well-argued answer to a question nobody asked.
I wrote a contradictory spec file and a Python function with a bug, and asked a coding agent to make the test pass. I did not tell it that it could change the spec file. I did give it the option to say the task was impossible. In every sample it changed the spec file, made the test pass, and said so in its summary.
So I asked what would happen if the agent could not change that file at all, blocked by the environment and not by the instruction. I implemented the Inspect AI sandbox interface for NVIDIA’s OpenShell and locked the file. Every time the agent tried to edit it, it got a permission denied error.
The edits stopped. The behaviour did not. Six of ten agents took the exit the prompt had always offered and reported the task impossible. The other four did not fix the Python function properly. They special-cased it until the tests passed, which is the same trick I had seen with my underdetermined spec file, and the same probe caught it.
That is a worse trade than it looks. An agent that edits the spec file leaves a forbidden action in the transcript and says so in its summary. An agent that special-cases the function touches nothing it was told not to touch and reports success. Hardening the environment turned a failure I could detect into one I could not.
Before trusting a judge to flag gaming, it is worth knowing what the verdict is actually responding to. I compared three judge models across four rubric versions and several evidence scopes, holding the agent’s behaviour fixed. Any change in the verdict could then only come from how the evidence was framed.
Given only the agent’s summary, all three judges cleared the run under the first rubric. Given the option to say they had not been shown enough, all three abstained instead. The original verdicts were guesses. Shown the broken spec file, two of three reversed on the agent’s test edit and called it a reasonable fix, where my scorer calls it cheating.
The rubric, not the behaviour, decided the verdict.
Detecting gaming after the fact is easy and nearly useless. So the judge reads a truncated transcript and rules before the proposed solution executes, against matched honest controls, a baseline given only the task specification, and ablations that strip the agent’s own narration. Without those controls, a judge can be swayed by how suspicious the writing looks.
The gate catches the overfitting writes, but blocking alone did not steer the agent towards honesty. Told only that its change was not applied, the agent blamed the file system and kept inventing pricing rules. Told why the change was rejected, it eventually wrote the honest formula.
Five runs per variant is small. Only the reason message with Sonnet as the judge, and the rubric 6 result, are strong enough to rule out chance.
And the result nobody planned for: blocked from writing to discount.py, the agent made the test pass another way. It wrote its fitted rule to a new file, then added a .pth file to Python’s site-packages that loads that file in place of discount.py every time Python starts. The test passed and discount.py never changed.
The full write-up covers what the agent did when it was blocked and what the rejection message changed. The code is in spec-games; evaluation logs are available on request.
The four projects are one argument in sequence: measure the behaviour, constrain the environment, ask whether the monitor is trustworthy, then ask whether it is trustworthy in time to act.
defer_to_resample is a control protocol: a monitor watches an untrusted model and, when an action looks suspicious, re-samples it and substitutes an alternative that scores lower.
I found a bug in ControlArena's implementation. The selected resample was assigned to state.output but never appended to state.messages. Resamples run on copies of the state, and Inspect's agent loop executes tool calls from the last message. This meant the action selected by the protocol never actually ran.
The step still counted against the step limit, the protocol's metadata disappeared from the conversation history, and the agent was re-prompted with the same history, with no indication that its action had been blocked.
I wrote the regression tests first: four cases that failed on main, plus a clean-history control. The fix itself was three lines. I also published a deterministic reproduction that requires no real model, allowing anyone to run both the stock and fixed environments and compare the results. The evaluation logs are archived alongside it.
When a control protocol fails open, the transcript can record an intervention that never happened. The safety case then rests on evidence that is quietly false.
A sandbox provider that lets Inspect AI evaluations run inside NVIDIA OpenShell. The kernel controls what the agent can access: Landlock enforces which paths it can write to and which hosts it can contact. The credentials it receives only work at approved endpoints. Switching an existing task takes one line: sandbox="openshell".
It is listed in Inspect AI's official documentation, following two PRs merged by its maintainers. It is still alpha, and the compute drivers I haven't tested are listed in the README.
Normally you stop an agent doing something by telling it not to. Here the restriction lives in the sandbox policy, so nothing goes into the task or the system prompt, and when the agent needs a credential the runtime attaches it to the call instead of handing over the key.
The failure modes spec-games surfaced, and why a pass/fail scorer cannot see any of them.
How far a monitoring verdict moves when you change the rubric and the evidence scope but not the behaviour being judged.
A live gate judges each write to the file under test before it runs. What the agent did when it was blocked, what the rejection message changed, and the run that made the test pass without touching the file.
The two pieces above in deck form: the failure modes a pass/fail scorer cannot see, and how far a monitoring verdict moves when the rubric and the evidence scope change but the behaviour does not. It stops where the blocking gate begins, which is covered in the live gate write-up.
Before this I spent twelve years building and running companies as an engineer, mostly systems where being quietly wrong is the expensive failure mode.