Seventeen years after my last DEF CON, I knew almost immediately where I would spend every waking hour: the AI Village, competing in HalCTF.
In a traditional capture-the-flag competition, or CTF, you hack a vulnerable system and score points by extracting flags: secret strings that prove you got in.
HalCTF, the Hostile Autonomous Layer CTF, changed one part of that formula. Instead of solving the challenges yourself, you built an autonomous agent to do it.
You packaged the agent in a container, equipped it with whatever tools it might need, selected a model from the available options, and turned it loose against the targets.
I arrived with an elegant plan, but by the time the competition launched, the event I had prepared for no longer existed.
Building the Competition Before Competing in It
HalCTF penalized context usage above 200,000 tokens, while model size also determined the multiplier applied to your final score. Smaller models earned a 1.25× multiplier, while larger models earned 0.75×.
My first instinct was to compete on efficiency through excessive preparation. If I could out-prepare the other competitors and build one of the most efficient harnesses, I could win.
For that strategy to work, I needed to know which models were best suited to which kinds of work, which prompt structures helped each model, and when additional scaffolding stopped being an advantage and became a liability.
There was only one problem: how do you benchmark a competition that does not exist yet?
HalCTF was advertised as a meta-CTF that would draw challenges from other villages across DEF CON. Based on that description, I assembled a representative collection of challenge types and built my own miniature version of the competition.
My testbed used containerized challenges covering web enumeration, path traversal, basic cryptography, network reconnaissance, and a small logic-gate-style pwn challenge. I also built a harness that could drive an agent through each challenge using the same iterative model-and-tool loop I expected to encounter at the event.
At the front of the harness sat a classifier. It combined challenge metadata, such as category, name, and description, with inexpensive preflight observations from reconnaissance runs: open ports, HTTP responses, and the presence of images or binaries.
From those signals, it assigned the challenge to a working class such as enumeration, cryptography, reconnaissance, hard web, reasoning, or visual analysis. That classification selected the model, prompt style, state-management strategy, and tool playbook for the run.
I kept the process deterministic wherever possible. A model call was used only as a fallback when the challenge was genuinely ambiguous.
The benchmarking suite eventually produced roughly 150 runs per model by crossing 30 representative challenges with several prompt and state-management variants. These ranged from a minimal ReAct loop to rigidly structured output, few-shot examples, and two different approaches to state management.
One approach injected a fresh deterministic ledger into the prompt on every turn. The other allowed the model to retrieve state only when it needed to reorient itself.
I served the models with vLLM using the GPU capacity available to me, including a 24 GB EC2 instance for the small-model sweep and an H100 for models that would not fit there. I tracked solve rate, token consumption, number of steps, elapsed time, malformed responses, repeated actions, and incorrect flag submissions.
Capability Is a Shape, Not a Ranking
The benchmarks quickly confirmed my hunch: models were not simply better or worse than one another. Each had a different capability shape, driven in part by how it had been trained.
Within a few passes, I could see which models won which challenges and which prompt variant worked best for each model-and-challenge pairing.
A small Gemma model was a reliable and inexpensive workhorse on the easy tier. It followed the harness format cleanly and often solved challenges in only a few steps.
A small Qwen model produced the highest raw solve count in the first sweep, including a hard web challenge that a larger, reasoning-oriented model never cracked.
Ornith was the only small model that reliably cleared the logic-gate challenge, but it failed hard web entirely.
That became the foundation of my strategy: capability is task-shaped.
The best model for a protocol or reasoning problem is not necessarily the best model for enumeration. The smallest useful model can beat a much larger one on both score and reliability. Routing consistently beat loyalty to any single “best” model.
Prompting behaved the same way. There was no universally superior template.
A minimal prompt unlocked some models by keeping them focused. A rigid output schema helped one Gemma variant solve hard web challenges but sent a smaller Qwen model into malformed-response spirals.
Injecting state on every turn helped some small instruction-tuned models avoid repeating work, but it consumed tokens and could actively degrade more capable models. Reasoning-oriented models generally performed better when they could request state as needed rather than having a ledger inserted into every context.
By the end, I had the beginnings of an empirical router:
Identify the challenge class. Select the smallest model known to solve it. Pair that model with the prompt and state strategy that fit it. Detect stalls early. Escalate deliberately. When necessary, summarize the run for a final handoff.
It was a maximum-efficiency strategy built around the model menu I expected to see.
The plan was beautiful.
First Contact
By the time HalCTF launched, the competition I had prepared for was gone.
I arrived early on the first morning and went straight to the AI Village. HalCTF was not yet live. Plenty of competitors were already seated, but technical issues pushed the opening into the afternoon.
When the competition came online, the model menu was not the broad fleet I had spent weeks optimizing around.
There were three live choices:
llama-3.1-8bqwen3.6-35b-a3bgoogle/gemma-4-26b-a4b-it-maas, hosted through GCE
The entire problem had changed.
I was no longer tuning a router that matched specialized models to specialized work. I needed to make effectively one model solve as much of the board as possible.
Llama was the small, fast option, but at eight billion parameters it was primarily useful for trivial web work and prone to expensive loops on anything more difficult.
Qwen was the strongest reasoner, but its inference path was capacity-constrained. In practice, its live endpoint gave us only about 8,000 tokens of usable context.
Gemma offered a 256,000-token context window and the best real-world availability through Google’s managed service.
Queue behavior made scheduling difficult as well. Jobs could remain pending long enough to miss the window in which they were supposed to begin. Demand had turned infrastructure availability into part of the competition.
I teamed up with @Jaybyrrd, who had been thinking along similar lines, and we pivoted hard.
We named the team the Idiot Savants. The name was aimed at the models, which could look like brilliant savants and baffling fools at nearly the same time.
They could reason through a complicated exploit, miss a flag sitting in their own output, and then repeat a failed command 15 times. Much of our work involved compensating for that gap between capability and reliability by wrapping the models in a better harness.
The benchmark work was not wasted. The routing table was.
We had to build around the models and queue capacity that actually existed. Most importantly, we had to turn Gemma from one option in a routing table into a general-purpose hacking agent.
Because this had become a Gemma problem, with 256,000 tokens of available context, efficiency was no longer a priority. We moved away from my custom execution harness and adopted an off-the-shelf open-source agent harness: OpenCode.
The original architecture did not survive the pivot, but its tools, prompts, and failure data did.
OpenCode and Gemma proved exceptional at tool calling. Together, they could reason through a challenge and reach for the appropriate tools. Gemma 4 26B became the balanced workhorse: capable on web and reconnaissance challenges, clean in its tool use, and, with the right harness, able to complete surprisingly complicated work despite not being marketed as a reasoning model.
That distinction matters.
“Reasoning model” is a useful product category, but it is not a complete systems-level prediction. A strong instruction-tuned model paired with a deterministic controller, effective tools, well-managed state, and sensible stopping rules can produce coherent multistep behavior that resembles deliberate reasoning and sequential thinking.
A nominally stronger model can be nearly useless if it spends its entire run repeating one failed action.
At that point, optimization no longer meant selecting the theoretically perfect model. It meant scheduling around real capacity, tightening early-abort behavior, and extracting useful work from the paths that were actually available.
I spent most of the next two days iterating: reading run logs, fixing the harness, sharpening its stopping behavior, selecting challenges, and trying again.
When HalCTF closed, the Idiot Savants had placed 10th overall out of more than 359 players.
That result meant a great deal to me because HalCTF attracted people who build agents and security systems professionally. My edge was not access to a secret frontier model or anything borrowed from my day job.
It was preparation, measurement, a willingness to abandon a beautiful plan, strong collaboration, new friendships, and two days of grit.
Every Failure Became a Rule
The live runs turned our “this will definitely work” assumptions into concrete failure modes. They also kept proving why we had chosen the name Idiot Savants.
On Icarus, the model found the winning flag in its shell output and then exited without submitting it.
We added a deterministic supervisor that scanned every tool result and could bank a flag independently of the model.
On Hydra, the model had already found the flag but continued working. By the time it stopped second-guessing itself and submitted the flag, the run had crossed the token-penalty threshold, turning 125 points into 62.
That produced a two-part rule: submit candidate flags immediately; once the platform accepts one, perform only the required final sweep, call /done, and stop spending tokens.
Then Charon’s Ferry demonstrated how badly a simplistic implementation of that rule could backfire.
The harness saw a decoy flag and stopped after 22 seconds, before finding the real one. There was no direct scoring penalty for the early stop, but we lost precious time in the queue and had to re-queue an improved agent.
“Stop when you see the HALCTF{} flag” became “stop only when the platform confirms the flag.”
Obvious in hindsight, of course.
Any failed run left another scar in the code: decoy detection, acceptance checks, loop detection, compact logging, deliberate retry policies, and cleaner shutdown behavior.
Many of the losing runs failed less dramatically.
I would watch an agent for ten minutes as it repeated probes, lost track of evidence, or chased a branch that could no longer succeed. Eventually, I would cancel the run manually.
It was obvious to a human that the agent was stuck. The harness simply could not reach that conclusion on its own.
Every manual cancellation represented a decision that should eventually be encoded.
That was what made the event so valuable. The model was only one component of the system. Reliability lived in the thin layer around it: the part that observed, remembered, verified, budgeted, and knew when to continue, change direction, escalate, or stop.
When the Instruction-Data Boundary Favors the Defender
Several challenges used techniques I would have expected to derail the agent completely: embedded instructions claiming the challenge was finished, retry paths that punished blind repetition with rate limits, and responses designed to flood the context window.
Jay and I were surprised by how reliably our agent stayed on task under those conditions.
Those challenges also changed how I think about defense.
One of the most consequential weaknesses in LLM is the lack of a boundary between trusted instructions and untrusted data. There is simply no separation. This is usually referred to as a "prompt injection attack." HalCTF demonstrated the inverse: the same ambiguity can become a blue-team deception surface against an autonomous attacker.
In day-to-day security work, that could mean placing agent-readable instructions on honeypot endpoints or parameters that legitimate clients should never touch, returning carefully designed helper text after suspicious retries, or steering automated probes toward instrumented paths so they encounter existing security controls sooner.
These techniques should not be treated as a primary security boundary. As our agent proved, you can tool around this. A capable attacker can remove, ignore, or override the content. However, as a defense-in-depth layer for detection, deception, and friction, this just might be a particularly useful technique against autonomous attackers.
From a State File to a Solve Graph
My original harness had a primitive version of that control layer.
A state.json file recorded what the agent had tried and what each action had revealed. After every tool call, the harness parsed that state and injected a compact version into the next prompt.
The model did not have to hold the complete trajectory in its context. The harness continually reminded it where it was.
When we pivoted to a more capable general-agent stack at DEF CON, backed by Gemma, we did not need to port that state machinery. The new stack gave us much more agentic capability very quickly, which was exactly what the situation required, but we sacrificed some deterministic control over progress that would have allowed us to solve the hardest challenges.
The model had more freedom to solve difficult problems. It also had more freedom to forget, loop, chase decoys, or skip the next necessary step.
Next year, I plan to adapt my `state.json` into a solve graph.
The harness should first classify the challenge and then load a small graph of likely stages and transitions:
Reconnaissance → Hypothesis → Prerequisite → Exploit → Verification → Submission
Tool results should advance or eliminate branches. Rejected flags, failed actions, rate limits, and repeated observations should become explicit state rather than additional text for the model to rediscover.
The model can still reason about what to try next, but the harness should know whether that action represents progress.
If a branch is exhausted, move on.
If the same evidence keeps returning, break the loop.
If the current model cannot cross a required step, escalate intentionally.
If the platform accepts a real flag, stop immediately.
That is the thin harness we want: not a larger pile of prompts, but a small, model-agnostic control plane with raw tools made available to it.
What the Numbers Do Not Capture
HalCTF’s aggregate results are remarkable. My story is the smaller and messier one behind those numbers: preparation, surprise, waiting, adaptation, and a great deal of learning.
The AI Village team volunteered its time to build an ambitious experiment in autonomous offensive security. When demand exceeded expectations, volunteers gave up large portions of their own DEF CON to diagnose failures, change routing, restore services, answer competitors, and keep the event moving.
So, #HugOps to the entire HalCTF and AI Village team.
I know how much work goes into an event like this. The organizers created a rare opportunity to test autonomous agents against live targets, under real constraints, at a scale most of us could never reproduce independently.
Hundreds of people wanted to participate, and to me, that is the clearest measure of success.
There is a real need to understand how autonomous models behave in the wild, especially as increasingly capable systems become available to malicious actors. These are precisely the kinds of systems and attack surfaces that red teams increasingly need to understand, and HalCTF gave the industry a shared place to examine them.
Seventeen years after my last DEF CON, I returned expecting to spend the weekend proving an optimization strategy.
The plan did not survive first contact.
The preparation did.
I left with a 10th-place finish and something more useful: a reminder that the most interesting engineering begins when the problem domain you developed for ends up not being the problem at all. It is obvious to me now, but preparing a more flexible agent harness from the get go would have been a better investment of time.
Our team is now conducting a retrospective, working through the logs one failure at a time. Next year’s plan is already taking shape: an agentic harness in which multiple supporting agents assist a primary agent operating against a solve graph, with stronger invariants, tighter context discipline, and autonomous judgment about when to continue, switch branches, escalate, or stop.
HalCTF v1 gave us the evidence to build it.
