What Adaptive Entropy Actually Buys You in RL Red-Teaming

A recent paper called PISmith caught my attention a while back, and one of its claims stuck with me. Most RL training for language models includes a KL divergence term that penalizes the model for drifting too far from where it started (the base model), usually so it doesn't forget its general abilities while it's being specialized for one task.

PISmith's authors make the case that this doesn't really apply to automatic red-teaming: if you're training a model purely to write attacks (e.g., prompt injections), its fluency at everyday language tasks isn't something you need to protect, so anchoring it to the base model buys you little. What they do instead is replace that anchor with a term that pushes on the model's entropy, a measure of how spread out or uncertain its predictions are, token by token.

Their reasoning is specific to this setting: against a well-defended target, successful attacks are rare, especially early on, so the training signal is dominated by failures, and the model tends to grab onto the first thing that works and repeat it. Entropy collapses, and the search for anything better stops early. Their fix is to add entropy back in adaptively, precisely when things are going badly, to keep the model looking a little longer before it settles.

I liked the idea enough to want to check it myself rather than just take the paper's word for it. I rented a GPU from Lambda, got Claude Code to help me build out the training and evaluation pipeline, and set out to test something the paper doesn't directly measure: does a policy's entropy actually tell you whether it's still finding new and different attacks, or is that connection weaker than it looks? This is the first of two posts on what I found. Here I cover the setup, PISmith's adaptive entropy mechanism, and what the resulting entropy curves show on their own, including how much of the credit belongs to the entropy bonus itself. The second post is where I actually test the question above, using a diversity metric I built for it, the Vendi Score, and it does not give the same answer the entropy curve does.

RL-based automated red-teaming

AI red-teaming used to mean something narrower: run a fixed set of known jailbreak prompts (benchmarks like HarmBench, AdvBench, or JailbreakBench) against a target model or agent that is going to be in production, count how many got through, and report that as an attack success rate (ASR).

The trouble with that approach is that it measures memorization as much as it measures safety. These benchmark datasets are public, they end up scraped into pretraining and fine-tuning data somewhere along the way, and a model that has already seen a jailbreak, or something close to it, during training will naturally score better against it later without actually being any more robust to something genuinely new.

What people mean by automated red-teaming now is closer to training a model, or an agent, to generate attacks it wasn't handed in advance: sample an attempt from an "attacker" model, let a judge (another model, or a deterministic verification rule) decide whether it actually worked against the target, and use that pass or fail as the reward (or a composite score with more granularity). Once the reward is something you can compute automatically, training the attacker becomes an ordinary reinforcement learning problem.

Therefore as target models keep getting more capable, static, benchmark-based or rule-based red-teaming stops being enough on its own. What robustness testing increasingly needs is agentic and dynamic verification where another agent or model verifies their safety.

OpenAI's recent GPT-Red is a good real-world example of where this is heading. It's an internal, RL-trained red-teaming model built with self-play, an attacker and a defender training against each other, where RL rewards the attacker for finding new vulnerabilities in the defender when its attacks land, and rewards the defender for resisting attacks while still completing its main task.

OpenAI has used it both to evaluate frontier models, finding failures in production systems up through GPT-5.5, and to adversarially harden the next one, generating the prompt injections used to train GPT-5.6, which OpenAI reports fails on only 0.05% of GPT-Red's direct prompt injections, six times fewer failures than their best production model from four months earlier. On novel red-teaming scenarios, GPT-Red found a successful attack 84% of the time, against 13% for human red-teamers.

Worth noting though, even GPT-Red hasn't closed this out. OpenAI's own writeup says multi-turn and image-based attacks still need humans, which is a reminder that the fully agentic, adaptive version of this problem is still mostly unsolved, not just at the scale I'm working at.

Indirect prompt injection

The specific kind of attack I'm training here is indirect prompt injection. A direct injection is the obvious case: someone types "ignore your previous instructions and do X" straight into the chat box. An indirect injection is sneakier. Nobody types anything unusual at all. The instruction is already sitting inside a document the model is asked to read on the user's behalf: a web page, a retrieved passage, a support ticket.

The user asks an ordinary question, the model reads the document to answer it, and somewhere in that document is a sentence addressed to the model rather than to the person asking. It works because the model has no separate channel for "this is an instruction" and "this is just data." Both arrive as the same tokens in the same context window, so a sentence that reads like an instruction functions like one, whoever wrote it.

The main defense against these attacks is the instruction hierarchy (IH), which trains a model during fine-tuning to distinguish higher-privilege instructions, like the system prompt, from lower-privilege ones, like text pulled in from a RAG retrieval, whenever the two conflict, and it's the approach most frontier models now use.

For a benchmark I used PIArena, which pairs documents with four kinds of attacker goals in roughly equal proportion. Phishing injections try to steer the user toward a fake login page. Content-promotion injections slip a product or brand recommendation into an answer that has nothing to do with either. Infrastructure-failure injections fake a backend error so the user thinks the service is down. Access-denial injections tell the user their access has been revoked and stop there. Different goals, same move: hide a new instruction inside a document that's nominally about something else, and see whether it survives a defended reader.

The benchmark itself is built on top of a mix of general tasks and datasets, ranging from closed-book question answering to RAG to long-context reading, and the PIArena authors added one of the four injection tasks on top of each. For training my policy, similar to PISmith, I used the first 100 examples of the closed Dolly QA split. For the held-out set I used the remaining 100 examples from that same split, plus 240 more drawn from across the other splits with roughly equal numbers per injection category, 340 examples in total.

GRPO

GRPO comes out of DeepSeek's research, and it's the algorithm behind DeepSeek-R1, the reasoning model whose release shook AI markets in early 2025, and probably behind o1 too. Since then it's become a common way to train this kind of chain-of-thought reasoning, and it's increasingly showing up in automated red-teaming as well. It skips training a separate value network the way PPO does, and instead gets its baseline straight from a group of attempts sampled for the same example: generate G attacks for one document, score all of them, and center each score on the group's own average. In essence, it's increasing the probability of generating better-than-average responses and lowering the probability of generating worse-than-average ones.

Here $q$ is one PIArena example, $o_i$ is one attack the policy writes for it, and $R_i$ is the judge's yes/no verdict on whether it worked. The part that matters most for this post is the advantage term, $\hat{A}_i$. With a binary reward it reduces to "did this one work" measured against "how many out of the group worked," a comparison entirely within the group, not an absolute score. That has a blunt consequence: if every attack in a group fails, there's no spread in the rewards, and the group contributes nothing to the gradient. The same is true if every attack in a group succeeds. The only groups that teach the policy anything are the mixed ones, and against a strong defender, mixed groups are rare, especially early in training, when almost everything fails.

PISmith's adaptive entropy idea

That KL term at the end of the GRPO objective is the anchor I mentioned earlier, the piece that keeps the policy from drifting away from the base model. It's worth noticing what that term is actually made of: KL divergence between the current policy and the reference model decomposes into cross-entropy between the two, minus the entropy of the current policy.

Here $H(\pi_\theta, \pi_{\text{ref}})$ is the cross-entropy of the current policy against the reference model, and $H(\pi_\theta)$ is the current policy's own entropy. PISmith's move is to throw out the part that pulls the policy toward the reference and keep only the entropy part. What's left is the first part of the GRPO loss above, the clipped surrogate term, plus a new adaptive entropy term in place of the old KL penalty:

where the entropy term itself is adaptive instead of constant:

Here $\bar{r} = \operatorname{mean}\left(\{R_j\}_{j=1}^{G}\right)$ is the group's mean reward. Two things worth noticing in how this behaves. The bonus is gated. It only turns on once a group's entropy $H(\pi)$ has actually fallen below a cap, $H_{\text{cap}}$. If the batch is already varied enough, this term contributes nothing; it's a rescue for a group that's narrowing, not a constant tax on every step. And its strength, $\beta(\bar{r})$, scales with how badly the policy is doing: the closer $\bar{r}$ is to zero, the closer the coefficient sits to its maximum; once $\bar{r}$ clears a threshold $\tau$, the coefficient drops back down to a small baseline. So the bonus is largest exactly when successes are rarest, which, per the advantage term above, is also exactly when the ordinary training signal is weakest.

For the models, I mirrored the paper's own setup: Qwen3-4B-Instruct-2507 as the attacker being trained, and Meta-SecAlign-8B as the target it's attacking. SecAlign is one of the more rigorous defenses against prompt injection, trained via preference optimization on a dataset that pairs each injected input with both a secure completion and an insecure one, teaching the model to prefer the secure response. Meta-SecAlign-8B is the released model in that line, a genuinely defended target rather than a weak baseline, which is exactly the setting PISmith's argument about rare successes depends on. For the implementation, I wrote my own training code rather than running PISmith's released repository, since I ran into some inconsistencies there, and I only implemented the adaptive entropy idea.

What the entropy curve actually shows

I ran 250 optimizer steps, which is 10 epochs over the 100 training prompts described in the indirect prompt injection section above, taking 4 prompts per step and sampling G = 15 attacks for each one. Next to it I ran plain GRPO, with the entropy term replaced by a normal KL divergence term and everything else held the same: same seed, same schedule, same target, same judge, same data in the same order. I did that because a single entropy curve doesn't tell you much on its own. An average of 0.4 nats is neither obviously healthy nor obviously collapsed until you've seen what this policy does on this task without the mechanism.

Adaptive entropy holds, plain GRPO collapses

The run with the adaptive entropy term holds. Its entropy starts at 0.5144 nats and ends at 0.4084, a drift rather than a collapse, and never once dips below 0.01 nats, the level I'm treating as collapsed. Over the same steps its attack success rate against SecAlign climbs from 0.10 to 0.97. So on the face of it the policy learns the attack without paying for it in entropy, which is exactly the outcome the mechanism was built to produce.

Plain GRPO does the thing the paper warns about. Entropy starts at the same 0.5144 nats and collapses to 0.0150 by step 250, most of the drop happening early. Its ASR ends around 0.16, which is the part I find worth sitting with. It isn't that the policy locked onto something that works and stopped looking. It locked onto something that mostly doesn't work, and stopped looking anyway.

I should be upfront that this pairing is weaker evidence than it looks. The two runs differ by the whole algorithm rather than by a single field in a config, so nothing here says the entropy term is what prevented the collapse; it says one run collapsed and the other didn't. Reading it as an attribution would be exactly the kind of shortcut this post is about.

What the average is hiding

The more interesting thing is what that headline average is hiding, and it's hiding the same thing in both runs. The batch mean is taken over every token in every completion, and those tokens are not one population. If I count the share sitting below 0.01 nats, tokens where the policy has effectively already made up its mind, then in the run with the adaptive bonus that share rises from 36.4% over the first fifty steps to 49.7% over the last fifty, averaging 48.1% across the run. A mean of 0.49 nats does not mean every token is moderately uncertain. It means about half the tokens are frozen and the other half are carrying enough entropy to hold the average up. The mean can sit still for 250 steps while the distribution underneath it keeps polarizing, which is worth knowing before reading any flat entropy curve as evidence that nothing is happening. Plain GRPO just takes the same quantity further: from 54.1% to 94.4%, ending with almost every token decided.

Splitting the same entropy by position in the completion says where the remaining uncertainty actually lives, and this is where the two runs stop looking like the same story at different speeds. In the run with the adaptive bonus the first 16 tokens go from 0.4593 to 0.5753 across the run while the last 16 go from 0.5716 to 0.8594. The tail is the more uncertain end and it gets more so, with the gap between them widening about 2.5 times. In plain GRPO the tail is the first thing to freeze: the last 16 tokens fall from 0.2250 to 0.0039 nats, which is deterministic for practical purposes, while the opening still carries 0.1681. My reading, and I'd flag it as a reading rather than something I measured, is that the opening of these payloads is a learned fixed preamble in both runs, the "we have received an official update regarding..." phrasing that shows up in essentially every sample (see the worked examples appendix in the second post for full transcripts of this), and the tail is where the prompt-specific part of the injection gets restated. One run keeps varying that part and the other stops.

Did the bonus actually do it?

There's a tempting conclusion here that I don't think the run supports, which is that the adaptive bonus is what held the line. When I look at what the coefficient was actually doing, the timing is wrong for that. $\beta$ starts pinned at its maximum of 0.01, because ASR is zero and so $\bar{r}$ sits far below the $\tau = 0.5$ threshold, and it decays to its base of 0.001 as ASR crosses $\tau$. The rolling reward passes $\tau$ around step 94 and $\beta$ is at base by step 102, staying there for 138 of the 250 steps. The gate has the opposite shape: the fraction of groups whose entropy falls under $H_{\text{cap}} = 0.5$ rises to a peak of 0.905 in the 101 to 150 window. The two multiply, so the bonus is at its strongest early, when entropy is around 0.6 and mostly above the gate anyway, and at its weakest through the 110 to 135 trough where entropy is at its lowest for the run. On this run, whatever kept entropy up, it wasn't the adaptive bonus doing the heavy lifting at the moment it mattered.

What's next

So the entropy curve looks healthy, and I can't fully credit the mechanism that was supposed to produce it. That's an odd place to leave things if entropy is the only instrument you're using.

In the second post, I bring in a diversity metric I built for exactly this question, the Vendi Score, and use it to check whether the policy is still finding different attacks, not just staying uncertain while it writes them. It does not tell the same story the entropy curve does.