Six LLMs Walk Into My Codebase

Six LLMs walked into my codebase last week. They left behind a list of bugs, a set of fixes, and the faint but unmistakable smell of several million matrix multiplications.

Which sounds like the setup to a joke, and in a way it is, because this is roughly where someone on a podcast will explain that language models cannot do security review and you should not be trusting them with your production code. That is correct, in the same careful way it is correct to say that a single philosopher alone in a room cannot do philosophy. One LLM does not do security review. One LLM does vibes, in the way a lone diner in an empty restaurant eventually begins to do vibes no matter how hungry they were when they sat down. What turns vibes back into review is a room of them, each taught politely but firmly to disagree with the others.

The council is nullhex's six-model audit workflow. It is the thing we reach for when an architecture decision has acquired too many legitimate shapes, or when a production surface has grown a new API endpoint and nobody is entirely sure it closes cleanly, or simply when we need to know whether a piece of code is as safe as the person who wrote it is currently convinced it is. Last week's run was against the freshly-expanded payment and voucher surface of one of the web applications we maintain. The council produced thirty-six findings across the severities that matter, one of the six models caught a path that the other five had politely overlooked, and the fix sprint that followed occupied most of the week.

This is the write-up. What the council is, how it actually works, and a handful of the findings that would not have happened in front of a single pair of eyes.

// the lineup

Six slots. Six models. Six different providers. That last rule is the one that people expect to be cosmetic and it is in fact the whole game.

The current seats are Claude Opus 4.7 from Anthropic, sitting as the origin; Gemini 3.1 Pro from Google; GPT-5.4 from OpenAI; Grok 4.20 from xAI; GLM-5.1 from Zhipu; and Kimi K2.6 from Moonshot. Six models, six vendors, no overlaps. The composition rule is deliberate and it is also the bit of the whole arrangement that is easiest to get wrong, in the way that it is easiest to arrange a dinner party by inviting six people from the same office. Two Anthropic models are one model with different settings. Two OpenAI models share a training lineage, a safety philosophy, and a set of opinions about how to end a sentence. If four of your six reviewers are drinking from the same training distribution, you are not getting six perspectives. You are getting one perspective with a majority vote that feels like consensus and is really just a long, self-satisfied echo.

The lineup rotates. Through April 14 the council was Opus 4.6, Gemini 3.1 Pro, GLM-5.1, DeepSeek V3.2 and Qwen 3.6 Plus. The April 21 refresh swapped DeepSeek and Qwen out for GPT-5.4, Grok 4.20 and Kimi K2.6, and bumped the origin model to Opus 4.7. The swaps were not based on benchmarks. They were based on watching, over a few weeks of real reviews, which models kept saying something the others had not. Models that drift into agreement with the room get retired. Models that keep pulling findings out from unexpected angles earn the seat.

// the rule of the room

Three rules govern how a council run works. They are the difference between a council review and six parallel chatbot prompts.

Blind review. When findings are sent to the five non-origin models, they are stripped of attribution. The models do not know which finding came from Claude's first pass, which came from the prior audit, which came from a human complaint. They do not know who else is in the room. Each model sees the same code context and the same finding list, and is asked to react to the code, not to the authorship of the finding. The effect is the same as single-blind review in academic publishing: a model cannot defer to, or argue with, the reputation of the reviewer upstream, because there is no reputation to argue with.

Mission not role. The prompt does not say "you are a senior security engineer." It states the mission and lets each model bring its own frame. "You are a senior X" prompting anchors capable models to the prior they associate with that role, and that prior is often wrong. Mission-framed prompting gets you the model's actual opinions; role-framed prompting gets you the model's impression of what a senior X would say. The first is usable. The second is a costume.

Post-fix re-review. This is the part most AI review setups do not do, largely because the part before it was remembering to review the fix at all. When a finding gets fixed, the fix itself gets its own council round. The models are shown the diff, asked to verify the fix closes the original vector, asked to flag new regressions introduced by the fix, asked to rule on scope discipline. Only fixes that come back APPROVE or APPROVE_WITH_FOLLOWUP from the second council go in. This is the step that turns the council from a bug finder into a review pipeline.

Every run writes its raw responses into ~/dev/council/raw/ and produces a session JSON of tallies, votes, costs and agreed fixes that syncs to the nullhex ops dashboard at /ops/council where dissenting votes stay in the record alongside the majority calls, because three months from now you will want to know which model flagged a finding LOW when it turned out to be a full credential-reset bypass.

// what convergence looks like

Some findings are easy. Every model in the room rates them HIGH, every model recommends the same class of fix, and the tally renders as a straight vertical bar in the dashboard. These are the findings you would have caught with any single one of the six models; the council's value on them is in the ceremony, not the discovery.

Last week's audit had a few of those. One was a voucher redemption that was not atomic. Two concurrent requests could redeem the same voucher twice under adversarial timing, because the "has this voucher been redeemed" read and the "mark this voucher redeemed" write were separate SQL calls inside a single request handler instead of a single RPC with row-level locking. Every model in the council said HIGH. Every model said the fix should be a Postgres RPC with a SELECT FOR UPDATE on the voucher row. We built the RPC, the postfix council came back unanimous APPROVE, and the finding closed.

That is the boring case and it is the case where the council is arguably overkill. We do it anyway, because the price of running the council on a boring finding is cents, and the price of missing an interesting one hiding next to a boring one is not.

// what divergence looks like

One of the low-severity findings was tiny. The code under review was constructing Stripe checkout redirect URLs and used the request's Origin header as the first-choice source of truth for what origin to redirect back to. A fallback to the app's configured public URL existed as an environment variable, but the header took priority. The initial severity was LOW. Most of the models shrugged. The Origin header is attacker-controllable, sure, but the redirect targets were the app's own success and cancel pages, and modern browsers block obvious nonsense, and the blast radius looked limited.

GLM-5.1 did not shrug. During the post-fix review for a different finding that had already closed the explicit open-redirect vectors on body-supplied success_url and cancel_url, GLM-5.1 pointed out that the header case reached the same ground through a different door. An attacker who skipped the body fields entirely and just spoofed the Origin header got their redirect URL into the Stripe checkout session's success path, which meant that after the victim paid, Stripe bounced them to https://evil.com/booking/success?payment=complete. Same outcome, different entry vector.

What happened next is the part where the council actually earns its keep. Because the finding was still on the board at LOW, and because GLM-5.1's elevation was logged alongside the original vote, the synthesis step saw the delta. The next postfix batch ran the finding back through the full council with GLM-5.1's argument attached. Four of the other five models agreed that the severity needed to go up. We reproduced the attack against the real handler with production env vars in a Node script and it worked. The fix, which in the end was stop trusting the Origin header, fall back to env, fail closed if env is missing, landed later the same day and went through its own postfix round.

This is the case that makes six models worth more than one. None of the other five raised the escalation on first pass. In a room of three Anthropic models and two OpenAI models, it is entirely possible that nobody would have raised it at all, and everyone would have gone home pleased with a thorough morning's work. GLM-5.1 is trained on a different distribution, maintained by a different team, nudged by a different safety philosophy, and that is the entire point.

// independent findings

When we ran the audit, the preliminary list from the first pass had thirty-six findings. Each of the five reviewing models was also asked to do an independent scan of the same code and flag anything the preliminary list had missed.

GPT-5.4 came back with seven independent findings. One was a HIGH: a publicly-accessible voucher checkout path was using the cookie-authenticated Supabase client for a flow that was meant to work anonymously, and anon-role RLS was silently blocking the writes the handler needed to make, namely the redemption insert and the payment-status update on the parent record. Nothing obvious broke, because the flow failed silently and looked like an RLS issue rather than a client-choice bug. The finding was real. The fix was a one-line swap to the service-role client on that handler's write path. It is the kind of bug that a grep for common patterns does not find, because the failure mode does not look like a bug, it looks like RLS working as intended.

Grok 4.20's independent list was shorter but flagged a mass-assignment on a package-create endpoint. The request body was being spread directly into the insert with no schema whitelist, which is a delicate way of saying a client could set any writable column it felt curious about, including the Stripe ID fields that were supposed to be server-controlled. Zod schemas existed for other endpoints. This one had simply been missed. Fixed with a whitelisting Zod parse on top of the insert.

Gemini 3.1 Pro's contributions were mostly calibrations rather than independents, but its calibration notes on the subscription checkout's fail-closed behaviour were what pushed us to align it with the parallel one-off-payment path. The followup fix closed a subtle divergence between two flows that should have been handled identically and were not.

The independent-finding side of the workflow is the part that makes the council useful even when the primary audit is thorough. Six sets of eyes, six different priors for what a bug looks like, produce a union of concerns wider than any single reviewer. The intersection of concerns is the HIGHs. The union is the LOWs and MEDIUMs that would otherwise have aged into next quarter's problem.

// who catches what

Model-by-model accuracy is a thing we track. Every finding in every session carries votes from every model that reviewed it, and the session record has a confirmed flag. When a dissenting vote is later validated as correct, because the code path turned out to be worse or better than the majority called it, the flag flips. Over enough sessions, the pattern becomes a scoreboard.

The current picture, with the usual caveats that the sample is small and the lineup just rotated:

• GLM-5.1 is the escalation specialist. It flags things other models mark LOW and keeps being right about it. The Origin-header case above is not the first time it has pulled a finding up a severity rank.
• GPT-5.4 is the independent-findings champion. It spots things that were not on the list; its false-positive rate on independents is not zero, but its true-positive rate is the highest of the six.
• Gemini 3.1 Pro is the conservative anchor. When Gemini calls a HIGH, it is almost always a HIGH. Its threshold for escalation is the highest in the room, which makes it the quietest but the most diagnostic voter.
• Grok 4.20 is the fastest. Latency under the others, useful for iteration, but with a slightly higher rate of confident-sounding wrong calls.
• Kimi K2.6 is the cheapest per million tokens and the newest addition. Too early to tell where its accent sits, but it has not yet pulled a finding out from underneath the others.
• Claude Opus 4.7 is the origin and does not get a review rank of its own here; it is the model that proposes, not disposes.

A more honest way to say this: each model has a direction it tends to drift in. The council is the system that makes the drifts cancel.

// what it cost

The entire audit week, from the initial severity round through every postfix re-review, cost about four US dollars in API fees across the five non-origin models. The origin model is free in the sense that it runs inside an existing Claude Code session; its token cost is tracked elsewhere. Four dollars. That is less than a pint in an unserious London pub, distributed across six models and the better part of a working week, producing a triage on every finding and a shipped fix on a healthy fraction of them, and one escalation that would not have happened with a single-model reviewer in the chair.

The cost discipline is the quiet argument against outsourcing security review to a vendor and for building the council yourself. Commercial AI review tools are priced per seat, per repo, per scan. The council is priced per-token, and the token cost per session is in the range of pocket change. The engineering investment is not zero, but the recurring cost is.

Every session's USD spend lands in the session JSON and rolls up to the dashboard. If you want the shape of it, the dashboard at nullhex.io/ops/council shows the cumulative totals.

// what it is not

The council catches a specific class of bug very well. The headers you should not have trusted, the atomic-looking writes that were not atomic, the anonymous flows that quietly failed because the wrong database client was wired up: all come off the list before a human eye ever opens the pull request.

What it cannot do is think about the product. Threat modelling, compliance reasoning, and judgement about which parts of the surface actually matter all require a mental model of the thing being built, and that stays firmly on the human side of the desk. The council clears the floor so the human can spend attention on the hard questions instead of on open-redirect checks and SQL atomicity.

Six models in a room, blind, mission-framed, with a postfix round on every fix.

It is not complicated. It just works better than one.