Sharpening the Workshop

A tool we are commercialising stopped being internal last week, and the workshop it was built in stopped being a private affair at the same moment. The substrate that produces a thing is the thing's quality ceiling. If the workshop is sloppy, no amount of polish on the artefact closes the gap.

Workflow spend runs in the low thousands of dollars a week when we are fully engaged. That is not an accident or a complaint; it is what serious AI-native engineering costs at our throughput, and the rate has been rising for six months. Against a baseline that size, an audit that locks in even single-digit-percent gains in substrate quality pays back in days. Sharpening is not a project that finishes. It is the highest-leverage recurring spend on the book.

This is the write-up of one cycle of that work, run over two days. Seven workstreams, nine councils, one large self-correction in the prompt construction layer, and a list of things deferred to the next cycle. The substrate now runs at 41 active skills, 18 self-gating hooks, 6 cron schedules, 240MB of substrate coverage, and a council mechanism with verdict vocabulary, value-tracking, plateau detection and a pre-flight fact-check rule that did not exist a fortnight ago. What it cost, what it changed, and why we will run another cycle next quarter.

// why there were 1,481 skills

The substrate started this cycle with 1,481 active skills. That is the number that a casual reader will read as drift; it is in fact the number that produces useful data.

The harness, for the last several months, has been running an experiment: can it learn to suggest skills mid-task at quality high enough to load into context only the things the orchestrator will actually use? The signal we wanted was not "is this skill good." The signal was "does the orchestrator's behaviour change as a function of catalog scale, and where does suggestion quality plateau." Answering that question requires a wide test corpus. We installed everything plausible, ran sessions across the full set, watched which skills got auto-suggested, watched which ones got loaded, watched which ones meaningfully changed the answer.

The plateau is real. Past a few hundred entries, suggestion quality flattens and noise rises. Past a thousand, the orchestrator starts loading skills the user did not need, including skills whose triggering descriptions were too generic. So 1,481 was the experiment. 41 is the conclusion.

The rule we wrote down afterwards is short. Before installing a new skill, ask which active workflow needs it. "Might be useful" is a rejection.

// 41 we kept, several we built

The curated set is 41 active, 11 quarantined, the rest removed from the active path. The recovery catalog at audits/skill-catalog-pre-prune.tsv covers all 1,481 pre-prune entries with descriptions, so nothing gets claimed-not-to-exist later. A wrapper called skill-mgr.sh (37 tests, atomic JSON updates against a canonical install ledger, drift detection between filesystem state and ledger) makes future add, remove and quarantine ops auditable. Skills that vanish without a record are how substrate forensics gets impossible after the fact.

What survived curation is one half of the picture. The other half is what we built ourselves to encode disciplines we want enforced automatically rather than remembered. The headline among those is /craft, a skill that turns vague delegation prompts into precise ones, with file paths, line numbers, and the exact change to make spelled out before any subagent runs. It is the operational expression of one rule: never delegate understanding. The default failure mode of subagent delegation is the prompt that says "based on your findings, fix the bug." That phrasing pushes synthesis onto the agent, which is exactly the work the orchestrator was supposed to do before delegating. /craft enforces the inversion. If you cannot say what the agent should change at line-number granularity, you have not yet understood the problem, and the agent is not going to understand it for you.

/craft is the headline but it is not alone. The council skills that drive the six-model audit workflow are in-house, and this cycle extended them with verdict vocabulary, value-tracking schema and a pre-flight fact-check rule. A silent-failure-hunter skill scans for empty catches, dangerous fallbacks and missing error propagation, the things that make a postmortem hard a year later. A pair of verification skills, one for Next.js projects and one for Expo projects, pin the pre-merge checklists for two of the most common shapes of work we ship. A config-protection skill blocks edits to lint, format and typecheck configs unless an explicit env override is set, because the bug class of "tests passed because the config was rewritten" is one we have seen too many times.

The skill catalog now exists for two purposes. Keep the tools from the wider community that earn their keep on a measurable workflow. Build the ones that encode our own disciplines, and keep them inside the workshop where they can evolve.

// GAN, on probation

Generator-evaluator loops, in the adversarial-network shape, are new to the toolkit this cycle. A council session picked the topology. A pre-condition was added before any pilot ran: the evaluator side has to be measurably useful as a critic before the loop is allowed to spend real money. The gate caught a load-bearing flaw in the first proposed variant before the £100 spend cap on the pilot was reached.

Two pilots ran. The first pilot, on UI generation, ran £8.90 and got demoted to a low priority. The generator side did not produce design quality the evaluator could meaningfully reject. The loop converged on bland competence, which is the known failure mode of evaluator-checked generation when the artefact is judged by aesthetics. The second pilot, on prose generation with a rules-aware specification, ran £13 and produced output the evaluator could rank against the spec. That variant graduated.

The reason the pattern earns space in the post is the gate, not the pilots. Most adversarial-generation setups skip the evaluator-validation step and produce competent-looking nothing. The discipline of evaluator-validation as a gate, not a checkpoint, is the part that makes the difference. Generator-evaluator belongs in the toolkit when the artefact has rules the evaluator can read. It does not belong when the artefact is judged by aesthetics.

// the council, reformed

The six-model council we wrote up in April has been the workhorse of every architectural decision since. This cycle changed how it runs.

The verdict vocabulary went from four options to five. The previous set biased structurally toward "pause and test", because there was no clean way to register "proceed, but verify in the test environment first." The new options separate "proceed as is", "proceed pre-test", "proceed with changes", "pause and test" and "fundamentally flawed." Verdict distributions on subsequent sessions stopped clumping artificially.

Each session now writes a value_tracking and outcome_followup record into ~/dev/council/sessions.json. Four fields: before_my_position, after_my_position, decision_changed, top_signal_model. The schema means a quarterly audit can compute decision-changed rate, identify synthesis-heavy framings for re-run, and track which models have been doing the load-bearing reasoning. Across post-reform sessions, the decision-changed rate is around 75%. That number is the case for the mechanism existing at all. Convened-then-overrule is what councils are supposed to produce. Convene-then-confirm is theatre. 75% is real signal.

A second rule landed in the skill body: read the concerns substantively, not the verdict tally. Convergence on top concerns is the load-bearing signal in any session. Verdict counts can be misleading, especially when the dissenting verdict is the right one and the four-versus-one split makes it look ignorable.

The runner now supports iteration. Two flags, --iterate N and --plateau-threshold M, let the council pressure-test a session when there is genuine disagreement worth pressing on, instead of one-shot voting. Plateau means both verdict stability and concern stability across iterations.

The criteria for when council fires at all are now written down. Architectural commitments, security boundaries, monthly cost commitments above £100, and decisions whose rollback would cost more than a day are non-negotiable. The cost of a council session is twenty pence to a pound and ten minutes wall. The cost of a wrong call in any of those four categories is days to weeks of cleanup. The math favours convening.

// the phantom in the prompt

The most consequential moment of this cycle was none of the above. It was a single question, asked on first read of a council recommendation, that exposed a class of failure invisible to the orchestrator's own self-check.

A council session ran on the prompt "should we migrate from SurrealDB to Postgres given the stability issues?" The council reasoned from the brief and recommended phased migration. The recommendation came back. The first reaction was the right one: why are we moving from SurrealDB? The answer was supposed to be "stability issues." The empirical answer was: one month and three days of continuous uptime, zero restart events, zero watchdog interventions across 8,640 consecutive health checks, zero client-side errors logged. The "stability issues" claim was a phantom. It came from a vague memory line that had been treated as fact when the orchestrator wrote the council prompt.

This is the part that matters. The council reasoned correctly. The brief was wrong. A council whose brief contains a synthesised premise cannot un-synthesise it from inside the room. The work the council does on a phantom claim is structurally indistinguishable from the work it does on a verified one. The pre-flight is the only place to catch it.

The fix moved fast because the audit pattern was already clear. With one instance proven, the next question was whether the failure had appeared anywhere else in the cycle. A second retro had the same shape. A vague memory line had been promoted to fact in the prompt construction step, and the council had reasoned from a position the prompt-writer had never actually verified. Two of eight councils run during this cycle were driven by synthesis-as-fact failures upstream of the council itself. A 25% bad-recommendation rate, not from model quality, but from prompt-construction. Correct reasoning over incorrect premises.

The rule that shipped tags every load-bearing claim in a council prompt as one of three things. VERIFIED means the claim was checked against logs, code, or test runs before the prompt ran. UNVERIFIED_OPERATOR_CLAIM means the operator stated it, no separate verification was done, and the council should reason accordingly. SYNTHESIZED means the orchestrator inferred it from existing context, and the council must either treat it as a hypothesis or refuse to reason from it. Synthesised claims that turn out to be load-bearing get verified before the prompt runs. The procedure lives in the council skill body, the canonical rule lives in the memory wiki, and a quarterly audit cadence re-runs synthesis-heavy sessions to catch any leakage of the rule.

The mechanism is real. The same council models, given the same data with the claims correctly tagged, produced a materially different recommendation on the second retro. Same room. Same data. Different brief. Different answer.

There is a meta-lesson, and it is not about the orchestrator drifting. Synthesis-as-fact was a class of error invisible to the orchestrator's own self-check, because the orchestrator does not know which of its own claims came from where without an explicit pass. The operator caught both instances on first read. The rule now closes the loop so the next case does not depend on vigilance from the human.

// the substrate, profiled

The plumbing improvements all serve a single thread: drift fixes that do not measure their own value just become new drift.

Hooks self-gate now. Eighteen pre-tool-use hooks read a profile flag at fire time and decide whether to run, instead of every hook firing on every tool call. CORTEX_HOOK_PROFILE=minimal fires only the five security-critical hooks. standard is the default. strict adds the rest. A CORTEX_DISABLED_HOOKS env override lets a session disable any specific hook by name without touching settings.json. 42 tests across the bash and node sides. Minimal profile cuts pre-tool-use overhead by around 70% for sensitive ops.

The memory store got a four-wave prune. The first wave alone removed 6,680 rows because the memory MCP server had ingested its own source tree at install time via a recursive ingest_codebase feature. A 51%-of-store self-citation that contributed nothing to recall and added latency to every query. Three more waves removed bash-noise capture rows, edit-noise capture rows excluding the wiki paths, and episodic duplicates. Total: 9,763 rows gone, 75.6% of the store, 64% of disk reclaimed. Full-text search execution post-prune sits between 0.3 and 1.0 milliseconds.

Cron consolidation took 13 schedules down to 6 via four wrapper scripts: hourly-state, nightly-maintenance, every-4h, daily-health-check. Each wrapper runs its sub-steps with their original log paths preserved. The shells use set -u and not set -e so one sub-step failure does not kill the rest. Failure isolation per sub-step matters more than fail-fast when the wrapper's job is to keep telemetry flowing.

One audit signal worth naming is the briefing system. Three briefings run on cron: a morning summary and an evening one, both pushed to Telegram, and a long-form briefing.md written to disk for pull-on-demand. The push-route briefings have been doing the job for months; they land on the device the moment we want them, and they get read. The pull-route file briefing was getting written hourly and pulled zero times across thirty days. That is not a "stale file" finding. It is the system telling us, with months of data, that the push route is the route we actually use, and the file route is solving for a use case we do not have. The fix was not deletion. The fix was tuning the file's content to match what we actually want when we do open it, and then measuring pull rate over the next seven days. If the rate stays low, the file format gets retired and the saved cycles go elsewhere. Continuous tuning informed by usage data, not a one-shot cleanup.

// the rename canary

A small bug class hit twice during this cycle and shipped a fix that closes the door on it for good. Cortex bumped a column name (heat to heat_base) at some point in a recent schema revision. Two scripts that queried the column kept running, kept exiting zero, kept writing their output, and kept producing empty results because the new column reference was wrong. The briefing generator and the freshness scanner both went quiet for an unknown duration before either was caught. Silent failure is the worst shape of failure: nothing alarms, nothing logs, the system looks healthy.

The fix is a daily regression check, cortex-schema-check.sh, that runs EXPLAIN on every Cortex-querying script's load-bearing query against the live schema. EXPLAIN parses and plans the query without executing, so missing columns, missing tables and renamed identifiers all surface as a parse error. Each script declares its load-bearing queries via a sentinel comment that the runner extracts and validates. New scripts added later add a sentinel; the canary list grows with the substrate. A failure sends a Telegram alert with the failing query name. Five scripts are wired in today. The next column rename in Cortex flips the canary the morning after the schema change, instead of waiting for someone to notice the briefings have gone empty.

The lesson is the one we keep relearning. A drift fix that does not measure its own value becomes new drift. The canary measures.

// the runbook

The off-site backup of the personal-assistant system has been running on a four-hour cadence for a long time. What this cycle did was extend the cadence to cover the rest of the workshop. The agent substrate at ~/.claude/, scripts, council artefacts, the planning directory, four local-only repos, the crontab itself and the shell environment now ride the same schedule and land in the same target. The tarball grew from 5MB to 240MB to match. The discipline was already in place; the surface area expanded to fit what the substrate now is.

Four local-only repos got pushed to private GitHub for redundancy. The agent's own toolchain has its source-of-truth in two places now.

A disaster-recovery runbook lives at DR-RUNBOOK.md in ten phases, from cold-bootstrap through restore through smoke tests, with exact commands, expected outputs and decision points. The phases are written for a future operator, possibly a future orchestrator, recovering against a fresh Ubuntu 24.04 container with nothing but the tarball.

The most useful change to the runbook came from how the council reframed the author's pre-position. The starting plan was a single-tier RTO, quarterly drills and weekly verification. The council unanimously rewrote it to a dual-tier RTO with separate scripted and manual targets, monthly drills for the first three to six months because that is when assumptions are likeliest to be wrong, and a daily-plus-weekly-plus-monthly verification cadence with four mandatory pre-conditions before authoring the runbook itself. The original draft would have shipped a runbook asserting recovery times nobody had measured. The reformed draft shipped a runbook stating recovery times we can prove. That is the council pattern earning its keep on yet another decision: not "is this a good runbook" but "is the framing of the author's confidence calibrated against the data."

The Docker dry-run of the runbook is deferred to an operator-supervised session. A runbook only proves itself when actually executed against a fresh container with a stopwatch.

// what it cost and what it bought

The cycle ran just under £300 of a £500 cap, plus time. Across nine council sessions, one demoted UI pilot, one graduated prose pilot, the subagent fan-out across the mechanical phases of every workstream, the backup extension, the memory migration tooling and the audit work that produced this post.

The economic translation is the part that matters more than the headline number. Workflow spend at full engagement runs in the low thousands of dollars per week. The audit cost rounds to a single working day's spend at that rate. A single fewer phantom-claim council, a single faster recovery from substrate corruption, a single avoided redo of a piece of work the orchestrator should have checked the first time, any one of those, once, makes the audit net-positive on its own. The compounded effect of all of them, every week the substrate is in use, is the actual return. We do not need a precise ROI number to know which side of the ledger we are on.

The audits also translate sideways, which is the second-order point. Every discipline that lands in the substrate ships into the products by virtue of being how the products get built. The pre-flight fact-check rule will catch synthesis-as-fact in product councils, not just substrate councils. The hook profiles will reduce overhead in customer-facing sessions, not just internal ones. The disaster-recovery runbook covers everything the substrate produces. There is no separation between "workshop work" and "product work." There is product work and the toolchain that ships it, and the toolchain is on the same audit cadence as the products.

A handful of items are deferred, named honestly. The Docker dry-run of the runbook needs an operator-supervised session with a stopwatch. The briefing format change waits on the seven-day pull-rate data before the next move. An auto-capture intake filter belongs in the next cycle, with scope larger than this round's memory prune.

The first quarterly audit of the pre-flight fact-check rule lands in Q3 2026. The audit measures bad-recommendation rate against the 25% baseline. The next sharpening cycle starts when the data says it should.

// the cadence

Sharpening is not a project. It is the maintenance pattern of an instrument expensive enough to operate that the maintenance pays for itself many times over per cycle. At our throughput, the workshop earns the audit without anyone having to argue for it. The argument was "are we doing this." The answer was "we already are; this is the write-up."

The substrate that builds a tool we are commercialising, and the in-house products that came before it, now runs in a state that can be inspected, measured, audited and recovered. The disciplines that produced the state are written down where they fire automatically. Hook profiles gated by env. Council mechanism with verdict vocabulary, value-tracking and pre-flight fact-check tagging. Skill management wrapper with drift detection. The runbook on disk. The next cycle will surface findings we did not plan for, gains we did not yet measure, and a class of error we have not yet named, and we will run that one too.

A workshop that does not periodically audit its own grinding wheels eventually finds out the hard way that the wheels are out of true. Ours stay sharp on a cadence. That is the whole bet.