Fast Experiments, Hard Evidence | Russ Watson-Thomas

The problem changed

Building things stopped being the bottleneck. Knowing whether I should trust them became the bottleneck.

For years, an experiment carried a natural speed limit. You could only write, configure, connect, test and deploy so much software in an evening or a weekend. AI changed that limit rather abruptly.

I could move from a thought to a working interface, API, game mechanic, migration or automation far faster than before. Very useful. It also meant a prototype could become a durable repository, a public URL and a thing another person depended on before I had properly noticed the transition.

The danger is not simply that AI writes bad code. Humans have been managing that without assistance for quite some time. The more interesting problem is that plausible output arrives quickly enough to make confidence feel like evidence.

Standardise once

I did not want eighteen little projects inventing eighteen slightly different versions of “probably fine”.

The Standards repository became the reusable operating layer across the estate. The current central inventory contains eighteen managed repositories, covering static sites, web applications, API-backed applications, games, documentation repositories, client work and experiments. That inventory is useful precisely because the estate does not politely stop changing after it has been counted.

That matters because a new project no longer starts by rediscovering branch rules, evidence language, deployment discipline, access assumptions, security expectations and what an AI agent is allowed to call “done”. It declares what it is, which capabilities it has and which providers it uses. The applicable baseline follows from that.

The point is not to make every project identical. Quite the opposite. The reusable bit is the mechanism for deciding what should apply. A static archive does not need API controls. A documentation-only repository should not suddenly become deployable. An API-backed product gets a deeper set of controls because it has more trust boundaries to get wrong.

18

Registered repositoriesShared baseline

Profile → inherit → extend.

Every registered repository carries a small project profile. Core controls apply everywhere. Capabilities such as runtime, web, API, Vercel or GitHub Actions add the relevant controls. Project-specific rules can extend that. Explicit exceptions are recorded rather than smuggled in through tribal knowledge.

Why it scales

Onboarding another experiment should mostly mean describing the thing accurately, not copying a pile of governance documents and hoping they remain vaguely related six months later.

The operating model

Policy for humans, controls for machines, evidence for decisions.

The mistake I wanted to avoid was writing a beautiful standards document that everybody, including the AI, could solemnly ignore.

So the system is split into different kinds of authority. Human-readable standards explain the intent. Machine-queryable controls state testable requirements and severity. Project profiles declare the shape of a repository. The inventory lets one audit ask whether those declarations still match reality. Templates standardise handovers, releases, audits and exceptions.

The loop is deliberately simple: profile → resolve controls → inspect current state → gather evidence → decide → publish. If a failure is real, fix it. If a deviation is deliberate, record it. If something cannot be proved, say so. This sounds slightly dull because it is. Dull is underrated when software is changing quickly.

AI inside the system

The agent gets rules too. Especially the agent.

AI is useful precisely because it can move quickly across code, documentation and infrastructure. That is also why I do not let its own confidence define completion.

01

Current stateBefore memory

Inspect the repository before changing the repository.

Conversation is useful context, but it is not durable project truth. Before a material change, the agent is expected to inspect current source, canonical documentation, relevant branches or pull requests, applicable controls and any external provider state the task actually depends on.

That became necessary because AI is exceptionally good at producing a coherent answer to yesterday’s version of a system.

02

Evidence languageNo imaginary green ticks

“Passed” has a deliberately boring definition.

It means the relevant check was actually run against the relevant current state. Older evidence is historical. An unexecuted command is not “expected to pass”. A deployment is not READY because the configuration looks correct. Assumptions and verified facts stay visibly different.

03

ScopeNarrow diffs

Capability does not grant permission.

The requested change and its necessary dependencies are the scope. Nearby redesigns, infrastructure changes, copy rewrites and “while I’m here” improvements require a reason and, where material, separate approval.

Testing

The test model is broader than “does the build go green?”.

The current estate has its deepest implemented coverage in unit and functional behaviour, repository and architecture contracts, type/lint/build checks, security-focused negative paths, and deployment or live-state verification. The deeper application test strategy extends that into integration, API, component and browser end-to-end coverage where the architecture needs it.

Not every repository runs every layer. That would be process theatre with a CI bill. The rule is cheaper evidence first, then move outward as the failure mode crosses a boundary the lower layer cannot prove.

A

Fast feedbackImplemented widely

Static, build, unit and functional checks.

Type checking, linting, production builds and deterministic unit or functional tests catch cheap failures early. In the Quality Platform, for example, the access gate is tested to fail closed when configuration is missing, reject weak configuration, validate only the correct passphrase, reject tampered tokens and expire sessions correctly.

Best for

Pure rules, calculations, validation, identity mapping, configuration, security invariants and anything that can be proved without starting half the internet.

B

ContractsArchitecture & API boundaries

Repository, integration and API contract testing.

Contracts protect things that should survive implementation changes: dependency direction, required routes, access boundaries, configuration assumptions, API shape, provider adapters and the seams between systems. They are particularly useful in AI-assisted work because they let the implementation move quickly while the boundary stays stubborn.

Where a service boundary exists, the fuller model adds integration tests around persistence, authentication, events, parsing and external adapters, plus API-level tests for request, response, authorisation and failure behaviour.

Best for

Proving that two separately sensible pieces of software still agree when they meet.

C

User journeysSelective depth

Component, feature and end-to-end testing.

Component or feature tests cover meaningful interaction without paying the full browser cost. Critical journeys can then move into browser end-to-end coverage. The Quality Platform strategy uses Chromium for the main breadth and reserves WebKit for Safari-specific risk rather than mechanically duplicating the entire suite.

Physical-device checks remain part of the model when touch, navigation, responsive behaviour or native lifecycle genuinely matters. DunDunSee, for example, has an explicit path through compile, XCTest, simulator and physical-iPhone evidence rather than pretending the simulator is the phone.

Best for

The places where technically correct parts can still combine into a broken experience.

D

Operational evidenceBeyond CI

Security, accessibility, manual, visual and live checks.

Some things remain awkward or irresponsible to infer from a unit test. Security negative paths, headers, access behaviour, visual regressions, accessibility, device behaviour and the state actually being served from production all need evidence at the surface where the claim is being made.

This is where automated checks and human inspection deliberately overlap. Automation proves the repeatable parts. A person still judges the bits where meaning, presentation or consequence matter more than a boolean.

Best for

Answering the irritating but useful question: “Fine, but does the real thing actually behave that way?”

Security hardening

Small projects still have credentials, trust boundaries and public doors.

The estate includes public static sites, restricted previews, API-backed applications, databases, external providers and deliberately non-deployable repositories. Treating all of them as “just experiments” would be a very efficient way to lose track of where secrets and authority live.

The baseline is intentionally unromantic. Secrets do not belong in source control. Private provider credentials stay behind trusted server-side boundaries. A private GitHub repository does not make a public deployment private. Restricted applications must enforce the restriction on provider-generated hostnames as well as the nice custom domain. Missing configuration should fail closed where the capability requires it.

For web surfaces, configuration is treated as intent and the live response as evidence. The estate-wide hardening work checks the controls an architecture can sensibly support, including HSTS, content-type protection, framing restrictions, referrer policy, permissions policy and appropriate CSP. Restricted previews also get indexing and cache behaviour checked.

The policies are not forced into fake uniformity. One Next.js preview, for example, was deliberately not given a generic CSP that would either break the runtime or need to be weakened into meaninglessness. A future policy there needs an architecture-appropriate nonce or hash design. Consistency is useful. Cargo-cult consistency is just matching wallpaper.

Security audit

An audit should be allowed to say “I cannot prove that”.

The security and production-readiness audit process starts by mapping the repository, architecture, trust boundaries and data flows before producing detailed conclusions. Depending on the surface, it can inspect code and configuration, lockfiles and advisories, Git history, CI permissions, deployment configuration, live headers, database migrations and RLS, authentication and authorisation, integration scopes, logging behaviour and AI execution boundaries.

A retained finding needs a real path back to evidence, a plausible failure route, the affected surface, impact, confidence, severity rationale, remediation and a way to verify the fix. “Confirmed”, “strongly indicated”, “possible” and “cannot confirm” are deliberately different states.

The AI-assisted audit also treats repository content as untrusted input. Source, comments, docs, fixtures and generated files can contain instructions. They are evidence to inspect, not authority to inherit. If the thing being audited can quietly rewrite the auditor’s instructions, we have built prompt injection into the control plane and probably deserve the resulting headache.

Practical evidence

Enough architecture diagrams. Here are some fingerprints.

These are deliberately small excerpts from standards, contracts and test rails rather than business logic. The interesting bit is not that code exists. It is what the code is allowed to prove, what it found, and what changed next.

Standards · profile resolution

One declaration changes the assurance baseline.

Standards
controls/profiles.yaml

api-backed-application:
  capabilities: [documentation, runtime, web, api, deployable]
  inherits: [core]

capabilities:
  api:
    controls: [SEC-002]

resolution:
  unknown_capability: fail_closed

What it proves

The reusable unit is not a copied checklist. The project describes its shape, then the resolver adds the relevant controls. An unknown capability does not quietly become “probably fine”.

What changed

Onboarding became profile-driven. Static sites, games, API-backed applications and documentation repositories can share one operating model without pretending they share one architecture.

Quality Platform · negative-path testing

The happy path is not the security test.

Quality Platform
tests/site-access.test.mjs

test("missing access secret fails closed", async () => {
  delete process.env[SECRET_NAME];
  assert.equal(isSiteAccessConfigured(), false);
  assert.equal(await isSiteAccessPasswordValid("anything"), false);
});

assert.equal(await isSiteAccessTokenValid(tamperedToken, now), false);
assert.equal(await isSiteAccessTokenValid(token, afterExpiry), false);

What it proves

The access rail is designed around failure: missing configuration, weak configuration, altered signatures and expired sessions. “The password works” would be a rather low bar for a private engineering prototype.

What changed

Fail-closed behaviour became a reusable expectation for restricted surfaces rather than a Quality Platform peculiarity. That same idea now appears in the wider standards model.

Clenchmere · public surface contract

The story can change without the exposure boundary changing with it.

Clenchmere
scripts/validate-public-surface.mjs

for (const forbidden of ['/reader-room/', '/api/reader-feedback']) {
  if (sitemap.includes(forbidden)) {
    throw new Error(`Restricted surface leaked into sitemap: ${forbidden}`);
  }
}

What it proves

The contract protects the boundary that matters, not the prose around it. Chapters, presentation and creative experiments can move without teaching CI to care what a paragraph says.

What changed

Public indexing and restricted reader/API surfaces became explicit machine-checkable facts. That is much more useful than relying on somebody remembering which routes were meant to stay quiet.

Dusty’s Favourite Foods · bounded automation

A workflow can have write access without being allowed to improvise.

Dusty’s Favourite Foods
automated-nutrition-pass.yml

on:
  workflow_dispatch:

permissions:
  contents: write

with:
  ref: feature/gold-standard-dashboard

# ...validate generated catalogue...
git push origin HEAD:feature/gold-standard-dashboard

What it proves

The write-capable path is manual-dispatch and branch-bounded. Automatic site checks can stay read-only while specialist automation is allowed to do useful work somewhere reviewable.

What changed

“The automation can write” stopped being equivalent to “the automation can publish”. That distinction now feeds directly into the wider agent and release rules.

DunDunSee™ · audit, remediation, re-audit

A real audit found things. Good. That was the point.

DunDunSee™
production closeout · 10 Aug 2026

Observed

The July audit retained four findings: incomplete browser-security headers, weak persistence/deletion wording, no permanent compact security rail, and automatic third-party runtime asset requests.

Result

The findings were remediated and re-verified against the exact production merge commit. The closeout kept the deployment ID, live headers, workflow runs, physical-iPhone acceptance and a known-good rollback deployment as evidence.

4 → 0retained findings after remediation

286unit and repository-contract tests

16Chromium breadth journeys

7Safari-sensitive WebKit journeys

1physical-iPhone acceptance gate

Zenabi Biltong · commerce contract

Once money and fulfilment appeared, “the page builds” stopped being remotely enough.

Zenabi Biltong
scripts/check-stripe-contract.mjs

files.stripe.includes('Idempotency-Key')
files.webhook.includes('request.headers.get("stripe-signature")')
files.stripe.includes('timingSafeEqual')

!files.checkout.includes('customer_email')
!files.checkout.includes('shipping_address')

files.session.includes('MAX_ATTEMPTS = 10')
files.session.includes('Cache-Control": "no-store"')

What it proves

The commerce branch checks server-authoritative totals, payment idempotency, Stripe webhook verification, throttling, no-store payment endpoints, reservation cleanup and a deliberate customer-PII boundary. Supabase migrations, admin operations and shipping/fulfilment now sit behind that same evidence model.

What changed

It also exposed a useful bit of drift. Zenabi’s current baseline profile still describes the earlier public web application, while the commerce branch has clearly crossed into database, API, payment and admin territory. That is not something to hide. It is exactly why inventory reconciliation exists before a branch like this earns promotion.

This paper is currently following the same rule. At the time of review, the article is being read from a temporary Vercel preview while its pull request remains unmerged and the production branch remains untouched. The preview proves presentation. It does not quietly promote itself into production.

Release control

A merge is an approved source state. It is not a production event.

This is probably the most opinionated part of the experiment. Automatic Git-triggered Vercel deployment is disabled across the governed estate.

A branch exists to make the change coherent. A pull request exists to review scope and evidence. Main represents an approved repository state. Publication is then a separate decision from that exact main state, followed by live verification against the deployed source SHA.

That separation initially feels slower than letting every push create another hosted environment. In practice it removes a noisy feedback loop, particularly for AI-assisted work. The agent cannot use production as its scratchpad. Source inspection and deterministic evidence come first. Hosted previews are there when a specific review question genuinely needs one.

It also gives “deployed” a useful meaning again. A GitHub merge says the source was approved. A successful provider deployment says publication succeeded. A live check says the intended hostname is actually serving the expected state. Three facts. Much nicer.

Gaps and extensions

There is a decent system here. There are also some very obvious places to make it less dependent on me remembering to be sensible.

The current model is useful partly because it is allowed to admit where the evidence is incomplete. The next step is not to bolt every possible tool onto every repository. It is to make the common assurance path cheaper and more automatic, then spend human attention on the places that genuinely differ.

01

Supply chainSecurity depth

History-aware secret scanning and stronger dependency evidence.

Current-source inspection is not proof that a secret never existed in Git history. That remains an explicit gap. The natural extension is history-aware secret scanning, stronger dependency and advisory checks, and a machine-readable software inventory or SBOM where the project risk justifies it.

02

Testing reuseLess reinvention

Reusable harnesses for the test layers that keep recurring.

Access gates, API authentication, security headers, route exposure, deployment state and provider configuration repeat across projects. Those are candidates for shared fixtures, helpers and contract suites so a new repository inherits useful tests rather than merely inheriting instructions to write them.

03

Journey depthSelective automation

Broader integration, API and end-to-end coverage where consequence warrants it.

The target model is already defined more deeply than it is implemented everywhere. The next useful expansion is not blanket browser testing. It is identifying the products whose main risks now live across service boundaries, APIs or complete journeys and adding the appropriate layer there.

04

Experience qualityNon-functional

Accessibility, performance and resilience as first-class evidence.

Accessibility and device checks are already part of the philosophy, but the reusable automation can go further. Accessibility rules, performance budgets, degraded-network behaviour and selected resilience tests should become profile-driven capabilities rather than one-off good intentions.

05

Evidence productQuality Platform

Turn assurance output into a first-class artefact.

The Standards repository already knows the controls and the inventory knows the projects. The obvious next move is a generated conformance view: applicable controls, evidence, exceptions, drift, last verification and release state in one place. That starts to turn the Quality Platform from a concept into the interface for the same philosophy used to build it.

06

OnboardingProject standardisation

Make “standard project” nearly boring to create.

Longer term, a new repository should be able to declare its profile and receive the normal structure automatically: standards profile, AGENTS rules, PR template, baseline workflows, test scaffolding, release record and the right security defaults. Then the interesting engineering effort starts with what is unusual about the product, not with retyping the same guardrails.

07

Risk-based securityBeyond static review

Threat modelling, dynamic testing and specialist review when the stakes rise.

The framework should know when it has reached the edge of its own competence. More consequential products need deliberate threat models, dynamic application testing, fuzzing or penetration testing where appropriate, and specialist review rather than another increasingly confident automated report.

The finding

Applied AI maturity is not how quickly you can make software appear.

That bit is getting cheaper by the month.

The more interesting capability is whether you can increase the rate of change without losing the ability to explain what changed, what evidence exists, which boundaries still hold, where uncertainty remains and who actually approved the thing now running.

I started this because a growing pile of side projects was heading toward operational soup. The result is more reusable than I expected: shared standards across the estate, explicit rules for agents, layered testing, security hardening, evidence-led audit and release controls that keep “source approved”, “deployed” and “verified live” separate.

The rule I would carry into a bigger organisation is fairly simple. Let AI make engineering faster. Make evidence cheaper and more reusable at the same time. If the first curve rises and the second does not, you have not really accelerated engineering. You have accelerated uncertainty.

Fast Experiments, Hard Evidence.

Building things stopped being the bottleneck. Knowing whether I should trust them became the bottleneck.

I did not want eighteen little projects inventing eighteen slightly different versions of “probably fine”.

Profile → inherit → extend.

Policy for humans, controls for machines, evidence for decisions.

The agent gets rules too. Especially the agent.

Inspect the repository before changing the repository.

“Passed” has a deliberately boring definition.

Capability does not grant permission.

The test model is broader than “does the build go green?”.

Static, build, unit and functional checks.

Repository, integration and API contract testing.

Component, feature and end-to-end testing.

Security, accessibility, manual, visual and live checks.

Small projects still have credentials, trust boundaries and public doors.

An audit should be allowed to say “I cannot prove that”.

Enough architecture diagrams. Here are some fingerprints.

One declaration changes the assurance baseline.

The happy path is not the security test.

The story can change without the exposure boundary changing with it.

A workflow can have write access without being allowed to improvise.

A real audit found things. Good. That was the point.

Once money and fulfilment appeared, “the page builds” stopped being remotely enough.

A merge is an approved source state. It is not a production event.

There is a decent system here. There are also some very obvious places to make it less dependent on me remembering to be sensible.

History-aware secret scanning and stronger dependency evidence.

Reusable harnesses for the test layers that keep recurring.

Broader integration, API and end-to-end coverage where consequence warrants it.

Accessibility, performance and resilience as first-class evidence.

Turn assurance output into a first-class artefact.

Make “standard project” nearly boring to create.

Threat modelling, dynamic testing and specialist review when the stakes rise.

Applied AI maturity is not how quickly you can make software appear.

If AI lets a team ship twice as much change, what has to become twice as easy to prove?