Rob Dull/ Tools/ AI ProdOps/ Rebuild retrospective
Build Process · Retrospective

Rebuilding eight separate tools into one workflow

This page covers the process behind the toolchain: the problem that forced a rebuild, what got built, which tools did the work, how instructions were structured, what got checked in code, how traceability was enforced end to end, where a human stayed in the loop, and what the process taught. The PetHealth case study shows the workflow running; this page covers how it got built.

The starting point

Eight separate AI-assisted tools spanning the product-delivery lifecycle: persona research, journey mapping, feature intake, business-case generation, prioritization, requirements docs, backlog building, and batch planning. Each one worked in isolation. Tracing the actual data flow between them told a different story.

8 → 9
Disconnected tools rebuilt into one nine-step workflow
2 → 1
Independent scoring implementations, collapsed to a shared source of truth
5
Kinds of checks that run in code instead of trusting the model
3
Bugs found only by a person testing, missed by scripted checks
01
The problemDiagnosis

Eight cards sat on the tools index page, each one claiming to hand its output to the next. The prioritization tool wrote no output another tool could use; it only exported a static file. The backlog tool downstream re-implemented its own prioritization math from scratch, so two independent scoring implementations could silently disagree. Persona data had to be manually copy-pasted into the journey-mapping tool. Epic-level and feature-level planning were conflated in the shared schema, so prioritization decisions and delivery-team sizing decisions were happening on the same object when they needed to be separable.

Behind the "automated" label, the handoffs were manual copy-paste.

02
What got builtRebuild

A restructured, nine-step workflow, deployed live: Persona → Journey Mapping → Initiative Intake → Business Cases → Epic Prioritizer → Roadmap & Milestones → Business Documents (the BRD) → Backlog Builder → Build Planner, ending in a CSV file for Jira's bulk import. Each step's output became the next step's input, passed along inside the browser.

The demo runs an insurance-claims initiative through the whole chain with both a front-stage persona and a backstage persona, deliberately, so the workflow has to handle customer-facing and infrastructure work correctly. That's where the original tools broke down: backstage work kept getting scoped as if it were customer-facing.

03
Tools usedStack
The build leaned on a deliberate set of tools, each doing a specific job.
ClaudeDifferent model tiers by phase: a faster model to scaffold a stage, and a stronger model for a separate adversarial review pass.
Claude CodeThe agentic build environment: multi-step builds, live browser verification, and file-level review in one loop.
React + TypeScript monorepoOne shared toolkit package for data types and scoring math, so the duplicate-implementation bug can't recur by construction.
Sample-evidence layerFictional records shaped like MCP responses, standing in for live connections, so every generated document has evidence to cite.
04
Structured instructionsPrompting

Nothing in the chain runs on free-text handoffs. Every stage has a typed input/output contract defined once in a shared package, and the model fills in that fixed schema.

Two instruction patterns did the most work. Cite only what's present: every generative step references evidence by a [REF] id and may only use ids included in its input; anything it lacks evidence for is marked ASSUMED. Keep independently-varying concepts decoupled: an early version conflated "is this epic customer-facing or infrastructure" with "does its business case rest on a customer metric or an engineering one." Two independent questions were collapsed into one field, which produced the wrong framing downstream. Splitting them into two fields, classified independently, was verified with a controlled test: two epics with identical infrastructure classification produced correctly divergent business-case framing.

05
Quality checksGuardrails
The rule running through the whole build: the model never gets to be the source of truth for anything checkable. Wherever an output can be computed or verified exactly, it is checked in code, after the model, every time.
★ Enforced in code
  • Prioritization scores recomputed from structured inputs in code
  • Requirement coverage computed: each business requirement is checked for at least one downstream feature
  • Dependency order rechecked: a story placed in the same batch as, or an earlier batch than, a story it depends on raises a visible warning, on screen and in every export
  • Output shape defended everywhere: one missing field from a model response can't take down a page
  • References checked: nothing can cite an id that doesn't exist upstream
06
TraceabilityProvenance
Every artifact downstream carries forward the ids of what produced it. Code checks that nothing references an id that doesn't exist upstream.
BR-04→ FEAT-02→ ST-07→ ST-09→ Jira: Blocks
A feature cites the business-requirement ids it satisfies; a story cites its parent feature and the other stories it depends on; the final export's dependency links are read directly off that same story graph. An exec summary generated at one stage seeds directly into the next stage's document.
07
Human reviewJudgment

Two tiers. First, a second AI pass on complex phases, framed as adversarial review, hunting for bugs. Second, and more important: a person tested every phase by entering realistic inputs, separate from the fixture data used for automated checks.

Scripted checks confirmed the code paths ran, but hand-authored test fixtures were too clean to surface these bugs.

★ Caught only by a person running the full workflow
  • A paste box silently accepted the wrong shape of data because two nearly-identical text boxes sat next to each other on the same screen
  • A rule displayed a correct warning on screen but silently vanished from the exported file
  • Generated documentation described a feature the prompt explicitly said shouldn't appear
✋ Human gate: every phase reviewed with realistic inputs before it counted as done
08
What the process taughtRetrospective
01

Scripted and human testing catch different bugs

Skip the human pass on a step with pasted or free-form input, and that's where bugs hide.

02

Never trust a model's own math or ordering

Recompute it in code afterward, every time, even after the model gets it right once.

03

Design for visible failure

A system that marks content ASSUMED or refuses below a threshold is more trustworthy than one that always sounds confident.

04

Test the connections end to end

Tools that each work on their own were the gap that caused the original failure and started this rebuild.

05

Decouple concepts that vary independently

Folding two real distinctions into one field for simplicity produces silently wrong downstream framing that's hard to trace back to its cause.

What this process demonstrates

  • Every step was tested against the next step's handoff, end to end
  • Checks in code ran after every AI step
  • Traceability held across the full chain: an id from an early stage is still checkable at the last one
  • A person testing with realistic inputs found bugs that scripted, fixture-based checks missed

Scope

  • This is a working prototype. Scope disclosures are on the case study and technical notes pages
  • Human judgment applies at each gate, alongside the guardrails
  • Model behavior varies run to run; the computed checks are the repeatable part
  • The second-model adversarial review ran on select complex phases