From chaos to clarity
A structured QE practice for a fast-moving AI platform. How we replaced informal Excel bug tracking, zero release visibility, and undocumented features with Jira, Confluence, defined release windows, and a structured QE practice across multiple enterprise client environments.
Defects raised
Peak quarter executions
Test cases built
Release cycles supported
Quality engineering for an enterprise agentic intelligence platform.
Capitol AI is an enterprise-grade agentic intelligence platform designed for regulated, high-stakes organisations. It converts expert judgment and proprietary data into trusted, auditable outputs — enabling data teams, financial services firms, professional services organisations, and government bodies to deploy sovereign AI workflows without vendor lock-in or data leakage.
The platform is model-agnostic, SOC 2 compliant, and supports fully customisable output artifacts — from reports and briefings to spreadsheets, code, and audio. It operates across multiple enterprise clients with independent environments per tenant, making QE across a multi-client, rapidly evolving AI product both critical and complex.
No process. No tracker. No release visibility.
When the QE team joined Capitol AI, there was no structured quality engineering process in place. Bug tracking was informal — Excel with no consistent format. Releases shipped without advance notice to QA, with no test cases, feature documents, or recordings. The brief: build the entire QE infrastructure from zero while testing a live, fast-moving multi-tenant AI platform — and change how the company thinks about quality along the way.
No QA process or structure
Bug tracking was informal and Excel-based with no consistent format, severity classification, or closure workflow. There was no structured way to raise, triage, or track defects through to resolution.
No release visibility
Releases went live with no advance notice to QA u2014 no release notes, no communication windows, no pre-release sign-off. Impossible to prepare regression cycles or validate what had changed.
No test documentation or feature specs
No test cases, no feature matrix, no test plans, and no recordings to explain new features. QA had to reverse-engineer the application to build coverage.
A rapidly evolving AI platform
Capitol AI ships features, deprecates old ones, and changes behaviour frequently across multiple client environments. Testing had to adapt continuously u2014 often with minimal notice
Test, document, advocate & in parallel.
Built test documentation from scratch
Created 251+ structured test cases across 13+ modules u2014 Login, Signup, Documents, Projects, Marketplace, Remix, Citations, Workflow Builder, Dashboard, Data Collections, Guardrails, LLM Playground, and AI Agents. A Feature Matrix mapped coverage across all functional areas.
Migrated defect tracking to Jira
Drove the move from informal Excel bug tracking to Jira u2014 introducing structured defect logging with clear descriptions, reproduction steps, evidence, and severity classification. Transformed defect reporting from noise into actionable intelligence.
Established release communication process
Through continuous feedback, drove a fundamental change in how releases are managed. Defined release windows, plans shared with QA in advance, weekly cycles with pre-release regression and post-release sanity.
Introduced feature documentation practice
Advocated for and achieved a new norm where developers create recordings and test plans before QA begins testing. Confluence is now the documentation hub u2014 replacing the previous state of nothing at all.
Structured Devu2013QA retest workflow
Implemented a clear process for how developers pick up, resolve, and return bug tickets for QA retesting. Reduced confusion, improved closure velocity, and made the Jira board a reliable source of truth across all environments.
Sustained high-volume manual testing
Delivered 4,373 test executions in a peak quarter u2014 spanning Dev, Staging, and Production across multiple enterprise clients (EY, Plexal). Daily regression, sanity checks, and new-feature testing executed within the allocated weekly hours.
Then vs. now. One practice, transformed.
The most significant outcome isn't just the volume of testing delivered — it's the structural transformation of how QE operates at Capitol AI, from unstructured beginnings to a traceable, documented, release-aware practice.
What the work delivered.
Defect tracking fundamentally transformed
From unstructured Excel to Jira u2014 with clear descriptions, severity classifications, reproduction steps, and a defined retest workflow developers now follow consistently.
Release process re-engineered
From zero advance notice to defined weekly release windows with QA involvement in pre-release regression sign-off and post-release production sanity.
251+ test cases built from scratch
Across 13+ modules u2014 a structured, expanding test asset library providing end-to-end coverage for all core product flows.
650+ defects raised across the engagement
High-quality documentation surfacing Medium and High severity issues that directly protected product quality across multiple enterprise deployments.
Documentation culture established
Developers now create recordings and Confluence test plans before features reach QA u2014 replacing the previous state of no documentation at all.
Multi-environment, multi-client coverage
QA now spans distinct enterprise client configurations (EY, Plexal, others), with tailored sanity and regression cycles per environment u2014 backed by QA Wolf automation collaboration.
The stack.
The Capitol AI engagement is a story of QE-led process transformation. The team didn't just test the product — they changed how quality engineering works at Capitol. From the Jira migration to release communication norms to documentation culture, every structural improvement was initiated by the QE team. The result is a quality practice that scales with a fast-moving AI platform and provides meaningful protection across every enterprise client it serves.