From Flaky to Flawless
Rebuilding a Cypress automation pipeline from the ground up — from 80% flaky tests to a fast, trustworthy CI signal.
Rebuilding trust in a Cypress automation pipeline.
The client is a leading demand and revenue management platform for the golf industry, trusted by over 800 courses nationwide, including many of the top public courses and multi-course operators in the country. Its product suite — waitlist management, booking confirmation, an AI pro-shop assistant, and lead management — helps golf course and resort operators automatically fill cancelled tee times, reduce no-shows, and deliver a better booking experience for golfers. The platform handles high-throughput, real-time automation across multiple chatbot integrations and AI-powered features, making quality engineering critical to product reliability and customer trust.
A case study in pipeline rehabilitation.
The numbers.
HIGH/MED FLAKINESS ELIMINATED
EXECUTION TIME REDUCTION
AUTOMATION SCRIPTS
CI PIPELINE DURATION
A fragile suite, a painful pipeline.
When the team joined, the Cypress automation suite was in a fragile state — the CI pipeline was slow, unreliable, and difficult to debug, making continuous delivery more painful than productive. Four interconnected problems needed solving.
Pervasive Test Flakiness
Over 80% of the Cypress test suite was flaky on arrival — including high and medium priority failures. Tests failed intermittently without code changes, making it impossible to trust CI results.
No On-Demand Execution
All Cypress tests ran only as part of deployment-triggered GitLab pipelines. The QA team had no way to trigger a test run independently — any manual validation required a code push.
No Failed-Test Rerun Mechanism
When tests failed, the only option was to re-run the entire suite from scratch. Without a Cypress Business plan, there was no built-in mechanism to rerun only failed tests — wasting CI minutes and slowing the development feedback loop significantly.
No Failed-Test Rerun Mechanism
When tests failed in a pipeline run, there was no way to rerun only the failing tests. The entire suite had to be re-executed, wasting time and slowing confirmation of fixes.
Slow CI Execution (12–15 Minutes)
The full Cypress CI job took 12–15 minutes per run. With frequent commits and pipeline-triggered execution, this delay slowed developer feedback loops and stacked up quickly.
Stabilise, then build for speed.
The team tackled each problem methodically — stabilising the existing suite first, then engineering new pipeline capabilities, and finally expanding coverage and introducing AI tooling to raise overall quality velocity.
Flaky Test Eradication
A systematic audit of all 70+ Cypress scripts identified the root causes of flakiness: back-to-back deployments leaving the server unresponsive, data collisions in parallel runs, and fragile locators. By mid-February, all high and medium priority flakiness was eliminated.
On-Demand GitLab Pipeline
A configurable manual trigger was built directly into the CI/CD pipeline, allowing the QA team to execute the full suite on demand against any environment or branch — without needing a deployment event.
Custom Failed-Test Rerun Script
Without access to a paid Cypress plan, a custom rerun script was written to extract and re-execute only the failed tests from a prior GitLab run. One click now reruns only the failures and reports results immediately.
Execution Time Reduction
Pipeline count was increased and Gmail session logins were made reusable through shared session storage — the full CI job went from 12–15 minutes to under 5 minutes, a 70%+ improvement.
Pipeline Failure Monitoring & Triage
A dedicated Slack channel was set up to receive all CI pipeline failure notifications in real time — the team monitored, classified, and reported on every failure, distinguishing genuine failures from environment noise.
Automation Coverage Expansion to 100+
New Cypress scripts were continuously added for new product features — including the AI Assistant, Lead Management, Analytics, Booking Engine, and Admin Dashboard — growing the suite from ~70 to 100+ scripts.
Audit, rehabilitate, enhance.
The engagement moved through three focused phases — a diagnostic audit that mapped every flaky test and identified root causes, a systematic rehabilitation phase that eliminated all high and medium priority flakiness, and an enhancement phase that delivered execution time improvements, coverage expansion, and the bonus Testberry LLM evaluation platform.
What the work delivered.
All high and medium priority flaky tests eliminated
from over 80% flakiness on arrival to zero high/medium priority failures, giving the team a trustworthy CI signal for the first time.
Full CI pipeline execution time cut from 12–15 minutes to under 5 minutes
a 70%+ reduction achieved through parallelisation, increased pipeline count, and shared session storage.
On-demand test execution capability delivered in GitLab
the QA team can now trigger the full Cypress suite against any environment or branch independently, without waiting for a deployment event.
Custom failed-test rerun mechanism built without a paid Cypress subscription
a bespoke GitLab-native script allows one-click rerun of only the failing tests, saving hours of re-execution.
Automation coverage grew from
70 to 100+ Cypress scripts, covering AI Assistant, Lead Management, Analytics, Booking Engine, Admin Dashboard, and additional modules.
A purpose-built LLM evaluation tool delivered
enabling structured A/B testing of AI chatbot accuracy with ground truth management, Slack integration, and a full run history dashboard.
Real-time pipeline failure monitoring via Slack
every CI failure was triaged, classified, and reported with root cause analysis and next steps, giving the client clear and actionable visibility.
The stack.
By addressing flakiness at its root, engineering custom tooling that worked within the client's existing subscription constraints, and delivering a purpose-built LLM evaluation platform on top, the team transformed a slow, unreliable CI environment into a fast, trustworthy foundation for continuous quality delivery.