A nine-month embedded Solutions Architect engagement, building an AI-powered incident analysis platform for over 2,000+ ServiceNow incidents, helping a global business services firm move from reactive firefighting to systematic quality improvement, cutting developer triage time by 80%.
The Call
It was supposed to be straightforward. A global business services firm needed help with their offshore development team. They had spent over £2 million on a legacy platform replacement. They were months behind schedule. Features that should have taken weeks were taking months. Production incidents were piling up. And nobody could quite explain why.
The Director of Engineering wanted a Solutions Architect to come in and run a diagnostic. “Come in, spend a couple of weeks, tell us what’s wrong with our delivery practices, give us some recommendations.” The classic diagnostic engagement.
I ended up staying for nine months.
This is the story of what I found, what I built, and why it took nine months to properly understand what was really going wrong.
What They Thought the Problem Was
The firm were 2-3 years into replacing a legacy low-code order management platform. The new system was live, handling real customer orders, generating real revenue. But something was deeply wrong.
The symptoms were obvious:
- Features taking 3-4 times longer than estimated
- Production incidents occurring weekly
- Multiple teams unable to deliver reliably
- Releases that everyone was afraid to deploy
- A growing sense that “we don’t know what we don’t know”
Their hypothesis? “Our requirements aren’t landing. The offshore team doesn’t understand what we want. We need better delivery practices.”
It seemed reasonable. I came in to assess the delivery function across five teams working on the platform.
Phase 1: The Initial Assessment (Weeks 1-4)
I started where you’d expect: observing how requirements were captured and refined, reviewing documentation, sitting in on refinement sessions, talking to developers and testers.
What I saw confirmed some of their concerns, but also revealed warning signs of something deeper.
The Knowledge Bottleneck
I watched a refinement session with 10 people: the lead analyst, five developers, three testers, and the Product Owner. It lasted eight hours.
One analyst told me: “Stories live and die with me. If I’m on holiday, nothing moves. If I’m sick, the team waits. They can’t start without me explaining everything.”
Another said: “I write the acceptance criteria, I explain them in refinement, I answer questions during development, I clarify during testing, I explain again during UAT. I’m involved in every conversation about every story.”
These weren’t bad analysts. They were experienced, knowledgeable people who had become single points of failure because the system forced them to be.
The Test Automation Debt
The test automation story was worse. One team had 1,500 manual regression test cases. An automation team (separate from the product teams) was working through them, automating where possible. They’d managed to automate about 50% after months of effort.
But here’s what nobody was tracking: while they automated 750 tests, the product teams had added 400 new manual tests. The backlog was growing faster than they could automate.
QAs were writing test scenarios in single-use Excel workbooks, with no version control and no clear link to what code they were testing. Manual regression testing was taking 50+ hours per release.
The Hidden Pattern
But there was something else. Something that did not fit the “delivery practices” hypothesis.
Production incidents were happening every week. Not edge cases or rare scenarios. Core functionality was breaking. Orders getting stuck. Users unable to complete basic tasks. The same problems recurring weeks and months apart.
The support team was firefighting. They had developed workarounds for common issues. They knew which problems needed which manual interventions. But nobody was analysing why these incidents kept happening or whether they were related.
When I asked to see incident data, they pointed me to ServiceNow. “It’s all in there,” they said. “We log everything.”
I looked. They did log everything. Over 2,000 incidents in the past four months alone. Unstructured text descriptions. Inconsistent categorisation. No obvious grouping or pattern analysis.
This was the moment I realised: this was not a requirements problem. It was a quality and release-confidence problem hiding inside the incident data.
The Extended Engagement
I delivered my initial findings after the first month. The report covered the surface issues: acceptance criteria quality, knowledge bottlenecks, documentation practices, and team collaboration problems. All accurate, all actionable.
But I also included something else: a proposal for deeper work.
The organisation lacked confidence in release readiness. When asked whether upcoming releases were safe to deploy, answers were qualified, uncertain, or based more on hope than evidence.
The Director of Engineering was direct: “We don’t trust what we’re being told. We can’t prove what we can’t see. When someone says a release is ready, I don’t know if that’s true or just hopeful.”
She approved the extended engagement. My brief evolved: help the organisation understand whether their releases were safe, and build the capability to answer that question confidently going forward.
Phase 2: Building the AI Platform (Months 2-4)
The incident data in ServiceNow was the key. It contained the truth about what was actually breaking in production. But the sheer volume and the unstructured nature of the data made human analysis impossible.
117 incidents for a single problem. Hundreds more for other issues. Spread across five teams, multiple services, different parts of the platform. Written by different support staff with different levels of detail.
If only we could analyse this data systematically, then we could:
- Identify which problems were actually the same problem
- Understand their real business impact
- Prioritise what to fix based on risk, not just noise
- Give developers and testers the information they needed to fix things properly
So I built an AI-powered incident analysis platform.
What the Platform Did
The tool ingested large volumes of ServiceNow incidents and:
- Automatically grouped related incidents - Using NLP and pattern-matching over the unstructured incident text, the platform identified when multiple reports were actually describing the same underlying problem
- Applied risk scoring - Against a framework I developed (more on this below) that weighted factors like user impact, recurrence patterns, detection methods, and mitigation complexity
- Generated summary reports - Showing all problem groups, their relative risk scores, and impact across the platform
- Produced detailed problem reports - For each problem group, these reports provided the analysis developers and testers needed: patterns, affected users, workaround effectiveness, root cause indicators — everything needed to take a problem from “this keeps happening” to “this is the fix”
Under the hood, the platform was C# .NET on Amazon Bedrock with Claude Sonnet as the core reasoning model. Development used Claude Code in a spec-driven loop with AI code reviews. The RAG layer worked over the ServiceNow incident corpus directly: no manual labelling, no fine-tuning, no months of preparation. The platform produced its first useful problem groupings within two weeks of starting work and improved continuously from there.
Why It Mattered
The platform made the invisible visible. And it cut developer triage time on those problems by 80%.
Take one problem I’ll call “the missing submit button issue” - 117 incidents logged over four months. Each one handled by support staff with a manual workaround. Nobody had done the analysis to see this was a systematic failure.
The platform’s analysis revealed:
- The problem was specific to users with “Access group” permissions
- It consistently occurred at the same workflow transition point
- Every incident required the same manual intervention
- A development fix existed but wasn’t prioritised for release
- This single issue was consuming significant support capacity
The platform identified dozens of problem groups like this. Issues that had been generating incidents for months, consuming support time, frustrating users, but never making it to the top of the priority list because nobody could see their cumulative impact.
The Risk Scoring Framework
The platform needed a way to score problems objectively. I developed a framework with 10 weighted criteria:
High Weight (3x):
- User Impact & Business Severity
- Detection & Reproducibility
- Mitigation Effectiveness
- Resolution Timeline
Medium Weight (2x):
- Observability Coverage
- Resource & Performance Impact
- Root Cause Analysis Status
- Recurrence Pattern
Low Weight (1x):
- Knowledge & Documentation
- Customer Communication
Each criterion scored 1-5 (higher = greater risk). The weighted calculation produced an overall risk score that helped Product Owners make informed decisions about what to fix and when.
This wasn’t theoretical. The scores directly influenced priority and budget decisions across the five teams.
Phase 3: The Hothouse Team (Months 5-6)
Having a platform that identified problems was one thing. Proving it could drive actual fixes was another.
We established a “hothouse team” - a pilot to test whether the AI platform’s analysis could accelerate problem resolution.
The Setup
Two experienced developers, one Product Owner, and direct access to the platform’s output. We selected one problem group from the risk-scored list - not the highest risk, but one that had been causing persistent issues for over 18 months, with several hundred logged incidents.
The hypothesis: if the detailed problem report gave developers everything they needed (affected users, failure patterns, reproduction steps, suspected root cause), they should be able to fix it faster than the traditional approach of reading individual incident reports and piecing together the pattern themselves.
The Result
One sprint. The problem was fixed.
An 18-month-old issue that had generated hundreds of support incidents, resolved in a single two-week sprint.
The developers later told me: “We spent 80% of our time in the past trying to understand the problem. The platform gave us that understanding on day one. We spent our time actually fixing it.”
This proved the model worked. The AI platform wasn’t just identifying problems - it was making them solvable.
Phase 4: The Real Problems Emerge (Months 6-9)
With the platform deployed and the hothouse team proving the concept, I spent the next several months embedded across the five teams, using the incident analysis to understand systemic patterns across the platform.
By month six, I was presenting findings to the Technology Director and the senior team. By month eight, I was having detailed calls with technical leads about what I was discovering.
The picture that emerged was not about delivery practices anymore. It was about systemic engineering problems that had accumulated over years of pressure to deliver.
The “Top Nine Things”
By the final months of the engagement, I had synthesised everything into nine critical problems that needed addressing:
Build:
- Code could be deemed “done” and released without unit tests. The pressure to deliver meant testing was treated as optional, creating technical debt that showed up later as production incidents
- Engineering best practices were undefined. No shared standards for what “quality” meant, no continuous improvement culture
Test:
- QAs wrote test scenarios in single-use Excel workbooks. Saved to SharePoint with no version control, no traceability, making it impossible to know what was actually being tested
- Manual regression test scenarios continued to grow. While automation was happening, new manual tests were being created faster, making the problem worse over time
- Automation tests were developed after release by a separate team. Not part of the critical path, meaning features were released without automated coverage
Metrics:
- Unclear how team initiatives related to labour cost of release. Teams were improving local metrics without understanding their impact on the broader cost structure
- Unclear how improvements related to service desk staffing. Nobody could see the connection between quality improvements and support load reduction
- Hard to track key release metrics over time. No longitudinal view of release date, feature volume, defect volume, SDLC stage, automation ratio, labour costs
Tech Debt:
- No plan to substantially reduce manual test effort. Just incrementally automating wasn’t enough; a strategic approach was needed
These weren’t individual problems. They were interconnected symptoms of an organisation that had optimised for feature delivery at the expense of sustainable quality.
What Changed
By the end of the nine-month engagement, several things had shifted:
Tactical Improvements
- The AI incident analysis platform was operational and being used by Product Owners for priority decisions
- Risk scoring was being applied to new incidents as they were logged
- The hothouse team model had proven that systematic problem analysis could dramatically accelerate resolution
- Several teams were showing measurable reductions in manual test effort
- The automation team had made progress (50% of regression tests automated), though the backlog continued to grow
Strategic Understanding
- Leadership now had language to discuss quality problems (“Can we release with confidence?” became a standard question)
- The “Top Nine Things” provided a roadmap for systematic improvement
- The connection between incident volumes, support costs, and underlying quality was now visible
- Teams understood that cross-functional metrics were needed
Cultural Shifts
- Developers in the hothouse team had experienced what “good” looked like
- Product Owners had objective data for priority decisions rather than relying on whoever shouted loudest
- The conversation shifted from “why are releases taking so long?” to “what quality standards should we maintain?”
The Lesson
This case study demonstrates something important about complex organisational problems: you can’t diagnose them in two weeks.
The initial assessment found real surface problems. The recommendations would have helped. But they would have been tactical improvements to a system with systemic issues.
It took:
- Four weeks to see the surface problems
- Six months to understand the deeper patterns
- Three months to document and synthesise the findings
- Building actual tools to make problems visible
- Establishing pilot teams to prove solutions worked
- Embedding across five teams to see the full picture
Regulated enterprise environments are risk-averse and complexity-heavy. Legacy platform replacements take years. When something is going wrong, it is rarely one thing: accumulated decisions, the technical debt a forensic assessment can quantify, organisational structure, and incentive misalignment.
Quick assessments can find symptoms. Deep engagement finds causes.
The AI platform now automatically processes thousands of incidents. The hothouse team model provided a blueprint for how other teams could tackle their highest-risk problems. The “Top Nine Things” gave leadership a framework for investment decisions.
But more importantly, the organisation now knows what they don’t know. They can see the problems. They can measure improvement. They can answer the question, “Can we release with confidence?” with data rather than hope.
That took nine months. It was worth every week.
Technical Details
AI Platform Stack:
- C# .NET for core processing
- Amazon Bedrock with Claude Sonnet as the reasoning model
- Claude Code in a spec-driven development loop with AI code reviews
- RAG over the ServiceNow incident corpus (unstructured data analysis, no labelling or fine-tuning)
- ServiceNow API integration for data ingestion
- Automated report generation (summary + detailed problem reports)
- Risk scoring engine based on 10-criterion framework
Teams Covered:
- 5 product teams across the order management platform
- 2,000+ ServiceNow incidents over a 4-month period analysed
- Multiple backend services and workflows
- Mix of direct-to-consumer and B2B functionality
Engagement Timeline:
- Month 1: Initial assessment delivered
- Months 2-4: AI platform development
- Months 5-6: Hothouse team pilot
- Month 6: Mid-engagement presentation to leadership
- Month 8: Deep-dive technical discussions
- Month 9: Final “Top Nine Things” synthesis and recommendations
Outcomes:
- 18-month production issue resolved in a single sprint using AI tool analysis
- 80% reduction in developer triage time on prioritised problem groups
- Measurable reductions in manual test effort for several teams
- Risk scoring framework adopted for incident prioritisation
- Systematic problem identification replacing reactive firefighting
Specifics are accurate to the engagement; the client is referred to here as a global business services firm to financial institutions.
This is what becomes possible when an AI-powered analysis layer is built over production data that no one has time to read. If your organisation has a regulated enterprise system in this shape, agentic AI techniques can turn that data into prioritised work in weeks rather than years.