Having spent the last three years working deep in the engine room of the public sector, from migrating legacy on-premise databases to the cloud, to implementing governance for RAG (Retrieval-Augmented Generation) models, I have come to a singular realisation about digital transformation in government.
The barrier isn’t talent. It’s the system we ask that talent to work within.
I have worked alongside some of the most intelligent, most ambitious people I’ve met in my career. These are civil servants who care deeply about their communities and are desperate to use data to improve lives.
Yet, despite this ambition, we constantly encounter the same hurdles: a "Waterfall" mindset masquerading as Agile, and governance structures that, while well-intentioned, can inadvertently paralyse innovation.
Here is what I’ve observed from the front lines in the public sector, and the practical changes we need to make to unblock delivery.
The “Agile” identity crisis in public sector delivery
We often start projects with the best Agile intentions: two-week sprints, stand-ups, and retrospectives. But all too often, the mechanics are Agile, while the mindset remains Waterfall.
I have seen minor changes to low-level ETL designs, changes that should take hours to implement and test, stall for days or weeks awaiting board approval in the public sector. This happens because leadership often fears the "feedback loop."
There is a pervasive worry about shipping code that isn't "100% finished."
The impact: this removes the quick development, ship, and feedback loop that Agile relies on. You cannot innovate if you are terrified of version 1.0. To fix this, we need to shift the metric of success from "Is it perfect?" to "Is it safe to learn from?"
The governance paradox in public sector data
One thing the public sector does better than any private organisation is security. Respect for data privacy is immense, and as a data engineer and architect, I will always advocate for that.
However, governance becomes a double-edged sword when it is applied as a blanket rule rather than a risk-based assessment.
I have worked on projects where penetration testing for extreme "edge case" scenarios in the public sector delayed production by weeks. The irony?
This delay didn't just hurt timelines; it actually hurt trust. Stakeholders, who aren't data experts, interpreted these delays not as "thoroughness" but as a sign that the product was inherently flawed or risky.
We often treat non-sensitive internal data with the same level of sensitivity as sensitive citizen PII. By trying to protect everything equally, we end up moving nothing efficiently.
The solution: From gatekeepers to guardrails
So, how do we unblock the pipes without compromising security? We need to move away from "Governance by Default" (checking everything manually) to "Governance by Exception."
This requires a two-pronged approach dealing with both Design and Implementation:
Phase 1: The design phase (harvesting the paved road)
The biggest delays often occur before a single line of code is written, as projects are stuck in the queue for a Design Authority or Architecture Board. The solution is a "Paved Road" of pre-approved patterns, but there is often resistance to the upfront effort required to create these standards.
The fix is to harvest patterns, not invent them.
Instead of writing a massive rulebook upfront, we should adopt a successful, approved project design as the standard.
The rule: If a new team copies that successfully deployed pattern, for example, the same ETL ingestion flow used by Team A, they get fast-track approval.
Maintenance: This keeps the standards "living." As technology changes, the first team to get a new, modern approach approved effectively "repaves" the road for everyone who follows.
Phase 2: The implementation phase (automated policy)
Once we are building, we need to stop relying on manual gatekeepers to check if code is safe.
Automated guardrails: Security and governance checks should be baked into the CI/CD pipeline. Instead of a person manually reviewing a deployment checklist, the system should automatically block any code that contains secrets, lacks documentation, or fails PII detection tests.
The Path to Faster Delivery
Fix forward: When the system (the guardrail) detects an error, the team can fix it instantly and redeploy, rather than waiting for a human reviewer to return from leave.
The priority for central government is clear: make services faster, safer, and more intuitive for citizens. We have the technology to do this. We have the AI tools to automate internal processes and the cloud infrastructure to scale.
The challenge now is to build an operating model that matches our ambition. We need to trust our teams to be Agile, and we need to right-size our governance so that it protects our citizens without preventing us from serving them.
