Digital government: problems on the horizon

The problems

The NAO’s library of reports highlights numerous problems over multiple decades, from the Home Office’s Digital Services at the Border to HM Revenue & Customs’ Making Tax Digital and the Government Digital Service’s flagship failure Gov.uk Verify.

Alongside the NAO’s compelling library of evidence, media and civil society reports reveal an interwoven and interdependent mix of policy and technical failures. They span everything from breaching the rule of law to the routine loss or alteration of data in mission critical national systems — a problem worryingly similar to the failed Post Office Horizon system.

Current examples include:

• Universal Credit: the digital system’s design breaches the rule of law, fails to properly implement legislation, and routinely leads to wrong amounts being awarded to claimants, often the most vulnerable, leaving some in acute hardship.
• Courts: staff in magistrates’ courts report that the flawed Common Platform routinely loses and alters case records.
• Home Office: the Atlas immigration system has left colleagues sobbing because official data can ‘evaporate’, with people logging in to find “photographs or visa details often relating to complete strangers attached to their records”.
• Ofsted: inspectors find the system crashes unexpectedly and loses all their notes from interviews.

Unlike the Post Office’s Horizon system, where supplier Fujitsu can be held responsible and pursued for compensation, it’s less clear who to hold to account for unreliability and poor performance when digital programmes are owned and designed by in-house teams. Yet as some of these examples demonstrate, in-house design and delivery is not necessarily any better at creating successful outcomes than more traditional contracting and outsourcing.

The causes

Public sector transformation relies on having the right leadership and team experience, capabilities, and culture in place — regardless of whether delivery happens in-house or is outsourced under government oversight.

However, appropriate skills remain in short supply, leaving government's modernising ambitions at continuing risk of failure. This makes it even more surprising that government recently published proposals that would have reduced its access to technical expertise by deterring smaller suppliers from bidding for contracts. Sensibly, these proposals appear to have been dropped after a supplier backlash.

The gap between the longstanding political aspiration for a digital transformation of the public sector and the reality suggests an enduring organisational and cultural challenge

The current problems being reported with government programmes appear similar to those unearthed in 2017 by internal assurance reviews, which found widely variable approaches to the quality of programme design and delivery. These included teams lacking practical experience of managing and designing scalable, secure, resilient systems. Some teams were found to be checking and approving their own code into production systems without any meaningful quality assurance.

One system presented a highly praised user interface and was successfully demonstrated to an enthusiastic minister. But it was all “smoke and mirrors,” lacking connectivity to any other systems. As remedial work was carried out to integrate the interface to existing departmental networks, systems, processes, and data, everything ground to a halt.

Fundamental design and engineering flaws caused the solution to become slower and slower and eventually lock up and hang. For a system intended to support thousands of simultaneous users across the UK in near real time, it was not fit for purpose.

Time and resources were also pointlessly displaced into hand-baking bespoke “special” infrastructure rather than consuming commodity services and products. Tactical design decisions would pass small-scale unit testing, but create bottlenecks and poor performance as systems were scaled up to meet real world demand. Solutions would be signed off as fit to enter service without any attempt to ensure they were legally compliant and not in breach of the rule of law.

This latter problem reflects the widespread practice of policy and digital teams inhabiting largely separate worlds. Yet policy and technology are complex, dynamic, and interdependent: they need to be co-designed — managed, developed, and adapted together to reflect and respond to an ever-changing political, legal, social, and economic landscape.

Problems of technical design adrift from policy and legal requirements aren’t a recent phenomenon. The Rural Payments Agency experienced similar issues in 2015 - the software locked up when multiple farmers (or even a single farmer using multiple browser windows) logged in. Legal deadlines were missed, fines were levied. Things got so bad that, according to the NAO, “in response to serious failings of the system, the online application system was withdrawn and replaced by ‘paper-assisted digital’ applications.”

The gap between the longstanding political aspiration for a digital transformation of the public sector and the reality suggests an enduring organisational and cultural challenge - a lack of capacity and capability in the leadership and management of large-scale policy initiatives and their associated design, development, and implementation. It’s a problem compounded by the absence of independent policy, legal, and technical assurance.

The fix

There’s insufficient data available to assess definitively whether in-house developed systems cause more, fewer, or the same level of problems as those implemented by external providers. But as the number of high-profile problems and failures make clear, without the right capabilities in place, bringing systems development in-house isn’t in itself a magical panacea.

Just as the forensic accounts Second Sight identified issues with Horizon, government needs an independent function to establish objective and ongoing policy, technical, and legal assurance of public sector programmes. It will need teeth to succeed, together with improved access to meaningful information to help analyse and identify programmes going adrift, or at risk of going adrift, to help flag and fix them quickly and effectively. And it needs the authority to terminate “no hope” programmes in a much more timely and effective way, rather than letting them limp on, undermining governments’ policy aspirations and bleeding precious public resources and credibility.

Although it may not be as glamorous and eye-catching as suggesting AI will somehow “revolutionise our public services for the better,” a cross-government, independent assurance function is essential to help support and reset digital government onto a better, and more successful, track. It will provide essential, baseline insight, exposing what’s going well and what isn’t, and helping inform decisions on where interventions are required to improve organisational culture, leadership, and skills to ensure better and more consistent results.

And, who knows? It might also prove an important step towards helping politicians deliver on their nearly 30 year old vision to use technology to modernise policy making, update our public institutions, and provide better outcomes for citizens and businesses.