Uncomfortable truth about agile transformations

Over the past few months, I’ve worked with an experienced clinical psychologist and a PhD-trained research team including a psychologist and an academic behavioural scientist to understand what separates the best-performing teams from those that struggle. Research shows that 81% of business decision-makers in the UK and 89% in the US are concerned about the on-time delivery of software projects in their organisations. 

This work has been done against a backdrop where the industry has typically focussed on improving the software testing process, one of the later stages in the software development lifecycle, to address these issues. Additionally, many companies have increasingly turned to studying the metrics of how software itself is written in order to attempt to improve the software engineering process. 

The challenges however seem even more deep-rooted than when code is either written or tested. 

When I wrote my last book, How to Protect Yourself from Killer Computers, I looked deeply into a number of catastrophic software failures, including aircraft entering ‘death dives’, fatal car crashes and killer radiation overdoses in hospitals. The book describes how psychological factors help us understand why software bugs snowball into catastrophic computer failures, however, an alarming recurring theme in many of the case studies is how the original technical cause of many issues was software design issues, or indeed the lack of robust design. 

This requirements-light approach was formalised in the Agile Manifesto, now over 23 years old, which advocated a software design approach centred around “responding to change over following a plan”. 

With the polling firm J L Partners, I asked 600 software engineers in the UK and US about successful and failed software development projects they’ve worked on. Our research found that in software projects where development started before clear requirements, where there was no complete specification document and where significant changes occurred late in development, projects failed 65% of the time. 

However, we identified just five engineering practices which reduced the failure rate to 10%. Having clear requirements before development started alone made projects 97% more likely to succeed and engineers having the freedom to discuss and address problems increased success rates by 87%. Other practices included: ensuring the requirements were accurate to the real-world problem (54%); having a complete specification before starting development (50%) and avoiding late requirements changes (7%). When combined, we refer to these practices as Impact Engineering. 

Interestingly, we did not find a statistically significant difference between those working on multiple projects concurrently versus those only working on one at a time, despite limiting work-in-progress being a key tenent of frameworks like Lean software development. This goes to highlight how early risks in software projects must be addressed. 

It was concerning to see that the largest difference in engineering practices between the UK and the USA was that software engineers in the UK were 13% less likely to feel they could discuss and address problems than those in the US. This is despite numerous studies conducted by me and others, including this research, highlighting how fundamentally important psychological safety is to the success of computer systems. 

During our research, we also looked into why transformation initiatives fail. Despite the promise of transformation methodologies, we still see 70% of digital transformations and 96% of agile transformations fail. Even amongst all companies regardless of industry, when a company enters administration in England and Wales (i.e. an insolvency practitioner takes over control), only 10% of businesses will survive despite the change in management. 

Our research has found that fundamental psychological factors play a determining role in whether such transformations succeed or fail. This has been shown in research by EY (Ernst & Young) and the University of Oxford’s business school which found that leaders who prioritise workforce emotions are 260% more likely to be successful in digital transformations. 

Much like when comforting a crying baby, it is essential to give emotional attention before addressing the actual problem. It is also important to ensure that emotional factors are not neglected in organisational transformations. 

Similar findings have been made by psychologist David DeSteno, who found that simply by study participants feeling the emotion of gratitude, they more than doubled their ability to delay gratification now for a bigger reward later. 

Through research like this, we developed a psychological framework enabling people to achieve significant personal and organisational transformations. These techniques are presented in my new book, Impact Engineering: Transforming Beyond Agile Project Management, which describes its application to real-world case studies alongside describing the scientific basis for these findings. 

However, I am keen to stress that it is vital that this research should not be abused to force others to change when they don’t want to. In many instances before, those advocating transformation have not given due respect to the absolute necessity for consent, both for individuals and organisations. 

The moral issues aside, depriving someone the ability to learn from their own mistakes deprives them of an important source of learning they can benefit from (to the extent they have informed consent to make those mistakes and they don’t pose an unmitigated risk to others). 

Additionally, as our research on agile has shown, those who have the highest conviction in their ideas can be mistaken, we should approach situations with an open mind regardless of how convinced we are that we are right. 

Nevertheless, by ensuring problems are well-defined before we seek to find solutions, and ensuring we design solutions before we start implementing them, we are able to improve the success of our software projects. Similarly, by ensuring we address emotional needs alongside solving problems, we are able to ensure that transformation initiatives are successful. 

Impact Engineering: Transforming Beyond Agile Project Management is now available on Amazon in eBook and paperback format. 

Dr Junade Ali CEng FIET is an experienced technologist with an interest in software engineering management, computer security research and distributed systems.