Avoid complexity in data storage decisions

A little while ago, I faced the challenge of migrating a digital asset management system to the cloud. As part of this, I had to migrate out terabytes of data on legacy hardware to a new provider. Add to this the fact that the storage hardware was based in a staffed office in a rural part of England with significantly limited internet speeds.

Ultimately, I managed to broker an agreement with a nearby datacentre to allow me to copy data physically to an external hard drive, plug it into a rented server in a third-party datacentre and then upload it to the cloud from the datacentre’s high-speed gigabit internet connection. Despite all the moving parts and security checks associated with accessing the datacentre, the plan worked surprisingly well.

Times have moved on and things have got easier. Nowadays, Amazon Web Services even offers a Snowball service in which it will physically send you hardware that you can load your data onto and send back to AWS for upload to the cloud. 

Cloud providers nowadays will offer many different solutions to seemingly the same problem. One example of this is hybrid cloud storage, which allows companies to simultaneously keep their data both in their own premises and in the cloud. When navigating through the seemingly ever complex landscape of new cloud products, it is important to bear simplicity in mind. Unnecessary complexity added now will simply result in greater headaches later on.

The remorseless pursuit of simplicity is a hugely advantageous trait for an engineer, but in many ways it flies in the face of human nature. In a recent study published in Nature, most participants favoured addition over subtraction when trying to solve a problem. For engineers, achieving simplicity rests in satisfying the business requirements without adding unnecessary complexity that makes future changes harder.

In the example of hybrid cloud storage, there are instances where such technology can be beneficial, for example where low-latency access speeds are needed on-site. Nevertheless, it remains essential to consider whether it is the simplest solution to meeting the requirements at hand.

Far too often, for different reasons, we adopt technologies that introduce far more complexity than we need, causing us headaches further down the road. By focusing on simplicity and fulfilling the business requirements at hand, we are able to build solutions that are better for both the business and for technologists.

Junade Ali is an experienced technologist with an interest in software engineering management, computer security research and distributed systems.