Best practices to beat container misconfiguration

Beltramini recommends IT leaders harness automation, encryption and native integrations, with all departments involved in authoring or changing shared code where possible. That means granular per process, container and pod policies, and detailed runtime configurations and resource accounting. Security profiles should be customised around system calls, file system access, binaries, libraries, capability restriction and so on. 

A systematic approach is needed to cover all the bases with this degree of complexity in the stack. For instance, just in each pod alone, considerations span images, applications, operating systems, users, any baked-in secrets and more, Beltramini says. 

Chris Jenkins, principal chief architect at Red Hat, also highlights thorough consideration of containers and their orchestration as well as what’s inside containers. Don’t forget that many security failures can come back to neglect of basics such as updating and patching, not to mention ill-considered use of private versus public keys.

Keep it simpler when you can 

Jenkins urges IT leaders to start clean, keeping it simple where possible. “You want to start by doing something right. Give them a [clean] base image on which they can start, and minimise pain,” he says. 

Jenkins urges software develpers only to use packages needed to run the application. For instance, if 50,000 packages are installed that are never used, they are likely to expose a vast number of potential vulnerabilities. Instead, Jenkins’ approach is to start with a blank page on which to base the application development work, where it is possible to understand the security status or health index of a specific image.  

“We produce base images every every six weeks,” he says. “Then we run code analysis to check all the code. Developers will also have dependencies of code which will call in other libraries.”

The final immutable bill of materials or recipe should equal and be verified against this specific image. If anything changes, if a clarification goes wrong or the key doesn’t match, then don’t deploy it, he says. 

Smaller organisations may obviously find this level of risk management challenging. It may be possibvle to look at splitting up all the tasks, for instance, by cycling through the different tasks on different days of the week, to get everything covered cyber security-wise. 

Plenty of different tools exist to help with this, particularly in the open-source arena.

Dinesh Majrekar, chief technology officer (CTO) at services provider Civo, says some tools can help organisations ‘shift-left’ on container security. They offer functionality such as scanning containers during the build and reporting into the continuous integration and continuous delivery (CI/CD) pipelines on known issues. Examples include on whether network ports are open, how libraries are defined, or whether a version of Java or Node.js is out of date. 

Giving developers continuous feedback is “really important”, Majrekar says. He recommends opting for least privilege and permissions, minimal installations and dependencies – although IT decision-makers need to balance this against debugging requirements.  

“You can make mistakes everywhere,” Majrekar adds. “Using Kubernetes, there are 100 different things you can change. You might not spot something you should have said ‘yes’ to which is currently set at ‘no’, and that isn’t necessarily the containers either, but the managed Kubernetes environment you’re running within. 

“You’ve got to now orchestrate changing the password and deploying your code into production at the same time, for example,” he says. “You need to use your experience as well in terms of secrets, storage and things like that instead. This comes back to a little bit of education, and I think specific security-focused training [such as certified Kubernetes security specialist (CKS)] and/or reading up.” 

Regular penetration and configuration testing can be key to reveal role-based access vulnerabilities and missing best practice, such as around network segmentation or secrets management – not least because the potential issues are constantly changing as well, he says. 

“Smaller companies might need to roll the dice more because of cost,” adds Civo chief executive Mark Boost. “You might sometimes just get things working, push them out, and before you know, it is out in production even though you didn’t get around to hardening it. So, the recommendation there as well is to just make sure you come back to it.” 

Organisations shoud double-check things for themselves. Do not simply leave it all to a managed services provider, assuming they’ve taken on that responsibility and all the security is handled.

“Containerisation can be massively powerful, but with great power comes great responsibility,” adds Majrekar.

Reduce public attack surfaces 

Crystal Morin, cyber security strategist at Sysdig, notes that “something like 66% of registries” are still public – not internal, not private, not vendor-managed. She says that these are often “just pulling something off the internet, throwing it into your infrastructure”. 

Scans are happening while updates are being pushed into the public-facing image, and the IT security team may be unaware it is happening. This is an issue Morin belives IT security and developer teams need to addresss

“Turning away from public resources for container images is going to reduce supply chain risk, and misconfigurations,” Morin adds. 

Shift left, shield right remains best practice: ensure enhanced security to begin with and continue protection on the back-end. Secure containers in production with real-time threat detection and policies for timely alerting and response, including automation of incident response. 

She notes that without automation, time, capacity or capability can be in short supply. 

“Cloud-native security focused organisations are realising now that security is an organisation’s problem. Everyone needs to be aware of what they do and how they can help,” Morin says. 

Integrating security across the entire organisation also delivers value to all users and ultimately to every customer. 

“If I go in and talk to others about security, they might understand maybe 40%,” Morin says. “We need to collaborate better across the business, coordinating together, not just as the technical nerdy guys behind the scenes. Shift left, make development more secure. And that’s obviously a much bigger picture than just containers.”