Improving DevOps Security Practices

When writing code, it’s essential to ensure the security of sensitive information such as passwords. The best time for DevOps engineers to implement proper security hygiene is before writing code; the worst time is after the code has been deployed.

Coding efficiency is gained through reusable modules. But if modules are written in an insecure manner, the security of any workflow using them is compromised. The more critical a module is, the harder it is to correct flaws after the fact.

It’s easier to build a solid foundation than to correct weaknesses after the house is built. In this article, I'll look at some of the places that security can be added during the development cycle.

Project workshops

Project workshops often focus specifically on the task which the workflow performs, with security discussions taking a back seat. The result is a project which does not take into account security complexities.

As a task is broken into modules, you can identify external system interactions. Now is the time to start raising questions about credential and access management, and also check for misuse of credentials. Here are some questions DevOps engineers should ask.

    • Where are the credentials stored?

    • How are certificates stored and secured?

    • What level of access is actually needed to interact with that system?

    • What is the audit process?

    • What is the company policy on security?

    • What sort of data is stored on that system?

Raising these questions early allows them to be placed as action items to be followed up throughout the project.

code-security.jpg

Security practices from the first line of code

During the initial build phases, the use of plain text passwords in code is almost guaranteed. However, even in test environments, this should not happen. Validating connection to a remote system at the very beginning is fine, but the next immediate step should be to test with proper security.

You might argue that you’ll correct the use of plain text passwords later. In my experience, either the project timeline prevents this from happening, or it’ll end up in the "too hard" basket.

There are times where a system does not have secure authentication methods. When you encounter this, don’t just add it to the risk register and accept the status quo. Raise the issue with the vendor and take required action where possible.

Credential management

Credentials should never be stored in plain text, nor should they be hard-coded into scripts. The use of credential management systems dramatically improves security. But it’s important to understand how securely stored credentials are.

There are many factors which need to be considered when accessing a credential management system. The list below covers many considerations, but is by no means complete.

    • What is the credential encryption mechanism?

    • Are credentials encrypted at rest, in transit, or in memory during execution?

    • How can the credential be rotated?

    • What does the interaction method look like?

    • How does the process fit within an operational model?

    • Is two-factor authentication supported?

    • Which external authentication systems are supported?

    • Failure scenarios and recovery options

The purpose of automation is to reduce human interaction and credentials have to come from somewhere. Answers to the above questions help to properly identify credential management and risks.

Watch your logging

Effective management and transport of credentials can come undone if logging is incorrectly configured. A recent example of this happened in May, when Twitter warned users to change their passwords after the company found that account passwords were being captured in its logs.

You will need to understand the logging mechanism beyond your own code and the data it captures. Commands run on a remote system could be captured by auditing tools or shell history.

The principle of least access

The principle of least access is providing accounts with only the access rights required to complete a task and no more. The goal is to limit the potential for impact if an account is compromised, not just between systems, but within a system as well.

Detailed discovery and validation of the permissions requirements takes time, but is required for successful implementation. This does result in higher operational overhead due to more sets of credentials that have to be managed. Credential management systems can mitigate some of the overhead.

Self-service deployment credentials

If you’re providing a self-service portal for users to deploy their own VMs or instances, credential management is a crucial consideration. In a self-service scenario, users are likely to require local administrative rights.

While users are likely to need local administrator rights, they should never have credentials to root accounts, nor the ability to modify those accounts. The same applies to any infrastructure service accounts.

Unfortunately, it is too common that users do not change default passwords after a system is deployed. Without initial random passwords, you’ll quickly find the same password used across many systems.

The steps given in this article are just the starting point for securing DevOps practices. You need to give a lot of thought to security from the beginning all the way through the end of an application's life in order to ensure that your environment isn't put at risk.