Understanding a DevOps Toolchain: Use Cases and Fundamentals

DevOps pipelines or CICD pipelines are a series of tools combined to ensure rapid and reliable application delivery.  A toolchain is a combination of tools used to construct the pipeline, where each tool fulfills a specific task or function.

A well-designed DevOps toolchain ensures efficient and reliable code deployment and management throughout the lifecycle. Creating toolchain which is flexible enough to meet changing demands requires independent and modular components.

Typically, developers do not directly consume a toolchain. Instead, a pipeline tool such as Jenkins consumes the toolchains components. A pipeline is one or more steps (or stages); at each step executes one or more workflows to perform a specific task. Pipeline workflows consume a relevant component from the toolchain.

One or more pipelines should cover end to end coverage of the Software Development Lifecycle (SDLC). There are seven stages in the SDLC:

Plan

-- Project planning

-- Requirement gathering

-- Business workshops

Create

-- Software design

-- Code development

-- Unit testing

Verify

-- Acceptance testing

-- Integration testing

-- Security scans

Package

-- Approvals

-- Package creation

Release

-- Provisioning

-- Deployment

Configure

-- Post software deployment configuration

Monitor

-- Performance monitoring

-- Reliability monitor

-- Feedback gathering

Some stages in the SDLC such as testing and packaging will form part of an automated CICD pipeline, while stages such as feedback gathering require manual input.

You’ll quickly find that there are a plethora of tools to choose from when building a toolchain, making it challenging to select tools to use. Some tools are fantastic for a particular purpose, while others are good for multiple purposes.

A toolchain comprised of many small tools, each with a specific can be highly modular and flexible which is great. However, management overhead increases with the number of tools used. It is essential to ensure that the toolchain contains just enough tools to reliable and efficient code shipping.

I have seen two common toolchain management approaches: Shared toolchain which is managed by an automation team and toolchains managed by application teams.

Shared toolchains are common in companies who are using automation for Infrastructure as Code (IaC) and have either have small development teams or no in-house application development. Multiple teams consume the same toolchain as a shared platform. This approach creates a standard toolchain for everyone and typically has a lower management overhead.

In my experience companies who use a shared toolchain are more likely to buy into an SDLC ecosystem from a single vendor such as Atlassian. The ecosystem choice is usually based on lowering management overhead by leveraging native integrations between components; opposed to writing integrations manually.

Companies with larger application development teams typically have a CICD pipeline and toolchain solution for each team, which is managed by the team. This approach to managing the SDLC allows application development teams to create a process which is best suited for their needs.

Application teams might have a set platform that they need to consume based on the relationship between the company and vendors, or other company requirements. The application teams incorporate additional purpose specific tooling to cover any SDLC stage gaps in the standard platform.

Ensuring integration between various tools is a time-consuming task and sometimes difficult during DevOps adoption as there are many unknowns to work through. Validate integration levels between each tool and other tools and the various endpoint types in the environment.

Some tools provide the facility to expand integration levels using plugins to simplify the process. Manual integration is required when plugins are not an option, which is typically in the form of using an SDK or REST API.

Do not overlook the security impact when integrating different tools; make sure that you have a good understanding of mechanisms in place to secure stored credentials. Additionally, validate the how inflight traffic security between tools; Is TLS on by default? Is client authentication supported?

Toolchains are an essential part of automation and DevOps but need to be managed and should remain modular and agile. Ensure that you understand the problems that the toolchain is required to solve and then look at the tools which can best solve those problems.