What Is CI/CD?
Continuous integration (CI) and continuous delivery or deployment (CD) are the main components of a CI/CD pipeline. CI is a software development process that involves frequent, incremental code changes in a reliable pipeline. The pipeline consists of automated steps to build and test code changes and ensure they are safe before being merged into the code repository.
The CD process is the next part of the pipeline, which delivers the code to production quickly and seamlessly. Software developers often refer to the CI/CD pipeline as the automated steps that allow them to push incremental code changes to the main production branch.
In this article:
- What Is CI/CD Security
- CI/CD Security Threats
- Securing the CI/CD Pipeline with Automated Tools
- Best Practices for Securing a CI/CD Pipeline
What Is CI/CD Security?
Typically, the main stages of a CI/CD pipeline do not address security. Development teams often build, test, and deploy applications without ensuring security at every CI/CD process stage.
However, this failure to incorporate security into the CI/CD exposes you to various risks. Your application might be inherently insecure, or you might waste time trying to fix security bugs later in the software development life cycle (SDLC). Security should be an integrated part of the software delivery workflow, not a separate process.
A growing DevOps trend that addresses this issue and makes it easier to detect and resolve security issues efficiently is DevSecOps. This approach integrates security into the core of DevOps, encouraging collaboration between developers, engineers, and security teams.
Integrating security into the CI/CD pipeline usually involves incorporating safeguarded and security checks at each pipeline stage:
- Scan rules and templates—another way to ensure security is to scan IAM rules and infrastructure as code (IaC) templates for vulnerable configurations in the environment that hosts your application.
- Scanning and analysis—developers can scan their source code for known security vulnerabilities as they write it. The DevOps team can use source composition analysis (SCA) to find security bugs introduced to the application through a third-party library or code.
- Security testing—you can run security tests together with application performance tests before deploying each new build. Security testing helps identify misconfiguration issues and vulnerabilities to various attack vectors.
- Monitoring security at runtime—the team should continue security testing and monitoring after deployment to identify and respond to new threats in production.
CI/CD security aims to detect and mitigate vulnerabilities efficiently to keep up with your overall CI/CD workflows. Robust CI/CD security allows teams to find and address security issues with minimal impact on the pipeline’s flow to prevent delays and rollbacks.
CI/CD Security Threats
Here is a list of common CI/CD security threats:
1. Insufficient Flow Control Mechanisms
A lack of effective mechanisms to control flow allows attackers with system permissions to push malicious artifacts and code into the CI/CD pipeline. These controls should protect artifact repositories, CI tools, and source code management (SCM) systems to enforce security, for example, by requiring additional review or approval steps.
2. Weak Identity and Access Management (IAM)
Weak IAM policies and controls expose the CI/CD system to attack. IAM risks are common, given the challenges of managing large numbers of identities across distributed environments. If the system has inadequately managed identities, including programs and human users, it is at risk of compromise.
3. Dependency Chain Exploits
This threat involves attackers exploiting flaws in how development environments and engineering workstations retrieve code dependencies. Dependency chain exploits can cause the system to fetch and execute malicious packages locally.
4. Poisoned Pipeline Execution (PPE)
PPE vulnerabilities allow attackers to access source control systems (but not the build environment), where they can manipulate the development process. For instance, attackers can inject malicious commands into build pipeline configurations to “poison” the pipeline. The malicious code executes as part of the CI/CD process.
5. Insufficient Pipeline-Based Access Controls (PBAC)
An execution node can access various systems and resources across the pipeline execution environment and beyond. Attackers can run malicious code in a pipeline by leveraging weak PBACs and abusing the pipeline’s access permissions to move laterally through the CI/CD environment.
6. Improperly Managed Credentials
Poor credential hygiene allows attackers to steal and exploit secrets and access tokens across the CI/CD pipeline. This risk usually results from poor secret management, excessive permissions, weak access controls to protect credentials, or insecure credential storage.
7. System Misconfiguration Issues
Inadequately secure system configurations include vulnerable security settings and insufficient hardening of various pipeline systems (i.e., CI, SCM, artifact repositories, etc.). These insecure configurations provide low-hanging fruit that attackers can exploit to infiltrate the CI/CD environment.
8. Ungoverned Third-Party Service Usage
The attack surface of a CI/CD includes the organization’s internal assets (e.g., CI and SCM) and third-party services with access to these assets. Many organizations provide easy access to third parties to use their critical systems, introducing a risk of ungoverned access and increasing the CI/CD system’s attack surface.
9. Inadequate Validation of Artifacts
Failure to properly validate the integrity of artifacts is another risk that allows attackers to access CI/CD systems. Without proper artifact and code validation mechanisms, attackers can push malicious artifacts into the pipeline undetected.
10. Insufficient Visibility and Logging
A lack of visibility allows attackers to execute malicious actions in the CI/CD pipeline and evade detection. Likewise, insufficient logs make it harder to identify and track attacks or investigate security incidents after they occur.
Securing the CI/CD Pipeline with Automated Tools
Source Composition Analysis (SCA)
Source/Software Composition Analysis (SCA) scans Open-Source Software (OSS) by analyzing the codebase. It is used for security and license compliance as well as for risk management, all by automation.
SCA tools scan source code, binaries, packages, manifest files, containers and all such components. These details are included as part of the Bill of Materials (BOM) which is further cross-checked against the National Vulnerability Database (NVD) and similar security databases. Shifting left through SCA is a very efficient way to optimize the security and quality of a code.
SCA can include the following processes:
- Inventory—an elaborate inventory of all open source components is maintained for thorough analysis.
- Security analysis—SCA analyzes the open-source components for all known security vulnerabilities by scanning the open-source libraries and where your code calls these libraries, and how to fix any issues with them.
- License Compliance—details about the licenses of all open source components are scanned to verify their credibility and also compare their compatibility with organization policies.
- Policy compliance—SCA tools can also perform security policy enforcement so as to ensure its compliance within the workflow.
Security Unit Tests
To strengthen the overall security of your CI/CD pipeline it is essential that each individual component is secure by itself before any integration. The best phase to ensure this is in the development stage.
Through Unit Level Security Testing developed by security engineers/analysts and verifying the security compliance of those tests before any integration or code commit, all component-level security can be properly managed.
Static Application Security Testing (SAST)
SAST is a testing method using which source code is scanned for any security vulnerabilities that is done before the code compilation. AKA White Box Testing, this testing does not need any code execution and can be done at the very early stages of SDLC, providing developers with real-time feedback on their code.
SAST limitations:
- SAST leads to an abundance of false positives
- Increased language dependency
- Unlike APIs or REST endpoints SAST is poor at interpreting frameworks or libraries
Dynamic Application Security Testing (DAST)
DAST is the testing process that scans the security of an application while it’s in the process of execution. AKA Black Box Testing, DAST tools perform automated scans, simulating various attacks to detect vulnerabilities and areas of improvement.
DAST limitations:
- DAST does not scan the code or detect any security vulnerabilities in them.
- DAST tools are usually used after the development stage, so improving the code, later on, becomes a challenge.
- Generates more false positives.
Interactive Application Security Testing (IAST)
IAST tools combine SAST and DAST techniques, overcoming both their drawbacks. IAST tools work dynamically to detect issues during application execution but it operates within the application server and scans the code like SAST. IAST tools are highly functional during the QA stage working alongside functional testing.
IAST limitations:
- Tools are mostly proprietary which creates a supplier-dependent support system.
- Language and technology support is quite limited.
- Contrary to SAST, IAST does not scan each line of the code.
Best Practices for Securing a CI/CD Pipeline
1. Threat Modeling
A key element in CI/CD of a DevSecOps cycle, threat modeling is the process of detecting all potential threats, security issues and vulnerabilities, and based on those, defining techniques or countermeasures to prevent, tackle and mitigate them.
There can be 6 steps in Threat Modeling:
- Defining Security Objectives
- Asset and dependency identification
- Attack Surface and Attack Vector Analysis
- Security Controls
- Threat identification
- Threat mitigation and validation
2. Establishing Secure Authentication
Authentication is the process of validating the identity of a user, device or application. As this is usually the entry point to an application, device or process, it is also the most targeted aspect by an attacker. Like an intruder would try to exploit the entrance to your home first, the same way a lackluster authentication process in place can easily be bypassed by malicious attackers. Attacks like brute force, dictionary attacks, password spraying, and credential stuffing, are mostly preferred by attackers.
Access control is a security method that determines who is provided with what access and to which degree. Through this, you can control all access privileges of users, processes and applications. So if elevated privileges are provided to unessential resources, this creates a huge backdoor for attackers. Just by bypassing authentication of a low-level user/API with high access privileges, all data and systems can be compromised.
Broken Access Control and Authentication Failure are still part of the OWASP Top 10 list, which shows how these factors are so often overlooked. Methods like Multi-Factor Authentication (MFA), strict credential policies, approval and need-to-know based access privileges, and application vulnerability scanning can help a lot to maintain proper security.
3. Credential Management
Secrets such as passwords, tokens, keys must always be stored in an encrypted format to prevent any leaks. An attacker can breach the CI/CD pipeline through a backdoor, and gain access to the exposed credentials of an admin-level user.
Most organizations prefer using a Credential Management System (CMS), which is used to issue and manage credentials within the public key infrastructure (PKI). Through its integration with other entities, it can handle the credentials of all devices, applications, users, and processes in your workflow.
4. Network Security
With your CI/CD pipeline, it is equally important to harden the security of your network. If your network is insecure or has risks in any form, all your security efforts in CI/CD will become worthless. Network security is an umbrella term that covers various areas related to Firewall, network isolation, VPN, Email security, Data Loss Prevention (DLP), Sandboxing, Intrusion Detection System (IDS) & Intrusion Prevention System (IPS) and so on.
5. Vulnerability Scanning
Vulnerability scanning is the procedure to detect security issues in your systems and applications. Through this, your security team can assess probable security vulnerabilities, how they could be exploited and how to remediate those issues. All systems and applications are thoroughly analyzed and then also compared with standard cyber security databases to cross-check compliance and vulnerabilities, like the CVE database.
Tools like Nessus, Nmap, Nexpose, OpenVAS, Wireshark, Burp Suite can be good options for vulnerability scanning.
6. Observability
Monitoring is used for capturing and displaying data usually in a visualized format, while observability is using metrics to determine the system’s health and analyze issue root causes. In basic terms, monitoring shows what is wrong in a system, whereas observability also shows why something is wrong in it.
7. Monitoring
A subset of observability, monitoring is an essential factor in maintaining the security of your CI/CD pipeline. The key areas you should be monitoring include system resources, network traffic, application health, individual components/tools, application delivery status.
You can use open source monitoring tools like ELK, Zabbix, Nagios, Prometheus-Grafana. As these are widely used in the CI/CD community, any issues can be resolved through their extensive official and community support.
8. Logging
It is quite essential to keep a track of all the records of all events, issues, access and other activities in your system. This will not only help you with the retrospection process but also with log analysis you can narrow down the cause of any issue and immediately look for its resolution. In case of any security breaches, through log analysis, the security team can figure out all user, authentication and access control activities.
9. Alerting
Usually, most monitoring systems have a default feature to alert whenever something goes wrong. Based on your specific requirements, parameters and conditions can be configured and when an event triggers your predefined condition or a preset threshold is crossed, an alert will be sent to configured users/applications.
Excessive alerts can lead to alert fatigue, wherein, amongst multiple irrelevant alerts or false positives, a genuinely concerning alert gets missed. Having multiple alerts spamming your notification channel regularly will only lead to alert fatigue, so beware of that while configuring alerts.
10. Auditing
Performing security auditing can reap benefits since during the auditing process, security engineers can analyze and retrospect all security related issues and events, their root causes, how it was mitigated, and what steps can be taken to prevent such mishaps in future.
11. Remediation/Mitigation
If you ever face a security breach in your CI/CD pipeline, you and your team should be ready with a defined set of actions that each one needs to perform. Creating and maintaining security and incident response SOPs (Standard Operating Procedures) are immensely critical when it comes to dealing with such situations. To avoid escalating it into a chaotic and panic-inducing situation, properly-outlined steps and processes that can be done to mitigate the issue should take the first priority. A few steps you can take are: all admin-level access to users should be temporarily revoked, logs should be traced continuously, use your IDS, and all systems and applications should be virtually isolated for deeper analysis.
Using all the data and experience, remediation actions can be set as part of SOP, to ensure not only such events or issues are never repeated in future, but also how to use the data to prevent similar attacks and safeguard your systems.