CI/CD pipelines exist in just about every company that does some kind of development. Some companies have more mature pipelines than others, but the rule still holds. Most companies with in house development have a Jenkins instance or some similar build orchestration software, code repositories, path to deployment, etc. Depending on your background you may be familiar with these environments to varying degrees. I started out doing somethin between running application servers and devops, maintaining deployment pipelines and helping developers troubleshoot issues. I even wrote my own share of code and deployed my own changes to a minor application that eventually went into production. This was done as an opportuntiy to get me more familiar with that side of things and I found the experience very valuable. Those experiences became invaluable when I went into red teaming and security research as I already had a baseline familiarity with these environments and how they worked, and more importantly, insight in how these environments could be exploited by an attacker.
From a security perspective, CI/CD environments are kind of a soft underbelly that got very little attention until recent years. CI/CD pipelines are often maintained by the developers using them. Their priority tends towards getting working code and features shipped to production. This often means CI/CD environments are bolted together as needs become apparent, some companies have a holistic process or setup, but that doesn’t mean the developers using them aren’t just trying to get their code pushed. This means security takes a back seat to functionality as it often does.
It’s important to note that there is usually always controls on CI/CD environments, but they had far more to do with change control and who can make what changes to code, or deploy to which environments, more or less controlling which employees can push code and make sure there was some kind of second set of eyes reviewing and approving changes. Very little of the controls were concerned with external threats. It was not uncommon to find credentials in code, source code readable by a large number of people, and developers had free range to change and customize the build processes.
Jenkins - RCE is a feature not a bug
Jenkins is a common avenue of attack for adversaries and often flagged in pentests. That’s because by default Jenkins is pretty insecure and lets you run code on the host as a feature. The intention is that you can run scripts or processes to automate your build processes, but anyone who gains access to the host can use it to execute their own code, gain access to the build pipeline and steal credentials that are often stored in Jenkins to be used to access code repositories, or push builds out, etc. This often includes powerful service accounts which may have overly broad access to the build pipeline. This can be done by an attacker by either creating their own build process or gaining access to the script console if this isn’t restricted to use only by admins. Even with limited access to Jenkins some information such as build process logs can often be a good source of information, and may even sometimes leak credentials as they’re called and utilized during a build process depending on logging levels, etc. If I had a dollar for every time I saw Jenkins flagged on a pentest, often one maintained by application development teams, I’d have enough to buy a beer at least. I myself have taken advantage of Jenkins environments to gain code execution and harvest credentials.
Container Registries
Another overlooked area of SDLC is container images, primarily how credentials and secrets are handled withing container images and how internally built images are maintained and stored. Often companies host their own internal registry, these can sometimes have lax controls on them, letting anyone in the company pull images without authentication. The problem is developers may have a bad habit of building their images with credentails stored in the images as config files, environment variables, etc. It’s trivial for someone to pull images and scrape them for credentials, if nothing else, the application code or artifacts are also available in the image. Many programming languages are easy to decompile or reverse, others like Python and NodeJS are scripting languages and require no effort at all for an attacker to gain insight into how an application works that can be used for exploitation.
Code and Artifact Repositories
Code repositories are the start of the SDLC chain, obviously all that code needs some place to go, and there’s a lot of solutions for that. Today you’ll find git (GitHub, GitLab, BitBucket, Gitea, etc), and Azure DevOps Server formally known as Team Foundation Server for Microsoft shops. But some places may still be using sub version and shudders CVS still. There’s even some odd ones like Mercurial. I don’t really need to go into depth but access to code repositories, especially the ability commit code has a lot of implications. The main concerns here is the ability for an attacker to gain access and be able to commit code to a main branch or merge to a main branch without some sort of peer review or even do so in a way that might get by the peer review process. There’s also often poor hygeine in some coding teams and credentials and other private information is stored in code repos or set in the code itself. In either case, this is nothing new, and code repos should receive a lot of scrutiny in terms of who has access, why they have access, how much access and if its needed to perform their duties (for instance a manager doesn’t need anything above read access if they’re not writing code).
What’s often ignored though is Artifact repositories (e.g. Artifactory), or where Artifacts (the output or results of all the code once its compiled or built) are kept. In terms of supply chain, these locations are even more valuable for attackers as if they can push a fake or a modified artifact to these locations, they can gain code execution in your environments when one of these artifacts is mistakenly deployed believing its something released by the dev team. This is the reason why a lot of companies implement code signing, etc, to verify the provinence of packages after they’ve been compiled. That is of course assuming the attacker hasn’t gained access to the keys necessary to sign the code themselves.
Deployment
At some point, the code gets deployed, either to a lower environment where it can be smoke tested, unit tested and put through quality assurance or user acceptance. Eventually, it will get deployed to production. Usually its the same system either way, some deploy straight from Jenkins, some use in house built systems, and others use a dedicated commercial or open source product. Who can deploy and when ideally is tightly controlled based on environment, some sort of change control process, although some start ups or newer operations may deploy to production multiple times a day or do blue/green deployments. Suffice it to say if you have access to the deployment systems, you have access to the systems its deploying to by association. The deployment system has to have accounts to deploy or make changes to the environments its deploying to, which lends an opporuntity to attackers who gain some level of access. Since this system is far closer to the end points its often easier to push changes with less peer review or eye balls who will notice or have check off on it before it goes forward, generally that is done far sooner in the pipeline, during code review, user acceptance, etc. If the attacker can disable notifactions that typically get sent out during deployments it can further hide their activity, even then such notifications often come in the form of emails or Slack channel notifications that often get sorted into a sub folder in email inboxes or just get overlooked because its one of multiple messages concerning deployments and builds a day.
Conclusion and further resources
Supply Chain attacks have been all the rage since SolarWinds and though some were already paying attention to that threat vector and maybe giving warning earlier, its now receiving a lot of overdue attention and part of that is the attention given to a CI/CD pipeline. Developers make an attractive target for threat actors, and while Developers are usually far more tech savy than your average office worker (not always true, some developers just know how to code and not much else), they’re often dealing with deadlines, scrum meetings, and writing/trouble shooting/debugging code every other moment. This does lend susceptability to phishing attacks, credential stuffing, etc that generally work on other people, since all it takes is just a weak but easy to remember password, clicking a seemingly useful link or site, etc. Developers are such attractive targets because they have access to the pipelines, and good hygeine in these environments will reduce the risk of exposure or blast radius if (or more like when) a developer is compromised.
If you want to get more familiar with CI/CD pipelines and the SDLC process, especially from a red/blue security perspective, I recommend Cider Security Research’s CICD Goat. It’s pretty easy to get setup on a system that supports Docker Compose and has a lot of simple exercises that illustrate the many risks or attacks that exist in a typical CI/CD pipeline. If you like things like OWASP Top 10 to base your testing or awareness off of, they also have a CICD Top 10 List that goes over the main ways a CI/CD pipeline can pose risks to an organization or provide opportunity for attackers. I have no association with Cider Security Research but in my own research and education I found they’ve been doing a good job of calling attention to the kinds of risks you might see in these environments.