Improving Docker Security with AWS ECR push-on-scan.

Docker Image Scanning tools play an important role in hardening base images and monitor changes in container-based workloads. Established tools like Clair, Anchore, and relative newcomers like Trivy, and Grype can be used to integrate with CI pipelines and provide docker image scanning functionality.

But since 2019, AWS offers a scan on push feature in the Elastic Container Register (ECR) which greatly simplifies scanning images. The feature performs a vulnerability scan every time an image is pushed to the repository. ECR push-on-scan uses the Clair vulnerability feed and does a good job on OS-level image vulnerabilities. It also produces less noise and questionable results than the enthusiastic newcomers Trivy and Grype.

You can create an ECR repository either via the Console, the CLI, or your provisioning tools (CloudFormation, Terraform, etc). In all cases the same underlying APIs are used and so we will use the AWS CLI v2 to drive our experiment and highlight the APIs used. At the end, we package it all in a bash script to help automate this task.

You need to have the AWS CLI v2 installed and configured. To follow this tutorial you also need to have docker installed in your environment.

Note that the account or role you use, needs to have permissions to create an ECR repository and manage it. For learning purposes, an admin user is a convenient way to explore the concepts without having to worry about permissions but make sure that for other environments, especially production ones, least privilege access is applied using IAM polices. The AWS documentation on ECR explains these permissions, lists the related AWS-managed policies, contains examples for applying policies to IAM identities.

Once these prerequisites are in place, create a sample docker image by creating a Dockerfile as follows (alternatively you can just pull a base image from docker hub and use that)

FROM ubuntu:18.04
# Install dependencies
RUN apt-get update && \
apt-get -y install apache2
# Install apache and write hello world message
RUN echo ‘Docker Image Scanning Sample’ > /var/www/html/index.html
# Configure apache
RUN echo ‘. /etc/apache2/envvars’ > /root/run_apache.sh && \
echo ‘mkdir -p /var/run/apache2’ >> /root/run_apache.sh && \
echo ‘mkdir -p /var/lock/apache2’ >> /root/run_apache.sh && \
echo ‘/usr/sbin/apache2 -D FOREGROUND’ >> /root/run_apache.sh && \
chmod 755 /root/run_apache.sh
EXPOSE 80
CMD /root/run_apache.sh

Save the file as dockerfile and build the image using

docker build -t sample-image .

Now that you have a base image let’s create an ECR repository, tag and push the image and retrieve the scan results.

aws ecr create-repository --repository-name aws-sample --image-scanning-configuration scanOnPush=true --region eu-west-2 --profile samples

(Region and profile are optional parameters, which you use if you want the repo created under a profile other than the default or/and a region other than the default one)

The command will return with the new repository details. Notice the repositoryUri, part of it is used for tagging and pushing images.

create

You can also modify existing repos to switch on or off the push on scan feature

aws ecr put-image-scanning-configuration --repository-name aws-sample --image-scanning-configuration scanOnPush=true --region eu-west-2 --profile samples

You now need to tag and push your image, but first you need to authenticate against the docker registry. The following command does it without showing the short-lived password by piping the generated password to the standard input

aws ecr get-login-password --region eu-west-2 --profile samples | docker login --username AWS --password --stdin 646178029367.dkr.ecr.eu-west-2.amazonaws.com

The command will store the generated password unencrypted in a json file, which is fine for learning purposes but for other environments you will want to configure a credentials helper and use one of the credential providers, the docker engine integrates with These are listed in https://docs.docker.com/engine/reference/commandline/login/#credentials-store

But now that we are authenticated we can tag the image using the repo name

docker tag sample-image:latest 646178029367.dkr.ecr.eu-west-2.amazonaws.com/aws-sample:latest

and then push

docker push 646178029367.dkr.ecr.eu-west-2.amazonaws.com/aws-sample:latest

The push runs and triggers the scan and when it completes it returns a digest.

We can see the results in the AWS console

We can also use the CLI to retrieve the findings for an image tag or the image digest we received from the docker push command

aws ecr describe-image-scan-findings --repository-name aws-sample --image-id imageTag=latest --region eu-west-2 --profile samples

You can also pass the image digest returned from the push command to get the results (see earlier screenshot)

aws ecr describe-image-scan-findings --repository-name aws-sample --image-id imageDigest=sha256:ac75c8d11ac3429b5fd81b71578c701a914cbcce53f8d10cf88320b8f31676df --region eu-west-2 --profile samples

The results are in JSON and you can use tools like jq to extract the relevant data and react appropriately. Typically you want to integrate all of the above in your CI pipeline and take action depending the scan results.

You can do that by creating a shell script to use in your pipeline. A working bash script sample is shown below that automates the entire chain of tagging, pushing, getting the scan results, extracting a threshold, and displaying a message.

Notice that because the docker push command returns immediately with the scan running in the background, we use aws ecr wait image-scan-complete to wait until the results are ready.

Save the code below as ecr-push-scan.sh, make it executable and run as

./ecr-push-scan.sh <account-id> <region> <image name> <image tag to apply>

The script will tag and push the image, wait for the scan to complete, retrieve the results, extracts the vulnerabilities and display a warning if CRITICAL, HIGH, or UNDEFINED vulnerabilities are found. It can be extended to take corrective action such as removing the image, if needed. For example, assuming that we have already created a python repo in the eu-west-1 region we can pull the official Python image and test it.

docker pull python
./ecr-push-scan.sh 646178029367 eu-west-1 python latest

This is an example of on-demand pushing and scanning an image from the command line, which can be integrated to a CD pipeline with minimal effort. A different approach is event-driven response to scan results, which we discuss in more details in this post.