Setting up a docker registry proxy – Linux

Docker hub recently announced an upcoming request limit to their registry. So in order to reduce the amount of requests issued there, the idea of a caching proxy solution comes to mind.

Possible solutions

Fortunately there are some projects already working on that problem. However there are different approaches.

Some of the projects I found during research are:

https://github.com/lhns/docker-registry-cache

https://github.com/rpardini/docker-registry-proxy

Comparison

Feature	docker-registry-cache	docker-registry-proxy
Central changes sufficient	no	yes
Changes of all docker hub names required	yes	no
Breaks end-to-end encryption (MITM)	no	yes
Caches manifest gets	???	yes
Uses mirror concept	yes	no
Acts as http proxy	no	yes
Containers required	N+2	1

docker-registry-cache

This project is used to create a docker compose environment based on several standard components to create a multi registry cache. Components are

multiple docker registries
traefik load balancer
redis
S3 storage (optional)

However this setup seems quite complex and I couldn’t find a single line about whether it’d really solve the problem of the upcoming docker hub limits.

So for now I decided to give docker-registry-proxy a try.

docker-registry-proxy

The docker-registry-proxy was explicitly build to get around docker’s access limits. It is acting as a man-in-the-middle (MITM) http proxy and in order to use it needs to be treated like one.

Startup is very easy as a docker container is available:

linux # docker run --rm --name docker_registry_proxy -it \
       -p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
       -v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
       -v $(pwd)/docker_mirror_certs:/ca \
       rpardini/docker-registry-proxy:0.6.5

In order to use it with docker just add some proxies to its configuration and restart docker:

linux # vi /etc/docker/daemon.json 
{
  "proxies": {
    "http-proxy": "http://127.0.0.1:3128",
    "https-proxy": "http://127.0.0.1:3128",
    "no-proxy": "registry.mydomain.de,127.0.0.0/8"
  }
}

A different approach is to modify some systemd files (this is shown below in the output of the curl command).

A classical http proxy has two main purposes: to connect to the outside world (in restricted networks) and to cache requests (to save network bandwidth).

However nowadays most of the http traffic is encrypted (and therefore can no longer be cached by a proxy – at least not without breaking the encryption and causing trouble).

How it works

But that’s exactly what this MITM setup does: it retrieves information from the docker registry via https (by acting like a http(s) client). It then decrypts the retrieved data, stores it in a cache, re-encrypts the data (with its own certificate) and sends this response back to the client.

As a result, the client receives encrypted data, but won’t accept the SSL certificate used for encryption (as the MITM proxy used its own local certificate authority (CA) that is not yet known by the client).

Client-side changes

So in order to make things work, we need to make all possible clients accept this CA. How to do that is either documented on the project page, but also within the software itself: you can just get a certain URL from the proxy to get the commands needed to

export the used MITM CA
include it into the systems accepted CA pool
restart systemd and docker to apply changes

The short version of things to do is this:

linux # curl -sS http://127.0.0.1:3128/ca.crt > /usr/share/ca-certificates/docker_registry_proxy.crt
linux # echo "docker_registry_proxy.crt" >> /etc/ca-certificates.conf
linux # update-ca-certificates --fresh
linux # systemctl daemon-reload
linux # systemctl restart docker.service

If everything succeeded you can now pull images that will be cached locally:

linux # docker pull ubuntu

Meanwhile you can observe what’s going on in a second terminal:

linux # docker logs -f docker_registry_proxy

Additional information

As mentioned above the basic information required to set things up can also be retrieved via URL. This contains some extra steps (and uses systemd configuration instead of docker’s daemon.js):

linux # curl http://127.0.0.1:3128/setup/systemd

set -e

if [ ! -d /etc/systemd ]; then
   echo "Not a systemd system"
   exit 1
fi

mkdir -p /etc/systemd/system/docker.service.d
cat << EOD > /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTPS_PROXY=http://127.0.0.1:3128/"
EOD

# Get the CA certificate from the proxy and make it a trusted root.
curl -sS http://127.0.0.1:3128/ca.crt > /usr/share/ca-certificates/docker_registry_proxy.crt
if fgrep -q "docker_registry_proxy.crt" /etc/ca-certificates.conf ; then
   echo "certificate refreshed"
else
   echo "docker_registry_proxy.crt" >> /etc/ca-certificates.conf
fi

update-ca-certificates --fresh

# Reload systemd
systemctl daemon-reload

# Restart dockerd
systemctl restart docker.service
echo "Docker configured with HTTPS_PROXY=http://127.0.0.1:3128/"

More detailed configuration options can be found on the project page.