OpenShift: Using oc-mirror to create image mirrors for air gapped environments
4 min read | by Jordi Prats
Combining oc-mirror with ImageContentSourcePolicy we can configure image mirrors for container images in OpenShift. We can use it to setup air gapped environments: The images won't be available for the source repository, just from the internal mirror. This way we can audit them before allowing our cluster to use them
To do so first we'll have to populate our internal mirror. To do so we can use oc-mirror
Populate internal mirror
We can download oc-mirror from RedHat, or use it's repository to compile it: openshift/oc-mirror
There are several ways to configure oc-mirror to copy container images, making it easy to download images from the different operators we might need. For this example we are going to use mirror.additionalImages to specify an arbitrary image from docker hub:
kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 storageConfig: registry: imageURL: repo.pet2cattle.com/os-mirror skipTLS: false mirror: additionalImages: - name: registry.hub.docker.com/jordiprats/flask-pet2cattle:5.30 This is a configuration file for oc-mirror, not an actual Kubernetes object. The OpenShift cluster is not going to accept this object.
Using oc mirror with this configuration and the target repository, we can sync the container images. We'll need to run it somewhere that can download the images from the source registry and push them to our internal mirror. This tool can be used to download images locally and then push them to the internal mirror using a different computer, but it's out of the scope of this post.
$ oc-mirror --config=configuration.yaml docker://repo.pet2cattle.com/os-mirror This command is going to create a folder named oc-mirror-workspace that will contain some files that we can use as reference:
$ find oc-mirror-workspace oc-mirror-workspace oc-mirror-workspace/publish oc-mirror-workspace/publish/.metadata.json oc-mirror-workspace/results-1687463421 oc-mirror-workspace/results-1687463421/release-signatures oc-mirror-workspace/results-1687463421/charts oc-mirror-workspace/results-1687463421/imageContentSourcePolicy.yaml oc-mirror-workspace/results-1687463421/mapping.txt oc-mirror-workspace/mapping.txt In the mapping.txt file we are going to see the list of container images that has sync, with it's internal equivalent:
registry.hub.docker.com/jordiprats/flask-pet2cattle@sha256:bf5823de6c97c10ced8e25787a355a92f08c2d83fbf91b3922bbf4d399510337=repo.pet2cattle.com/os-mirror/jordiprats/flask-pet2cattle:5.30 Configure container image mirrors
The imageContentSourcePolicy.yaml contains the object that we need to persist into the OpenShift cluster to be able to tell it where it to look for container images.
apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: generic-0 spec: repositoryDigestMirrors: - mirrors: - repo.pet2cattle.com/os-mirror source: registry.hub.docker.com/jordiprats To demostrate that it is using the mirrors, we can update it so we tell OpenShift that a certain image container from a non-existing registry can be found in our mirror: This way the only way it can fetch the image is using our mirror:
apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: name: generic-0 spec: repositoryDigestMirrors: - mirrors: - repo.pet2cattle.com/os-mirror source: non-existing-repo.com/jordiprats To test it, we can use kubectl run using the cryptographic hash of the image, it won't work with tags:
kubectl run testpull \ --image='non-existing-repo.com/jordiprats/flask-pet2cattle@sha256:bf5823de6c97c10ced8e25787a355a92f08c2d83fbf91b3922bbf4d399510337' --command -- sleep 24h We can now check that it has been able to download the image anyway using kubectl describe:
$ kubectl describe pod testpull Name: testpull Namespace: test-jordi (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10m default-scheduler Successfully assigned test-jordi/testpull to ip-100-75-36-20.eu-central-1.compute.internal Normal AddedInterface 10m multus Add eth0 [10.128.2.67/23] from openshift-sdn Normal Pulling 10m kubelet Pulling image "non-existing-repo.com/jordiprats/flask-pet2cattle@sha256:bf5823de6c97c10ced8e25787a355a92f08c2d83fbf91b3922bbf4d399510337" Normal Pulled 10m kubelet Successfully pulled image "non-existing-repo.com/jordiprats/flask-pet2cattle@sha256:bf5823de6c97c10ced8e25787a355a92f08c2d83fbf91b3922bbf4d399510337" in 6.181838823s Normal Created 10m kubelet Created container testpull Normal Started 10m kubelet Started container testpull We can check what's the mirror we are using by checking the status.containerStatuses attribute. When we are using a container image fetched from the source, the image and imageID are going to point to the same registry:
$ kubectl get pod apiVersion: v1 kind: Pod (...) - containerID: cri-o://807336c48df5ed81c3ea970303c2a06d47cb4f51bf719086f21d0311850f4dc2 image: registry.hub.docker.com/jordiprats/flask-pet2cattle:5.30 imageID: registry.hub.docker.com/jordiprats/flask-pet2cattle@sha256:1590c0b9c38d565b3e5f65e78a3e05e47d8eab8d81cc5e7a04793fc1d9e62f67 (...) When we have a ImageContentSourcePolicy configured and the Pod is using a cryptographic hash, they won't match: The imageID is going to show us the actual source it used to download the container:
$ kubectl get pod apiVersion: v1 kind: Pod (...) containerStatuses: - containerID: cri-o://5e6f1aa90d0f63f5ebd74a804b750e02b0f6ad60aa419eca84acb5fa9838d11f image: non-existing-repo.com/jordiprats/flask-pet2cattle@sha256:bf5823de6c97c10ced8e25787a355a92f08c2d83fbf91b3922bbf4d399510337 imageID: repo.pet2cattle.com/os-mirror/jordiprats/flask-pet2cattle@sha256:1590c0b9c38d565b3e5f65e78a3e05e47d8eab8d81cc5e7a04793fc1d9e62f67 (...) Posted on 16/01/2023