Linux and Nagios‎ > ‎

MicroK8s (Kubernetes) - Raspberry Pi Ubuntu Linux Basic Setup Guide - Part 4 (Image Repositories)

posted 6 Aug 2020, 14:13 by Tristan Self

https://microk8s.io/docs/registry-images


So images. A container is based on an image. You can pull these images in from the outside world, e.g. from a Public, Private registry, or from the built-in registry.


For this copy the "hello-python" directory we created in Part 2 into one called "hello-python3", you can delete the .tar file in there for now, its not needed.


Push Image into Kubernetes Image Repository


So in the earlier steps, we basically injected the image straight into the Kubernetes image cache, so not using the built-in registry.


Now we’ll do the same but this time working with the Kubernetes registry, so firstly lets enable it if not already done:


microk8s enable registry

Right, so the registry is now available at localhost:32000, when we upload an image we need to tag it accordingly with localhost:32000/your-image-name.


So let’s inject the image we used before, but this time tag with another name:


docker build . -t localhost:32000/hello-python:registry

Build the image:


root@k8s-master:/home/ubuntu/application/hello-python/app# docker build . -t localhost:32000/hello-python:registry

Sending build context to Docker daemon  99.71MB (updates….)


Wait for the image to upload into the docker images library.


docker images


root@k8s-master:/home/ubuntu/application/hello-python/app# docker images

REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE

localhost:32000/hello-python   registry            03e7eaf7b301        13 seconds ago      1.77GB

hello-python                   local               a5aba8ae8f7d        38 hours ago        874MB

python                         3.7                 42cd7c61ce0e        2 days ago          863MB


Now we want to push this into the Kubernetes registry:


docker push localhost:32000/hello-python


root@k8s-master:/home/ubuntu/application/hello-python/app# docker push localhost:32000/hello-python

The push refers to repository [localhost:32000/hello-python]

32963f3bbf70: Pushing [==================>                                ]  3.899MB/10.74MB

0059fe46108e: Pushing [>                                                  ]  9.997MB/897.8MB

8c51df724705: Pushed

fc5072a676ca: Pushing [==========================>                        ]  3.773MB/7.235MB

bfc730b5d6a1: Pushed

c2047c5c6a3d: Pushing [===>                                               ]  6.636MB/86.97MB

ad87fe9fbf86: Pushing [===============>                                   ]  5.625MB/18.08MB

de08a49275ea: Waiting

e6eb43d220d2: Waiting

a76bedae40bf: Waiting

9184b9a70c9e: Waiting

9276caf83dc1: Waiting


At this point the image is then within the Kubernetes image repository and ready to be used in your application deployments.


Deploy Container using Registry Image


So lets create a cheeky application, create a deployment.yaml and service.yaml as follows:


apiVersion: apps/v1

kind: Deployment

metadata:

  name: hello-python3

spec:

  selector:

    matchLabels:

      app: hello-python3

  replicas: 1

  template:

    metadata:

      labels:

        app: hello-python3

    spec:

      containers:

      - name: hello-python3

        image: localhost:32000/hello-python:registry

        ports:

        - containerPort: 5000


And an service.yaml:


apiVersion: v1

kind: Service

metadata:

  name: hello-python3-service

spec:

  selector:

    app: hello-python3

  ports:

  - port: 8080

    targetPort: 5000

  type: LoadBalancer



Then deploy the application and service with:


kubectl apply -f deployment.yaml

kubectl apply -f service.yaml


root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get pods -o wide

NAME                                  READY   STATUS             RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES

hello-python-6bfc96894d-qhv8h         1/1     Running            1          38h   10.1.54.21   k8s-master      <none>           <none>

hello-python-redux-7849b9b844-55sxb   1/1     Running            0          19h   10.1.54.40   k8s-master      <none>           <none>

hello-python3-5fdcb55ccd-dvpnx        0/1     ImagePullBackOff   0          47s   10.1.49.2    k8s-worker-02   <none>           <none>


Oh look, we have an error, excellent. So let's look at why this happened.


So so far, we’ve deployed the image registry on the master node, the worker nodes have no configuration for them to reach this registry at the moment. Now if we were to have built and pushed this registry in when we would have only had one node, then the only place the pod could have run would have been on the master node, which can get to the Kubernetes image registry. To fix this we need to complete the next section, as you can see the “ImagePullBackOff” basically means that the pods has been scheduled to a node which is unable to download the image.

If we look at the log for the pod: "hello-python3-5fdcb55ccd-dvpnx" we see:


root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl logs hello-python3-7b6df7676b-sp2mf

Error from server (BadRequest): container "hello-python3" in pod "hello-python3-7b6df7676b-sp2mf" is waiting to start: image can't be pulled


So we make the change to the deployment.yaml:


apiVersion: apps/v1

kind: Deployment

metadata:

  name: hello-python3

spec:

  selector:

    matchLabels:

      app: hello-python3

  replicas: 1

  template:

    metadata:

      labels:

        app: hello-python3

    spec:

      containers:

      - name: hello-python3

        image: 192.168.1.164:32000/hello-python:registry

        ports:

        - containerPort: 5000


And then deploy again with:

kubectl apply -f deployment.yaml


We still are having problems.


If we run this on the node where the pod is trying to start:


tail -f /var/log/syslog  | grep hello

We see:


Aug  2 14:48:49 k8s-worker-02 microk8s.daemon-kubelet[1324]: E0802 14:48:49.023084    1324 pod_workers.go:191] Error syncing pod 9490b872-87cd-48c9-90b6-952c7dc7b452 ("hello-python3-7b6df7676b-vt4v5_default(9490b872-87cd-48c9-90b6-952c7dc7b452)"), skipping: failed to "StartContainer" for "hello-python3" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"192.168.1.164:32000/hello-python:registry\": failed to resolve reference \"192.168.1.164:32000/hello-python:registry\": failed to do request: Head \"https://192.168.1.164:32000/v2/hello-python/manifests/registry\": http: server gave HTTP response to HTTPS client"

So this exact error is described here: https://microk8s.io/docs/registry-private under the section “Configuring MicroK8s”. We need to make some changes to the configuration of the MicroK8s nodes.


Allowing Insecure Registry


Edit the file: /var/snap/microk8s/current/args/containerd-template.toml on ONLY the worker nodes, so in our case k8s-worker-01 and k8s-worker-02, right at the bottom of the file look for the following section:


[plugins] -> [plugins."io.containerd.grpc.v1.cri".registry] -> [plugins."io.containerd.grpc.v1.cri".registry.mirrors]:


Create a new section underneath to match the IP address of your master node where the image repository lies for example the below, in my case the k8s-master node is on 192.168.1.164.


[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.1.164:32000"]

          endpoint = ["http://192.168.1.164:32000"]


The Microk8s documentation then talks about just restarting microk8s with a “microk8s stop” followed by a “microk8s start”, however I found this was not enough, so rebooted the worker and master nodes, once restarted, i attempted the deployment again:


kubectl apply -f deployment.yaml

Then a quick check on the progress:


root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get pods -o wide

NAME                                  READY   STATUS    RESTARTS   AGE     IP           NODE            NOMINATED NODE   READINESS GATES

hello-python-6bfc96894d-qhv8h         1/1     Running   4          40h     10.1.54.63   k8s-master      <none>           <none>

hello-python-redux-7849b9b844-55sxb   1/1     Running   3          21h     10.1.54.59   k8s-master      <none>           <none>

hello-python3-7b6df7676b-qk2mq        1/1     Running   0          8m11s   10.1.81.4    k8s-worker-01   <none>           <none>


As you can see it is there, the pod is running on the k8s-worker-01 node, lets have a quick look at the logs:


cat /var/log/syslog | grep hello-python3

Now this time we see all is well:


Aug  2 15:11:21 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T15:11:21.254616306+01:00" level=info msg="CreateContainer within sandbox \"8f4012258d385007f7a6523b627ca1fa27f4ea21d01e3ac6bf3024b4ffe1fca0\" for container &ContainerMetadata{Name:hello-python3,Attempt:0,}"

Aug  2 15:11:25 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T15:11:25.561390309+01:00" level=info msg="CreateContainer within sandbox \"8f4012258d385007f7a6523b627ca1fa27f4ea21d01e3ac6bf3024b4ffe1fca0\" for &ContainerMetadata{Name:hello-python3,Attempt:0,} returns container id \"bc93c29f081139990e98558d065e6b459fe26b34e3f17e49b7c7ecbc3768785e\""


Now lets create the service to access the application, so back on the Master node, we run:


kubectl apply -f service.yaml

And have a quick check:


root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get services -o wide

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)          AGE   SELECTOR

hello-python3-service        LoadBalancer   10.152.183.149   192.168.1.22   8080:31272/TCP   16s   app=hello-python3

kubernetes                   ClusterIP      10.152.183.1     <none>         443/TCP          40h   <none>


Yes, there it is: http://192.168.1.22:8080, so lets try to access it:


Good stuff, we’ve done it.


Doing it All Again! Well Why Not!?


Lets run this command:


microk8s kubectl create deployment hello-python4 --image=192.168.1.164:32000/hello-python:registry

That will just create a deployment of that image as a new container and then schedule it on one of the worker nodes:


root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get pods -o wide

NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE            NOMINATED NODE   READINESS GATES

hello-python-6bfc96894d-qhv8h         1/1     Running   4          40h   10.1.54.63   k8s-master      <none>           <none>

hello-python-redux-7849b9b844-55sxb   1/1     Running   3          21h   10.1.54.59   k8s-master      <none>           <none>

hello-python3-7b6df7676b-qk2mq        1/1     Running   0          13m   10.1.81.4    k8s-worker-01   <none>           <none>

hello-python4-55bdb58666-sk8wg        1/1     Running   0          5s    10.1.81.5    k8s-worker-01   <none>           <none>


Cool, there it is “hello-python4-55bdb58666-sk8wg” working as expected on the k8s-worker-01 node, thats good that proves that the registry is working.


Verifying the Registry Image is Used


Okay so lets copy the existing directory:


cp -r hello-python hello-python5

Now within there, if there is an image .tar file delete it, we don’t need it anymore.


Let’s edit the application to show we are getting a new image.


cd hello-python5/app

Edit the “main.py”, and add something to the line saying “Hello from Python”, e.g. “This is a test page using Hello-python5 application!”


Build the image, and tag it as the image name for the Kubernetes image registry when put into the docker image library.


docker build . -t localhost:32000/hello-python5:registry


docker images

root@k8s-master:/home/ubuntu/application/hello-python5/app# docker images

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE

localhost:32000/hello-python5   registry            e699191159f2        6 seconds ago       874MB


Okay so now get docker to push it into the Kubernetes image library.


docker push localhost:32000/hello-python5

Now edit the deployment.yaml and service.yaml, and where you have “hello-python3…” change it to “hello-python5”, including the image location in the deployment.yaml file, so we make sure we use our new image!


So now deploy the service and deployment with:


kubectl apply -f deployment.yaml

kubectl apply -f service.yaml


If we run a quick: “tail -f /var/log/syslog | grep hello-python5” we can see the container image being deployed and started.


Aug  2 15:44:32 k8s-worker-02 microk8s.daemon-containerd[1300]: time="2020-08-02T15:44:32.322560841+01:00" level=info msg="PullImage \"192.168.1.164:32000/hello-python5:registry\" returns image reference \"sha256:e699191159f229e8e6aeae167a32c37a830d7c238182ee3f8825e8d5000f0d3d\""

Aug  2 15:44:32 k8s-worker-02 microk8s.daemon-containerd[1300]: time="2020-08-02T15:44:32.328968944+01:00" level=info msg="CreateContainer within sandbox \"35c4ee9ebdd4b90744f276d066ba7c75abe1f856cfe6e2e795100c39411789f4\" for container &ContainerMetadata{Name:hello-python5,Attempt:0,}"

Aug  2 15:44:33 k8s-worker-02 microk8s.daemon-containerd[1300]: time="2020-08-02T15:44:33.337282018+01:00" level=info msg="CreateContainer within sandbox \"35c4ee9ebdd4b90744f276d066ba7c75abe1f856cfe6e2e795100c39411789f4\" for &ContainerMetadata{Name:hello-python5,Attempt:0,} returns container id \"59ff7a8553651222ed6f3e8a13801c2ae850f995d133fa5e79bb37da5b297db9\""


kubectl get services


Gives us:

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)          AGE

hello-python5-service        LoadBalancer   10.152.183.154   192.168.1.23   8080:32278/TCP   23s


There it is running at http://192.168.1.23:8080, so let’s give it a connect in a browser:


And you can see we really are running the expected image, because its the hello-python5 one as we wanted.


Using an External Public Registry (e.g. DockerHub)


https://microk8s.io/docs/registry-public


Create an account at Docker Hub.


You’ll also need to create a repository. On your master node run:


Docker login

Enter your logon details.


Now lets create a new image and push it up into the Docker Hub.


First make a copy of the existing application image definition.


cp -r hello-python5 hello-python6


cd hello-python6/app


Make a change to the image’s main.py, so you know its something different for when you run it. Build the image locally, then we’ll push it up.


docker build . -t geekmungus/private:hello-python6

Where “geekmungus” is my DockerHub username, “private” is the repository name and “hello-python6” is the tag (in this case the name of the application).


root@k8s-master:/home/ubuntu/application/hello-python6/app# docker images

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE

geekmungus/private              hello-python6       1dfc3b05a3ca        5 seconds ago       874MB


Okay now we can push that repository up.


docker push geekmungus/private

Cool, and there it is:



Let’s delete the local copy to ensure we only ever use the pulled version, when we now do a deployment.


docker images

root@k8s-master:/home/ubuntu/application/hello-python6/app# docker images

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE

geekmungus/private              hello-python6       1dfc3b05a3ca        4 minutes ago       874MB


docker image rm 1dfc3b05a3ca 

So now that image has gone and is not available locally, so it should pull from DockerHub now when we refer to it.


Change the deployment.yaml and service.yaml in the hello-python6/app folder, so everywhere where it did say “hello-python5” it now says “hello-python6”.


Then ensure you put the image name in as: geekmungus/private:hello-python6


kubectl apply -f deployment.yaml


kubectl apply -f service.yaml


Watching the logs I got this error:


Aug  2 20:59:45 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T20:59:45.554847607+01:00" level=info msg="PullImage \"geekmungus/private:hello-python6\""

Aug  2 20:59:47 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T20:59:47.366927939+01:00" level=error msg="PullImage \"geekmungus/private:hello-python6\" failed" error="failed to pull and unpack image \"docker.io/geekmungus/private:hello-python6\": failed to resolve reference \"docker.io/geekmungus/private:hello-python \": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"


https://stackoverflow.com/questions/56642311/microk8s-cannot-pull-from-private-registry


https://github.com/ubuntu/microk8s/issues/512


So run this to create a secret to be used when you connect to the external public repository.


kubectl create secret docker-registry regcred --docker-username=<yourdockerhubusername> --docker-password=<password> --docker-email=<youremailaddress>

And then we see:

root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl create secret docker-registry regcred --docker-username=<yourdockerhubusername> --docker-password=<password> --docker-email=<youremailaddress> --docker-server=https://index.docker.io/v1/

secret/regcred created


We now have a secret called "regcred" we can use in our deployments.


Now within the deployment.yaml, you need to refer to it so your deployment.yaml will end up looking like this:


apiVersion: apps/v1

kind: Deployment

metadata:

  name: hello-python6

spec:

  selector:

    matchLabels:

      app: hello-python6

  replicas: 1

  template:

    metadata:

      labels:

        app: hello-python6

    spec:

      containers:

      - name: hello-python6

        image: geekmungus/private:hello-python6

        ports:

        - containerPort: 5000

      imagePullSecrets:

      - name: regcred


Now redeploy with:

Kubectl apply -f deployment.yaml

It might take a little while because it will be downloading the image from dockerhub, but then:


root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl get pods -o wide

NAME                                  READY   STATUS              RESTARTS   AGE     IP           NODE            NOMINATED NODE   READINESS GATES

hello-python6-7ccb5c4bf5-4z9sf        0/1     ContainerCreating   0          8s      <none>       k8s-worker-01   <none>           <none>


root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl get pods -o wide

NAME                                  READY   STATUS    RESTARTS   AGE     IP           NODE            NOMINATED NODE   READINESS GATES

hello-python6-7ccb5c4bf5-4z9sf        1/1     Running   0          67s     10.1.81.11   k8s-worker-01   <none>           <none>


Finally deploy the service:


kubectl apply -f service.yaml

root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl get services

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)          AGE

hello-python6-service        LoadBalancer   10.152.183.229   192.168.1.24   8080:32475/TCP   6s

kubernetes                   ClusterIP      10.152.183.1     <none>         443/TCP          46h



And we are done! We can see the application running on that port in our browser and it is the image we expected to see.


Comments