How to configure a kubernetes cluster to use a local (insecure) registry

 
1
0
-1

In my WindRiver environment, I have the following hosts:

  • dgl-rancher  - local source code for oom and other projects; also a docker registry; also where I run helm commands
  • dgl-k8s-1, dgl-k8s-2, dgl-k8s-3  - a 3 node kubernetes cluster

I am able to deploy helm charts successfully as long as they reference images already in nexus3.  However, I want to be able to create new containers and test them prior to pushing to nexus3.  But when I update the value of the repository: attribute in values.yaml, it doesn't seem to work.   No matter if I leave the repository attribute as null, or put the IP address:Port of my docker registry, it fails without identifying the registry it is trying to pull from.  e.g.  I get:

Failed to pull image "onap/aaf/aaf_agent:2.1.2-SNAPSHOT": rpc error: code = Unknown desc = Error response from daemon: repository onap/aaf/aaf_agent not found: does not exist or no pull access
Error syncing pod

I suspect this has to do with needing to configure kubernetes to add an insecure registry.   What are the general instructions for doing this on a kubernetes cluster?

  1. Dominic Lunanuova

    I had two problems:

    1. chart template not properly referencing the repository attribute!
    2. missing insecure-registry config on k8s cluster nodes

    the first problem was masking the behavior of the 2nd problem.  But if you avoid making that first mistake, at least you will see clearer errors that you really don't have access to the registry.

  2. Dominic Lunanuova

    Note to self on the docker cmds needed on the build server.

    docker build . --tag ${image}:${ver}
    docker tag ${image}:${ver} 10.12.5.45:5000/${image}:${ver}
    docker push 10.12.5.45:5000/${image}:${ver}

CommentAdd your comment...

3 answers

  1.  
    1
    0
    -1

    If you are deploying ONAP component independently (not coupled with ONAP chart) then you can create secret like below


    kubectl create secret docker-registry onap-docker-registry-key --docker-server=nexus3.onap.org:10001 --docker-username=docker --docker-password=docker --docker-email=@ --namespace onap


    Note: here I am using onap namespace to deploy components


    Take a look into onap chart template directory. 

    onap/templates/secrets.yaml

    onap/templates/clusterrolebinding.yaml

     

      CommentAdd your comment...
    1.  
      1
      0
      -1

      Dominic Lunanuova

      On Ubuntu-16.04 following would work

      cat /etc/systemd/system/docker.service.d/docker.conf

      [Service]

      ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001 --insecure-registry=xx.xx.xx.xx:5000

      if you want you can also include --max-concurrent-downloads=9 (however this depends on system CPUs)


      systemctl daemon-reload

      systemctl restart docker


      Later check docker info


      if you want insecure docker registry to act as a proxy to nexus3 then you need to start the docker as follows


      docker run -d -p 5000:5000 --restart=unless-stopped --name registry -v /opt/docker-registry:/var/lib/registry -e REGISTRY_PROXY_REMOTEURL=https://nexus3.onap.org:10001 registry:2

      Note: in my case -v (volume is externally mounted). if you don't have an external disk then you don't have to use volume option 

        CommentAdd your comment...
      1.  
        1
        0
        -1


        Steps to add a local insecure docker registry to your kubernetes cluster:

        1. Create your local docker registry.  In my case, it was on dgl-rancher VM (10.12.5.45) on port 5000
        2. Push your dev version to the registry in step 1.
        3. On each kubernetes node, repeat the following steps:
          1. sudo service docker stop
          2. Add insecure registries to docker daemon configuration.  Edit /etc/docker/daemon.json to add the registry from step 1 to the insecure-registries set.  e.g.

            {
            "insecure-registries": [
            "10.12.5.45:5000"
            ]
            }

          3. sudo service docker start
          4. I found that the socket lock file often was left in the wrong state and dockerd did not start.  Confirm that the docker daemon restarted:  ps -ef | grep dockerd
          5. If dockerd did not start:
            1. The workaround was:  sudo rm -rf /var/run/docker.sock
            2. Go to step 3c
        4. Edit your chart template to refer to the local repo.
        5. Remake your helm charts
        6. Install the chart, and kubernetes should be able to pull your image from the local repo.


        Other notes:

        • Most of the docker documentation I found assumed the docker service was controlled by systemctl.  But on ubuntu 16.04, it is the service command, so you have to adjust accordingly.
        • My install already had a file /etc/systemd/system/docker.service.d/docker.conf which had an existing setting for --insecure-registry.   i.e.

          $ cat /etc/systemd/system/docker.service.d/docker.conf

          [Service]

          ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001


        • My attempts to simply add another value to this file and restart the service were unsuccessful, possibly due to pilot error.  Editing this file seems like it would be the natural place to introduce another insecure registry.


        References:

        https://giovanni.wordpress.com/2016/03/16/how-to-use-a-private-docker-registry-from-kubernetes/

        https://docs.docker.com/config/daemon/systemd/#start-the-docker-daemon

        https://docs.docker.com/engine/reference/commandline/dockerd//#daemon-configuration-file

          CommentAdd your comment...