Building Controllers with Kubebuilder with Docker in VS Code

Fri, Sep 15, 2023 It is better some days to keep development environments separate from a workstation’s environment. Language versions, kubernetes configuration files, among other things can get pretty borked up. Developing in a container environment keeps the host workstation clean.

The environment set out here is based on the Go Environment, since Kubebuilder depends on Go. However, it also needs docker for image building as well as the kubectl and kubebuilder binaries.

One should not run a docker daemon inside a docker container, so the host workstation’s docker socket will be mounted. This is reflected in the docker-init.sh script.

B. Ingredients

Ensure Docker or Docker Desktop is running.
Download and install Microsoft Visual Studio Code on Mac, Linux, or PC.
Install the following extensions:
- Docker
- Remote Development. This pack includes a bunch tools, including the absolutely vital Dev Containers.
- Rewrap (optional: keeps your comments easy to read)
- Markdownlint (optional: keeps your documentation clean)

C. Project Setup

The developer will want to set this section up as a template so it can be used over and over again for new projects.

C.1. Directory structure

The Go project layout at https://github.com/golang-standards/project-layout is overkill for beginning projects, especially since kubebuilder sets up its own go.mod, cmd/ and internal/ directories. Until the developer knows which modules he or she will create on top of kubebuilder’s boilerplate, there is no point in adding too many directories and files. The baseline should be:

<project name>
|- .devcontainer/
|  |- Dockerfile
|  |- devcontainer.json
|  |- config/
|  |  |- requirements.txt  # for mkdocs or other Python tools.
|  |  |- docker-init.sh    # Sets up docker-outside-docker for Mac, Linux & Win
|- docs/
|- .gitignore
|- README.md

If using mkdocs, add mkdocs.yaml and VERSION.cfg to the root as well.

C.2 The Python requirements

The Python requirements file config/requirements.txt is for setting up mkdocs. If you are not using mkdocs, just keep the Python requirements file empty. If you do not need Python at all, get rid of the requirements.txt file and the Python references in the Dockerfile below.

mkdocs
mkdocs-autorefs
mkdocs-mermaid2-plugin
mkdocs-material
mkdocs-material-extensions
mkdocstrings

C.3 docker-init.sh

Rather than running Docker inside a container (yuck), we use the workstation’s already running Docker by mounting its UNIX socket. The following code is modified from https://github.com/microsoft/vscode-dev-containers/tree/main/containers/docker-from-docker as our container is based on Alpine, not Ubuntu.

The socket on the host is called “/var/run/docker.socket”. It is mounted as “/var/run/docker-host.socket” in the container to allow access to both root- and nonroot-owned sockets.

Basically, if the socket GID is non-root, we add the vscode user to the appropriate group. If the socket GID is root, then we need to use socat to pipe from a new socket owned by vscode to the root-owned docker socket.

#!/bin/sh
sudoIf() {
    if [ "$(id -u)" -ne 0 ];
    then
      sudo "$@"
    else
      "$@"
    fi
}
SOCKET_GID=$(stat -c '%g' /var/run/docker-host.sock)

# If socket GID is not owned by root, just add vscode to the socket group
if [ "${SOCKET_GID}" != '0'];
then
    sudoIf ln -s /var/run/docker-host.sock /var/run/docker.sock
    # Create a group called docker-host that conforms to the socket GID if it
    # does not already exist.
    if [ "$(cat /etc/group | grep :${SOCKET_GID}:)" = ''];
    then
        sudoIf addgroup -g ${SOCKET_GID} docker-host
    fi
    # Add vscode to the docker-host group if it isn't already there
    if [ "$(id vscode) | grep -E \"groups=.*(=|,)${SOCKET_GID}\(\"" == '' ];
    then
        groupname=$(getent group ${SOCKET_GIT} | cut -d: -f1)
        sudoIf addGroup vscode $groupname
    fi
# If socket GID is owned by root, pipe with _socat_ in the background.
else
    sudoIf rm -rf /var/run/docker.sock
    sudoIf socat UNIX-LISTEN:/var/run/docker.sock,fork,mode=660,user=vscode UNIX-CONNECT:/var/run/docker-host.sock &
fi
exec "$@"

C.4 Dockerfile

Here are the contents of the Dockerfile. This installs a decent terminal, Python for documentation tools, Go Visual Studio tools, a series of useful Go binaries and linters, and the Kubernetes binaries. This is almost identical to the Go environment discussed above, but it also adds docker and bash. Bash, it turns out, is needed for some of the building.

ARG GO_VERSION=1.20
ARG ALPINE_VERSION=3.17

FROM golang:$GO_VERSION-alpine${ALPINE_VERSION}
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=1000
ARG GOPLS_VERSION=latest
ARG KUBECTL_VERSION=1.27.4

RUN adduser $USERNAME -s /bin/sh -D -u $USER_UID $USER_GID && \
    mkdir -p /etc/sudoers.d && \
    echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME && \
    chmod 0440 /etc/sudoers.d/$USERNAME

RUN apk add -q --update --progress --no-cache \
    git sudo openssh-client zsh curl zsh-vcs make gpg graphviz \
    python3 yamllint jq curl unzip git docker bash socat

RUN python3 -m ensurepip
RUN pip3 install --no-cache --upgrade pip setuptools

RUN go install golang.org/x/tools/gopls@${GOPLS_VERSION}
RUN for tool in tools/gopls tools/cmd/goimports lint/golint; \
    do go install golang.org/x/${tool}@latest; \
    done

# Detect shadowing bugs
RUN go install golang.org/x/tools/go/analysis/passes/shadow/cmd/shadow@latest

# Visual Studio Code tools
RUN go install github.com/cweill/gotests/gotests@latest
RUN go install github.com/fatih/gomodifytags@latest
RUN go install github.com/josharian/impl@latest
RUN go install github.com/haya14busa/goplay/cmd/goplay@latest
RUN go install github.com/go-delve/delve/cmd/dlv@latest
RUN go install honnef.co/go/tools/cmd/staticcheck@latest

# Kubernetes and Kubebuilder
RUN curl -LO https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/$(go env GOARCH)/kubectl
RUN chmod +x kubectl && mv kubectl /usr/local/bin/
RUN curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
RUN chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

# Provide docker-outside-docker functionality
COPY config/docker-init.sh /usr/local/share/docker-init.sh
RUN chmod +x /usr/local/share/docker-init.sh

# Allow the developer to download third-party modules
RUN chown -R $USER_UID:$USER_GID /go/pkg

# Setup shell
USER $USERNAME
RUN sh -c "$(wget -O- https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)" "" --unattended &> /dev/null
ENV ENV="/home/$USERNAME/.ashrc" \
    ZSH=/home/$USERNAME/.oh-my-zsh \
    EDITOR=vi \
    LANG=en_US.UTF-8 \
    PATH=/home/vscode/.local/bin:$PATH
RUN printf 'ZSH_THEME="agnoster"\nENABLE_CORRECTION="false"\nplugins=(git copyfile extract colorize dotenv encode64 golang)\nsource $ZSH/oh-my-zsh.sh\nsource <(kubectl completion zsh)\nalias k=kubectl' > "/home/$USERNAME/.zshrc"
RUN echo "exec `which zsh`" > "/home/$USERNAME/.ashrc"

# To run mkdocs server as per Eitri standards
COPY config/* .
RUN python3 -m pip install -r requirements.txt

USER root
ENTRYPOINT ["/usr/local/share/docker-init.sh"]
CMD ["sleep", "infinity"]

Notes

There must be a Go image available with both the Alpine and Go versions the developer wants. Some Alpine/Go combinations have had trouble, so one should do their research. The default listed in the first two ARG commands I know personally to work. Rather than change the default here, change the devcontainer.json file.
Zsh and the Oh-my-zsh are my personal preferences. By all means, throw them out if you do not want them. Since the build stage is separate from the release and run stages, it does not matter.
Likewise, you may not want Python tools in your build environment. Since I use mkdocs, I keep the Python and its config/requirements.txt.

C.5 The Devcontainer configuration

Once the developer has created the devcontainer.json file, then the environment is ready to be built and executed.

Note the ALPINE_VERSION, GO_VERSION, and KUBECTL_VERSION fields. These will override the defaults in the Dockerfile.
Note also the Git SSH key mappings. Change this to reflect your own settings.
Lastly, notive the vscode.vim local extension. If you are not a vi or vim user, get rid of it.

{
    "name": "Kubebuilder Environment",
    "build": {
        "dockerfile": "Dockerfile",
        "args": {
            "ALPINE_VERSION": "3.17",
            "GO_VERSION": "1.20",
            "KUBECTL_VERSION": "1.27.4"
        }
    },
    "forwardPorts": [8000],
    "remoteUser": "vscode",
    "customizations": {
        "vscode": {
            "extensions": [
                "davidanson.vscode-markdownlint",
                "golang.Go",
                "ms-azuretools.vscode-docker",
                "ms-vscode.go",
                "ms-vscode.makefile-tools",
                "vscodevim.vim"
            ]
        },
        "settings": {
            "go.useLanguageServer": true
        }
    },
    "runArgs": [
        "-u",
        "vscode",
        "--cap-add=SYS_PTRACE",
        "--security-opt",
        "seccomp=unconfined",
    ],
    "overrideCommand": false,
    "mounts": [
        // map SSH keys for Git
        "source=${env:HOME}/.ssh,target=/home/vscode/.ssh,type=bind,readonly",
        // Kube directory for Kubernetes control. Autocreates.
        "source=${env:HOME}/.kube,target=/home/vscode/.kube,type=bind,readonly",
        // Docker-outside-Docker for image building
        "source=/var/run/docker.sock,target=/var/run/docker-host.sock,type=bind"
    ]
}

The “overrideCommand” set to false is a requirement of a custom Docker ENTRYPOINT.

C.5. Gitignore

It is a bad thing to check in binaries, coverage reports and secrets. The gitignore here will ensure that Git does not push certain files or directories up.

curl https://raw.githubusercontent.com/github/gitignore/main/Go.gitignore \
  > .gitignore
echo '# Binaries' >> .gitignore
echo bin/ >> .gitignore

D. Use the Environment

D.1 Copy the Template

Using cp -a <template> <project_name>, create a copy of the template directory you set up in section B.

D.2 Build Container and Test

Either open VS Code and open the new directory (CTRL-K, CTRL-O) or cd into the new directory and execute code . VS Code will ask to re-open the folder in a container, to which you naturally respond ‘Yes.’

After a few minutes (you can look at the logs for a progress report), the container will be ready. To test, open the Terminal tab in VS Code then "+" > zsh. To make absolutely sure, type:

go version
kubectl version --short
docker image ls

D.3 Initialize Kubebuilder Project

Since Go and Docker work, start your own work!

mkdir <projectname> && cd <projectname>
kubebuilder init --domain <my.domain> --repo <my.domain/projectname>

This creates boilerplate code. See https://book.kubebuilder.io/introduction for further instructions.

E. References

https://code.visualstudio.com/docs/devcontainers/containers