Dockerising a Go Service

Level: Intermediate

Containers and specifically Docker containers have been playing a very significant role in development and operations arena for quite a while. They have made their ways from our local dev machines all the way to the build and production servers thanks to their insane level of simplicity and the huge amount of value they bring to our tables by massively increasing our productivity.

Whether you want to run Redis or MySQL instances locally  in just a few seconds during development, or you need to package and deploy your final product on a language agnostic build server, or even run your service as an isolated “stateless” unit in production, containers are probably going to be one of the most efficient options on your table.

In this article, I am going to show you how to wrap a Go binary inside a Docker container and how to use the beauty of Go modules to reproduce the build inside the container.

The service

To keep things simple we are going to build a very basic HTTP api with a single GET endpoint which is going to be used as a centralised string encryption service within our company. The service will encrypt the {data} it receives on /encrypt/{data} route and returns the encoded string to the user. We are going to use gorilla mux package to register the aforementioned parameterised route.

Here is how the API looks like:

import (
	"crypto/aes"
	"crypto/cipher"
	"crypto/rand"
	"encoding/base64"
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/gorilla/mux"
)

func main() {
	r := mux.NewRouter()
	r.HandleFunc("/encrypt/{data}", encrypt)
	const port = ":8080"
	fmt.Printf("The server is running on port %s\n", port)
	err := http.ListenAndServe(port, r)
	if err != nil {
		log.Fatal(err)
	}
}

func encrypt(w http.ResponseWriter, req *http.Request) {
	vars := mux.Vars(req)
	data, ok := vars["data"]
	if !ok || len(data) == 0 {
		http.Error(w, "Data is not provided", http.StatusBadRequest)
		return
	}

	encrypted, err := encryptData([]byte(data))
	if err != nil {
		http.Error(w, "Failed to encrypt the data", http.StatusInternalServerError)
		return
	}
	_, err = w.Write([]byte(encrypted))
	if err != nil {
		http.Error(w, "Failed to write the response", http.StatusInternalServerError)
		return
	}
}

func encryptData(input []byte) (string, error) {

	// ** ATTENTION **
	// NEVER EVER EVER commit the encryption key into the source control
	// This is for demonstration purposes ONLY
	key := []byte("16 bytes AES key")

	block, err := aes.NewCipher(key)
	if err != nil {
		return "", err
	}

	cipherText := make([]byte, aes.BlockSize+len(input))
	iv := cipherText[:aes.BlockSize]
	if _, err := io.ReadFull(rand.Reader, iv); err != nil {
		return "", err
	}
	stream := cipher.NewCFBEncrypter(block, iv)
	stream.XORKeyStream(cipherText[aes.BlockSize:], input)

	return base64.URLEncoding.EncodeToString(cipherText), nil
}

Disclaimer

The way the encrypt function is implemented is not the main focus of this article. I am nowhere near being a security expert and this article is not about cryptography in Go. If you are interested to learn how cryptography works in Go, there are so many awesome resources available online, but I strongly recommend the book Practical Cryptography With Go by Kyle Isom. 

Nothing fancy here, but let’s have a look at the module’s dependency chain which is stored in the go.mod file:

module go.xitonix.io/dockered

require (
	github.com/gorilla/context v1.1.1 // indirect
	github.com/gorilla/mux v1.6.2
)

Link

If you are not familiar with the concept of go modules, I would recommend reading Introduction to Go Modules by Roberto Selbach before going further. I will be here waiting for you, I promise.

The aim is to build a Docker image which internally uses go mod to fetch all the required dependencies and eventually go build to compile the code. This is how the image is going to look like:

The Docker file

The life of a Docker image begins from a Docker file. A docker file is nothing but a series of instructions which will get executed from top to bottom. We usually follow the standard naming convention of calling the file Dockerfile, so that it automatically gets picked up by Docker CLI. But if you need to do it differently for whatever reason, you can name the file as you please and explicitly ask Docker to load that file using -f switch of build command.

Let’s add the file to our project:

 

FROM golang:latest

WORKDIR /src
RUN mkdir /api
COPY ./ ./
RUN CGO_ENABLED=0 GOOS=linux go build -o /api/server
EXPOSE 8080
CMD ["/api/server"]

On the first line, we specify the base image. As the above diagram shows, since we are going to need the Go compiler to build our code, the image needs to have the Go toolchain installed. It also needs git in order for go mod to be able to download the dependencies internally.

One option could be using a vanilla Docker image, such as ubuntu or alpine, and install the required tools ourselves before compiling the code. But why bother when there is already a pre-built Go image available to us, right?

On the next line, we set the current working directory inside the image by using WORKDIR command. It’s important to understand that the working directory is internal to the image. That means /src doesn’t need to exist on the host machine. Also, if the working directory does not exist on the image’s filesystem, WORKDIR command will create it for us. 

On line four, we create a directory called api in the root of the image’s filesystem to which we are going to copy the final binary file. 

Next step is to copy everything from the current directory on the host machine (the project directory) into the image by running the COPY command. Once everything is copied into the image, the next step is going to be running go build inside the image exactly the same way we run it hundred times a day on our machines. That will create the binary in the path, specified by -o flag of the build command.

Alright! We are almost there. One more thing to remember is that our web server runs on port 8080. It’s important to understand that a running Docker container (off the image we are building), has an absolutely isolated execution scope. A completely different execution lifecycle, a different filesystem and different everything from the host machine.  Whatever happens inside a Docker image, stays in the Docker image. That means, if we need to talk to the app running inside the container from outside world, we need to make sure that the inbound communication channel is open. To make that happen, we need to ask Docker to expose port 8080, so that any external party could talk to the service. That happens on line seven, where we use EXPOSE command to open the desired port.

Alright! Now that we have got everything ready, the final step is to tell Docker which command to run by default when it runs the container. Since we copied the binary into the /api directory, all we need to do is to let Docker know, where to find it using the CMD command: 

CMD ["/api/server"]

Building the image

It’s time to test our Docker file now. To build the image, we need to use the Docker build command:

docker build .

Note

Don’t forget the dot at the end to specify the context in which you want to build the image, otherwise Docker is not going to be happy.

Perfect! You can see that the image has been successfully created and go mod, has managed to find and install all the dependencies before compiling the code. But there is a tiny little problem here. Let’s run the same build command a few more times, but make sure that you change something in main.go before building the image every time.

As you might have noticed, every single time we build the image, go build downloads all the dependencies over and over again. But is this really necessary? Given that we have neither changed the version of any of the dependencies nor added or removed any packages, we definitely don’t need to refetch them with every build.

Let’s see how to solve this problem and decrease the build time:

FROM golang:latest

WORKDIR /src
RUN mkdir /api
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY ./ ./
RUN CGO_ENABLED=0 GOOS=linux go build -o /api/server
EXPOSE 8080
CMD ["/api/server"]

The trick is to break the build process into two steps. Step one is to explicitly ask go mod to download the dependencies, right at the beginning, and step two is to build the code as usual. With this technique, go build will be able to find what it needs and it won’t try to download them with every build anymore.

Even if we change the source code, as long as go.mod and go.sum files are not altered, the go runtime will use the cached dependencies inside the image without downloading them again.

Here is the output of building the new image for the second time:

As you can see, Docker is smart enough to realise that go.mod and go.sum files have not been changed since the last build, so it loads the output of COPY and RUN go mod download commands from the cache.

On the other hand, since I changed the source code in main.go before I rebuild the image, we can see that go build is not coming from the cache!

TIP

In case you wonder where the image name xitonix/enc-server is coming from, the answer is the image tag. You can tag your image at the time of building by using -t switch.

docker build -t xitonix/enc-server:latest .

The naming convention is [namespace]/[image name]:tag, in which the namespace is usually your docker hub account name and the tag is the version of the image. If you don’t specify the tag, Docker will choose latest by default. 

You can also use docker tag command to tag the image after building.

Private Github Repositories

Before we use the image to run the container I would like to show you a technique about how to enable go modules when go mod needs to fetch some or all of your dependencies from a private github repository.

Before go modules, we used to have all the dependencies downloaded locally into the vendor directory (most likely you used a ssh key or your AD credentials on your machine to access the private repo and vendorise the dependencies). That means at the time of building the image, running COPY . . would have copied all the requirements into the file, making everything ready for go build to proceed.

But with go modules, this is not the case anymore. You need to pull the dependencies from within the image context where the Github SSH key or the credentials on the host machine won’t work anymore (remember that a Docker container is a fully isolated box). A workaround is to use Github’s personal tokens and configure git inside the image before running go mod download:

COPY go.mod .
COPY go.sum .
ARG TOKEN
RUN git config --global url."https://$TOKEN:[email protected]/".insteadOf "https://git.mycompany.com" 
RUN go mod download COPY ./ ./ 

Dockerfile with Variables

As you would hopefully agree with me, it’s always a good practice not to push any sensitive information- such as the Github token above- into the source code. As you can see in the snippet above, we used another nice feature of Docker to parameterise the Docker file. Wherever we choose to store the token (A Teamcity environment variable for example), we need to use the docker build‘s --build-arg flag to pass the value into the Docker file at the time of building:

docker build -t xitonix/enc-server --build-arg TOKEN=abcd .

If you are interested to know more about Docker build arguments, please click here.

Running the container

Let’s see if the image that we just built together works as we expect. Cross your fingers fellas! 

In order to run the container, we need to use the Docker’s run command and ask it to execute our image. Here we go:

docker run --rm -p 8080:8080 xitonix/enc-server:latest

The Command Explained

--rm: Tells Docker to delete the container once the execution is finished. This is to keep our docker cache clean and make sure that we don’t end up having container zombies dangling around, taking the precious space on our hard drive. We need it for the next season of Game of Thrones soon ;).

-p will setup the port mapping between the container and our local machine. Given that we exposed port 8080 inside the container, we can use the same port on our machine (if not taken by another applications) to forward the traffic from the host to the container. It could be any available port on your machine. For example to use 5000, you need to setup the binding as -p 5000:8000. Once the container is up and running, we can test our server by simply curling into the external port:

curl localhost:8080/encrypt/my_password
rSzKt7Ax-3Qk7e-kXROTv9eMtPAxGC_z6DDm

Bingo! It works like a charm! Excellent!

What’s next?

We got the image ready and the container running on our machine. Now it’s time to share it with our friend to run and test it on his, before shipping it to production. Let’s have a look at our local image registry and see our masterpiece first! Run docker image ls to see the list of the images you have on your machine:

Hmmmm!? Seriously? That’s almost 1 GB for running a simple HTTP API service with one endpoint only! That’s not good, is it?

Well… considering that the image contains a fully fledged operating system, it’s not that bad. But maybe we can do something to shrink the size a little bit. The truth is, once we build the binary, we certainly don’t need the artefacts which were originally included in the golang base image. Things like the Go toolchain, git, etc. To be honest, we can also drop a lot of OS level binaries and tools, but how?

The answer to that question is Docker’s multi-stage builds. Using multi-stage builds we can break the build into two different stages. Stage one, is to use the same golang image to compile the code and create the binary, and stage two is to use a minimal image called scratch as the base, and copy the binary from the first stage into the final image.

Here is the new multi-stage Docker file:

FROM golang:latest AS compile

WORKDIR /src
RUN mkdir /api
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY ./ ./
RUN CGO_ENABLED=0 GOOS=linux go build -o /api/server

FROM scratch
COPY --from=compile /api/server /api/
EXPOSE 8080
CMD ["/api/server"]

All we did differently was to label the first stage using an AS clause, and reference that label at the second stage to copy the binary file --from the previous stage.

Let’s rebuild our image and see if the image size has been reduced:

Holy Gopher! What did just happen? Did the image size really shrink from 800 MB to 6 MB?!! That’s mind blowing, isn’t it?

We can probably shave a few more bytes off the image by removing the debug symbols from the binary at the time of compiling the code using  ldflags :

go build -ldflags '-w -s' -o /api/server

And here is the final image:

NOTE

One thing to keep in mind when building production-ready Docker images, is that the image size is not always the main concern. Some people would argue that deploying such a minimal image to production will introduce security risks. Depends on your requirements, you might instead choose to go with a vanilla ubuntu image or any other base images that suits your needs the best.

Summary

In this tutorial we learned how to,

  • Run basic Docker commands to build an image and run a container
  • Use golang base image to build a Go service.
  • Configure git to fetch packages from a private Github repository by utilising Docker build arguments
  • And finally use Docker multi-stage builds and Go ldflags to reduce the final image size.

You can download the source code for this tutorial from here.

The final Docker image is also available in my Docker hub. To pull the image and run the container:

docker run --rm -p 8080:8080 xitonix/enc-server

4 thoughts on “Dockerising a Go Service

  1. >>Some people would argue that deploying such a minimal image to production
    >>will introduce security risks.
    I’d like to understand how we can introduce security risk by removing not needed apps and operating system dependencies?

    1. Using a minimal image not only would not increase the security risk perse, but also reduces the attack surface. The scratch image for example has completely removed the user land from the image which reduces the risk of impersonation exploits. The argue is mainly about how people may use scratch. People might get a bit distracted with having a small image and forget about the small things such as SSL support which is removed from the base image. Focusing too much on the image size might make it easier to overlook something and get it wrong. I am not a security expert but I thought this was something to be aware of when Dockerising our services. More than happy to hear other thoughts on this. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *