How to set up OpenTelemetry Collector on Kubernetes

Transcript

In the last video we went through how to configure an OpenTelemetry Collector locally using Docker. In this video you'll learn how to configure the OpenTelemetry Operator, how to configure instances of OpenTelemetry Collectors, how to automatically inject OpenTelemetry Collectors as sidecars, and how to do automatic instrumentation of your applications where the language is supported.

Now before we begin, it is important to note that you do need cert-manager installed prior to installing the Kubernetes OpenTelemetry Operator. There are instructions on how to install cert-manager in the description below. There are two different ways that we can install the Kubernetes Operator for OpenTelemetry. The first is by a standard kubectl apply and the second is via the Helm chart, which I would recommend. To keep things simple for this tutorial however, I'm going to use the kubectl approach. With each release of the Kubernetes Operator for OpenTelemetry, the OpenTelemetry project provides a YAML file that can be downloaded and run automatically using kubectl apply -f and the path to the installation URL. This URL is also in the description below. Once this has been applied, we are now ready to start configuring our cluster.

But before we continue, what's your preferred way of configuring Kubernetes clusters? Let me know in the comments below.

The OpenTelemetry Operator installs a Custom Resource Definition called OpenTelemetryCollector. This lives under the opentelemetry.io API version. This resource allows us to configure two different things. The first of these is the deployment mode. This can either be run as a DaemonSet, automatically injected as a sidecar, or a normal Deployment, depending on your use case. The other thing it allows you to do is specify the configuration that will be used for that collector. If required, you can have multiple different resources each specifying a different OpenTelemetry Collector configuration and then you can select which configuration to use in your applications, as we'll see a little bit later on.

For every OpenTelemetry Collector resource that you create, you should see one or more collectors being spun up as pods depending on your configuration. If you have selected sidecar injection, then we need to add an annotation onto our pods. Alternatively you can add it onto the namespace for a namespace-wide sidecar injection. The annotation that we need to add is sidecar.opentelemetry.io/inject. The value can be one of three things: the string "false" which prevents any sidecar injection, the string "true" which will pick one of the registered collector configurations, or you can specify the name of the collector configuration itself. It is worth noting that the OpenTelemetry Operator does use a hierarchy of where to search. It will start searching on the local pod first before falling back to the namespace. This means you can have two different configurations, one for your namespace and then override it on your local pod.

One of the other things that we can set up is auto-instrumentation. At the time of this video, only three languages are supported: Java, Node.js, and Python. There is an open GitHub issue for .NET support for auto-instrumentation. I'll put a link to this in the description below should you wish to subscribe to it and want to be notified when that auto-instrumentation is being released.

Before we add the auto-instrumentation annotation to the pod, we first need to create the instrumentation configuration. To do this we create a new Kubernetes resource from the opentelemetry.io namespace called Instrumentation. This resource provides the auto-instrumentation configuration such as the endpoint of the OpenTelemetry Collector. In this case I am running a sidecar so I use the container name as the host inside of the configuration. After this you are free to choose which propagators and samplers you're going to have inside of your configuration.

If you are already on one of these supported languages, then you can add an annotation in the exact same way as before. Instead, this time the annotation needs to be instrumentation.opentelemetry.io/inject- and then the language name. So for Node.js this will be /inject-nodejs. Once this is done, OpenTelemetry will add a new image to your pod which will automatically collect things like traces and send them through to your OpenTelemetry Collector. If you need to, you can restrict which container names inside your pod are collected by auto-instrumentation.

If you enjoyed this video, consider subscribing to the YouTube channel for more content like this.