Running an OpenTelemetry Collector locally with Docker

Transcript

In my other videos on OpenTelemetry, we've been taking a look at how to export logs, metrics, and traces by using the console exporter. In a production environment, what you're more likely to want to do is to send all of this telemetry to one or more different destinations, depending on your use case. In this video, you're going to learn the differences between the two different types of collector that are available, how to run the collector inside of Docker, and how to configure the OpenTelemetry Collector.

The OpenTelemetry Collector has two different flavors: the normal flavor and the contributor flavor. Depending on your use case depends on which flavor you want to use. The links of both of these will be in the description below. Both flavors of the collector have three distinct parts: receivers, processors, and exporters.

The standard OpenTelemetry Collector has a very limited set, all based around the OpenTelemetry Protocol. This means that the only receiver that's available is the gRPC and HTTP version of the OpenTelemetry Protocol. Likewise, the exporter only has an OpenTelemetry Protocol version. The processors are also somewhat limited. There is a basic batch processor and a basic memory limiting processor.

The contributor version of the collector has a lot more options for you to pick from. For example, you have the Prometheus receiver, the StatsD receiver. You also have more processors such as attribute processing, and exporters such as Datadog. This is the version of the collector that we'll be using in this tutorial.

So if you could only have one of logs, metrics, or traces, which one would you have in your applications? Let me know in the comments below.

So if you now go to the Docker file, you'll be able to see that we use two distinct images here. The first is Zipkin. This is where we're going to send all of our traces to in a bit. And secondly is the OpenTelemetry Contrib image. With the OpenTelemetry Contrib image, there are a couple of different points here that we need to be paying attention to. The first one is the config flag that we're passing in the command. This tells OpenTelemetry where to go and look for the configuration file that we're going to pass in. The next part is the volume mount that we're going to have from our local file with our OpenTelemetry configuration. This is going to be mounted to the location that we specified in the config parameter. Once this is spun up, we'll have two different images that are running on our local Docker instance. Any traces that are sent to our OpenTelemetry Collector will automatically be forwarded to our Zipkin collector.

So let's go and take a look at our configuration file, see how this all works. Inside the configuration file, each top level element falls under one of two categories: declaration or usage. We have four main top level declaration types: a receiver, an exporter, a processor, or an extension. Under each of these four sections, you put down the different elements that you want. So for example, under the receivers section, I've put two different receivers: Zipkin and OTLP. As you can see from the configuration on screen, I'm not doing anything special with them, but this is where you would also be able to set things like port numbers.

Next, under the processors section, I'm adding a simple batch processor with a timeout of 10 seconds. This is so that I batch everything for up to 10 seconds on the collector before sending it over to Zipkin. Under my extensions section, I've just added the health check option. This is useful for when we're running in production environments and we want to make sure that the collector is alive. Again here, I'm not specifying any other options. Finally, under the exporters section, I've added Zipkin. As you can see from my configuration here, I've configured Zipkin with a local endpoint.

So far, this configuration doesn't do much. What we need to do now is configure the service element. The service element has two parts to it: extensions and the pipeline. Under the extensions element, we configure an array referencing the elements that we previously declared under the extensions section. Only when the extension is added into this section will it be applied when the collector starts.

Underneath the pipeline section, we have three elements that we need to configure: receivers, processors, and exporters. Here we follow exactly the same method as we did for the extensions. We again specify an array referencing the declarations that we had earlier. Again, unless added to one of these sections, it will not be used.

So now we've gone through our configuration. I'm going to spin up an application that I built earlier which just uses the OpenTelemetry Protocol exporter on its default port. After a few attempts of running the web application, we wait a few seconds for the batch processor to work and then we go and check our Zipkin UI. You'll be able to see all of the traces have been forwarded.

If you enjoyed this video, consider subscribing to the YouTube channel for more content like this.