Get Started With OpenTelemetry Tracing and ASP.NET Core
Transcript
Today we're going to be taking a look at distributed tracing using multiple ASP.NET Core applications. Tracing is one of the main three pillars of an observable system, with the others being metrics and logging. Tracing tells us which components and services are being used in our application through a process called instrumentation. Distributed tracing is just tracing across multiple services under a central trace ID. The most popular way to have distributed tracing is through the OpenTelemetry project.
The OpenTelemetry project provides a specification with four key areas that we need to be aware of. These are spans, events, attributes, and baggage. Spans uniquely identify a request with a start and end time that may also have a parent. In the .NET world we use the Activity type to represent a span, so the terms span and activity are interchangeable. In other languages this is not the case and it should be referred to as a span. Events are a simple text-based element that's added to a trace to denote that something happened at a specific point in time. An example of this may be a cache miss. Attributes are a collection of name-value pairs that you can record as part of a span, for example adding the HTTP status code. Again in the .NET space, attributes are not called attributes, they're actually called tags. This is because we have prior art in the space with the Activity type. Baggage is also a name-value pair collection, but this time baggage travels across the process boundaries whereas attributes do not. For example, you may want to have a user ID flowing all the way through the system, which is where baggage can come in handy.
So let's take a look at the projects that we're going to be using today. I've got two projects based off of the ASP.NET Core web template: Project A and Project B. Project A has a single endpoint which returns "hello from Project A" and the result it gets from Project B. Currently Project B only says "hello from Project B". So in theory, when we call our Project A website it should say "hello from Project A" followed by "hello from Project B". Project B for brevity has already had distributed tracing set up, but we're going to go through and set up Project A with distributed tracing as well. The other thing that we have is a Docker Compose file which has the Jaeger all-in-one image already set up and ready for us, and this is running in the background already.
Our objective for this video is to take the traces from Project A and Project B and have them report into the Jaeger all-in-one image. We'll be able to search this through the UI as well. So if I go to the startup, just to clarify what we have: we can set up .NET to use one of many different activity ID formats, so here we're just going to use the W3C spec. I also give out the request ID in the header just to make it really easy for us to go and grab the traces later. And as I mentioned, we have an HTTP client that calls Project B and just writes the content to the response stream. It's worth pointing out that these are both .NET 5 projects. And just to show you it working, if I hit F5 now we see our "hello from Project A" and "hello from Project B" as we're expecting.
So the first thing we need to do is add four different NuGet packages to our Project A solution. So I'm going to right-click on the project and click Manage NuGet Packages. I'm going to select the include pre-release because at the current time OpenTelemetry is a pre-release project as some parts of the specifications such as metrics are being finalized. And in search I'm going to search for OpenTelemetry and then I'm going to install the Hosting package, the Instrumentation.Http package, the Instrumentation.AspNetCore package, and the Jaeger Exporter package.
The next thing we need to do is make sure the services are added to our service collection. To do this we need to type services.AddOpenTelemetryTracing and this takes a builder inside of it. Inside of this builder is where you register the extensions that you have, such as the Jaeger exporter and the two instrumentation packages that we installed. So first of all I'm going to add the ASP.NET Core instrumentation which monitors all of the incoming HTTP requests. Next we want to monitor the outbound HTTP requests, so we add the HTTP client instrumentation as well. This comes from the HTTP client extension package that we installed. Lastly we need to add the Jaeger exporter, and for now we can leave everything with the defaults. With all the defaults set, the Jaeger exporter is going to look for something on the local machine using port 6831. All of the outbound HTTP requests are going to be captured and all of the inbound HTTP requests are going to be captured.
With our services registered we can now hit F5 to see this all in action. As you can see we've got Project A and Project B loaded up here and I'm just going to refresh a few times just so we get some traffic. Now if we go across to our Jaeger UI, you can see that we have three services underneath the service tab: Project A, Project B, and some Jaeger internal services. So if I select Project A from here and then go down and click Find Traces, just make this full screen for you, then you can see that we've made a few requests on the part of the system. And you can click on any one of these requests to have a look at it in a bit more detail. You can see the total duration of the entire span, how many services were called, and how many spans in general. If I start to click on one of the spans you can start to see different parts of the process, such as the tags that we add here like HTTP status code 200. If I go down to Project B I can see a similar kind of thing.
At this point you may be wondering why do we have three spans when we've only made two requests. This first level one is the incoming request to our web server. This second span is basically saying that we are going to call another service. This service may or may not have distributed tracing capabilities. And finally this last one is the view from the internal service. From my perspective this is actually a really good thing because you get to see any network delays between the two services.
So now let's take a look at how to get the trace ID out of the system and use it as a response header. As I mentioned earlier, in the .NET ecosystem we use the Activity type to represent the spans in the OpenTelemetry specification. The current running activity is always available through the Activity.Current property. However, if you don't have the automatic instrumentation available, we may not always be able to get the current activity as one may not be started, so we need to safeguard this with the null coalescing operator. As you can see, grabbing the trace ID is as simple as calling ToString on the TraceId property on the currently running activity. Then we simply add this as a request ID.
So let's now see how we can put this request ID to use. If we hit F5 again and quickly load up Jaeger as well, what we can do is use the dev tools inside of Chrome that I have here and I'm going to refresh the page to grab the header. So I'm going to click on localhost, I'm going to look down for the response header section, and I'm going to grab the request ID. Exactly as it says, we can see that this trace took approximately 250 milliseconds to complete. I'm going to take this request ID now and take it over to the Jaeger UI and put it into the search box in the top right hand corner and press enter. And now I can see the full trace of my system.
And that pretty much wraps it up for today's video. There is a lot behind the OpenTelemetry project such as different instrumentations, different exporters, and a whole bunch of configurability that you can do as well. What I've shown you in today's video is the easiest way to get up and running really quickly so you get a lot of bang for your buck.
If you enjoyed this video, consider subscribing to the YouTube channel for more content like this.