Micronaut Tracing

Adds Distributed Tracing Support

Version:

1 Introduction

When operating Microservices in production it can be challenging to troubleshoot interactions between Microservices in a distributed architecture.

To solve this problem, a way to visualize interactions between Microservices in a distributed manner can be critical. Currently, there are various distributed tracing solutions, the most popular of which are Zipkin and Jaeger, both of which provide different levels of support for the Open Tracing API.

Micronaut features integration with both Zipkin and Jaeger (via the Open Tracing API).

Tracing Annotations

The io.micronaut.tracing.annotation package contains annotations that can be declared on methods to create new spans or continue existing spans.

The available annotations are:

  • The @NewSpan annotation creates a new span, wrapping the method call or reactive type.

  • The @ContinueSpan annotation continues an existing span, wrapping the method call or reactive type.

  • The @SpanTag annotation can be used on method arguments to include the value of the argument within a Span’s tags. When you use @SpanTag on an argument, you must either annotate the method with @NewSpan or @ContinueSpan.

The following snippet presents an example of using the annotations:

Using Trace Annotations
@Singleton
class HelloService {

    @NewSpan("hello-world") (1)
    public String hello(@SpanTag("person.name") String name) { (2)
        return greet("Hello " + name);
    }

    @ContinueSpan (3)
    public String greet(@SpanTag("hello.greeting") String greet) {
        return greet;
    }
}
1 The @NewSpan annotation starts a new span
2 Use @SpanTag to include method arguments as tags for the span
3 Use the @ContinueSpan annotation to continue an existing span and incorporate additional tags using @SpanTag

Tracing Instrumentation

In addition to explicit tracing tags, Micronaut includes a number of instrumentations to ensure that the Span context is propagated between threads and across Microservice boundaries.

These instrumentations are found in the io.micronaut.tracing.instrument package and include Client Filters and Server Filters to propagate the necessary headers via HTTP.

Tracing Beans

If the Tracing annotations and existing instrumentations are not sufficient, Micronaut’s tracing integration registers a io.opentracing.Tracer bean which exposes the Open Tracing API and can be dependency-injected as needed.

Depending on the implementation you choose, there are also additional beans. For example for Zipkin brave.Tracing and brave.SpanCustomizer beans are available too.

2 Release History

For this project, you can find a list of releases (with release notes) here:

3 Tracing with Jaeger

Jaeger is a distributed tracing system developed at Uber that is more or less the reference implementation for Open Tracing.

Running Jaeger

The easiest way to get started with Jaeger is with Docker:

$ docker run -d \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.6

Navigate to http://localhost:16686 to access the Jaeger UI.

See Getting Started with Jaeger for more information.

Sending Traces to Jaeger

Using the CLI

If you create your project using the Micronaut CLI, supply the tracing-jaeger feature to include Jaeger tracing in your project:

$ mn create-app my-app --features tracing-jaeger

To send tracing spans to Jaeger, add the micronaut-tracing-jaeger dependency in your build:

implementation("io.micronaut.tracing:micronaut-tracing-jaeger")
<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-jaeger</artifactId>
</dependency>

Then enable Jaeger tracing in your configuration (potentially only your production configuration):

application.yml
tracing:
  jaeger:
    enabled: true

By default, Jaeger will be configured to send traces to a locally running Jaeger agent.

Jaeger Configuration

There are many configuration options available for the Jaeger client that sends Spans to Jaeger, and they are generally exposed via the JaegerConfiguration class. Refer to the Javadoc for available options.

Below is an example of customizing JaegerConfiguration configuration:

Customizing Jaeger Configuration
tracing:
  jaeger:
    enabled: true
    sampler:
      probability: 0.5
    sender:
      agentHost: foo
      agentPort: 5775
    reporter:
      flushInterval: 2000
      maxQueueSize: 200
    codecs: W3C,B3,JAEGER

You can also optionally dependency-inject common configuration classes into JaegerConfiguration such as io.jaegertracing.Configuration.SamplerConfiguration just by defining them as beans. Likewise, a custom io.opentracing.ScopeManager can be injected into JaegerTracerFactory. See the API for JaegerConfiguration and JaegerTracerFactory for available injection points.

Filtering HTTP spans

It may be useful to exclude health-checks and other HTTP requests to your service. This can be achieved by adding a list of regular expression patterns to your configuration:

Filtering HTTP request spans
tracing:
  jaeger:
    enabled: true
  exclusions:
    - /health
    - /env/.*

4 Tracing with Zipkin

Zipkin is a distributed tracing system. It helps gather timing data to troubleshoot latency problems in microservice architectures. It manages both the collection and retrieval of this data.

Running Zipkin

The quickest way to get up and started with Zipkin is with Docker:

Running Zipkin with Docker
$ docker run -d -p 9411:9411 openzipkin/zipkin

Navigate to http://localhost:9411 to view traces.

Sending Traces to Zipkin

Using the CLI

If you create your project using the Micronaut CLI, supply the tracing-zipkin feature to include Zipkin tracing in your project:

$ mn create-app my-app --features tracing-zipkin

To send tracing spans to Zipkin, add the micronaut-tracing-zipkin dependency in your build:

implementation("io.micronaut.tracing:micronaut-tracing-zipkin")
<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-zipkin</artifactId>
</dependency>

Then enable ZipKin tracing in your configuration (potentially only your production configuration):

application.yml
tracing:
  zipkin:
    enabled: true

Customizing the Zipkin Sender

To send spans you configure a Zipkin sender. You can configure a HttpClientSender that sends Spans asynchronously using Micronaut’s native HTTP client with the tracing.zipkin.http.url setting:

Configuring Multiple Zipkin Servers
tracing:
  zipkin:
    enabled: true
    http:
      url: http://localhost:9411

It is unlikely that sending spans to localhost will be suitable for production deployment, so you generally need to configure the location of one or more Zipkin servers for production:

Configuring Multiple Zipkin Servers
tracing:
  zipkin:
    enabled: true
    http:
      urls:
        - http://foo:9411
        - http://bar:9411
In production, setting TRACING_ZIPKIN_HTTP_URLS environment variable with a comma-separated list of URLs also works.

Alternatively, to use a different zipkin2.reporter.Sender implementation, you can define a bean of type zipkin2.reporter.Sender and it will be used instead.

Zipkin Configuration

There are many configuration options available for the Brave client that sends Spans to Zipkin, and they are generally exposed via the BraveTracerConfiguration class. Refer to the Javadoc for available options.

Below is an example of customizing Zipkin configuration:

Customizing Zipkin Configuration
tracing:
  zipkin:
    enabled: true
    traceId128Bit: true
    sampler:
      probability: 1

You can also optionally dependency-inject common configuration classes into BraveTracerConfiguration such as brave.sampler.Sampler just by defining them as beans. See the API for BraveTracerConfiguration for available injection points.

Filtering HTTP spans

It may be useful to exclude health-checks and other HTTP requests to your service. This can be achieved by adding a list of regular expression patterns to your configuration:

Filtering HTTP request spans
tracing:
  zipkin:
    enabled: true
  exclusions:
    - /health
    - /env/.*

5 Tracing with OpenTelemetry

The Micronaut Open Telemetry module uses Open Telemetry Autoconfigure SDK to configure Open Telemetry for tracing. For some functionalities you have to add additional dependencies. The Default values that are defined inside the Micronaut which values might be different from the default ones inside Open Telemetry Autoconfigure SDK module are:

  • otel.traces.exporter = none

  • otel.metrics.exporter = none

  • otel.logs.exporte = none

  • otel.service.name = value of the application.name

OpenTelemetry annotations

The io.micronaut.tracing.opentelemetry.processing package contains transformers and mappers that enables usage of Open Telemetry annotations.

5.1 OpenTelemetry Annotations

The io.micronaut.tracing.opentelemetry.processing package contains transformers and mappers that enables usage of Open Telemetry annotations.

To enable Open telemetry annotations you have to add next annotation processor in your dependency block:

annotationProcessor("io.micronaut.tracing:micronaut-tracing-opentelemetry-annotation:4.2.1")
<annotationProcessorPaths>
    <path>
        <groupId>io.micronaut.tracing</groupId>
        <artifactId>micronaut-tracing-opentelemetry-annotation</artifactId>
        <version>4.2.1</version>
    </path>
</annotationProcessorPaths>

The Open Tracing annotations that are defined inside the The io.micronaut.tracing.annotation package are also available for usage inside Open Telemetry.

5.2 OpenTelemetry Exporters

In your project you can specify exporters that you want to use. The default one is set to "none" value which means by default there are no exporter registered. The available values are defined on the Open Telemetry Autoconfigure SDK documentation.

For each exporter that you want to use you have to specify it inside configuration, and you have to add required dependency:

  • OpenTelemetry Protocol exporter: otlp

    implementation("io.opentelemetry:opentelemetry-exporter-otlp")
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>

  • Logging exporter: logging

    implementation("io.opentelemetry:opentelemetry-exporter-logging")
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-logging</artifactId>
    </dependency>

  • Jaeger exporter: jaeger

    implementation("io.opentelemetry:opentelemetry-exporter-jaeger")
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-jaeger</artifactId>
    </dependency>

  • Google Cloud Trace: google_cloud_trace

    implementation("com.google.cloud.opentelemetry:exporter-auto")
    <dependency>
        <groupId>com.google.cloud.opentelemetry</groupId>
        <artifactId>exporter-auto</artifactId>
    </dependency>

  • Zipkin exporter: zipkin

    implementation("io.opentelemetry:opentelemetry-exporter-zipkin")
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-zipkin</artifactId>
    </dependency>

Example configuration for the Zipkin exporter:

application.yml
otel:
  traces:
    exporter: zipkin

5.3 OpenTelemetry Propagators

In your project you can specify propagators that you want to use. The default one is set to "tracecontext, baggage". The available values are defined on the Open Telemetry Autoconfigure SDK documentation.

To use AWS X-Ray propagator inside your application you have to add next dependency in your project:

implementation("io.opentelemetry:opentelemetry-extension-aws")
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-extension-aws</artifactId>
</dependency>

And the "xray" has to be added inside configuration file.

application.yml
otel:
  traces:
    exporter: otlp
  propagators: tracecontext, baggage, xray

5.4 ID Generator

ID Generator

Some custom vendor may require the span traceId in different format from the default one. You can provide your own bean of type IdGenerator. For an example, AWS X-Ray requires a specific format for their tracing identifiers. Add the following dependency, to register an instance of AwsXrayIdGenerator as a bean of type IdGenerator.

implementation("io.opentelemetry.contrib:opentelemetry-aws-xray")
<dependency>
    <groupId>io.opentelemetry.contrib</groupId>
    <artifactId>opentelemetry-aws-xray</artifactId>
</dependency>

To successfully export traces to the AWS X-Ray you have to run AWS Open Telemetry Collector that will periodically send traces to the AWS.

5.5 HTTP Server and Client

To enable creating span objects on the every HTTP server request, client request, server response and client response you have to add next depedency:

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")
<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
</dependency>

Filtering HTTP spans

It may be useful to exclude health-checks and other HTTP requests to your service. This can be achieved by adding a list of regular expression patterns to your configuration:

Filtering HTTP request spans
otel:
  exclusions:
    - /health
    - /env/.*

Add HTTP Headers into request spans

If you want you can add additional Http Headers inside your span objects. You can specify different headers for client request, client response, server request and server response.

Adding HTTP Headers into request spans
otel:
  http:
    client:
      request-headers:
        - X-From-Client-Request
      response-headers:
        - X-From-Client-Response
    server:
      request-headers:
        - X-From-Server-Request
      response-headers:
        - X-From-Server-Response

5.6 gRPC Server and Client

To enable creating span objects on the every GRPC server request, client request, server response and client response. You have to add next depedency:

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-grpc")
<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-grpc</artifactId>
</dependency>

5.7 AWS SDK Instrumentation

Include the following dependency to instrument the AWS SDK:

implementation("io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2")
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-aws-sdk-2.2</artifactId>
</dependency>

Additionally, include Micronaut AWS SDK v2 dependency:

implementation("io.micronaut.aws:micronaut-aws-sdk-v2")
<dependency>
    <groupId>io.micronaut.aws</groupId>
    <artifactId>micronaut-aws-sdk-v2</artifactId>
</dependency>

micronaut-aws-sdk-v2 dependency creates a bean of type SdkClientBuilder. To instrument the AWS SDK, Micronaut OpenTelemetry registers a tracing interceptor via a bean creation listener for the bean of type SdkClientBuilder.

5.8 AWS Resource Detectors

AWS Resource detectors enrich traces with AWS infrastructure information.

To use AWS resource detectors include the following dependency:

implementation("io.opentelemetry:opentelemetry-sdk-extension-aws")
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-extension-aws</artifactId>
</dependency>

and provide a bean of type ResourceProvider

package io.micronaut.tracing.opentelemetry;

import io.micronaut.core.annotation.NonNull;
import io.opentelemetry.sdk.resources.Resource;
import jakarta.inject.Singleton;
import io.opentelemetry.sdk.extension.aws.resource.Ec2Resource;

@Singleton
public class AwsResourceProvider implements ResourceProvider {
    @Override
    @NonNull
    public Resource resource() {
        return Resource.getDefault()
            .merge(Ec2Resource.get());
    }
}

6 Repository

You can find the source code of this project in this repository: