To solve this problem, a way to visualize interactions between Microservices in a distributed manner can be critical. Currently, there are various distributed tracing solutions, the most popular of which are Zipkin and Jaeger, both of which provide different levels of support for the Open Tracing API.

Micronaut features integration with both Zipkin and Jaeger (via the Open Tracing API).

Tracing Annotations

The io.micronaut.tracing.annotation package contains annotations that can be declared on methods to create new spans or continue existing spans.

The available annotations are:

The @NewSpan annotation creates a new span, wrapping the method call or reactive type.
The @ContinueSpan annotation continues an existing span, wrapping the method call or reactive type.
The @SpanTag annotation can be used on method arguments to include the value of the argument within a Span’s tags. When you use @SpanTag on an argument, you must either annotate the method with @NewSpan or @ContinueSpan.

The following snippet presents an example of using the annotations:

Using Trace Annotations

@Singleton
class HelloService {

    @NewSpan("hello-world") (1)
    public String hello(@SpanTag("person.name") String name) { (2)
        return greet("Hello " + name);
    }

    @ContinueSpan (3)
    public String greet(@SpanTag("hello.greeting") String greet) {
        return greet;
    }
}

1	The @NewSpan annotation starts a new span
2	Use @SpanTag to include method arguments as tags for the span
3	Use the @ContinueSpan annotation to continue an existing span and incorporate additional tags using `@SpanTag`

Tracing Instrumentation

In addition to explicit tracing tags, Micronaut includes a number of instrumentations to ensure that the Span context is propagated between threads and across Microservice boundaries.

These instrumentations are found in the io.micronaut.tracing.instrument package and include Client Filters and Server Filters to propagate the necessary headers via HTTP.

Tracing Beans

If the Tracing annotations and existing instrumentations are not sufficient, Micronaut’s tracing integration registers a io.opentracing.Tracer bean which exposes the Open Tracing API and can be dependency-injected as needed.

Depending on the implementation you choose, there are also additional beans. For example for Zipkin brave.Tracing and brave.SpanCustomizer beans are available too.

2 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-tracing/releases

3 Breaking Changes

This section will document breaking changes that may happen during milestone or release candidate releases, as well as major releases eg (1.x.x → 2.x.x).

Micronaut Tracing 4.5.0 breaking changes

In version 1.19.0 of OpenTelemetry annotations have moved from io.opentelemetry.extension.annotations to io.opentelemetry.instrumentation.annotations. If you want to continue to use them, you have to update package name.

Micronaut Tracing 5.0.0-M2 breaking changes

The Micronaut Tracing Zipkin module (io.micronaut.tracing:micronaut-tracing-zipkin) has been renamed and separated in two new modules:

Micronaut Tracing Brave (io.micronaut.tracing:micronaut-tracing-brave)
Micronaut Tracing Brave HTTP (io.micronaut.tracing:micronaut-tracing-brave-http)

if you are using OpenTracing and Micronaut Tracing Zipkin module, you have to change io.micronaut.tracing:micronaut-tracing-zipkin dependency to io.micronaut.tracing:micronaut-tracing-brave-http. The Micronaut Tracing Brave HTTP brings HTTP filters for auto instrumentation of your requests. If you don’t need HTTP filters you can only add io.micronaut.tracing:micronaut-tracing-brave dependency.

Micronaut Tracing 7.0.0 Breaking changes

The following dependencies have been updated:

Open Telemetry 1.36
Open Telemetry Instrumentation 1.33.1

The Open Telemetry updates required updating the following Zipkin dependencies to a new major version:

Zipkin Reporter 3.3.0
Zipkin Brave 6.0.2

The only change due to the above dependency updates that might affect end user code is that the errorParser property has been removed from io.micronaut.tracing.brave.BraveTracerConfiguration as the brave.ErrorParser type no longer exists.

The io.micronaut.tracing.zipkin.http.client.HttpClientSender class is now deprecated as it implements a deprecated Zipkin API. This class will be replaced with an implementation based on the Zipkin 3 APIs in a future release.

4 Tracing with Jaeger

Jaeger is a distributed tracing system developed at Uber that is more or less the reference implementation for Open Tracing.

Running Jaeger

The easiest way to get started with Jaeger is with Docker:

$ docker run -d \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.6

Navigate to http://localhost:16686 to access the Jaeger UI.

See Getting Started with Jaeger for more information.

Sending Traces to Jaeger

Using the CLI

If you create your project using the Micronaut CLI, supply the tracing-jaeger feature to include Jaeger tracing in your project:

$ mn create-app my-app --features tracing-jaeger

To send tracing spans to Jaeger, add the micronaut-tracing-jaeger dependency in your build:

implementation("io.micronaut.tracing:micronaut-tracing-jaeger")

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-jaeger</artifactId>
</dependency>

Then enable Jaeger tracing in your configuration (potentially only your production configuration):

tracing.jaeger.enabled=true

tracing:
  jaeger:
    enabled: true

[tracing]
  [tracing.jaeger]
    enabled=true

tracing {
  jaeger {
    enabled = true
  }
}

{
  tracing {
    jaeger {
      enabled = true
    }
  }
}

{
  "tracing": {
    "jaeger": {
      "enabled": true
    }
  }
}

By default, Jaeger will be configured to send traces to a locally running Jaeger agent.

Jaeger Configuration

There are many configuration options available for the Jaeger client that sends Spans to Jaeger, and they are generally exposed via the JaegerConfiguration class. Refer to the Javadoc for available options.

Below is an example of customizing JaegerConfiguration configuration:

Customizing Jaeger Configuration

tracing.jaeger.enabled=true
tracing.jaeger.sampler.probability=0.5
tracing.jaeger.sender.agentHost=foo
tracing.jaeger.sender.agentPort=5775
tracing.jaeger.reporter.flushInterval=2000
tracing.jaeger.reporter.maxQueueSize=200
tracing.jaeger.codecs=W3C,B3,JAEGER

tracing:
  jaeger:
    enabled: true
    sampler:
      probability: 0.5
    sender:
      agentHost: foo
      agentPort: 5775
    reporter:
      flushInterval: 2000
      maxQueueSize: 200
    codecs: W3C,B3,JAEGER

[tracing]
  [tracing.jaeger]
    enabled=true
    codecs="W3C,B3,JAEGER"
    [tracing.jaeger.sampler]
      probability=0.5
    [tracing.jaeger.sender]
      agentHost="foo"
      agentPort=5775
    [tracing.jaeger.reporter]
      flushInterval=2000
      maxQueueSize=200

tracing {
  jaeger {
    enabled = true
    sampler {
      probability = 0.5
    }
    sender {
      agentHost = "foo"
      agentPort = 5775
    }
    reporter {
      flushInterval = 2000
      maxQueueSize = 200
    }
    codecs = "W3C,B3,JAEGER"
  }
}

{
  tracing {
    jaeger {
      enabled = true
      sampler {
        probability = 0.5
      }
      sender {
        agentHost = "foo"
        agentPort = 5775
      }
      reporter {
        flushInterval = 2000
        maxQueueSize = 200
      }
      codecs = "W3C,B3,JAEGER"
    }
  }
}

{
  "tracing": {
    "jaeger": {
      "enabled": true,
      "sampler": {
        "probability": 0.5
      },
      "sender": {
        "agentHost": "foo",
        "agentPort": 5775
      },
      "reporter": {
        "flushInterval": 2000,
        "maxQueueSize": 200
      },
      "codecs": "W3C,B3,JAEGER"
    }
  }
}

You can also optionally dependency-inject common configuration classes into JaegerConfiguration such as io.jaegertracing.Configuration.SamplerConfiguration just by defining them as beans. Likewise, a custom io.opentracing.ScopeManager can be injected into JaegerTracerFactory. See the API for JaegerConfiguration and JaegerTracerFactory for available injection points.

Filtering HTTP spans

It may be useful to exclude health-checks and other HTTP requests to your service. This can be achieved by adding a list of regular expression patterns to your configuration:

Filtering HTTP request spans

tracing.jaeger.enabled=true
tracing.exclusions[0]=/health
tracing.exclusions[1]=/env/.*

tracing:
  jaeger:
    enabled: true
  exclusions:
    - /health
    - /env/.*

[tracing]
  exclusions=[
    "/health",
    "/env/.*"
  ]
  [tracing.jaeger]
    enabled=true

tracing {
  jaeger {
    enabled = true
  }
  exclusions = ["/health", "/env/.*"]
}

{
  tracing {
    jaeger {
      enabled = true
    }
    exclusions = ["/health", "/env/.*"]
  }
}

{
  "tracing": {
    "jaeger": {
      "enabled": true
    },
    "exclusions": ["/health", "/env/.*"]
  }
}

Jaeger Guides

See the following guides to learn more about distributed tracing with Jaeger in the Micronaut Framework:

5 Tracing with Zipkin

Zipkin is a distributed tracing system. It helps gather timing data to troubleshoot latency problems in microservice architectures. It manages both the collection and retrieval of this data.

Running Zipkin

The quickest way to get up and started with Zipkin is with Docker:

Running Zipkin with Docker

$ docker run -d -p 9411:9411 openzipkin/zipkin

Navigate to http://localhost:9411 to view traces.

Sending Traces to Zipkin

Using the CLI

If you create your project using the Micronaut CLI, supply the tracing-zipkin feature to include Zipkin tracing in your project:

$ mn create-app my-app --features tracing-zipkin

To send tracing spans to Zipkin, add the micronaut-tracing-brave-http dependency in your build:

implementation("io.micronaut.tracing:micronaut-tracing-brave-http")

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-brave-http</artifactId>
</dependency>

Then enable ZipKin tracing in your configuration (potentially only your production configuration):

tracing.zipkin.enabled=true

tracing:
  zipkin:
    enabled: true

[tracing]
  [tracing.zipkin]
    enabled=true

tracing {
  zipkin {
    enabled = true
  }
}

{
  tracing {
    zipkin {
      enabled = true
    }
  }
}

{
  "tracing": {
    "zipkin": {
      "enabled": true
    }
  }
}

Customizing the Zipkin Sender

To send spans you configure a Zipkin sender. You can configure a HttpClientSender that sends Spans asynchronously using Micronaut’s native HTTP client with the tracing.zipkin.http.url setting:

Configuring Multiple Zipkin Servers

tracing.zipkin.enabled=true
tracing.zipkin.http.url=http://localhost:9411

tracing:
  zipkin:
    enabled: true
    http:
      url: http://localhost:9411

[tracing]
  [tracing.zipkin]
    enabled=true
    [tracing.zipkin.http]
      url="http://localhost:9411"

tracing {
  zipkin {
    enabled = true
    http {
      url = "http://localhost:9411"
    }
  }
}

{
  tracing {
    zipkin {
      enabled = true
      http {
        url = "http://localhost:9411"
      }
    }
  }
}

{
  "tracing": {
    "zipkin": {
      "enabled": true,
      "http": {
        "url": "http://localhost:9411"
      }
    }
  }
}

It is unlikely that sending spans to localhost will be suitable for production deployment, so you generally need to configure the location of one or more Zipkin servers for production:

Configuring Multiple Zipkin Servers

tracing.zipkin.enabled=true
tracing.zipkin.http.urls[0]=https://foo:9411
tracing.zipkin.http.urls[1]=https://bar:9411

tracing:
  zipkin:
    enabled: true
    http:
      urls:
        - https://foo:9411
        - https://bar:9411

[tracing]
  [tracing.zipkin]
    enabled=true
    [tracing.zipkin.http]
      urls=[
        "https://foo:9411",
        "https://bar:9411"
      ]

tracing {
  zipkin {
    enabled = true
    http {
      urls = ["https://foo:9411", "https://bar:9411"]
    }
  }
}

{
  tracing {
    zipkin {
      enabled = true
      http {
        urls = ["https://foo:9411", "https://bar:9411"]
      }
    }
  }
}

{
  "tracing": {
    "zipkin": {
      "enabled": true,
      "http": {
        "urls": ["https://foo:9411", "https://bar:9411"]
      }
    }
  }
}

In production, setting TRACING_ZIPKIN_HTTP_URLS environment variable with a comma-separated list of URLs also works.

Alternatively, to use a different zipkin2.reporter.Sender implementation, you can define a bean of type zipkin2.reporter.Sender and it will be used instead.

Zipkin Configuration

There are many configuration options available for the Brave client that sends Spans to Zipkin, and they are generally exposed via the BraveTracerConfiguration class. Refer to the Javadoc for available options.

Below is an example of customizing Zipkin configuration:

Customizing Zipkin Configuration

tracing.zipkin.enabled=true
tracing.zipkin.traceId128Bit=true
tracing.zipkin.sampler.probability=1

tracing:
  zipkin:
    enabled: true
    traceId128Bit: true
    sampler:
      probability: 1

[tracing]
  [tracing.zipkin]
    enabled=true
    traceId128Bit=true
    [tracing.zipkin.sampler]
      probability=1

tracing {
  zipkin {
    enabled = true
    traceId128Bit = true
    sampler {
      probability = 1
    }
  }
}

{
  tracing {
    zipkin {
      enabled = true
      traceId128Bit = true
      sampler {
        probability = 1
      }
    }
  }
}

{
  "tracing": {
    "zipkin": {
      "enabled": true,
      "traceId128Bit": true,
      "sampler": {
        "probability": 1
      }
    }
  }
}

You can also optionally dependency-inject common configuration classes into BraveTracerConfiguration such as brave.sampler.Sampler just by defining them as beans. See the API for BraveTracerConfiguration for available injection points.

Filtering HTTP spans

It may be useful to exclude health-checks and other HTTP requests to your service. This can be achieved by adding a list of regular expression patterns to your configuration:

Filtering HTTP request spans

tracing.zipkin.enabled=true
tracing.exclusions[0]=/health
tracing.exclusions[1]=/env/.*

tracing:
  zipkin:
    enabled: true
  exclusions:
    - /health
    - /env/.*

[tracing]
  exclusions=[
    "/health",
    "/env/.*"
  ]
  [tracing.zipkin]
    enabled=true

tracing {
  zipkin {
    enabled = true
  }
  exclusions = ["/health", "/env/.*"]
}

{
  tracing {
    zipkin {
      enabled = true
    }
    exclusions = ["/health", "/env/.*"]
  }
}

{
  "tracing": {
    "zipkin": {
      "enabled": true
    },
    "exclusions": ["/health", "/env/.*"]
  }
}

Zipkin Guides

See the following guides to learn more about distributed tracing with Zipkin in the Micronaut Framework:

6 Tracing with OpenTelemetry

The Micronaut Open Telemetry module uses Open Telemetry Autoconfigure SDK to configure Open Telemetry for tracing. For some functionalities you have to add additional dependencies. The Default values that are defined inside the Micronaut which values might be different from the default ones inside Open Telemetry Autoconfigure SDK module are:

otel.traces.exporter = none
otel.metrics.exporter = none
otel.logs.exporter = none
otel.service.name = value of the application.name

OpenTelemetry annotations

The io.micronaut.tracing.opentelemetry.processing package contains transformers and mappers that enables usage of Open Telemetry annotations.

6.1 OpenTelemetry Annotations

The io.micronaut.tracing.opentelemetry.processing package contains transformers and mappers that enables usage of Open Telemetry annotations.

To enable Open telemetry annotations you have to add next annotation processor in your dependency block:

annotationProcessor("io.micronaut.tracing:micronaut-tracing-opentelemetry-annotation:6.5.2-SNAPSHOT")

<annotationProcessorPaths>
    <path>
        <groupId>io.micronaut.tracing</groupId>
        <artifactId>micronaut-tracing-opentelemetry-annotation</artifactId>
        <version>6.5.2-SNAPSHOT</version>
    </path>
</annotationProcessorPaths>

The Open Tracing annotations that are defined inside the io.micronaut.tracing.annotation package are also available for usage inside Open Telemetry.

6.2 OpenTelemetry Exporters

In your project you can specify exporters that you want to use. The default one is set to "none" value which means by default there are no exporter registered. The available values are defined on the Open Telemetry Autoconfigure SDK documentation.

For each exporter that you want to use you have to specify it inside configuration, and you have to add required dependency:

OpenTelemetry Protocol exporter: otlp

implementation("io.opentelemetry:opentelemetry-exporter-otlp")

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Logging exporter: logging

implementation("io.opentelemetry:opentelemetry-exporter-logging")

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-logging</artifactId>
</dependency>

Google Cloud Trace: google_cloud_trace

implementation("com.google.cloud.opentelemetry:exporter-auto")

<dependency>
    <groupId>com.google.cloud.opentelemetry</groupId>
    <artifactId>exporter-auto</artifactId>
</dependency>

6.2.1 OpenTelemetry Jaeger Exporter

For Jaeger export use the OpenTelemetry Protocol exporter: otlp with the following dependency:

implementation("io.opentelemetry:opentelemetry-exporter-otlp")

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

The latest binary release of the OpenTelemetry Collector no longer includes exporters for the native Jaeger format. Jaeger has support for OTLP out of the box. If you had in your application the dependency io.opentelemetry:opentelemetry-exporter-jaeger replace it with io.opentelemetry:opentelemetry-exporter-otlp. Also, in your application configuration, replace otel.tracer.exporter=jaeger with otel.tracer.exporter=otlp and supply the otlp endpoint.

To configure Micronaut Tracing for OpenTelemetry OTLP exporter and Jager, add the following configuration:

otel.traces.exporter=otlp
otel.exporter.otlp.endpoint=http://localhost:4317

otel:
  traces:
    exporter: otlp
  exporter:
    otlp:
      endpoint: http://localhost:4317

[otel]
  [otel.traces]
    exporter="otlp"
  [otel.exporter]
    [otel.exporter.otlp]
      endpoint="http://localhost:4317"

otel {
  traces {
    exporter = "otlp"
  }
  exporter {
    otlp {
      endpoint = "http://localhost:4317"
    }
  }
}

{
  otel {
    traces {
      exporter = "otlp"
    }
    exporter {
      otlp {
        endpoint = "http://localhost:4317"
      }
    }
  }
}

{
  "otel": {
    "traces": {
      "exporter": "otlp"
    },
    "exporter": {
      "otlp": {
        "endpoint": "http://localhost:4317"
      }
    }
  }
}

In the previous configuration snippet, replace http://localhost:4317 with your Jaeger endpoint.

Checkout the guide Use OpenTelemetry with Jaeger and the Micronaut Framework for Microservice distributed tracing.

6.2.2 OpenTelemetry Zipkin Exporter

Zipkin exporter: zipkin

implementation("io.opentelemetry:opentelemetry-exporter-zipkin")

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

Example configuration for the Zipkin exporter:

otel.traces.exporter=zipkin

otel:
  traces:
    exporter: zipkin

[otel]
  [otel.traces]
    exporter="zipkin"

otel {
  traces {
    exporter = "zipkin"
  }
}

{
  otel {
    traces {
      exporter = "zipkin"
    }
  }
}

{
  "otel": {
    "traces": {
      "exporter": "zipkin"
    }
  }
}

Micronaut provides Zipkin exporter that will use Micronaut’s HTTP client instead of OKHttp client. That will reduce dependency graph and will make your native executable smaller. To use it add next dependency:

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-zipkin-exporter")

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-zipkin-exporter</artifactId>
</dependency>

To configure Micronaut Zipkin exporter add

otel.exporter.zipkin.url=<url-to-zipkin-server>

otel:
  exporter:
    zipkin:
      url: <url-to-zipkin-server>

[otel]
  [otel.exporter]
    [otel.exporter.zipkin]
      url="<url-to-zipkin-server>"

otel {
  exporter {
    zipkin {
      url = "<url-to-zipkin-server>"
    }
  }
}

{
  otel {
    exporter {
      zipkin {
        url = "<url-to-zipkin-server>"
      }
    }
  }
}

{
  "otel": {
    "exporter": {
      "zipkin": {
        "url": "<url-to-zipkin-server>"
      }
    }
  }
}

Micronaut Zipkin exporter requires otel.traces.exporter property not to be defined to avoid conflicts with Open Telemetry default implementation of zipkin exporter.

6.3 OpenTelemetry Propagators

In your project you can specify propagators that you want to use. The default one is set to "tracecontext, baggage". The available values are defined on the Open Telemetry Autoconfigure SDK documentation.

To use AWS X-Ray propagator inside your application you have to add next dependency in your project:

implementation("io.opentelemetry.contrib:opentelemetry-aws-xray-propagator")

<dependency>
    <groupId>io.opentelemetry.contrib</groupId>
    <artifactId>opentelemetry-aws-xray-propagator</artifactId>
</dependency>

And the "xray" has to be added inside configuration file.

otel.traces.exporter=otlp
otel.propagators=tracecontext, baggage, xray

otel:
  traces:
    exporter: otlp
  propagators: tracecontext, baggage, xray

[otel]
  propagators="tracecontext, baggage, xray"
  [otel.traces]
    exporter="otlp"

otel {
  traces {
    exporter = "otlp"
  }
  propagators = "tracecontext, baggage, xray"
}

{
  otel {
    traces {
      exporter = "otlp"
    }
    propagators = "tracecontext, baggage, xray"
  }
}

{
  "otel": {
    "traces": {
      "exporter": "otlp"
    },
    "propagators": "tracecontext, baggage, xray"
  }
}

6.4 ID Generator

ID Generator

Some custom vendor may require the span traceId in different format from the default one. You can provide your own bean of type IdGenerator. For an example, AWS X-Ray requires a specific format for their tracing identifiers. Add the following dependency, to register an instance of AwsXrayIdGenerator as a bean of type IdGenerator.

implementation("io.opentelemetry.contrib:opentelemetry-aws-xray")

<dependency>
    <groupId>io.opentelemetry.contrib</groupId>
    <artifactId>opentelemetry-aws-xray</artifactId>
</dependency>

To successfully export traces to the AWS X-Ray you have to run AWS Open Telemetry Collector that will periodically send traces to the AWS.

6.5 HTTP Server and Client

To enable creating span objects on the every HTTP server request, client request, server response and client response you have to add next depedency:

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
</dependency>

Filtering HTTP spans

It may be useful to exclude health-checks and other HTTP requests to your service. This can be achieved by adding a list of regular expression patterns to your configuration:

Filtering HTTP request spans

otel.exclusions[0]=/health
otel.exclusions[1]=/env/.*

otel:
  exclusions:
    - /health
    - /env/.*

[otel]
  exclusions=[
    "/health",
    "/env/.*"
  ]

otel {
  exclusions = ["/health", "/env/.*"]
}

{
  otel {
    exclusions = ["/health", "/env/.*"]
  }
}

{
  "otel": {
    "exclusions": ["/health", "/env/.*"]
  }
}

Add HTTP Headers into request spans

If you want you can add additional Http Headers inside your span objects. You can specify different headers for client request, client response, server request and server response.

Adding HTTP Headers into request spans

otel.http.client.request-headers[0]=X-From-Client-Request
otel.http.client.response-headers[0]=X-From-Client-Response
otel.http.server.request-headers[0]=X-From-Server-Request
otel.http.server.response-headers[0]=X-From-Server-Response

otel:
  http:
    client:
      request-headers:
        - X-From-Client-Request
      response-headers:
        - X-From-Client-Response
    server:
      request-headers:
        - X-From-Server-Request
      response-headers:
        - X-From-Server-Response

[otel]
  [otel.http]
    [otel.http.client]
      request-headers=[
        "X-From-Client-Request"
      ]
      response-headers=[
        "X-From-Client-Response"
      ]
    [otel.http.server]
      request-headers=[
        "X-From-Server-Request"
      ]
      response-headers=[
        "X-From-Server-Response"
      ]

otel {
  http {
    client {
      requestHeaders = ["X-From-Client-Request"]
      responseHeaders = ["X-From-Client-Response"]
    }
    server {
      requestHeaders = ["X-From-Server-Request"]
      responseHeaders = ["X-From-Server-Response"]
    }
  }
}

{
  otel {
    http {
      client {
        request-headers = ["X-From-Client-Request"]
        response-headers = ["X-From-Client-Response"]
      }
      server {
        request-headers = ["X-From-Server-Request"]
        response-headers = ["X-From-Server-Response"]
      }
    }
  }
}

{
  "otel": {
    "http": {
      "client": {
        "request-headers": ["X-From-Client-Request"],
        "response-headers": ["X-From-Client-Response"]
      },
      "server": {
        "request-headers": ["X-From-Server-Request"],
        "response-headers": ["X-From-Server-Response"]
      }
    }
  }
}

6.6 gRPC Server and Client

To enable creating span objects on the every GRPC server request, client request, server response and client response, add the following dependency:

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-grpc")

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-grpc</artifactId>
</dependency>

6.7 Kafka

To enable creating span objects on the every Kafka message, add the following depedency:

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-kafka")

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-kafka</artifactId>
</dependency>

Add Kafka Message Headers into request spans

The way Kafka Message Headers are added inside your span objects may be configured as follows:

Adding Kafka Message Headers into request spans

otel.instrumentation.kafka.wrapper=true
otel.instrumentation.kafka.headers-as-lists=false
otel.instrumentation.kafka.attribute-with-prefix=true
otel.instrumentation.kafka.attribute-prefix=myPrefix
otel.instrumentation.kafka.captured-headers[0]=myHeader1
otel.instrumentation.kafka.captured-headers[1]=myHeader2
otel.instrumentation.kafka.included-topics[0]=topic1
otel.instrumentation.kafka.included-topics[1]=topic2
otel.instrumentation.kafka.excluded-topics[0]=topic1
otel.instrumentation.kafka.excluded-topics[1]=topic2

otel:
  instrumentation:
    kafka:
      wrapper: true
      headers-as-lists: false
      attribute-with-prefix: true
      attribute-prefix: myPrefix
      captured-headers: # list of headers which need to send as span attributes
        - myHeader1
        - myHeader2
      included-topics:
        - topic1
        - topic2
      excluded-topics:
        - topic1
        - topic2

[otel]
  [otel.instrumentation]
    [otel.instrumentation.kafka]
      wrapper=true
      headers-as-lists=false
      attribute-with-prefix=true
      attribute-prefix="myPrefix"
      captured-headers=[
        "myHeader1",
        "myHeader2"
      ]
      included-topics=[
        "topic1",
        "topic2"
      ]
      excluded-topics=[
        "topic1",
        "topic2"
      ]

otel {
  instrumentation {
    kafka {
      wrapper = true
      headersAsLists = false
      attributeWithPrefix = true
      attributePrefix = "myPrefix"
      capturedHeaders = ["myHeader1", "myHeader2"]
      includedTopics = ["topic1", "topic2"]
      excludedTopics = ["topic1", "topic2"]
    }
  }
}

{
  otel {
    instrumentation {
      kafka {
        wrapper = true
        headers-as-lists = false
        attribute-with-prefix = true
        attribute-prefix = "myPrefix"
        captured-headers = ["myHeader1", "myHeader2"]
        included-topics = ["topic1", "topic2"]
        excluded-topics = ["topic1", "topic2"]
      }
    }
  }
}

{
  "otel": {
    "instrumentation": {
      "kafka": {
        "wrapper": true,
        "headers-as-lists": false,
        "attribute-with-prefix": true,
        "attribute-prefix": "myPrefix",
        "captured-headers": ["myHeader1", "myHeader2"],
        "included-topics": ["topic1", "topic2"],
        "excluded-topics": ["topic1", "topic2"]
      }
    }
  }
}

wrapper - if "true" tracing will be done using object proxies over consumers and producers. If wrappers are disabled, then tracing can be done through a kafka listener instead.
If you want to set headers as lists, set headers-as-lists to true.
attribute-with-prefix - whether to add a prefix to span attribute names (default: false)
attribute-prefix - custom prefix for span attributes (header names). Default: messaging.header.
captured-headers is the list of headers to add as span attributes. By default, all headers are added as span attributes. To set no headers as span attributes, specify null or an empty string.
included-topics - list of topics to trace
excluded-topics - list of topics to exclude from tracing

You can’t use included-topics and excluded-topics properties together as they are mutually exclusive - choose one or the other.

6.8 AWS SDK Instrumentation

Include the following dependency to instrument the AWS SDK:

implementation("io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2")

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-aws-sdk-2.2</artifactId>
</dependency>

Additionally, include Micronaut AWS SDK v2 dependency:

implementation("io.micronaut.aws:micronaut-aws-sdk-v2")

<dependency>
    <groupId>io.micronaut.aws</groupId>
    <artifactId>micronaut-aws-sdk-v2</artifactId>
</dependency>

micronaut-aws-sdk-v2 dependency creates a bean of type SdkClientBuilder. To instrument the AWS SDK, Micronaut OpenTelemetry registers a tracing interceptor via a bean creation listener for the bean of type SdkClientBuilder.

6.9 AWS Resource Detectors

AWS Resource detectors enrich traces with AWS infrastructure information.

To use AWS resource detectors include the following dependency:

implementation("io.opentelemetry.contrib:opentelemetry-aws-resources")

<dependency>
    <groupId>io.opentelemetry.contrib</groupId>
    <artifactId>opentelemetry-aws-resources</artifactId>
</dependency>

and provide a bean of type ResourceProvider

package io.micronaut.tracing.opentelemetry;

import io.micronaut.core.annotation.NonNull;
import io.opentelemetry.contrib.aws.resource.Ec2Resource;
import io.opentelemetry.sdk.resources.Resource;

import jakarta.inject.Singleton;

@Singleton
public class AwsResourceProvider implements ResourceProvider {

    @Override
    @NonNull
    public Resource resource() {
        return Resource.getDefault()
            .merge(Ec2Resource.get());
    }
}

6.10 OpenTelemetry Guides

See the following guides to learn more about distributed tracing with OpenTelemetry in the Micronaut Framework:

7 Micronaut Tracing Guides

See the following list of guides to learn more about working with Distributed Tracing in the Micronaut Framework:

https://guides.micronaut.io/latest/tag-distributed_tracing.html

8 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-tracing

Table of Contents

Micronaut Tracing

1 Introduction

Tracing Annotations

Tracing Instrumentation

Tracing Beans

2 Release History

3 Breaking Changes

Micronaut Tracing 4.5.0 breaking changes

Micronaut Tracing 5.0.0-M2 breaking changes

Micronaut Tracing 7.0.0 Breaking changes

4 Tracing with Jaeger

Running Jaeger

Sending Traces to Jaeger

Jaeger Configuration

Filtering HTTP spans

Jaeger Guides

5 Tracing with Zipkin

Running Zipkin

Sending Traces to Zipkin

Customizing the Zipkin Sender

Zipkin Configuration

Filtering HTTP spans

Zipkin Guides

6 Tracing with OpenTelemetry

OpenTelemetry annotations

6.1 OpenTelemetry Annotations

6.2 OpenTelemetry Exporters

6.2.1 OpenTelemetry Jaeger Exporter

6.2.2 OpenTelemetry Zipkin Exporter

6.3 OpenTelemetry Propagators

6.4 ID Generator

ID Generator

6.5 HTTP Server and Client

Filtering HTTP spans

Add HTTP Headers into request spans

6.6 gRPC Server and Client

6.7 Kafka

Add Kafka Message Headers into request spans

6.8 AWS SDK Instrumentation

6.9 AWS Resource Detectors

6.10 OpenTelemetry Guides

7 Micronaut Tracing Guides

8 Repository