OpenTelemetry Tracing with Oracle Cloud and the Micronaut Framework

Use Oracle Cloud to investigate the behavior of your Micronaut applications.

Authors: Nemanja Mikic, John Shingler

Micronaut Version: 3.9.2

1. Getting Started

In this guide, we will create a Micronaut application written in Groovy.

In this guide, you will discover how simple it is to add tracing to a Micronaut application.

Tracing allows you to track service requests in a single application or a distributed one. Trace data shows the path, time spent in each section (called a span), and other information collected along the way. Tracing gives you observability into what is causing bottlenecks and failures.

2. What you will need

To complete this guide, you will need the following:

Some time on your hands
A decent text editor or IDE
JDK 1.8 or greater installed with JAVA_HOME configured appropriately

An Oracle Cloud account (create a free trial account at signup.oraclecloud.com)

3. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example.

Download and unzip the source

4. Writing the Application

Create an application using the Micronaut Command Line Interface or with Micronaut Launch.

mn create-app example.micronaut.micronautguide \
    --features=tracing-opentelemetry-zipkin,tracing-opentelemetry-http \
    --build=maven
    --lang=groovy

If you don’t specify the --build argument, Gradle is used as the build tool.
If you don’t specify the --lang argument, Java is used as the language.

The previous command creates a Micronaut application with the default package example.micronaut in a directory named micronautguide.

If you use Micronaut Launch, select Micronaut Application as application type and add tracing-opentelemetry-zipkin, and tracing-opentelemetry-http features.

If you have an existing Micronaut application and want to add the functionality described here, you can view the dependency and configuration changes from the specified features and apply those changes to your application.

Setup IntelliJ IDEA to develop Micronaut Applications.

4.1. OpenTelemetry

The Micronaut framework uses OpenTelemetry to generate and export tracing data.

OpenTelemetry provides two annotations: one to create a span and another to include additional information to the span.

@WithSpan: Used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@SpanAttribute: Used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead.

@WithSpan and @SpanAttribute can be used only on non-private methods.

If these annotations are not enough, or if you want to add tracing to a private method, the Micronaut framework’s tracing integration registers a io.opentelemetry.api.trace.Tracer bean, which exposes the OpenTelemetry API and can be dependency-injected as needed.

The following `io.micronaut.tracing.annotation`s are available if you prefer to use them or if you are working on an existing Micronaut application that already uses them.

@NewSpan : Identical to @WithSpan, it is used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@ContinueSpan : Used on methods to continue an existing span; primarily used in conjunction with @SpanTag.
@SpanTag : Similar to @SpanAttribute, it is used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead. To use the @SpanTag on a method argument, the method must be annotated with either @NewSpan or @ContinueSpan.

4.2. Inventory Service

src/main/groovy/example/micronaut/InventoryService.groovy

package example.micronaut

import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan
import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.Tracer
import jakarta.inject.Singleton

import java.util.Collection
import java.util.Map
import java.util.concurrent.ConcurrentHashMap

@Singleton
class InventoryService {

    private final Tracer tracer
    private final WarehouseClient warehouse
    private final Map<String, Integer> inventory = new ConcurrentHashMap<>()

    private static final String storeName = 'my_store'

    InventoryService(Tracer tracer, WarehouseClient warehouse) { (1)
        this.tracer    = tracer
        this.warehouse = warehouse

        inventory << [laptop:4, desktop:2, monitor:11]
    }

    Collection<String> getProductNames() {
        inventory.keySet()
    }

    @WithSpan('stock-counts') (2)
    Map<String, Integer> getStockCounts(@SpanAttribute('inventory.item') String item) { (3)
        def counts = [:]
        if(inventory.containsKey(item)) {
            int count = inventory[item]
            counts.'store' = count

            if(count < 10) {
                counts.'warehouse' = inWarehouse(storeName, item)
            }
        }

        counts
    }

    private int inWarehouse(String store, String item) {
        Span.current().setAttribute('inventory.store-name', store) (4)

        warehouse.getItemCount(store, getUPC(item))
    }

    void order(String item, int count) {
        orderFromWarehouse(item, count)

        inventory[item] = count + inventory[item] ?: 0
    }

    private void orderFromWarehouse(String item, int count) {
        Span span = tracer.spanBuilder('warehouse-order') (5)
                .setAttribute('item', item)
                .setAttribute('count', count)
                .startSpan()

        def json = [store: storeName, product: item, amount: count, upc: getUPC(item)]

        warehouse.order(json)

        span.end()
    }

    private int getUPC(String item) {
        return Math.abs(item.hashCode())
    }
}

1	Inject Tracing bean into class
2	Creates a new span called "stock-counts"
3	Adds a label, or tag, called "inventory.item" that will contain the value contained in `item`
4	Same as `@SpanAttribute("inventory.store-name")`
5	Same as `@WithSpan("warehouse-order")` with `@SpanAttributes` for item and count

4.3. Store Controller

This class demonstrates use of the io.micronaut.tracing.annotation instead of the OpenTelemetry annotations.

If you have the following dependency declared, all HTTP Server methods (those annotated with @Get, @Post, etc.) will create spans automatically.

pom.xml

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
    <scope>compile</scope>
</dependency>

src/main/groovy/example/micronaut/StoreController.groovy

package example.micronaut

import io.micronaut.http.HttpStatus
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.http.annotation.Status
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn
import io.micronaut.tracing.annotation.ContinueSpan
import io.micronaut.tracing.annotation.NewSpan
import io.micronaut.tracing.annotation.SpanTag

import java.util.ArrayList
import java.util.HashMap
import java.util.List
import java.util.Map

@ExecuteOn(TaskExecutors.IO)
@Controller('/store')
class StoreController {
    private final InventoryService inventory

    StoreController(InventoryService inventory) {
        this.inventory = inventory
    }

    @Post('/order')
    @Status(HttpStatus.CREATED)
    @NewSpan('store.order') (1)
    void order(@SpanTag('order.item') String item, @SpanTag int count) { (2)
        inventory.order(item, count)
    }

    @Get('/inventory') (3)
    List<Map<String, Object>> getInventory() {
        List<Map<String, Object>> currentInventory = []

        inventory.productNames.each{ product ->
            currentInventory << getInventory(product)
        }

        currentInventory
    }

    @Get('/inventory/{item}')
    @ContinueSpan (4)
    Map<String, Object> getInventory(@SpanTag('item') String item) { (5)
        def counts = inventory.getStockCounts(item)
        if(!counts) {
            counts.'note' = "Not available at store"
        }

        counts.'item' = item

        counts
    }
}

1	Equivalent to `@WithSpan("store.order")`
2	Same as @SpanAttribute("order.item")`and `@SpanAttribute
3	Span created automatically if `micronaut-tracing-opentelemetry-http` is declared
4	Required for `@SpanTag`
5	Tag is only added to the span if `micronaut-tracing-opentelemetry-http` is declared; otherwise ignored

4.4. Warehouse Client

You can also mix OpenTelemetry and Micronaut Tracing annotations in the same class.

If you have the following dependency declared, all HTTP Client methods (those annotated with @Get, @Post, etc.) will create spans automatically.

pom.xml

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
    <scope>compile</scope>
</dependency>

src/main/groovy/example/micronaut/WarehouseClient.groovy

package example.micronaut

import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.http.annotation.QueryValue
import io.micronaut.http.client.annotation.Client
import io.micronaut.tracing.annotation.ContinueSpan
import io.micronaut.tracing.annotation.SpanTag
import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan

import java.util.Map;

@Client("/warehouse") (1)
interface WarehouseClient {

    @Post("/order")
    @WithSpan
    void order(@SpanTag("warehouse.order") Map<String, ?> json)

    @Get("/count")
    @ContinueSpan
    int getItemCount(@QueryValue String store, @SpanAttribute @QueryValue int upc)

}

1	Some external service without tracing

4.5. Warehouse Controller

The WarehouseController class represents external service that will be called by WarehouseClient.

src/main/groovy/example/micronaut/WarehouseController.groovy

package example.micronaut

import io.micronaut.http.HttpResponse
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn

import java.util.Random


@ExecuteOn(TaskExecutors.IO) (1)
@Controller("/warehouse") (2)
class WarehouseController {

    @Get("/count") (3)
    HttpResponse getItemCount() {
        HttpResponse.ok(new Random().nextInt(10)+1)
    }

    @Post("/order") (4)
    HttpResponse order() {
        try {
            //To simulate an external process taking time
            Thread.sleep(500)
        } catch (ex) {
        }

        HttpResponse.accepted()
    }
}

1	It is critical that any blocking I/O operations (such as fetching the data from the database) are offloaded to a separate thread pool that does not block the Event loop.
2	The class is defined as a controller with the @Controller annotation mapped to the path `/warehouse`.
3	The @Get annotation maps the `getItemCount` method to an HTTP GET request on `/warehouse/count`.
4	The @Get annotation maps the `order` method to an HTTP GET request on `/warehouse/order`.

4.6. Create APM Domain

Open the Oracle Cloud Menu and click "Observability & Management", and then "Administration" under "Application Performance…":

Click "Create APM Domain":

Name your domain, choose a compartment, and enter a description.

Once the domain is created, view the domain details. Here you’ll need to grab a few values, so copy the data upload endpoint (#1) and public key (#2).

Now we have what we need to construct a URL to plug in to our application config files. The Collector URL format requires us to construct a URL by using the data upload endpoint as our base URL and generating the path based on some choices, including values from our private or public key. The format is documented here. Once we’ve constructed the URL path, we can add it to our application.yml config.

4.7. Configure Tracer

Use Micronaut CLI or Launch to create your application. You will see that the necessary OpenTelemetry configuration are automatically added to your application.yml file. You will have to change the value of the zipkin endpoint configuration variable.

src/main/resources/application.yml

otel:
  traces:
    exporter: zipkin
  exporter:
    zipkin:
      endpoint: https://[redacted].apm-agt.us-phoenix-1.oci.oraclecloud.com/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=[public key] (1)

1	The zipkin exporter URL mentioned in the previous step

5. Run the Application

The application can be deployed to Oracle Cloud.

Traces can be sent to the Oracle Cloud APM Trace Explorer outside the Oracle Cloud.

This allows us to run the application locally and see the traces in the Oracle Cloud APM Trace Explorer.

To run the application, use the ./mvnw mn:run command, which starts the application on port 8080.

6. Traces

Open the Oracle APM Tracing Explorer. Once the page is loaded, select the compartment that you selected in above step (#1), choose the APM domain that you created (#2), and run the query (#3).

If your traces aren’t displayed yet, give it a moment. It takes a few seconds for the traces to show up in the explorer. If no spans show up, run the query again by pressing the "Run" button.

6.1. Get Item Counts

curl http://localhost:8080/store/inventory/laptop

Each span is represented by a blue bar.

6.2. Order Item

curl -X "POST" "http://localhost:8080/store/order" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{"item":"laptop", "count":5}'

Selecting a different span will show you the labels (a.k.a. attributes/tags) and other details of the span.

6.3. Get Inventory

curl http://localhost:8080/store/inventory

Looking at the trace, we can conclude that retrieving the items sequentially might not be the best design choice.

Table of Contents