OpenTelemetry Tracing with Oracle Cloud and the Micronaut Framework

Use Oracle Cloud to investigate the behavior of your Micronaut applications.

Authors: Nemanja Mikic, John Shingler

Micronaut Version: 3.9.2

1. Getting Started

In this guide, we will create a Micronaut application written in Kotlin.

In this guide, you will discover how simple it is to add tracing to a Micronaut application.

Tracing allows you to track service requests in a single application or a distributed one. Trace data shows the path, time spent in each section (called a span), and other information collected along the way. Tracing gives you observability into what is causing bottlenecks and failures.

2. What you will need

To complete this guide, you will need the following:

Some time on your hands
A decent text editor or IDE
JDK 1.8 or greater installed with JAVA_HOME configured appropriately

An Oracle Cloud account (create a free trial account at signup.oraclecloud.com)

3. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example.

Download and unzip the source

4. Writing the Application

Create an application using the Micronaut Command Line Interface or with Micronaut Launch.

mn create-app example.micronaut.micronautguide \
    --features=tracing-opentelemetry-zipkin,tracing-opentelemetry-http \
    --build=maven
    --lang=kotlin

If you don’t specify the --build argument, Gradle is used as the build tool.
If you don’t specify the --lang argument, Java is used as the language.

The previous command creates a Micronaut application with the default package example.micronaut in a directory named micronautguide.

If you use Micronaut Launch, select Micronaut Application as application type and add tracing-opentelemetry-zipkin, and tracing-opentelemetry-http features.

If you have an existing Micronaut application and want to add the functionality described here, you can view the dependency and configuration changes from the specified features and apply those changes to your application.

Setup IntelliJ IDEA to develop Micronaut Applications.

4.1. OpenTelemetry

The Micronaut framework uses OpenTelemetry to generate and export tracing data.

OpenTelemetry provides two annotations: one to create a span and another to include additional information to the span.

@WithSpan: Used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@SpanAttribute: Used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead.

@WithSpan and @SpanAttribute can be used only on non-private methods.

If these annotations are not enough, or if you want to add tracing to a private method, the Micronaut framework’s tracing integration registers a io.opentelemetry.api.trace.Tracer bean, which exposes the OpenTelemetry API and can be dependency-injected as needed.

The following `io.micronaut.tracing.annotation`s are available if you prefer to use them or if you are working on an existing Micronaut application that already uses them.

@NewSpan : Identical to @WithSpan, it is used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@ContinueSpan : Used on methods to continue an existing span; primarily used in conjunction with @SpanTag.
@SpanTag : Similar to @SpanAttribute, it is used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead. To use the @SpanTag on a method argument, the method must be annotated with either @NewSpan or @ContinueSpan.

4.2. Inventory Service

src/main/kotlin/example/micronaut/InventoryService.kt

package example.micronaut

import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.Tracer
import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan
import jakarta.inject.Singleton
import java.util.concurrent.ConcurrentHashMap
import kotlin.math.abs

@Singleton
open class InventoryService(private val tracer: Tracer, (1)
                            private val warehouse: WarehouseClient) {

    private val inventory : ConcurrentHashMap<String, Int> = ConcurrentHashMap()

    private val storeName : String = "my_store"

    init {
        inventory["laptop"]  = 4
        inventory["desktop"] = 2
        inventory["monitor"] = 11
    }

    fun getProductNames(): Collection<String> = inventory.keys

    @WithSpan("stock-counts") (2)
    open fun getStockCounts(@SpanAttribute("inventory.item") item : String): Map<String, Int> { (3)
        val counts = mutableMapOf<String, Int>()
        if(inventory.containsKey(item)) {
            val count = inventory[item]!!
            counts["store"] = count

            if(count < 10) {
                counts["warehouse"] = inWarehouse(storeName, item)
            }
        }

        return counts
    }

    private fun inWarehouse(store: String, item: String): Int {
        Span.current().setAttribute("inventory.store-name", store) (4)

        return warehouse.getItemCount(store, getUPC(item))
    }

    fun order(item : String, count: Int) {
        orderFromWarehouse(item, count)

        inventory[item] = count + (inventory[item] ?:  0)
    }

    private fun orderFromWarehouse(item: String, count: Int) {
        val span = tracer.spanBuilder("warehouse-order") (5)
            .setAttribute("item", item)
            .setAttribute("count", count.toLong())
            .startSpan()

        val json = mapOf("store"   to storeName,
            "product" to item,
            "amount" to count,
            "upc" to getUPC(item))

        warehouse.order(json)

        span.end()
    }

    private fun getUPC(item: String): Int = abs(item.hashCode())

}

1	Inject Tracing bean into class
2	Creates a new span called "stock-counts"
3	Adds a label, or tag, called "inventory.item" that will contain the value contained in `item`
4	Same as `@SpanAttribute("inventory.store-name")`
5	Same as `@WithSpan("warehouse-order")` with `@SpanAttributes` for item and count

4.3. Store Controller

This class demonstrates use of the io.micronaut.tracing.annotation instead of the OpenTelemetry annotations.

If you have the following dependency declared, all HTTP Server methods (those annotated with @Get, @Post, etc.) will create spans automatically.

pom.xml

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
    <scope>compile</scope>
</dependency>

src/main/kotlin/example/micronaut/StoreController.kt

package example.micronaut

import io.micronaut.http.HttpStatus
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.http.annotation.Status
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn
import io.micronaut.tracing.annotation.ContinueSpan
import io.micronaut.tracing.annotation.NewSpan
import io.micronaut.tracing.annotation.SpanTag

@ExecuteOn(TaskExecutors.IO)
@Controller("/store")
open class StoreController(private val inventory: InventoryService) {

    @Post("/order")
    @Status(HttpStatus.CREATED)
    @NewSpan("store.order") (1)
    open fun order(@SpanTag("order.item") item: String, @SpanTag count: Int) = inventory.order(item, count) (2)

    @Get("/inventory") (3)
    fun getInventory(): List<Map<String, Any>> = inventory.getProductNames().map { getInventory(it) }

    @Get("/inventory/{item}")
    @ContinueSpan (4)
    open fun getInventory(item: String) : Map<String, Any> { (5)

        val counts : MutableMap<String, Any> = inventory.getStockCounts(item)
            .toMutableMap<String, Any>()
            .ifEmpty{ mutableMapOf("note" to "Not available at store") }

        counts["item"] = item

        return counts
    }
}

1	Equivalent to `@WithSpan("store.order")`
2	Same as @SpanAttribute("order.item")`and `@SpanAttribute
3	Span created automatically if `micronaut-tracing-opentelemetry-http` is declared
4	Required for `@SpanTag`
5	Tag is only added to the span if `micronaut-tracing-opentelemetry-http` is declared; otherwise ignored

4.4. Warehouse Client

You can also mix OpenTelemetry and Micronaut Tracing annotations in the same class.

If you have the following dependency declared, all HTTP Client methods (those annotated with @Get, @Post, etc.) will create spans automatically.

pom.xml

<dependency>
    <groupId>io.micronaut.tracing</groupId>
    <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
    <scope>compile</scope>
</dependency>

src/main/kotlin/example/micronaut/WarehouseClient.kt

package example.micronaut

import io.micronaut.http.HttpResponse
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.http.annotation.QueryValue
import io.micronaut.http.client.annotation.Client
import io.micronaut.tracing.annotation.ContinueSpan
import io.micronaut.tracing.annotation.SpanTag
import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan

@Client("/warehouse") (1)
interface WarehouseClient {

    @Post("/order")
    @WithSpan
    fun order(@SpanTag("warehouse.order") json: Map<String, Any>) : HttpResponse<Any>

    @Get("/count")
    @ContinueSpan
    fun getItemCount(@QueryValue store: String, @SpanAttribute @QueryValue upc: Int) : Int

}

1	Some external service without tracing

4.5. Warehouse Controller

The WarehouseController class represents external service that will be called by WarehouseClient.

src/main/kotlin/example/micronaut/WarehouseController.kt

package example.micronaut

import io.micronaut.http.HttpResponse
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn
import java.util.Random

@ExecuteOn(TaskExecutors.IO) (1)
@Controller("/warehouse") (2)
class WarehouseController {

    @Get("/count") (3)
    fun getItemCount() : HttpResponse<Int> = HttpResponse.ok(Random().nextInt(11))


    @Post("/order") (4)
    fun order() : HttpResponse<Any> {
        try {
            //To simulate an external process taking time
            Thread.sleep(500)
        } catch (e: InterruptedException) {
        }

        return HttpResponse.accepted()
    }

}

1	It is critical that any blocking I/O operations (such as fetching the data from the database) are offloaded to a separate thread pool that does not block the Event loop.
2	The class is defined as a controller with the @Controller annotation mapped to the path `/warehouse`.
3	The @Get annotation maps the `getItemCount` method to an HTTP GET request on `/warehouse/count`.
4	The @Get annotation maps the `order` method to an HTTP GET request on `/warehouse/order`.

4.6. Create APM Domain

Open the Oracle Cloud Menu and click "Observability & Management", and then "Administration" under "Application Performance…":

Click "Create APM Domain":

Name your domain, choose a compartment, and enter a description.

Once the domain is created, view the domain details. Here you’ll need to grab a few values, so copy the data upload endpoint (#1) and public key (#2).

Now we have what we need to construct a URL to plug in to our application config files. The Collector URL format requires us to construct a URL by using the data upload endpoint as our base URL and generating the path based on some choices, including values from our private or public key. The format is documented here. Once we’ve constructed the URL path, we can add it to our application.yml config.

4.7. Configure Tracer

Use Micronaut CLI or Launch to create your application. You will see that the necessary OpenTelemetry configuration are automatically added to your application.yml file. You will have to change the value of the zipkin endpoint configuration variable.

src/main/resources/application.yml

otel:
  traces:
    exporter: zipkin
  exporter:
    zipkin:
      endpoint: https://[redacted].apm-agt.us-phoenix-1.oci.oraclecloud.com/20200101/observations/public-span?dataFormat=zipkin&dataFormatVersion=2&dataKey=[public key] (1)

1	The zipkin exporter URL mentioned in the previous step

5. Run the Application

The application can be deployed to Oracle Cloud.

Traces can be sent to the Oracle Cloud APM Trace Explorer outside the Oracle Cloud.

This allows us to run the application locally and see the traces in the Oracle Cloud APM Trace Explorer.

To run the application, use the ./mvnw mn:run command, which starts the application on port 8080.

6. Traces

Open the Oracle APM Tracing Explorer. Once the page is loaded, select the compartment that you selected in above step (#1), choose the APM domain that you created (#2), and run the query (#3).

If your traces aren’t displayed yet, give it a moment. It takes a few seconds for the traces to show up in the explorer. If no spans show up, run the query again by pressing the "Run" button.

6.1. Get Item Counts

curl http://localhost:8080/store/inventory/laptop

Each span is represented by a blue bar.

6.2. Order Item

curl -X "POST" "http://localhost:8080/store/order" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{"item":"laptop", "count":5}'

Selecting a different span will show you the labels (a.k.a. attributes/tags) and other details of the span.

6.3. Get Inventory

curl http://localhost:8080/store/inventory

Looking at the trace, we can conclude that retrieving the items sequentially might not be the best design choice.

Table of Contents