OpenTelemetry Tracing with Google Cloud Trace and the Micronaut Framework

Use Google Cloud Trace to investigate the behavior of your Micronaut applications.

Authors: John Shingler

Micronaut Version: 3.9.2

1. Getting Started

In this guide, we will create a Micronaut application written in Kotlin.

In this guide, you will discover how simple it is to add tracing to a Micronaut application.

Tracing allows you to track service requests in a single application or a distributed one. Trace data shows the path, time spent in each section (called a span), and other information collected along the way. Tracing gives you observability into what is causing bottlenecks and failures.

2. What you will need

To complete this guide, you will need the following:

Some time on your hands
A decent text editor or IDE
JDK 1.8 or greater installed with JAVA_HOME configured appropriately
A Google Cloud Platform (GCP) account and a GCP project.

3. Costs

This guide uses paid services; you may need to enable Billing in Google Cloud to complete some steps in this guide.

4. Google Cloud Platform

Signup for the Google Cloud Platform

4.1. Cloud SDK

Install the Cloud SDK CLI for your operating system.

Cloud SDK includes the gcloud command-line tool. Run the init command in your terminal:

gcloud init

gcloud auth login

4.2. Google Cloud Platform Project

Create a new project with a unique name (replace xxxxxx with alphanumeric characters of your choice):

gcloud projects create micronaut-guides-xxxxxx

In GCP, project ids are globally unique, so the id you used above is the one you should use in the rest of this guide.

Change your project:

gcloud config set project micronaut-guides-xxxxxx

If you forget the project id, you can list all projects:

gcloud projects list

5. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example.

Download and unzip the source

6. Writing the Application

Create an application using the Micronaut Command Line Interface or with Micronaut Launch.

mn create-app example.micronaut.micronautguide \
    --features=tracing-opentelemetry-gcp,tracing-opentelemetry-http \
    --build=gradle
    --lang=kotlin

If you don’t specify the --build argument, Gradle is used as the build tool.
If you don’t specify the --lang argument, Java is used as the language.

The previous command creates a Micronaut application with the default package example.micronaut in a directory named micronautguide.

If you use Micronaut Launch, select Micronaut Application as application type and add tracing-opentelemetry-gcp, and tracing-opentelemetry-http features.

If you have an existing Micronaut application and want to add the functionality described here, you can view the dependency and configuration changes from the specified features and apply those changes to your application.

Setup IntelliJ IDEA to develop Micronaut Applications.

6.1. OpenTelemetry

The Micronaut framework uses OpenTelemetry to generate and export tracing data.

OpenTelemetry provides two annotations: one to create a span and another to include additional information to the span.

@WithSpan: Used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@SpanAttribute: Used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead.

@WithSpan and @SpanAttribute can be used only on non-private methods.

If these annotations are not enough, or if you want to add tracing to a private method, the Micronaut framework’s tracing integration registers a io.opentelemetry.api.trace.Tracer bean, which exposes the OpenTelemetry API and can be dependency-injected as needed.

The following `io.micronaut.tracing.annotation`s are available if you prefer to use them or if you are working on an existing Micronaut application that already uses them.

@NewSpan : Identical to @WithSpan, it is used on methods to create a new span; defaults to the method name, but a unique name may be assigned instead.
@ContinueSpan : Used on methods to continue an existing span; primarily used in conjunction with @SpanTag.
@SpanTag : Similar to @SpanAttribute, it is used on method parameters to assign a value to a span; defaults to the parameter name, but a unique name may be assigned instead. To use the @SpanTag on a method argument, the method must be annotated with either @NewSpan or @ContinueSpan.

6.2. Inventory Service

src/main/kotlin/example/micronaut/InventoryService.kt

package example.micronaut

import io.opentelemetry.api.trace.Span
import io.opentelemetry.api.trace.Tracer
import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan
import jakarta.inject.Singleton
import java.util.concurrent.ConcurrentHashMap
import kotlin.math.abs

@Singleton
open class InventoryService(private val tracer: Tracer, (1)
                            private val warehouse: WarehouseClient) {

    private val inventory : ConcurrentHashMap<String, Int> = ConcurrentHashMap()

    private val storeName : String = "my_store"

    init {
        inventory["laptop"]  = 4
        inventory["desktop"] = 2
        inventory["monitor"] = 11
    }

    fun getProductNames(): Collection<String> = inventory.keys

    @WithSpan("stock-counts") (2)
    open fun getStockCounts(@SpanAttribute("inventory.item") item : String): Map<String, Int> { (3)
        val counts = mutableMapOf<String, Int>()
        if(inventory.containsKey(item)) {
            val count = inventory[item]!!
            counts["store"] = count

            if(count < 10) {
                counts["warehouse"] = inWarehouse(storeName, item)
            }
        }

        return counts
    }

    private fun inWarehouse(store: String, item: String): Int {
        Span.current().setAttribute("inventory.store-name", store) (4)

        return warehouse.getItemCount(store, getUPC(item))
    }

    fun order(item : String, count: Int) {
        orderFromWarehouse(item, count)

        inventory[item] = count + (inventory[item] ?:  0)
    }

    private fun orderFromWarehouse(item: String, count: Int) {
        val span = tracer.spanBuilder("warehouse-order") (5)
            .setAttribute("item", item)
            .setAttribute("count", count.toLong())
            .startSpan()

        val json = mapOf("store"   to storeName,
            "product" to item,
            "amount" to count,
            "upc" to getUPC(item))

        warehouse.order(json)

        span.end()
    }

    private fun getUPC(item: String): Int = abs(item.hashCode())

}

1	Inject Tracing bean into class
2	Creates a new span called "stock-counts"
3	Adds a label, or tag, called "inventory.item" that will contain the value contained in `item`
4	Same as `@SpanAttribute("inventory.store-name")`
5	Same as `@WithSpan("warehouse-order")` with `@SpanAttributes` for item and count

6.3. Store Controller

This class demonstrates use of the io.micronaut.tracing.annotation instead of the OpenTelemetry annotations.

If you have the following dependency declared, all HTTP Server methods (those annotated with @Get, @Post, etc.) will create spans automatically.

build.gradle

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")

src/main/kotlin/example/micronaut/StoreController.kt

package example.micronaut

import io.micronaut.http.HttpStatus
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.http.annotation.Status
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn
import io.micronaut.tracing.annotation.ContinueSpan
import io.micronaut.tracing.annotation.NewSpan
import io.micronaut.tracing.annotation.SpanTag

@ExecuteOn(TaskExecutors.IO)
@Controller("/store")
open class StoreController(private val inventory: InventoryService) {

    @Post("/order")
    @Status(HttpStatus.CREATED)
    @NewSpan("store.order") (1)
    open fun order(@SpanTag("order.item") item: String, @SpanTag count: Int) = inventory.order(item, count) (2)

    @Get("/inventory") (3)
    fun getInventory(): List<Map<String, Any>> = inventory.getProductNames().map { getInventory(it) }

    @Get("/inventory/{item}")
    @ContinueSpan (4)
    open fun getInventory(item: String) : Map<String, Any> { (5)

        val counts : MutableMap<String, Any> = inventory.getStockCounts(item)
            .toMutableMap<String, Any>()
            .ifEmpty{ mutableMapOf("note" to "Not available at store") }

        counts["item"] = item

        return counts
    }
}

1	Equivalent to `@WithSpan("store.order")`
2	Same as @SpanAttribute("order.item")`and `@SpanAttribute
3	Span created automatically if `micronaut-tracing-opentelemetry-http` is declared
4	Required for `@SpanTag`
5	Tag is only added to the span if `micronaut-tracing-opentelemetry-http` is declared; otherwise ignored

6.4. Warehouse Client

You can also mix OpenTelemetry and Micronaut Tracing annotations in the same class.

If you have the following dependency declared, all HTTP Client methods (those annotated with @Get, @Post, etc.) will create spans automatically.

build.gradle

implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")

src/main/kotlin/example/micronaut/WarehouseClient.kt

package example.micronaut

import io.micronaut.http.HttpResponse
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.http.annotation.QueryValue
import io.micronaut.http.client.annotation.Client
import io.micronaut.tracing.annotation.ContinueSpan
import io.micronaut.tracing.annotation.SpanTag
import io.opentelemetry.instrumentation.annotations.SpanAttribute
import io.opentelemetry.instrumentation.annotations.WithSpan

@Client("/warehouse") (1)
interface WarehouseClient {

    @Post("/order")
    @WithSpan
    fun order(@SpanTag("warehouse.order") json: Map<String, Any>) : HttpResponse<Any>

    @Get("/count")
    @ContinueSpan
    fun getItemCount(@QueryValue store: String, @SpanAttribute @QueryValue upc: Int) : Int

}

1	Some external service without tracing

6.5. Warehouse Controller

The WarehouseController class represents external service that will be called by WarehouseClient.

src/main/kotlin/example/micronaut/WarehouseController.kt

package example.micronaut

import io.micronaut.http.HttpResponse
import io.micronaut.http.annotation.Controller
import io.micronaut.http.annotation.Get
import io.micronaut.http.annotation.Post
import io.micronaut.scheduling.TaskExecutors
import io.micronaut.scheduling.annotation.ExecuteOn
import java.util.Random

@ExecuteOn(TaskExecutors.IO) (1)
@Controller("/warehouse") (2)
class WarehouseController {

    @Get("/count") (3)
    fun getItemCount() : HttpResponse<Int> = HttpResponse.ok(Random().nextInt(11))


    @Post("/order") (4)
    fun order() : HttpResponse<Any> {
        try {
            //To simulate an external process taking time
            Thread.sleep(500)
        } catch (e: InterruptedException) {
        }

        return HttpResponse.accepted()
    }

}

1	It is critical that any blocking I/O operations (such as fetching the data from the database) are offloaded to a separate thread pool that does not block the Event loop.
2	The class is defined as a controller with the @Controller annotation mapped to the path `/warehouse`.
3	The @Get annotation maps the `getItemCount` method to an HTTP GET request on `/warehouse/count`.
4	The @Get annotation maps the `order` method to an HTTP GET request on `/warehouse/order`.

6.6. Tracer Configuration

If you used Micronaut CLI or Launch to create your application, the OpenTelemetry exporter will automatically be added to your application.yml configurations.

src/main/resources/application.yml

otel:
  traces:
    exporter: google_cloud_trace

7. Run the Application

The application can be deployed to Google Cloud Run, Cloud Function, Cloud Compute, or App Engine.

Traces can be sent to Google Cloud Trace outside the Google Cloud.

This allows us to run the application locally and see the traces in Google Cloud Trace.

To run the application, use the ./gradlew run command, which starts the application on port 8080.

You might get this error message when running your application:
"Message: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information."

If you are developing locally you can do:

gcloud auth application-default login

However, it is strongly recommended that you set up a service account. Follow the instructions in the link above and Micronaut GCP setup instructions for creating and configuring the service account.

8. Traces

Open the GCP Tracing Console.

If the data is not displayed, give it a minute. It takes a few seconds for the traces to show up in the console.

8.1. Get Item Counts

curl http://localhost:8080/store/inventory/laptop

Each span is represented by a blue bar.

8.2. Order Item

curl -X "POST" "http://localhost:8080/store/order" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{"item":"laptop", "count":5}'

Selecting a different span will show you the labels (a.k.a. attributes/tags) and other details of the span.

8.3. Get Inventory

curl http://localhost:8080/store/inventory

Looking at the trace, we can conclude that retrieving the items sequentially might not be the best design choice.

9. Cleaning Up

After you’ve finished this guide, you can clean up the resources you created on Google Cloud Platform so you won’t be billed for them in the future. The following sections describe how to delete or turn off these resources.

9.1. Deleting the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

Deleting a project has the following consequences:

If you used an existing project, you’ll also delete any other work you’ve done in the project.
You can’t reuse the project ID of a deleted project. If you created a custom project ID that you plan to use in the future, you should delete the resources inside the project instead. This ensures that URLs that use the project ID, such as an appspot.com URL, remain available.
If you are exploring multiple tutorials and quickstarts, reusing projects instead of deleting them prevents you from exceeding project quota limits.

9.1.1. Via the CLI

To delete the project using the Cloud SDK, run the following command, replacing YOUR_PROJECT_ID with the project ID:

gcloud projects delete YOUR_PROJECT_ID

9.1.2. Via the Cloud Platform Console

In the Cloud Platform Console, go to the Projects page.

In the project list, select the project you want to delete and click Delete project. After selecting the checkbox next to the project name, click Delete project

In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting or turning off specific resources

You can individually delete or turn off some of the resources that you created during the tutorial.

Table of Contents