[ + ]

Table of Contents

Micronaut Kubernetes

Integration between Micronaut and Kubernetes

Version: 6.2.0-SNAPSHOT

1 Introduction

This project eases Kubernetes integration with Micronaut.

It adds support for the following features:

  • Service Discovery.

  • Configuration client for config maps and secrets.

  • Kubernetes blocking and non-blocking clients built on top of official Kubernetes Java SDK

To use the BUILD-SNAPSHOT version of this library, check the documentation to use snapshots.

Namespace configuration

When a Micronaut application with this module is running within a Pod in a Kubernetes cluster, it will infer automatically the namespace it’s running from by reading it from the service account secret (which will be provisioned at /var/run/secrets/kubernetes.io/serviceaccount/namespace).

However, the namespace can still be overridden via configuration in bootstrap.yml:

kubernetes:
  client:
    namespace: other-namespace

2 What's New?

The Micronaut Kubernetes module 3.0.0 includes the following changes:

Official K8s JAVA SDK client with reactive support

Micronaut Kubernetes is now using the official Kubernetes Java SDK client instead of the in-house client.

Apart this new module micronaut-kubernetes-client there are two additional modules micronaut-client-reactor and micronaut-client-rxjava2 that extends the client API classes by the reactive support of respective reactive framework.

3 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-kubernetes/releases

4 Service Discovery

The Service Discovery module allows Micronaut HTTP clients to discover Kubernetes services.

To get started, you need to declare the following dependency:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-discovery-client")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-discovery-client</artifactId>
</dependency>

By default in any client you can use as Service ID the Kubernetes Endpoints name generated by a Kubernetes Service for the configured namespace.

Consider the following Kubernetes service definition:

my-service.yml
kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9376

This specification will create a new Service object named my-service, as well as an Endpoints object also named my-service.

In your HTTP client, you can use my-service as Service ID: @Client("my-service").

Note that service discovery is enabled by default in Micronaut. To disable it, set kubernetes.client.discovery.enabled to false.

Service specific client configuration

Kubernetes Service is a complex resource that can handle various use cases by providing specific configuration. For this Micronaut Kubernetes supports a manual service discovery configuration per Service http client that allows you to configure custom:

Key Description

name

name of the resource in Kubernetes in case it is different than the Service ID

namespace

namespace of the resource in case it’s different than the configured namespace

port

port name in case the target resource is a Multi-Port Service

mode

service specific discovery mode in case it’s different than the globally configured discovery mode

Examples of service configurations

Multi-port service

For the following Multi-Port Service:

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9376
    - name: https
      protocol: TCP
      port: 443
      targetPort: 9377

the manual service configuration for http port will be:

bootstrap.yml
kubernetes:
  client:
    discovery:
      services:
        my-service:
          port: http

Headless service with selector

For the following Headless service with selector:

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  clusterIP: None
  selector:
    app: MyApp
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9376

the manual service configuration will be:

bootstrap.yml
kubernetes:
  client:
    discovery:
      services:
        my-service:
          mode: endpoint

ExternalName service type

For the following ExternalName service:

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: prod
spec:
  type: ExternalName
  externalName: launch.micronaut.io

the manual service configuration will be:

bootstrap.yml
kubernetes:
  client:
    discovery:
      services:
        my-service:
          mode: service

Service discovery modes

Service discovery mode is a mechanism that allows to support different strategies for the actual service discovery in Kubernetes by implementing KubernetesServiceInstanceProvider interface.

Currently Micronaut Kubernetes implements two discovery modes:

  • endpoint mode uses the Kubernetes Endpoins API for the service discovery. Note that the service load balancing is handled by Microunat application.

  • service mode uses the Kubernetes Service API for the service discovery. The service mode extracts the service ClusterIP address from the Service status.

Both discovery modes are using the metadata.name for the Service ID identificator.

The discovery mode can be configured globally for all Service IDs or per service. Note that endpoint is the default global discovery mode. That can be overridden via configuration in bootstrap.yml:

bootstrap.yml
kubernetes:
  client:
    discovery:
      mode: service

Watching for changes

Both discovery modes support watching for changes of their respective resources. To enable it, set kubernetes.client.discovery.mode-configuration.endpoint.watch.enabled to true for the endpoint mode. For the service mode set kubernetes.client.discovery.mode-configuration.service.watch.enabled to true.

Kubernetes API authentication

Micronaut authenticates to the Kubernetes API using the token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. Note that by default, the service account used may only have permissions over the kube-system namespace. The service discovery functionality requires some additiona read permissions. Refer to the Kubernetes documentation for more information about Role-based access control (RBAC).

One of the options is to create the following Role and RoleBinding (make sure to apply them to the service account used, if not default):

auth.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-discoverer
  namespace: micronaut-kubernetes
rules:
  - apiGroups: [""]
    resources: ["services", "endpoints", "configmaps", "secrets", "pods"]
    verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: default-service-discoverer
  namespace: micronaut-kubernetes
subjects:
  - kind: ServiceAccount
    name: default
    namespace: micronaut-kubernetes
roleRef:
  kind: Role
  name: service-discoverer
  apiGroup: rbac.authorization.k8s.io
In Google Cloud’s Kubernetes Engine, in order to create the above, you must grant your user the ability to create roles in Kubernetes by running the following command: 
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user yourGoogleAccount@gmail.com

Connecting to services using HTTPS

There are three ways for this library to determine whether a service should be connected to using SSL (the following examples assume there is a Deployment named secure-deployment).

Using https as port name

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: secure-service-port-name
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 1234
      protocol: TCP
      name: https

Using a port ending in 443

Port numbers like 443, 8443, etc. will match.

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: secure-service-port-number
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 443
      protocol: TCP

Using labels

Set a label named secure with value true to have the client use HTTPS.

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: secure-service-labels
  labels:
    secure: "true"
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 1234
      protocol: TCP

Service filtering

You can filter the services discovered by using kubernetes.client.discovery.includes or kubernetes.client.discovery.excludes:

kubernetes:
  client:
    discovery:
      includes:
        - my-service
        - other-service

Or:

kubernetes:
  client:
    discovery:
      excludes: not-this-service

In addition to that, Kubernetes labels can be used to better match the services that should be available for service discovery:

kubernetes:
  client:
    discovery:
      labels:
        - app: my-app
        - env: prod

Note that the filtering is not applied on manually configured service configurations.

Service discovery guides

See the following guides to learn more about Kubernetes service discovery using the Micronaut Framework:

5 Configuration Client

The Configuration client will read Kubernetes' ConfigMaps and Secrets instances and make them available as PropertySources instances in your application.

Then, in any bean you can read the configuration values from the ConfigMap or Secret using @Value or any other way to read configuration values.

Configuration parsing happens in the bootstrap phase. Therefore, to enable distributed configuration clients, define the following in bootstrap.yml (or .properties, .json, etc):

micronaut:
  config-client:
    enabled: true

ConfigMaps

Supported formats for ConfigMaps are:

  • Java .properties.

  • YAML.

  • JSON.

  • Literal values.

The configuration client by default will read all the ConfigMaps for the configured namespace. You can further filter which config map names are processed by defining kubernetes.client.config-maps.includes or kubernetes.client.config-maps.excludes:

kubernetes:
  client:
    config-maps:
      includes:
        - my-config-map
        - other-config-map

Or:

kubernetes:
  client:
    config-maps:
      excludes: not-this-config-map

In addition to that, Kubernetes labels can be used to better match the config maps that should be available as property sources. This can be done by defining the label and value directly:

kubernetes:
  client:
    config-maps:
      labels:
        - app: my-app
        - env: prod

Or by including every config map that has the same Kubernetes label as the pod the Micronaut application is running in. This is handy if you use a package manager like Helm. A good example would be app.kubernetes.io/instance which is a unique label identifying the instance of an application:

kubernetes:
  client:
    config-maps:
      pod-labels:
        - "app.kubernetes.io/instance"

Additionally, you can also enable exception-on-pod-labels-missing property in case you want an exception to be thrown if at least one of the labels, specified in pod-labels section is not found among the application’s pod labels. This can be useful if you want to prevent loading all config maps available in namespace in case of mistyping or misconfiguration:

kubernetes:
  client:
    config-maps:
      exception-on-pod-labels-missing: true
      pod-labels:
        - "app.kubernetes.io/instance"

Note that on the resulting config maps, you can still further filter them with includes/excludes properties.

Reading ConfigMaps from mounted volumes

In the case of ConfigMapss, reading them from the Kubernetes API requires additional permissions, as stated above. Therefore, you may want to read them from mounted volumes in the pod.

Given the following ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mounted-config
data:
  mounted.yml: |-
    foo: bar

It can be mounted as a volume in a pod or deployment definition:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: micronautapp
    volumeMounts:
    - name: configuration
      mountPath: /etc/configuration
      readOnly: true
  volumes:
  - name: configuration
    configMap:
      name: mounted-config

This will make Kubernetes to create file per data entry:

  • /etc/configuration/mounted.yml

While you could potentially use the java.io or java.nio APIs to read the contents yourself, this configuration module can convert them into a PropertySource so that you can consume the values much more easily. In order to do so, define the following configuration:

kubernetes:
  client:
    config-maps:
      paths:
        - /etc/configuration

Each file in the directory will become a property source file. The file format will be automatically deduced based on the file suffix. Supported formats for mounted ConfigMap files are:

  • Java .properties

  • YAML

  • JSON

When kubernetes.client.config-maps.paths is defined, the Kubernetes API will not be used to read any other config maps. If you still want to read the remaining config maps from the API, set the following configuration:

kubernetes:
  client:
    config-maps:
      use-api: true
      excludes: mounted-config  # Because it will be read as a mounted volume
      paths:
        - /etc/configuration

In this scenario, if there are property keys defined in both type of config maps, the ones coming from mounted volumes will take precedence over the ones coming from the API.

Watching for changes in ConfigMaps

By default, this configuration module will watch for ConfigMaps added/modified/deleted, and provided that the changes match with the above filters, they will be propagated to the Environment and refresh it.

This means that those changes will be immediately available in your application without a restart.

If you want to disable watching for ConfigMap changes, set kubernetes.client.config-maps.watch to false. This should be done in the bootstrap.yml configuration file because the configuration client is initialized during the bootstrap phase, which happens before evaluating the application.yml configuration file.

When kubernetes.client.config-maps.use-api is set to false, watching for the changes won’t be started.

Examples

You can create a Kubernetes ConfigMap off an existing file with the following command:

kubectl create configmap my-config --from-file=my-config.properties

Or:

kubectl create configmap my-config --from-file=my-config.yml

Or:

kubectl create configmap my-config --from-file=my-config.json

You can also create a ConfigMap from literal values:

kubectl create configmap my-config --from-literal=special.how=very --from-literal=special.type=charm

Secrets

Secrets read from the Kubernetes API will be base64-decoded and made available as PropertySource s, so that they can be also read with @Value, @ConfigurationProperties, etc.

Only Opaque secrets will be considered.

By default, secrets access is diabled. To enable them, set in bootstrap.yml:

kubernetes:
  client:
    secrets:
      enabled: true

The configuration client, by default, will read all the Secrets for the configured namespace. You can further filter which config map names are processed by defining kubernetes.client.secrets.includes or kubernetes.client.secrets.excludes:

kubernetes:
  client:
    secrets:
      enabled: true
      includes: this-secret

Or:

kubernetes:
  client:
    secrets:
      enabled: true
      excludes: not-this-secret

Similarly to ConfigMaps, labels can also be used to match the desired secrets:

kubernetes:
  client:
    secrets:
      enabled: true
      labels:
        - app: my-app
        - env: prod

This also works for pod labels:

kubernetes:
  client:
    secrets:
      enabled: true
      pod-labels:
        - "app.kubernetes.io/instance"

As well as exception-on-pod-labels-missing property:

kubernetes:
  client:
    secrets:
      enabled: true
      exception-on-pod-labels-missing: true
      pod-labels:
        - "app.kubernetes.io/instance"

Reading Secrets from mounted volumes

In the case of Secrets, reading them from the Kubernetes API requires additional permissions, as stated above. Therefore, you may want to read them from mounted volumes in the pod.

Given the following secret:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

It can be mounted as a volume in a pod or deployment definition:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

This will make Kubernetes to create 2 files:

  • /etc/foo/username.

  • /etc/foo/password.

Their content will be the decoded strings from the original base-64 encoded values.

While you could potentially use the java.io or java.nio APIs to read the contents yourself, this configuration module can convert them into a PropertySource so that you can consume the values much more easily. In order to do so, define the following configuration:

kubernetes:
  client:
    secrets:
      enabled: true
      paths:
        - /etc/foo

Each file in the directory will become the property key, and the file contents, the property value.

When kubernetes.client.secrets.paths is defined, the Kubernetes API will not be used to read any other secret. If you still want to read the remaining secrets from the API, set the following configuration:

kubernetes:
  client:
    secrets:
      enabled: true
      use-api: true
      excludes: mysecret  # Because it will be read as a mounted volume
      paths:
        - /etc/foo

In this scenario, if there are property keys defined in both type of secrets, the ones coming from mounted volumes will take precedence over the ones coming from the API.

6 Health Checks

Health Indicators

This configuration module provides a KubernetesHealthIndicator that probes communication with the Kubernetes API, and provides some information about the pod where the application is running from.

The service discovery client will also display all the services that were resolved from Kubernetes.

An example output of a /health request would be:

{
  "name": "micronaut-service",
  "status": "UP",
  "details": {
    "kubernetes": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "namespace": "default",
        "podName": "example-service-786cd45b78-bzfw5",
        "podPhase": "Running",
        "podIP": "10.1.3.124",
        "hostIP": "192.168.65.3",
        "containerStatuses": [
          {
            "name": "example-service",
            "image": "registry.hub.docker.com/alvarosanchez/example-service:latest",
            "ready": true
          }
        ]
      }
    },
    "compositeDiscoveryClient(kubernetes)": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "services": {
          "example-service": [
            "http://10.1.3.124:8081",
            "http://10.1.3.126:8081"
          ],
          "non-secure-service": [
            "http://10.1.3.127:1234"
          ],
          "kubernetes": [
            "https://kubernetes:443"
          ],
          "secure-service-port-name": [
            "https://10.1.3.127:1234"
          ],
          "example-client": [
            "http://10.1.3.125:8082"
          ],
          "secure-service-port-number": [
            "https://10.1.3.127:443"
          ],
          "secure-service-labels": [
            "https://10.1.3.127:1234"
          ]
        }
      }
    },
    "diskSpace": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "total": 109702647808,
        "free": 69758287872,
        "threshold": 10485760
      }
    }
  }
}

Health checks require the following dependency:

implementation("io.micronaut:micronaut-management")
<dependency>
    <groupId>io.micronaut</groupId>
    <artifactId>micronaut-management</artifactId>
</dependency>

Also note that in order to see the full details of the health checks you may need additional configuration. Check the documentation of the Health Endpoint for more information about how to configure it.

7 Kubernetes Client

The micronaut-kubernetes-client module gives you the ability to use official Kubernetes Java SDK apis objects as a regular Micronaut beans.

The complete list of available beans is declared in the Apis#values annotation value.

First you need add a dependency on the micronaut-kubernetes-client module:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-client")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-client</artifactId>
</dependency>

Then you can simply use Micronaut injection to get configured apis object from package io.kubernetes.client.openapi.apis :

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.apis.CoreV1Api;
import io.kubernetes.client.openapi.models.V1PodList;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1Api coreV1Api;

    public MyService(CoreV1Api coreV1Api) {
        this.coreV1Api = coreV1Api;
    }

    public void myMethod(String namespace) throws ApiException {
        V1PodList v1PodList = coreV1Api.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null, false);
    }
}

Authentication

The Kubernetes client source of authentication options is automatically detected by the ClientBuilder#standard(), specifically:

Creates a builder which is pre-configured in the following way

  • If $KUBECONFIG is defined, use that config file.

  • If $HOME/.kube/config can be found, use that.

  • If the in-cluster service account can be found, assume in cluster config.

  • Default to localhost:8080 as a last resort.

Also for specific cases you can update the authentication configuration options by using kubernetes.client properties listed below:

Name

Description

basePath

Kubernetes API base path. Example: https://localhost:8081

caPath

CA file path.

tokenPath

Token file path.

kubeConfigPath

Kube config file path.

verifySsl

Boolean if the api should verify ssl. Default: true

Reactive Support

In addition to the official Kubernetes Java SDK Async clients, this module provides clients that use RxJava or Reactor to allow reactive programming with Micronaut for each Api.

RxJava 2 Reactive Support

For RxJava 2 add the following dependency:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-rxjava2")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-client-rxjava2</artifactId>
</dependency>

The module contains all official Kubernetes API beans in format <ApiClassName>RxClient, for example the CoreV1Api class is injected as CoreV1ApiRxClient.

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.rxjava2.CoreV1ApiRxClient;
import io.reactivex.Single;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1ApiRxClient coreV1ApiRxClient;

    public MyService(CoreV1ApiRxClient coreV1ApiRxClient) {
        this.coreV1ApiRxClient = coreV1ApiRxClient;
    }

    public void myMethod(String namespace) {
        Single<V1PodList> v1PodList = coreV1ApiRxClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
    }
}

RxJava 3 Reactive Support

For RxJava 3 add the following dependency:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-rxjava3")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-client-rxjava3</artifactId>
</dependency>

The module contains all official Kubernetes API beans in format <ApiClassName>RxClient, for example the CoreV1Api class is injected as CoreV1ApiRxClient.

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.rxjava3.CoreV1ApiRxClient;
import io.reactivex.Single;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1ApiRxClient coreV1ApiRxClient;

    public MyService(CoreV1ApiRxClient coreV1ApiRxClient) {
        this.coreV1ApiRxClient = coreV1ApiRxClient;
    }

    public void myMethod(String namespace) {
        Single<V1PodList> v1PodList = coreV1ApiRxClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
    }
}

Reactor Reactive Support

For Reactor add the following dependency:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-reactor")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-client-reactor</artifactId>
</dependency>

The module contains all official Kubernetes API beans in format <ApiClassName>ReactiveClient, for example the CoreV1Api class is injected as CoreV1ApiReactiveClient.

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.reactive.CoreV1ApiReactiveClient;
import reactor.core.publisher.Mono;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1ApiReactiveClient coreV1ApiReactiveClient;

    public MyService(CoreV1ApiReactiveClient coreV1ApiReactiveClient) {
        this.coreV1ApiReactiveClient = coreV1ApiReactiveClient;
    }

    public void myMethod(String namespace) {
        Mono<V1PodList> v1PodList = coreV1ApiReactiveClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
    }
}

Advanced Configuration

For advanced configuration options of ApiClient that are not suitable to provide via application.yml, you can declare a BeanCreatedEventListener bean that listens for ApiClient bean creation, and apply any further customisation to OkHttpClient there:

@Singleton
public class ApiClientListener implements BeanCreatedEventListener<ApiClient> {

    @Override
    public ApiClient onCreated(BeanCreatedEvent<ApiClient> event) {
        ApiClient apiClient = event.getBean();
        OkHttpClient okHttpClient = apiClient.getHttpClient().newBuilder()
                .readTimeout(5345, TimeUnit.MILLISECONDS)
                .build();
        apiClient.setHttpClient(okHttpClient);
        return apiClient;
    }
}

8 Kubernetes Informer

The micronaut-kubernetes-informer module integrates the SharedIndexInformer that is part of official Kubernetes SDK and simplifies its creation. The Informer is similar to a Watch but an Informer tries to re-list and re-watch to hide failures from the caller and provides a store of all the current resources. Note that the default official implementation SharedInformerFactory creates shared informers per the Kubernetes resource type. However the module micronaut-kubernetes-informer creates namespace scoped informers of the Kubernetes resource type, meaning that the informer is shared per specified namespace and kind.

First you need add a dependency on the micronaut-kubernetes-informer module:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-informer")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-informer</artifactId>
</dependency>

Then create a bean that implements ResourceEventHandler with the Kubernetes resource type of your choice and add the @Informer annotation trough which you provide the configuration for the SharedIndexInformer.

The example below illustrates the declaration of the @Informer with the ResourceEventHandler for handling the changes of the Kubernetes V1ConfigMap resource.

@Informer(apiType = V1ConfigMap.class, apiListType = V1ConfigMapList.class) // (1)
public class ConfigMapInformer implements ResourceEventHandler<V1ConfigMap> { // (2)

    @Override
    public void onAdd(V1ConfigMap obj) {
    }

    @Override
    public void onUpdate(V1ConfigMap oldObj, V1ConfigMap newObj) {
    }

    @Override
    public void onDelete(V1ConfigMap obj, boolean deletedFinalStateUnknown) {
    }
}
1 The @Informer annotation defines the Kubernetes resource type and resource list type
2 The ResourceEventHandler interface declares method handlers for added, updated and deleted resource

To create an Informer for non-namespaced resource like V1ClusterRole, configure the @Informer the same way like it is done for namespaced resource.

The ResourceEventHandlerConstructorInterceptor logic takes care of automated evaluation of the resource apiGroup and resourcePlural by fetching the API resource details from the Kubernetes API by using Discovery. The API resource discovery can be disabled by: kubernetes.client.api-discovery.enabled: false. In the case the discovery is disabled the resourcePlural and apiGroup needs to be provided manually in the @Informer annotation.

The @Informer annotation provides several configuration options:

Table 1. @Informer attributes

Element

Description

apiType

The resource api type that must extend from the KubernetesObject. For Kubernetes core resources the types can be found in io.kubernetes.client.openapi.models package. For example io.kubernetes.client.openapi.models.V1ConfigMap.

apiListType

The resource api list type. For example io.kubernetes.client.openapi.models.V1ConfigMapList.

apiGroup

The resource api group. For example some of the Kubernetes core resources has no group like V1ConfigMap, on the contrary the V1ClusterRole has rbac.authorization.k8s.io.

resourcePlural

The resource plural that identifies the Api. For example for the resource V1ConfigMap it is configmaps.

namespace

Namespace of the watched resource. If left empty then namespace is resolved by NamespaceResolver. To watch resources from all namespaces configure this parameter to Informer.ALL_NAMESPACES.

namespaces

List of the namespaces of the watcher resource. The SharedIndexInformer will be created for every namespace and all events will be delivered to the specified ResourceEventHandler.

namespacesSupplier

Supplier class that provides the list of namespaces to watch. Note that the supplier class needs to be a bean in the application context and it is intended for dynamic evaluation of the namesapces to watch. Finally the namespace, namespaces and namespacesSupplier can be used in combination.

labelSelector

Informer label selector, see Label selectors for more information. By default there is no label selector.

labelSelectorSupplier

Supplier class for the label selector. Note that the supplier class needs to be a bean in the application context. Finally the labelSelector and labelSelectorSupplier can be used in combination.

resyncCheckPeriod

How often to check the need for resync of resources. If left empty the default resync check period is used.
The concept of shared informer means that the SharedIndexInformer for the respective Kubernetes resource type is registered just once for the given namespace. The next request to register another informer of the same Kubernetes resource type within the same namespace will result in returning of the previously created informer. In practice if you create two ResourceEventHandler<V1ConfigMap> but the @Informer annotation will have different optional configuration for labelSelector then the SharedInformerFactory creates just one SharedInformer, meaning the other @Informer configuration will be ignored. If the labelSelector resp. labelSelectorSupplier differs then create one labelSelector that matches both cases.

Programmatic creation of SharedIndexInformer

Use the bean SharedIndexInformerFactory to create the SharedIndexInformer programmatically:

        SharedIndexInformer<V1ConfigMap> sharedIndexInformer = factory.sharedIndexInformerFor(
                V1ConfigMap.class, // (1)
                V1ConfigMapList.class, // (2)
                "configmaps", // (3)
                "",  // (4)
                "default",  // (5)
                null,
                null,
                true
        );
1 The resource api type that must extend from the KubernetesObject. For Kubernetes core resources the types can be found in io.kubernetes.client.openapi.models package. For example io.kubernetes.client.openapi.models.V1ConfigMap.
2 The resource api list type that must extend from the KubernetesListObject. For example io.kubernetes.client.openapi.models.V1ConfigMapList.
3 The resource plural that identifies the Api. For example for the resource V1ConfigMap it is configmaps.
4 The resource api group.
5 The namespace to watch.

By using the SharedIndexInformerFactory bean you can also get existing informer:

        SharedIndexInformer<V1ConfigMap> sharedIndexInformer = factory.getExistingSharedIndexInformer(
                "default", // (1)
                V1ConfigMap.class); // (2)
1 The informer namespace.
2 The informer resource type.

SharedIndexInformer local cache

The SharedIndexInformer has internal cache that is eventually consistent with the authoritative state. The local cache starts out empty, and gets populated and updated. For the detailed description of SharedIndexInformer internals visit the reference implementation pkg.go.dev/k8s.io/client-go#SharedIndexInformer in Go language.

The cache is exposed by SharedIndexInformer#getIndexer() method:

@Singleton
public class SharedInformerCache {

    private final SharedIndexInformerFactory sharedIndexInformerFactory;

    public SharedInformerCache(SharedIndexInformerFactory sharedIndexInformerFactory) {
        this.sharedIndexInformerFactory = sharedIndexInformerFactory;
    }

    /**
     * Get all config maps from informer from namespace.
     */
    List<V1ConfigMap> getConfigMaps(String namespace) {
        SharedIndexInformer<V1ConfigMap> sharedIndexInformer = sharedIndexInformerFactory.getExistingSharedIndexInformer(namespace, V1ConfigMap.class);
        if (sharedIndexInformer != null) {
            Indexer<V1ConfigMap> indexer = sharedIndexInformer.getIndexer();
            return indexer.list();
        } else {
            return null;
        }
    }
}

9 Kubernetes Operator

The micronaut-kubernetes-operator module integrates the official Kubernetes Java SDK controller support that is part of the extended client module. The micronaut-kubernetes-operator module is build on top of the micronaut-kubernetes-informer module and allows you to easily create the reconciler for both native and custom resources.

First you need add a dependency on the micronaut-kubernetes-operator module:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-operator")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-operator</artifactId>
</dependency>

Then create a bean that implements ResourceReconciler with the Kubernetes resource type of your choice. Then add the @Operator annotation trough the which you provide the configuration for the controllers that will be created by Micronaut for your reconciler.

The example below illustrates the use of the @Operator with the ResourceReconciler that reconciles the Kubernetes V1ConfigMap resource.

import io.kubernetes.client.extended.controller.reconciler.Request;
import io.kubernetes.client.extended.controller.reconciler.Result;
import io.kubernetes.client.openapi.models.V1ConfigMap;
import io.kubernetes.client.openapi.models.V1ConfigMapList;
import io.micronaut.core.annotation.NonNull;
import io.micronaut.kubernetes.client.informer.Informer;
import java.util.Optional;

@Operator(informer = @Informer(apiType = V1ConfigMap.class, apiListType = V1ConfigMapList.class)) // (1)
public class ConfigMapResourceReconciler implements ResourceReconciler<V1ConfigMap> { // (2)

    @Override
    @NonNull
    public Result reconcile(@NonNull Request request, @NonNull OperatorResourceLister<V1ConfigMap> lister) { // (3)
        Optional<V1ConfigMap> resource = lister.get(request); // (4)
        // .. reconcile  (5)
        return new Result(false); // (6)
    }
}
1 The @Operator annotation defines the resource type which is the subject of reconciliation. The definition is done by using the @Informer annotation. Both annotations provide other object specific configuration options.
2 The ResourceReconciler interface declares the reconcile method that is main point of interaction in between the Micronaut and your application logic.
3 The reconciliation input consists from the Request that uniquely identifies the reconciliation resource and the OperatorResourceLister trough which the actual subject of reconciliation can be retrieved.
4 Retrieval of the resource for the reconciliation.
5 This is where the idempotent reconciliation logic should be placed. Note that the reconciliation logic is responsible for the update of the resource status stanza.
6 The return value of reconciliation method is the Result.

The @Operator annotation provides several configuration options:

Table 1. @Operator attributes

Element

Description

name

The name of the operator controller thread. Defaults to the Operator<resource-type>.

informer

The @Informer annotation used for the configuration of the Operator’s informer. This value is required.

onAddFilter

The java.util.function.Predicate decides what newly created resources are subject for the reconciliation.

onUpdateFilter

The java.util.function.BiPredicate decides what updated resources are subject for the reconciliation

onDeleteFilter

The java.util.function.BiPredicate decides what deleted resources are subject for the reconciliation

Leader election

The LeaderElectingController is responsible for the leader election of the application replica that will reconcile the resources. Generally if the lock is not renewed within the specified amount of time, other replicas may try to acquire the lock and become the leader.

You can adjust the leader elector configuration by using Micronaut configuration properties kubernetes.client.operator.leader-election.lock:

Table 2. Leader election properties

Element

Description

lease-duration

The lock lease duration. Defaults to 10s.

renew-deadline

The lock renew deadline. If the LeaderElector fails to renew the lock within the deadline then the controller looses the lock. Defaults to 8s.

retry-period

The lock acquire retry period. Defaults to 5s.

For example:

Example of custom lock acquisition properties. Note that the value is of type Duration:
kubernetes:
  client:
    operator:
      leader-election:
        lock:
          lease-duration: 60s
          renew-deadline: 50s
          retry-period: 20s

Additionally, when the lock is acquired the LeaseAcquiredEvent is emitted. Similarly on a lost lease the LeaseLostEvent event is emitted.

Lock identity

The lock identity is used to uniquely identify the application that holds the lock and thus is responsible for reconciling the resources. By default, the POD name the application runs within is the source for the lock identity. This means the application must run in the Kubernetes cluster.

To create custom lock identity, create a bean that implements the LockIdentityProvider interface:

Lock resource types

The LeaderElectingController uses the native Kubernetes resource to store the lock information. Currently supported resources are V1ConfigMap, V1Endpoints and V1Lease.

By default, the micronaut-kubernetes-operator module uses the V1Lease. This can be changed to V1Endpoints by configuring the property kubernetes.client.operator.leader-election.lock.resource-kind to endpoints , resp. to configmaps in case the V1ConfigMap resource is requested.

Note that the resource for the lock is created in the application namespace. Then the application name is used as the lock resource name. This can be changed by using Micronaut configuration properties:

Example on how to configure custom lock resource name and namespace:
kubernetes:
  client:
    operator:
      leader-election:
        lock:
          resource-name: custom-name
          resource-namespace: custom-namespace

Note that in case the RBAC authorization is enabled in your Kubernetes cluster, your application needs to have properly configured role with respect to the lock resource type.

Example of role for the V1Lease lock resource type:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: operator-lease-role
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---

Resource filtering

The @Operator annotation allows you to configure the resource filters that are executed before the resource is added into the reconciler work queue. You can configure three types of filters that are distinguished by the resource lifecycle: onAddFilter, onUpdateFilter and onDeleteFilter.

The onAddFilter predicate processes the newly created resources. Create a bean that implements java.util.function.Predicate with the same Kubernetes resource type like the operator is. Example below illustrates such filter for the V1ConfigMap resource:

import io.kubernetes.client.openapi.models.V1ConfigMap;
import jakarta.inject.Singleton;

import java.util.function.Predicate;

@Singleton
public class OnAddFilter implements Predicate<V1ConfigMap> {

    @Override
    public boolean test(V1ConfigMap v1ConfigMap) {
        if (v1ConfigMap.getMetadata().getAnnotations() != null) {
            return v1ConfigMap.getMetadata().getAnnotations().containsKey("io.micronaut.operator");
        }
        return false;
    }
}

The onUpdateFilter bi-predicate processes modified resources. Create a bean that implements java.util.function.BiPredicate with the same Kubernetes resource type like the operator is. Example below illustrates such filter for the V1ConfigMap resource:

import io.kubernetes.client.openapi.models.V1ConfigMap;
import jakarta.inject.Singleton;

import java.util.function.BiPredicate;

@Singleton
public class OnUpdateFilter implements BiPredicate<V1ConfigMap, V1ConfigMap> {

    @Override
    public boolean test(V1ConfigMap oldObj, V1ConfigMap newObj) {
        if (newObj.getMetadata().getAnnotations() != null) {
            return newObj.getMetadata().getAnnotations().containsKey("io.micronaut.operator");
        }
        return false;
    }
}

The onDeleteFilter bi-predicate processes deleted resources. Create a bean that implements java.util.function.BiPredicate with the same Kubernetes resource type like the operator is and the Boolean as second type. Example below illustrates such filter for the V1ConfigMap resource:

import io.kubernetes.client.openapi.models.V1ConfigMap;
import jakarta.inject.Singleton;

import java.util.function.BiPredicate;

@Singleton
public class OnDeleteFilter implements BiPredicate<V1ConfigMap, Boolean> {

    @Override
    public boolean test(V1ConfigMap v1ConfigMap, Boolean deletedFinalStateUnknown) {
        if (v1ConfigMap.getMetadata().getAnnotations() != null) {
            return v1ConfigMap.getMetadata().getAnnotations().containsKey("io.micronaut.operator");
        }
        return false;
    }
}
Note that in case of onDeleteFilter the predicate receives the resource for the test method, but when the ResouceReconciler’s reconcile method is executed the lister will return Optional.empty since the resource was already deleted. To properly reconcile the resource on it’s removal, use finalizers.

Example below illustrates the configuration of the filters:

import io.kubernetes.client.extended.controller.reconciler.Request;
import io.kubernetes.client.extended.controller.reconciler.Result;
import io.kubernetes.client.openapi.models.V1ConfigMap;
import io.kubernetes.client.openapi.models.V1ConfigMapList;
import io.micronaut.core.annotation.NonNull;
import io.micronaut.kubernetes.client.informer.Informer;

@Operator(informer = @Informer(apiType = V1ConfigMap.class, apiListType = V1ConfigMapList.class),
        onAddFilter = OnAddFilter.class, // (1)
        onUpdateFilter = OnUpdateFilter.class, // (2)
        onDeleteFilter = OnDeleteFilter.class) // (3)
public class ConfigMapResourceReconcilerWithFilters implements ResourceReconciler<V1ConfigMap> {

    @Override
    @NonNull
    public Result reconcile(@NonNull Request request, @NonNull OperatorResourceLister<V1ConfigMap> lister) {
        // .. reconcile
        return new Result(false);
    }
}
1 Configuration of onAddFilter.
2 Configuration of onUpdateFilter.
3 Configuration of onAddFilter.

10 Logging and debugging

If you need to debug the Micronaut Kubernetes module, you need to set the io.micronaut.kubernetes logger level to DEBUG:

<logger name="io.micronaut.kubernetes" level="DEBUG"/>

By configuring the logger level to TRACE, the module will produce detailed responses from the Kubernetes API.

<logger name="io.micronaut.kubernetes" level="TRACE"/>

Other package that might produce relevant logging is io.micronaut.discovery, which belongs to Micronaut Core.

In addition to that, another source of information is the Environment Endpoint, which outputs all the resolved PropertySources from ConfigMaps, and their corresponding properties.

11 Breaking Changes

Micronaut Kubernetes 4.x

Micronaut Kubernetes 4.0.0 updates to Kubernetes Client v18 (major). This major upgrade of the Kubernetes Client addresses several security CVEs.

Micronaut Kubernetes 3.x

This section documents breaking changes between Micronaut Kubernetes 2.x and Micronaut Kubernetes 3.x:

In-house Kubernetes client removed

The in-house Kubernetes client io.micronaut.kubernetes.client.v1.* was deprecated and removed. Instead use the new module micronaut-kubernetes-client or the reactive alternatives micronaut-client-reactor or micronaut-client-rxjava2 that extends the client API classes by the reactive support of respective reactive framework.

12 Guides

See the following list of guides to learn more about working with Kubernetes in the Micronaut Framework:

https://guides.micronaut.io/latest/tag-kubernetes.html

13 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-kubernetes