The documentation you are viewing is not the latest documentation of Micronaut Kubernetes
[ + ]

Table of Contents

Micronaut Kubernetes

Integration between Micronaut and Kubernetes

Version:

1 Introduction

This project eases Kubernetes integration with Micronaut.

It adds support for the following features:

  • Service Discovery.

  • Configuration client for config maps and secrets.

  • Kubernetes blocking and non-blocking clients built on top of official Kubernetes Java SDK

To use the BUILD-SNAPSHOT version of this library, check the documentation to use snapshots.

Namespace configuration

When a Micronaut application with this module is running within a Pod in a Kubernetes cluster, it will infer automatically the namespace it’s running from by reading it from the service account secret (which will be provisioned at /var/run/secrets/kubernetes.io/serviceaccount/namespace).

However, the namespace can still be overridden via configuration in bootstrap.yml:

kubernetes:
  client:
    namespace: other-namespace

2 What's New?

The Micronaut Kubernetes module 3.0.0 includes the following changes:

Official K8s JAVA SDK client with reactive support

Micronaut Kubernetes is now using the official Kubernetes Java SDK client instead of the in-house client.

Apart this new module micronaut-kubernetes-client there are two additional modules micronaut-client-reactor and micronaut-client-rxjava2 that extends the client API classes by the reactive support of respective reactive framework.

3 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-kubernetes/releases

4 Service Discovery

The Service Discovery module allows Micronaut HTTP clients to discover Kubernetes services.

To get started, you need to declare the following dependency:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-discovery-client:3.0.1")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-discovery-client</artifactId>
    <version>3.0.1</version>
</dependency>

Note that this configuration module requires at least Micronaut 2.

By default in any client you can use as Service ID the Kubernetes Endpoints name generated by a Kubernetes Service for the configured namespace.

Consider the following Kubernetes service definition:

my-service.yml
kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9376

This specification will create a new Service object named my-service, as well as an Endpoints object also named my-service.

In your HTTP client, you can use my-service as Service ID: @Client("my-service").

Note that service discovery is enabled by default in Micronaut. To disable it, set kubernetes.client.discovery.enabled to false.

Service specific client configuration

Kubernetes Service is a complex resource that can handle various use cases by providing specific configuration. For this Micronaut Kubernetes supports a manual service discovery configuration per Service http client that allows you to configure custom:

Key Description

name

name of the resource in Kubernetes in case it is different than the Service ID

namespace

namespace of the resource in case it’s different than the configured namespace

port

port name in case the target resource is a Multi-Port Service

mode

service specific discovery mode in case it’s different than the globally configured discovery mode

Examples of service configurations

Multi-port service

For the following Multi-Port Service:

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9376
    - name: https
      protocol: TCP
      port: 443
      targetPort: 9377

the manual service configuration for http port will be:

bootstrap.yml
kubernetes:
  client:
    discovery:
      services:
        my-service:
          port: http

Headless service with selector

For the following Headless service with selector:

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  clusterIP: None
  selector:
    app: MyApp
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9376

the manual service configuration will be:

bootstrap.yml
kubernetes:
  client:
    discovery:
      services:
        my-service:
          mode: endpoint

ExternalName service type

For the following ExternalName service:

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: prod
spec:
  type: ExternalName
  externalName: launch.micronaut.io

the manual service configuration will be:

bootstrap.yml
kubernetes:
  client:
    discovery:
      services:
        my-service:
          mode: service

Service discovery modes

Service discovery mode is a mechanism that allows to support different strategies for the actual service discovery in Kubernetes by implementing KubernetesServiceInstanceProvider interface.

Currently Micronaut Kubernetes implements two discovery modes:

  • endpoint mode uses the Kubernetes Endpoins API for the service discovery. Note that the service load balancing is handled by Microunat application.

  • service mode uses the Kubernetes Service API for the service discovery. The service mode extracts the service ClusterIP address from the Service status.

Both discovery modes are using the metadata.name for the Service ID identificator.

The discovery mode can be configured globally for all Service IDs or per service. Note that endpoint is the default global discovery mode. That can be overridden via configuration in bootstrap.yml:

bootstrap.yml
kubernetes:
  client:
    discovery:
      mode: service

Kubernetes API authentication

Micronaut authenticates to the Kubernetes API using the token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. Note that by default, the service account used may only have permissions over the kube-system namespace. The service discovery functionality requires some additiona read permissions. Refer to the Kubernetes documentation for more information about Role-based access control (RBAC).

One of the options is to create the following Role and RoleBinding (make sure to apply them to the service account used, if not default):

auth.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-discoverer
  namespace: micronaut-kubernetes
rules:
  - apiGroups: [""]
    resources: ["services", "endpoints", "configmaps", "secrets", "pods"]
    verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: default-service-discoverer
  namespace: micronaut-kubernetes
subjects:
  - kind: ServiceAccount
    name: default
    namespace: micronaut-kubernetes
roleRef:
  kind: Role
  name: service-discoverer
  apiGroup: rbac.authorization.k8s.io
In Google Cloud’s Kubernetes Engine, in order to create the above, you must grant your user the ability to create roles in Kubernetes by running the following command: 
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user yourGoogleAccount@gmail.com

Connecting to services using HTTPS

There are three ways for this library to determine whether a service should be connected to using SSL (the following examples assume there is a Deployment named secure-deployment).

Using https as port name

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: secure-service-port-name
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 1234
      protocol: TCP
      name: https

Using a port ending in 443

Port numbers like 443, 8443, etc. will match.

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: secure-service-port-number
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 443
      protocol: TCP

Using labels

Set a label named secure with value true to have the client use HTTPS.

my-service.yml
apiVersion: v1
kind: Service
metadata:
  name: secure-service-labels
  labels:
    secure: "true"
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 1234
      protocol: TCP

Service filtering

You can filter the services discovered by using kubernetes.client.discovery.includes or kubernetes.client.discovery.excludes:

kubernetes:
  client:
    discovery:
      includes:
        - my-service
        - other-service

Or:

kubernetes:
  client:
    discovery:
      excludes: not-this-service

In addition to that, Kubernetes labels can be used to better match the services that should be available for service discovery:

kubernetes:
  client:
    discovery:
      labels:
        - app: my-app
        - env: prod

Note that the filtering is not applied on manually configured service configurations.

5 Configuration Client

The Configuration client will read Kubernetes' ConfigMaps and Secrets instances and make them available as PropertySources instances in your application.

Then, in any bean you can read the configuration values from the ConfigMap or Secret using @Value or any other way to read configuration values.

Configuration parsing happens in the bootstrap phase. Therefore, to enable distributed configuration clients, define the following in bootstrap.yml (or .properties, .json, etc):

micronaut:
  config-client:
    enabled: true

ConfigMaps

Supported formats for ConfigMaps are:

  • Java .properties.

  • YAML.

  • JSON.

  • Literal values.

The configuration client by default will read all the ConfigMaps for the configured namespace. You can further filter which config map names are processed by defining kubernetes.client.config-maps.includes or kubernetes.client.config-maps.excludes:

kubernetes:
  client:
    config-maps:
      includes:
        - my-config-map
        - other-config-map

Or:

kubernetes:
  client:
    config-maps:
      excludes: not-this-config-map

In addition to that, Kubernetes labels can be used to better match the config maps that should be available as property sources. This can be done by defining the label and value directly:

kubernetes:
  client:
    config-maps:
      labels:
        - app: my-app
        - env: prod

Or by including every config map that has the same Kubernetes label as the pod the Micronaut application is running in. This is handy if you use a package manager like Helm. A good example would be app.kubernetes.io/instance which is a unique label identifying the instance of an application:

kubernetes:
  client:
    config-maps:
      pod-labels:
        - "app.kubernetes.io/instance"

Note that on the resulting config maps, you can still further filter them with includes/excludes properties.

Watching for changes in ConfigMaps

By default, this configuration module will watch for ConfigMaps added/modified/deleted, and provided that the changes match with the above filters, they will be propagated to the Environment and refresh it.

This means that those changes will be immediately available in your application without a restart.

If you want to disable watching for ConfigMap changes, set kubernetes.client.config-maps.watch to false. This should be done in the bootstrap.yml configuration file because the configuration client is initialized during the bootstrap phase, which happens before evaluating the application.yml configuration file.

Examples

You can create a Kubernetes ConfigMap off an existing file with the following command:

kubectl create configmap my-config --from-file=my-config.properties

Or:

kubectl create configmap my-config --from-file=my-config.yml

Or:

kubectl create configmap my-config --from-file=my-config.json

You can also create a ConfigMap from literal values:

kubectl create configmap my-config --from-literal=special.how=very --from-literal=special.type=charm

Secrets

Secrets read from the Kubernetes API will be base64-decoded and made available as PropertySource s, so that they can be also read with @Value, @ConfigurationProperties, etc.

Only Opaque secrets will be considered.

By default, secrets access is diabled. To enable them, set in bootstrap.yml:

kubernetes:
  client:
    secrets:
      enabled: true

The configuration client, by default, will read all the Secrets for the configured namespace. You can further filter which config map names are processed by defining kubernetes.client.secrets.includes or kubernetes.client.secrets.excludes:

kubernetes:
  client:
    secrets:
      enabled: true
      includes: this-secret

Or:

kubernetes:
  client:
    secrets:
      enabled: true
      excludes: not-this-secret

Similarly to ConfigMaps, labels can also be used to match the desired secrets:

kubernetes:
  client:
    secrets:
      enabled: true
      labels:
        - app: my-app
        - env: prod

This also works for pod labels:

kubernetes:
  client:
    secrets:
      enabled: true
      pod-labels:
        - "app.kubernetes.io/instance"

Reading Secrets from mounted volumes

In the case of Secrets, reading them from the Kubernetes API requires additional permissions, as stated above. Therefore, you may want to read them from mounted volumes in the pod.

Given the following secret:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

It can be mounted as a volume in a pod or deployment definition:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

This will make Kubernetes to create 2 files:

  • /etc/foo/username.

  • /etc/foo/password.

Their content will be the decoded strings from the original base-64 encoded values.

While you could potentially use the java.io or java.nio APIs to read the contents yourself, this configuration module can convert them into a PropertySource so that you can consume the values much more easily. In order to do so, define the following configuration:

kubernetes:
  client:
    secrets:
      enabled: true
      paths:
        - /etc/foo

Each file in the directory will become the property key, and the file contents, the property value.

When kubernetes.client.secrets.paths is defined, the Kubernetes API will not be used to read any other secret. If you still want to read the remaining secrets from the API, set the following configuration:

kubernetes:
  client:
    secrets:
      enabled: true
      use-api: true
      excludes: mysecret  # Because it will be read as a mounted volume
      paths:
        - /etc/foo

In this scenario, if there are property keys defined in both type of secrets, the ones coming from mounted volumes will take precedence over the ones coming from the API.

6 Health Checks

Health Indicators

This configuration module provides a health check that probes communication with the Kubernetes API, and provides some information about the pod where the application is running from.

The service discovery client will also display all the services that were resolved from Kubernetes.

An example output of a /health request would be:

{
  "name": "micronaut-service",
  "status": "UP",
  "details": {
    "kubernetes": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "namespace": "default",
        "podName": "example-service-786cd45b78-bzfw5",
        "podPhase": "Running",
        "podIP": "10.1.3.124",
        "hostIP": "192.168.65.3",
        "containerStatuses": [
          {
            "name": "example-service",
            "image": "registry.hub.docker.com/alvarosanchez/example-service:latest",
            "ready": true
          }
        ]
      }
    },
    "compositeDiscoveryClient(kubernetes)": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "services": {
          "example-service": [
            "http://10.1.3.124:8081",
            "http://10.1.3.126:8081"
          ],
          "non-secure-service": [
            "http://10.1.3.127:1234"
          ],
          "kubernetes": [
            "https://kubernetes:443"
          ],
          "secure-service-port-name": [
            "https://10.1.3.127:1234"
          ],
          "example-client": [
            "http://10.1.3.125:8082"
          ],
          "secure-service-port-number": [
            "https://10.1.3.127:443"
          ],
          "secure-service-labels": [
            "https://10.1.3.127:1234"
          ]
        }
      }
    },
    "diskSpace": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "total": 109702647808,
        "free": 69758287872,
        "threshold": 10485760
      }
    }
  }
}

Health checks require the following dependency:

implementation("io.micronaut:micronaut-management")
<dependency>
    <groupId>io.micronaut</groupId>
    <artifactId>micronaut-management</artifactId>
</dependency>

Also note that in order to see the full details of the health checks you may need additional configuration. Check the documentation of the Health Endpoint for more information about how to configure it.

7 Kubernetes client

The micronaut-kubernetes-client module gives you the ability to use official Kubernetes Java SDK apis objects as a regular Micronaut beans.

The complete list of available beans is declared in the Apis#values annotation value.

First you need add a dependency on the micronaut-kubernetes-client module:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-client:3.0.1")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-client</artifactId>
    <version>3.0.1</version>
</dependency>

Then you can simply use Micronaut injection to get configured apis object from package io.kubernetes.client.openapi.apis :

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.apis.CoreV1Api;
import io.kubernetes.client.openapi.models.V1PodList;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1Api coreV1Api;

    public MyService(CoreV1Api coreV1Api) {
        this.coreV1Api = coreV1Api;
    }

    public void myMethod(String namespace) throws ApiException {
        V1PodList v1PodList = coreV1Api.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null, false);
    }
}

Authentication

The Kubernetes client source of authentication options is automatically detected by the ClientBuilder#standard(), specifically:

Creates a builder which is pre-configured in the following way

  • If $KUBECONFIG is defined, use that config file.

  • If $HOME/.kube/config can be found, use that.

  • If the in-cluster service account can be found, assume in cluster config.

  • Default to localhost:8080 as a last resort.

Also for specific cases you can update the authentication configuration options by using kubernetes.client properties listed below:

Name

Description

basePath

Kubernetes API base path. Example: https://localhost:8081

caPath

CA file path.

tokenPath

Token file path.

kubeConfigPath

Kube config file path.

verifySsl

Boolean if the api should verify ssl. Default: true

Reactive Support

In addition to the official Kubernetes Java SDK Async clients, this module provides clients that use RxJava or Reactor to allow reactive programming with Micronaut for each Api.

RxJava 2 Reactive Support

For RxJava 2 add the following dependency:

implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-rxjava2:3.0.1")
<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-client-rxjava2</artifactId>
    <version>3.0.1</version>
</dependency>

The module contains all official Kubernetes API beans in format <ApiClassName>RxClient, for example the CoreV1Api class is injected as CoreV1ApiRxClient.

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.rxjava2.CoreV1ApiRxClient;
import io.reactivex.Single;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1ApiRxClient coreV1ApiRxClient;

    public MyService(CoreV1ApiRxClient coreV1ApiRxClient) {
        this.coreV1ApiRxClient = coreV1ApiRxClient;
    }

    public void myMethod(String namespace) {
        Single<V1PodList> v1PodList = coreV1ApiRxClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
    }
}

Reactor Reactive Support

For Reactor add the following dependency:

dependency:io.micronaut.kubernetes:micronaut-kubernetes-client-1

The module contains all official Kubernetes API beans in format <ApiClassName>ReactiveClient, for example the CoreV1Api class is injected as CoreV1ApiReactiveClient.

import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.reactive.CoreV1ApiReactiveClient;
import reactor.core.publisher.Mono;
import jakarta.inject.Singleton;

@Singleton
public class MyService {

    private final CoreV1ApiReactiveClient coreV1ApiReactiveClient;

    public MyService(CoreV1ApiReactiveClient coreV1ApiReactiveClient) {
        this.coreV1ApiReactiveClient = coreV1ApiReactiveClient;
    }

    public void myMethod(String namespace) {
        Mono<V1PodList> v1PodList = coreV1ApiReactiveClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
    }
}

Advanced Configuration

For advanced configuration options of ApiClient that are not suitable to provide via application.yml, you can declare a BeanCreatedEventListener bean that listens for ApiClient bean creation, and apply any further customisation to OkHttpClient there:

@Singleton
public class ApiClientListener implements BeanCreatedEventListener<ApiClient> {

    @Override
    public ApiClient onCreated(BeanCreatedEvent<ApiClient> event) {
        ApiClient apiClient = event.getBean();
        OkHttpClient okHttpClient = apiClient.getHttpClient().newBuilder()
                .readTimeout(5345, TimeUnit.MILLISECONDS)
                .build();
        apiClient.setHttpClient(okHttpClient);
        return apiClient;
    }
}

8 Logging and debugging

If you need to debug the Micronaut Kubernetes module, you need to set the io.micronaut.kubernetes logger level to DEBUG:

<logger name="io.micronaut.kubernetes" level="DEBUG"/>

By configuring the logger level to TRACE, the module will produce detailed responses from the Kubernetes API.

<logger name="io.micronaut.kubernetes" level="TRACE"/>

Other package that might produce relevant logging is io.micronaut.discovery, which belongs to Micronaut Core.

In addition to that, another source of information is the Environment Endpoint, which outputs all the resolved PropertySources from ConfigMaps, and their corresponding properties.

9 Breaking Changes

This section documents breaking changes between Micronaut Kubernetes 2.x and Micronaut Kubernetes 3.x:

In-house Kubernetes client removed

The in-house Kubernetes client io.micronaut.kubernetes.client.v1.* was deprecated and removed. Instead use the new module micronaut-kubernetes-client or the reactive alternatives micronaut-client-reactor or micronaut-client-rxjava2 that extends the client API classes by the reactive support of respective reactive framework.

10 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-kubernetes