kubernetes:
client:
namespace: other-namespace
Table of Contents
Micronaut Kubernetes
Integration between Micronaut and Kubernetes
Version: 6.3.0-SNAPSHOT
1 Introduction
This project eases Kubernetes integration with Micronaut.
It adds support for the following features:
-
Service Discovery.
-
Configuration client for config maps and secrets.
-
Kubernetes blocking and non-blocking clients built on top of official Kubernetes Java SDK
To use the BUILD-SNAPSHOT
version of this library, check the
documentation to use snapshots.
Namespace configuration
When a Micronaut application with this module is running within a Pod in a Kubernetes cluster, it will
infer automatically the namespace it’s running from by reading it from the service account secret (which will be
provisioned at /var/run/secrets/kubernetes.io/serviceaccount/namespace
).
However, the namespace can still be overridden via configuration in bootstrap.yml
:
2 What's New?
The Micronaut Kubernetes module 3.0.0 includes the following changes:
Official K8s JAVA SDK client with reactive support
Micronaut Kubernetes is now using the official Kubernetes Java SDK client instead of the in-house client.
Apart this new module micronaut-kubernetes-client
there are two additional modules micronaut-client-reactor
and micronaut-client-rxjava2
that extends the client API classes by the reactive support of respective reactive framework.
3 Release History
For this project, you can find a list of releases (with release notes) here:
4 Service Discovery
The Service Discovery module allows Micronaut HTTP clients to discover Kubernetes services.
To get started, you need to declare the following dependency:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-discovery-client")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-discovery-client</artifactId>
</dependency>
By default in any client you can use as Service ID the Kubernetes Endpoints
name generated by a Kubernetes Service
for the configured namespace.
Consider the following Kubernetes service definition:
my-service.yml
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
This specification will create a new Service
object named my-service
, as well as an Endpoints
object also named my-service
.
In your HTTP client, you can use my-service
as Service ID: @Client("my-service")
.
Note that service discovery is enabled by default in Micronaut. To disable it, set kubernetes.client.discovery.enabled
to false
.
Service specific client configuration
Kubernetes Service is a complex resource that can handle various use cases by providing specific configuration. For this Micronaut Kubernetes supports a manual service discovery configuration per Service http client that allows you to configure custom:
Key | Description |
---|---|
|
name of the resource in Kubernetes in case it is different than the Service ID |
|
namespace of the resource in case it’s different than the configured namespace |
|
port name in case the target resource is a Multi-Port Service |
|
service specific discovery mode in case it’s different than the globally configured discovery mode |
Examples of service configurations
Multi-port service
For the following Multi-Port Service:
my-service.yml
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9376
- name: https
protocol: TCP
port: 443
targetPort: 9377
the manual service configuration for http port will be:
bootstrap.yml
kubernetes:
client:
discovery:
services:
my-service:
port: http
Headless service with selector
For the following Headless service with selector:
my-service.yml
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
clusterIP: None
selector:
app: MyApp
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9376
the manual service configuration will be:
bootstrap.yml
kubernetes:
client:
discovery:
services:
my-service:
mode: endpoint
ExternalName service type
For the following ExternalName service:
my-service.yml
apiVersion: v1
kind: Service
metadata:
name: my-service
namespace: prod
spec:
type: ExternalName
externalName: launch.micronaut.io
the manual service configuration will be:
bootstrap.yml
kubernetes:
client:
discovery:
services:
my-service:
mode: service
Service discovery modes
Service discovery mode is a mechanism that allows to support different strategies for the actual service discovery in Kubernetes by implementing KubernetesServiceInstanceProvider interface.
Currently Micronaut Kubernetes implements two discovery modes:
-
endpoint
mode uses the KubernetesEndpoins
API for the service discovery. Note that the service load balancing is handled by Microunat application. -
service
mode uses the KubernetesService
API for the service discovery. Theservice
mode extracts the serviceClusterIP
address from the Service status.
Both discovery modes are using the metadata.name
for the Service ID identificator.
The discovery mode can be configured globally for all Service IDs or per service.
Note that endpoint
is the default global discovery mode. That can be overridden via configuration in bootstrap.yml
:
bootstrap.yml
kubernetes:
client:
discovery:
mode: service
Watching for changes
Both discovery modes support watching for changes of their respective resources. To enable it, set kubernetes.client.discovery.mode-configuration.endpoint.watch.enabled
to true
for the endpoint
mode. For the service
mode set kubernetes.client.discovery.mode-configuration.service.watch.enabled
to true
.
Kubernetes API authentication
Micronaut authenticates to the Kubernetes API using the token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token
.
Note that by default, the service account used may only have permissions over the kube-system
namespace. The service discovery
functionality requires some additiona read permissions. Refer to the Kubernetes documentation for more information
about Role-based access control (RBAC).
One of the options is to create the following Role
and RoleBinding
(make sure to apply them to the service account used, if not default
):
auth.yml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: service-discoverer
namespace: micronaut-kubernetes
rules:
- apiGroups: [""]
resources: ["services", "endpoints", "configmaps", "secrets", "pods"]
verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: default-service-discoverer
namespace: micronaut-kubernetes
subjects:
- kind: ServiceAccount
name: default
namespace: micronaut-kubernetes
roleRef:
kind: Role
name: service-discoverer
apiGroup: rbac.authorization.k8s.io
In Google Cloud’s Kubernetes Engine, in order to create the above, you must grant your user the ability to create roles in Kubernetes by running the following command: |
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user yourGoogleAccount@gmail.com
Connecting to services using HTTPS
There are three ways for this library to determine whether a service should be connected to using SSL (the following
examples assume there is a Deployment
named secure-deployment
).
Using https
as port name
my-service.yml
apiVersion: v1
kind: Service
metadata:
name: secure-service-port-name
spec:
selector:
app: secure-deployment
type: NodePort
ports:
- port: 1234
protocol: TCP
name: https
Using a port ending in 443
Port numbers like 443, 8443, etc. will match.
my-service.yml
apiVersion: v1
kind: Service
metadata:
name: secure-service-port-number
spec:
selector:
app: secure-deployment
type: NodePort
ports:
- port: 443
protocol: TCP
Using labels
Set a label named secure
with value true
to have the client use HTTPS.
my-service.yml
apiVersion: v1
kind: Service
metadata:
name: secure-service-labels
labels:
secure: "true"
spec:
selector:
app: secure-deployment
type: NodePort
ports:
- port: 1234
protocol: TCP
Service filtering
You can filter the services discovered by using kubernetes.client.discovery.includes
or
kubernetes.client.discovery.excludes
:
kubernetes:
client:
discovery:
includes:
- my-service
- other-service
Or:
kubernetes:
client:
discovery:
excludes: not-this-service
In addition to that, Kubernetes labels can be used to better match the services that should be available for service discovery:
kubernetes:
client:
discovery:
labels:
- app: my-app
- env: prod
Note that the filtering is not applied on manually configured service configurations.
Service discovery guides
See the following guides to learn more about Kubernetes service discovery using the Micronaut Framework:
5 Configuration Client
The Configuration client will read Kubernetes' ConfigMap
s and Secret
s instances and make them available as PropertySource
s
instances in your application.
Then, in any bean you can read the configuration values from the ConfigMap
or Secret
using @Value
or
any other way to read configuration values.
Configuration parsing happens in the bootstrap phase. Therefore, to enable distributed configuration clients, define the
following in bootstrap.yml
(or .properties
, .json
, etc):
micronaut:
config-client:
enabled: true
ConfigMaps
Supported formats for ConfigMap
s are:
-
Java
.properties
. -
YAML.
-
JSON.
-
Literal values.
The configuration client by default will read all the ConfigMap
s for the configured namespace. You can further filter
which config map names are processed by defining kubernetes.client.config-maps.includes
or
kubernetes.client.config-maps.excludes
:
kubernetes:
client:
config-maps:
includes:
- my-config-map
- other-config-map
Or:
kubernetes:
client:
config-maps:
excludes: not-this-config-map
In addition to that, Kubernetes labels can be used to better match the config maps that should be available as property sources. This can be done by defining the label and value directly:
kubernetes:
client:
config-maps:
labels:
- app: my-app
- env: prod
Or by including every config map that has the same Kubernetes label as the pod the Micronaut application is running in.
This is handy if you use a package manager like Helm.
A good example would be app.kubernetes.io/instance
which is a unique label identifying the instance of an application:
kubernetes:
client:
config-maps:
pod-labels:
- "app.kubernetes.io/instance"
Additionally, you can also enable exception-on-pod-labels-missing
property in case you want an exception to be thrown if at least
one of the labels, specified in pod-labels
section is not found among the application’s pod labels. This can be useful if you want
to prevent loading all config maps available in namespace in case of mistyping or misconfiguration:
kubernetes:
client:
config-maps:
exception-on-pod-labels-missing: true
pod-labels:
- "app.kubernetes.io/instance"
Note that on the resulting config maps, you can still further filter them with includes/excludes properties.
Reading ConfigMap
s from mounted volumes
In the case of ConfigMaps
s, reading them from the Kubernetes API requires additional permissions, as stated above.
Therefore, you may want to read them from mounted volumes in the pod.
Given the following ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: mounted-config
data:
mounted.yml: |-
foo: bar
It can be mounted as a volume in a pod or deployment definition:
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: micronautapp
volumeMounts:
- name: configuration
mountPath: /etc/configuration
readOnly: true
volumes:
- name: configuration
configMap:
name: mounted-config
This will make Kubernetes to create file per data entry:
-
/etc/configuration/mounted.yml
While you could potentially use the java.io
or java.nio
APIs to read the contents yourself, this configuration module
can convert them into a PropertySource
so that you can consume the values much more easily. In order to do so, define
the following configuration:
kubernetes:
client:
config-maps:
paths:
- /etc/configuration
Each file in the directory will become a property source file. The file format will be automatically deduced based on the file suffix. Supported formats for mounted ConfigMap
files are:
-
Java
.properties
-
YAML
-
JSON
When
In this scenario, if there are property keys defined in both type of config maps, the ones coming from mounted volumes will take precedence over the ones coming from the API. |
Watching for changes in ConfigMaps
By default, this configuration module will watch for ConfigMap
s added/modified/deleted, and provided that the changes
match with the above filters, they will be propagated to the Environment
and refresh it.
This means that those changes will be immediately available in your application without a restart.
If you want to disable watching for ConfigMap changes, set kubernetes.client.config-maps.watch
to false
.
This should be done in the bootstrap.yml
configuration file because the configuration client is initialized during the bootstrap phase, which happens before evaluating the application.yml
configuration file.
When |
Examples
You can create a Kubernetes ConfigMap
off an existing file with the following command:
kubectl create configmap my-config --from-file=my-config.properties
Or:
kubectl create configmap my-config --from-file=my-config.yml
Or:
kubectl create configmap my-config --from-file=my-config.json
You can also create a ConfigMap
from literal values:
kubectl create configmap my-config --from-literal=special.how=very --from-literal=special.type=charm
Secrets
Secrets read from the Kubernetes API will be base64-decoded and made available as PropertySource
s, so that they can be
also read with @Value
, @ConfigurationProperties
, etc.
Only Opaque secrets will be considered.
|
By default, secrets access is diabled. To enable them, set in bootstrap.yml
:
kubernetes:
client:
secrets:
enabled: true
The configuration client, by default, will read all the Secret
s for the configured namespace. You can further filter
which config map names are processed by defining kubernetes.client.secrets.includes
or kubernetes.client.secrets.excludes
:
kubernetes:
client:
secrets:
enabled: true
includes: this-secret
Or:
kubernetes:
client:
secrets:
enabled: true
excludes: not-this-secret
Similarly to ConfigMap
s, labels can also be used to match the desired secrets:
kubernetes:
client:
secrets:
enabled: true
labels:
- app: my-app
- env: prod
This also works for pod labels:
kubernetes:
client:
secrets:
enabled: true
pod-labels:
- "app.kubernetes.io/instance"
As well as exception-on-pod-labels-missing
property:
kubernetes:
client:
secrets:
enabled: true
exception-on-pod-labels-missing: true
pod-labels:
- "app.kubernetes.io/instance"
Reading Secret
s from mounted volumes
In the case of Secret
s, reading them from the Kubernetes API requires additional permissions, as stated above.
Therefore, you may want to read them from mounted volumes in the pod.
Given the following secret:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: YWRtaW4=
password: MWYyZDFlMmU2N2Rm
It can be mounted as a volume in a pod or deployment definition:
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
This will make Kubernetes to create 2 files:
-
/etc/foo/username
. -
/etc/foo/password
.
Their content will be the decoded strings from the original base-64 encoded values.
While you could potentially use the java.io
or java.nio
APIs to read the contents yourself, this configuration module
can convert them into a PropertySource
so that you can consume the values much more easily. In order to do so, define
the following configuration:
kubernetes:
client:
secrets:
enabled: true
paths:
- /etc/foo
Each file in the directory will become the property key, and the file contents, the property value.
When
In this scenario, if there are property keys defined in both type of secrets, the ones coming from mounted volumes will take precedence over the ones coming from the API. |
Watching for changes in Secrets
If watch is enabled, this configuration module will watch for Secret
s added/modified/deleted, and provided that the
changes match with the above filters, they will be propagated to the Environment
and refresh it.
This means that those changes will be immediately available in your application without a restart.
If you want to enable watching for Secret changes, set kubernetes.client.secrets.watch
to true
.
This should be done in the bootstrap.yml
configuration file because the configuration client is initialized during the
bootstrap phase, which happens before evaluating the application.yml
configuration file.
When |
6 Health Checks
Health Indicators
This configuration module provides a KubernetesHealthIndicator that probes communication with the Kubernetes API, and provides some information about the pod where the application is running from.
The service discovery client will also display all the services that were resolved from Kubernetes.
An example output of a /health
request would be:
{
"name": "micronaut-service",
"status": "UP",
"details": {
"kubernetes": {
"name": "micronaut-service",
"status": "UP",
"details": {
"namespace": "default",
"podName": "example-service-786cd45b78-bzfw5",
"podPhase": "Running",
"podIP": "10.1.3.124",
"hostIP": "192.168.65.3",
"containerStatuses": [
{
"name": "example-service",
"image": "registry.hub.docker.com/alvarosanchez/example-service:latest",
"ready": true
}
]
}
},
"compositeDiscoveryClient(kubernetes)": {
"name": "micronaut-service",
"status": "UP",
"details": {
"services": {
"example-service": [
"http://10.1.3.124:8081",
"http://10.1.3.126:8081"
],
"non-secure-service": [
"http://10.1.3.127:1234"
],
"kubernetes": [
"https://kubernetes:443"
],
"secure-service-port-name": [
"https://10.1.3.127:1234"
],
"example-client": [
"http://10.1.3.125:8082"
],
"secure-service-port-number": [
"https://10.1.3.127:443"
],
"secure-service-labels": [
"https://10.1.3.127:1234"
]
}
}
},
"diskSpace": {
"name": "micronaut-service",
"status": "UP",
"details": {
"total": 109702647808,
"free": 69758287872,
"threshold": 10485760
}
}
}
}
Health checks require the following dependency:
Also note that in order to see the full details of the health checks you may need additional configuration. Check the documentation of the Health Endpoint for more information about how to configure it. |
By default the KubernetesHealthIndicator is enabled. To disable it use the following configuration:
endpoints:
health:
kubernetes:
enabled: false
7 Kubernetes Client
The micronaut-kubernetes-client
module gives you the ability to use official Kubernetes Java SDK apis objects as a regular Micronaut beans.
The complete list of available beans is declared in the Apis#values annotation value.
First you need add a dependency on the micronaut-kubernetes-client
module:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client</artifactId>
</dependency>
Then you can simply use Micronaut injection to get configured apis object from package io.kubernetes.client.openapi.apis
:
import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.apis.CoreV1Api;
import io.kubernetes.client.openapi.models.V1PodList;
import jakarta.inject.Singleton;
@Singleton
public class MyService {
private final CoreV1Api coreV1Api;
public MyService(CoreV1Api coreV1Api) {
this.coreV1Api = coreV1Api;
}
public void myMethod(String namespace) throws ApiException {
V1PodList v1PodList = coreV1Api.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null, false);
}
}
Authentication
The Kubernetes client source of authentication options is automatically detected by the ClientBuilder#standard(), specifically:
Creates a builder which is pre-configured in the following way
-
If
$KUBECONFIG
is defined, use that config file. -
If
$HOME/.kube/config
can be found, use that. -
If the in-cluster service account can be found, assume in cluster config.
-
Default to
localhost:8080
as a last resort.
Also for specific cases you can update the authentication configuration options by using kubernetes.client
properties listed below:
Name |
Description |
basePath |
Kubernetes API base path. Example: |
caPath |
CA file path. |
tokenPath |
Token file path. |
kubeConfigPath |
Kube config file path. |
verifySsl |
Boolean if the api should verify ssl. Default: |
Reactive Support
In addition to the official Kubernetes Java SDK Async
clients, this module provides clients that use RxJava or Reactor to allow reactive programming with Micronaut for each Api.
RxJava 2 Reactive Support
For RxJava 2 add the following dependency:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-rxjava2")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client-rxjava2</artifactId>
</dependency>
The module contains all official Kubernetes API beans in format <ApiClassName>RxClient
,
for example the CoreV1Api
class is injected as CoreV1ApiRxClient
.
import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.rxjava2.CoreV1ApiRxClient;
import io.reactivex.Single;
import jakarta.inject.Singleton;
@Singleton
public class MyService {
private final CoreV1ApiRxClient coreV1ApiRxClient;
public MyService(CoreV1ApiRxClient coreV1ApiRxClient) {
this.coreV1ApiRxClient = coreV1ApiRxClient;
}
public void myMethod(String namespace) {
Single<V1PodList> v1PodList = coreV1ApiRxClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
}
}
RxJava 3 Reactive Support
For RxJava 3 add the following dependency:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-rxjava3")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client-rxjava3</artifactId>
</dependency>
The module contains all official Kubernetes API beans in format <ApiClassName>RxClient
,
for example the CoreV1Api
class is injected as CoreV1ApiRxClient
.
import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.rxjava3.CoreV1ApiRxClient;
import io.reactivex.Single;
import jakarta.inject.Singleton;
@Singleton
public class MyService {
private final CoreV1ApiRxClient coreV1ApiRxClient;
public MyService(CoreV1ApiRxClient coreV1ApiRxClient) {
this.coreV1ApiRxClient = coreV1ApiRxClient;
}
public void myMethod(String namespace) {
Single<V1PodList> v1PodList = coreV1ApiRxClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
}
}
Reactor Reactive Support
For Reactor add the following dependency:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-reactor")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client-reactor</artifactId>
</dependency>
The module contains all official Kubernetes API beans in format <ApiClassName>ReactiveClient
,
for example the CoreV1Api
class is injected as CoreV1ApiReactiveClient
.
import io.kubernetes.client.openapi.ApiException;
import io.kubernetes.client.openapi.models.V1PodList;
import io.micronaut.kubernetes.client.reactive.CoreV1ApiReactiveClient;
import reactor.core.publisher.Mono;
import jakarta.inject.Singleton;
@Singleton
public class MyService {
private final CoreV1ApiReactiveClient coreV1ApiReactiveClient;
public MyService(CoreV1ApiReactiveClient coreV1ApiReactiveClient) {
this.coreV1ApiReactiveClient = coreV1ApiReactiveClient;
}
public void myMethod(String namespace) {
Mono<V1PodList> v1PodList = coreV1ApiReactiveClient.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null);
}
}
Advanced Configuration
For advanced configuration options of ApiClient
that are not suitable to provide via application.yml
, you can declare a BeanCreatedEventListener bean that listens for ApiClient
bean creation, and apply any further customisation to OkHttpClient
there:
@Singleton
public class ApiClientListener implements BeanCreatedEventListener<ApiClient> {
@Override
public ApiClient onCreated(BeanCreatedEvent<ApiClient> event) {
ApiClient apiClient = event.getBean();
OkHttpClient okHttpClient = apiClient.getHttpClient().newBuilder()
.readTimeout(5345, TimeUnit.MILLISECONDS)
.build();
apiClient.setHttpClient(okHttpClient);
return apiClient;
}
}
8 Kubernetes Client OpenAPI
The Micronaut Kubernetes Client OpenApi is a kubernetes client which uses Micronaut Netty HTTP Client and generated apis and modules from the OpenApi Spec of the official Java client library for Kubernetes.
Advantages of this client over the official Java client library for Kubernetes:
-
No extra dependencies needed (OkHttp, Bouncy Castle, Kotlin etc.)
-
Unified configuration with Micronaut HTTP client
-
Support for plugging in filters
-
Native Image compatibility
The client comes in two flavors:
-
micronaut-kubernetes-client-openapi
module implements a blocking client -
micronaut-kubernetes-client-openapi-reactor
module implements a reactive client using Project Reactor
The micronaut-kubernetes-client-openapi-common
module contains code that is shared between both above mentioned clients.
First you need to add a dependency on the micronaut-kubernetes-client-openapi
module:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-openapi")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client-openapi</artifactId>
</dependency>
Then you can simply use Micronaut injection to get configured apis object from package io.micronaut.kubernetes.client.openapi.api
:
package micronaut.client;
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;
import io.micronaut.kubernetes.client.openapi.api.CoreV1Api;
import io.micronaut.kubernetes.client.openapi.model.V1Pod;
import io.micronaut.kubernetes.client.openapi.model.V1PodList;
import io.micronaut.scheduling.TaskExecutors;
import io.micronaut.scheduling.annotation.ExecuteOn;
import jakarta.inject.Inject;
import jakarta.validation.constraints.NotNull;
import java.util.Map;
import java.util.stream.Collectors;
@Controller("/pods")
@ExecuteOn(TaskExecutors.BLOCKING)
public class PodController {
@Inject
CoreV1Api coreV1Api;
@Get("/{namespace}/{name}")
public String getPod(final @NotNull String namespace, final @NotNull String name) {
V1Pod v1Pod = coreV1Api.readNamespacedPod(name, namespace, null);
return v1Pod.getStatus().getPhase();
}
@Get("/{namespace}")
public Map<String, String> getPods(final @NotNull String namespace) {
V1PodList v1PodList = coreV1Api.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null, null, null);
return v1PodList.getItems().stream()
.filter(p -> p.getStatus() != null)
.collect(Collectors.toMap(
p -> p.getMetadata().getName(),
p -> p.getStatus().getPhase()));
}
}
First you need to add a dependency on the micronaut-kubernetes-client-openapi-reactor
module:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-openapi-reactor")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client-openapi-reactor</artifactId>
</dependency>
Then you can simply use Micronaut injection to get configured apis object from package io.micronaut.kubernetes.client.openapi.api.reactor
:
package micronaut.client;
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;
import io.micronaut.kubernetes.client.openapi.reactor.api.CoreV1ApiReactor;
import jakarta.inject.Inject;
import jakarta.validation.constraints.NotNull;
import reactor.core.publisher.Mono;
import java.util.Map;
import java.util.stream.Collectors;
@Controller("/pods")
public class PodController {
@Inject
CoreV1ApiReactor coreV1ApiReactor;
@Get("/{namespace}/{name}")
public Mono<String> getPod(final @NotNull String namespace, final @NotNull String name) {
return coreV1ApiReactor.readNamespacedPod(name, namespace, null)
.map(it -> it.getStatus().getPhase());
}
@Get("/{namespace}")
public Mono<Map<String, String>> getPods(final @NotNull String namespace) {
return coreV1ApiReactor.listNamespacedPod(namespace, null, null, null, null, null, null, null, null, null, null, null)
.map(it -> it.getItems().stream()
.filter(p -> p.getStatus() != null)
.collect(Collectors.toMap(
p -> p.getMetadata().getName(),
p -> p.getStatus().getPhase())));
}
}
kubernetes.client.enabled=true
kubernetes.client.kube-config-path=file:/path/to/kubeconfig
kubernetes.client.service-account.enabled=true
kubernetes.client.service-account.certificate-authority-path=file:/path/to/certificate/authority/file
kubernetes.client.service-account.token-path=file:/path/to/token/file
kubernetes.client.service-account.token-reload-interval=2m
kubernetes:
client:
enabled: true
kube-config-path: file:/path/to/kubeconfig
service-account:
enabled: true
certificate-authority-path: file:/path/to/certificate/authority/file
token-path: file:/path/to/token/file
token-reload-interval: 2m
[kubernetes]
[kubernetes.client]
enabled=true
kube-config-path="file:/path/to/kubeconfig"
[kubernetes.client.service-account]
enabled=true
certificate-authority-path="file:/path/to/certificate/authority/file"
token-path="file:/path/to/token/file"
token-reload-interval="2m"
kubernetes {
client {
enabled = true
kubeConfigPath = "file:/path/to/kubeconfig"
serviceAccount {
enabled = true
certificateAuthorityPath = "file:/path/to/certificate/authority/file"
tokenPath = "file:/path/to/token/file"
tokenReloadInterval = "2m"
}
}
}
{
kubernetes {
client {
enabled = true
kube-config-path = "file:/path/to/kubeconfig"
service-account {
enabled = true
certificate-authority-path = "file:/path/to/certificate/authority/file"
token-path = "file:/path/to/token/file"
token-reload-interval = "2m"
}
}
}
}
{
"kubernetes": {
"client": {
"enabled": true,
"kube-config-path": "file:/path/to/kubeconfig",
"service-account": {
"enabled": true,
"certificate-authority-path": "file:/path/to/certificate/authority/file",
"token-path": "file:/path/to/token/file",
"token-reload-interval": "2m"
}
}
}
}
Name |
Description |
enabled |
Whether to enable the client. Default: |
kube-config-path |
Absolute path to the kube config file. Default: |
service-account.enabled |
Whether to enable the service account authentication. Default: |
service-account.certificate-authority-path |
Absolute path to the certificate authority file. Default: |
service-account.token-path |
Absolute path to the token file. Default: |
service-account.token-reload-interval |
Token reload interval. Default: |
Authentication
The client supports the following authentication strategies:
-
client certificate authentication (certificate and private key provided in the kube config file)
-
basic authentication (username and password provided in the kube config file)
-
token authentication (token provided in the kube config file or by executing the command from the kube config file)
-
service account authentication (used only if the kube config file not provided and running inside the kubernetes cluster)
Customization
There are several interfaces that can be implemented to change default implementations:
-
KubeConfigLoader - a custom implementation can be used when a kube config file needs to be loaded from a cloud service. There is also an option of extending AbstractKubeConfigLoader which caches the loaded kube config data and provides a few helper methods.
-
KubernetesTokenLoader - a custom implementation can be used for loading a bearer token. KubernetesHttpClientFilter iterates through a list of implementations of this interface and creates the Authorization header using the token from the first implementation which returns it. The following implementations are currently used:
-
ExecCommandCredentialLoader - implementation which executes the command from the kube config file to get the token.
-
KubeConfigTokenLoader - implementation which uses the token from the kube config file.
-
ServiceAccountTokenLoader - implementation which uses the service account token.
-
-
KubernetesPrivateKeyLoader - a custom implementation can be used if there is a need for usage of third party libraries (for example, Bouncy Castle) to load the private key when the client certificate authentication is used.
9 Kubernetes Client OpenAPI Watcher
The Micronaut Kubernetes Client OpenApi Watcher implements streaming of Kubernetes events using the Micronaut Kubernetes Client OpenApi client.
First you need to add a dependency on the micronaut-kubernetes-client-openapi-watcher
module:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-client-openapi-watcher")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-client-openapi-watcher</artifactId>
</dependency>
Then you can simply use Micronaut injection to get configured apis object from package io.micronaut.kubernetes.client.openapi.watcher.api
:
package micronaut.client;
import io.micronaut.http.annotation.Controller;
import io.micronaut.http.annotation.Get;
import io.micronaut.kubernetes.client.openapi.watcher.api.CoreV1ApiWatcher;
import jakarta.inject.Inject;
import jakarta.validation.constraints.NotNull;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@Controller("/secrets")
public class SecretController {
private static final Logger LOG = LoggerFactory.getLogger(SecretController.class);
@Inject
CoreV1ApiWatcher coreV1ApiWatcher;
@Get("/{namespace}")
public void startWatchingSecrets(final @NotNull String namespace) {
coreV1ApiWatcher.listNamespacedSecret(namespace, null, null, null, null, null, null, null, null, null, null, true)
.subscribe(event -> LOG.info(event.toString()));
}
}
The streaming will work only if the watch parameter is set to true .
|
The Kubernetes streamed events are deserialized into the WatchEvent class:
@Serdeable
public record WatchEvent<T>(
@NonNull String type,
@Nullable T object,
@Nullable V1Status status
) {
}
Name |
Description |
type |
The type of the streamed event. At the moment the type can be one of the following values: ADDED, MODIFIED, DELETED, ERROR, BOOKMARK |
object |
The data for the watched object (V1Namespace, V1Pod etc.). The value will be empty only if the event type is |
status |
The instance of V1Status which won’t be empty only if the event type is |
10 Kubernetes Informer
The micronaut-kubernetes-informer
module integrates the SharedIndexInformer that is part of official Kubernetes SDK and simplifies its creation. The Informer is similar to a Watch but an Informer tries to re-list and re-watch to hide failures from the caller and provides a store of all the current resources. Note that the default official implementation SharedInformerFactory creates shared informers per the Kubernetes resource type. However the module micronaut-kubernetes-informer
creates namespace scoped informers of the Kubernetes resource type, meaning that the informer is shared per specified namespace and kind.
First you need add a dependency on the micronaut-kubernetes-informer
module:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-informer")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-informer</artifactId>
</dependency>
Then create a bean that implements ResourceEventHandler with the Kubernetes resource type of your choice and add the @Informer annotation trough which you provide the configuration for the SharedIndexInformer.
The example below illustrates the declaration of the @Informer with the ResourceEventHandler for handling the changes of the Kubernetes V1ConfigMap
resource.
@Informer(apiType = V1ConfigMap.class, apiListType = V1ConfigMapList.class) // (1)
public class ConfigMapInformer implements ResourceEventHandler<V1ConfigMap> { // (2)
@Override
public void onAdd(V1ConfigMap obj) {
}
@Override
public void onUpdate(V1ConfigMap oldObj, V1ConfigMap newObj) {
}
@Override
public void onDelete(V1ConfigMap obj, boolean deletedFinalStateUnknown) {
}
}
1 | The @Informer annotation defines the Kubernetes resource type and resource list type |
2 | The ResourceEventHandler interface declares method handlers for added, updated and deleted resource |
To create an Informer for non-namespaced resource like V1ClusterRole
, configure the @Informer
the same way like it is done for namespaced resource.
The ResourceEventHandlerConstructorInterceptor logic takes care of automated evaluation of the resource apiGroup
and resourcePlural
by fetching the API resource details from the Kubernetes API by using Discovery. The API resource discovery can be disabled by: kubernetes.client.api-discovery.enabled: false
. In the case the discovery is disabled the resourcePlural
and apiGroup
needs to be provided manually in the @Informer
annotation.
The @Informer annotation provides several configuration options:
Element |
Description |
|
The resource api type that must extend from the |
|
The resource api list type. For example |
|
The resource api group. For example some of the Kubernetes core resources has no group like |
|
The resource plural that identifies the Api. For example for the resource |
|
Namespace of the watched resource. If left empty then namespace is resolved by NamespaceResolver. To watch resources from all namespaces configure this parameter to |
|
List of the namespaces of the watcher resource. The SharedIndexInformer will be created for every namespace and all events will be delivered to the specified |
|
|
|
Informer label selector, see Label selectors for more information. By default there is no label selector. |
|
|
|
How often to check the need for resync of resources. If left empty the default resync check period is used. |
The concept of shared informer means that the SharedIndexInformer for the respective Kubernetes resource type is registered just once for the given namespace. The next request to register another informer of the same Kubernetes resource type within the same namespace will result in returning of the previously created informer. In practice if you create two ResourceEventHandler<V1ConfigMap> but the @Informer annotation will have different optional configuration for labelSelector then the SharedInformerFactory creates just one SharedInformer , meaning the other @Informer configuration will be ignored. If the labelSelector resp. labelSelectorSupplier differs then create one labelSelector that matches both cases.
|
Programmatic creation of SharedIndexInformer
Use the bean SharedIndexInformerFactory to create the SharedIndexInformer programmatically:
SharedIndexInformer<V1ConfigMap> sharedIndexInformer = factory.sharedIndexInformerFor(
V1ConfigMap.class, // (1)
V1ConfigMapList.class, // (2)
"configmaps", // (3)
"", // (4)
"default", // (5)
null,
null,
true
);
1 | The resource api type that must extend from the KubernetesObject . For Kubernetes core resources the types can be found in io.kubernetes.client.openapi.models package. For example io.kubernetes.client.openapi.models.V1ConfigMap . |
2 | The resource api list type that must extend from the KubernetesListObject . For example io.kubernetes.client.openapi.models.V1ConfigMapList . |
3 | The resource plural that identifies the Api. For example for the resource V1ConfigMap it is configmaps . |
4 | The resource api group. |
5 | The namespace to watch. |
By using the SharedIndexInformerFactory bean you can also get existing informer:
SharedIndexInformer<V1ConfigMap> sharedIndexInformer = factory.getExistingSharedIndexInformer(
"default", // (1)
V1ConfigMap.class); // (2)
1 | The informer namespace. |
2 | The informer resource type. |
SharedIndexInformer local cache
The SharedIndexInformer has internal cache that is eventually consistent with the authoritative state. The local cache starts out empty, and gets populated and updated. For the detailed description of SharedIndexInformer internals visit the reference implementation pkg.go.dev/k8s.io/client-go#SharedIndexInformer in Go language.
The cache is exposed by SharedIndexInformer#getIndexer() method:
@Singleton
public class SharedInformerCache {
private final SharedIndexInformerFactory sharedIndexInformerFactory;
public SharedInformerCache(SharedIndexInformerFactory sharedIndexInformerFactory) {
this.sharedIndexInformerFactory = sharedIndexInformerFactory;
}
/**
* Get all config maps from informer from namespace.
*/
List<V1ConfigMap> getConfigMaps(String namespace) {
SharedIndexInformer<V1ConfigMap> sharedIndexInformer = sharedIndexInformerFactory.getExistingSharedIndexInformer(namespace, V1ConfigMap.class);
if (sharedIndexInformer != null) {
Indexer<V1ConfigMap> indexer = sharedIndexInformer.getIndexer();
return indexer.list();
} else {
return null;
}
}
}
11 Kubernetes Operator
The micronaut-kubernetes-operator
module integrates the official Kubernetes Java SDK controller support that is part of the extended client module. The micronaut-kubernetes-operator
module is build on top of the micronaut-kubernetes-informer
module and allows you to easily create the reconciler for both native and custom resources.
First you need add a dependency on the micronaut-kubernetes-operator
module:
implementation("io.micronaut.kubernetes:micronaut-kubernetes-operator")
<dependency>
<groupId>io.micronaut.kubernetes</groupId>
<artifactId>micronaut-kubernetes-operator</artifactId>
</dependency>
Then create a bean that implements ResourceReconciler with the Kubernetes resource type of your choice. Then add the @Operator annotation trough the which you provide the configuration for the controllers that will be created by Micronaut for your reconciler.
The example below illustrates the use of the @Operator with the ResourceReconciler that reconciles the Kubernetes V1ConfigMap
resource.
import io.kubernetes.client.extended.controller.reconciler.Request;
import io.kubernetes.client.extended.controller.reconciler.Result;
import io.kubernetes.client.openapi.models.V1ConfigMap;
import io.kubernetes.client.openapi.models.V1ConfigMapList;
import io.micronaut.core.annotation.NonNull;
import io.micronaut.kubernetes.client.informer.Informer;
import java.util.Optional;
@Operator(informer = @Informer(apiType = V1ConfigMap.class, apiListType = V1ConfigMapList.class)) // (1)
public class ConfigMapResourceReconciler implements ResourceReconciler<V1ConfigMap> { // (2)
@Override
@NonNull
public Result reconcile(@NonNull Request request, @NonNull OperatorResourceLister<V1ConfigMap> lister) { // (3)
Optional<V1ConfigMap> resource = lister.get(request); // (4)
// .. reconcile (5)
return new Result(false); // (6)
}
}
1 | The @Operator annotation defines the resource type which is the subject of reconciliation. The definition is done by using the @Informer annotation. Both annotations provide other object specific configuration options. |
2 | The ResourceReconciler interface declares the reconcile method that is main point of interaction in between the Micronaut and your application logic. |
3 | The reconciliation input consists from the Request that uniquely identifies the reconciliation resource and the OperatorResourceLister trough which the actual subject of reconciliation can be retrieved. |
4 | Retrieval of the resource for the reconciliation. |
5 | This is where the idempotent reconciliation logic should be placed. Note that the reconciliation logic is responsible for the update of the resource status stanza. |
6 | The return value of reconciliation method is the Result. |
The @Operator annotation provides several configuration options:
Element |
Description |
|
The name of the operator controller thread. Defaults to the |
|
The @Informer annotation used for the configuration of the Operator’s informer. This value is required. |
|
The |
|
The |
|
The |
Leader election
The LeaderElectingController is responsible for the leader election of the application replica that will reconcile the resources. Generally if the lock is not renewed within the specified amount of time, other replicas may try to acquire the lock and become the leader.
You can adjust the leader elector configuration by using Micronaut configuration properties kubernetes.client.operator.leader-election.lock
:
Element |
Description |
|
The lock lease duration. Defaults to |
|
The lock renew deadline. If the LeaderElector fails to renew the lock within the deadline then the controller looses the lock. Defaults to |
|
The lock acquire retry period. Defaults to |
For example:
kubernetes:
client:
operator:
leader-election:
lock:
lease-duration: 60s
renew-deadline: 50s
retry-period: 20s
Additionally, when the lock is acquired the LeaseAcquiredEvent is emitted. Similarly on a lost lease the LeaseLostEvent event is emitted.
Lock identity
The lock identity is used to uniquely identify the application that holds the lock and thus is responsible for reconciling the resources. By default, the POD name the application runs within is the source for the lock identity. This means the application must run in the Kubernetes cluster.
To create custom lock identity, create a bean that implements the LockIdentityProvider interface:
Lock resource types
The LeaderElectingController uses the native Kubernetes resource to store the lock information. Currently supported resources are V1ConfigMap
, V1Endpoints
and V1Lease
.
By default, the micronaut-kubernetes-operator
module uses the V1Lease
. This can be changed to V1Endpoints
by configuring the property kubernetes.client.operator.leader-election.lock.resource-kind
to endpoints
, resp. to configmaps
in case the V1ConfigMap
resource is requested.
Note that the resource for the lock is created in the application namespace. Then the application name is used as the lock resource name. This can be changed by using Micronaut configuration properties:
kubernetes:
client:
operator:
leader-election:
lock:
resource-name: custom-name
resource-namespace: custom-namespace
Note that in case the RBAC authorization is enabled in your Kubernetes cluster, your application needs to have properly configured role with respect to the lock resource type.
V1Lease
lock resource type:---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: operator-lease-role
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- create
- update
---
Resource filtering
The @Operator annotation allows you to configure the resource filters that are executed before the resource is added into the reconciler work queue. You can configure three types of filters that are distinguished by the resource lifecycle: onAddFilter
, onUpdateFilter
and onDeleteFilter
.
The onAddFilter
predicate processes the newly created resources. Create a bean that implements java.util.function.Predicate
with the same Kubernetes resource type like the operator is. Example below illustrates such filter for the V1ConfigMap
resource:
import io.kubernetes.client.openapi.models.V1ConfigMap;
import jakarta.inject.Singleton;
import java.util.function.Predicate;
@Singleton
public class OnAddFilter implements Predicate<V1ConfigMap> {
@Override
public boolean test(V1ConfigMap v1ConfigMap) {
if (v1ConfigMap.getMetadata().getAnnotations() != null) {
return v1ConfigMap.getMetadata().getAnnotations().containsKey("io.micronaut.operator");
}
return false;
}
}
The onUpdateFilter
bi-predicate processes modified resources. Create a bean that implements java.util.function.BiPredicate
with the same Kubernetes resource type like the operator is. Example below illustrates such filter for the V1ConfigMap
resource:
import io.kubernetes.client.openapi.models.V1ConfigMap;
import jakarta.inject.Singleton;
import java.util.function.BiPredicate;
@Singleton
public class OnUpdateFilter implements BiPredicate<V1ConfigMap, V1ConfigMap> {
@Override
public boolean test(V1ConfigMap oldObj, V1ConfigMap newObj) {
if (newObj.getMetadata().getAnnotations() != null) {
return newObj.getMetadata().getAnnotations().containsKey("io.micronaut.operator");
}
return false;
}
}
The onDeleteFilter
bi-predicate processes deleted resources. Create a bean that implements java.util.function.BiPredicate
with the same Kubernetes resource type like the operator is and the Boolean
as second type. Example below illustrates such filter for the V1ConfigMap
resource:
import io.kubernetes.client.openapi.models.V1ConfigMap;
import jakarta.inject.Singleton;
import java.util.function.BiPredicate;
@Singleton
public class OnDeleteFilter implements BiPredicate<V1ConfigMap, Boolean> {
@Override
public boolean test(V1ConfigMap v1ConfigMap, Boolean deletedFinalStateUnknown) {
if (v1ConfigMap.getMetadata().getAnnotations() != null) {
return v1ConfigMap.getMetadata().getAnnotations().containsKey("io.micronaut.operator");
}
return false;
}
}
Note that in case of onDeleteFilter the predicate receives the resource for the test method, but when the ResouceReconciler’s reconcile method is executed the lister will return Optional.empty since the resource was already deleted. To properly reconcile the resource on it’s removal, use finalizers.
|
Example below illustrates the configuration of the filters:
import io.kubernetes.client.extended.controller.reconciler.Request;
import io.kubernetes.client.extended.controller.reconciler.Result;
import io.kubernetes.client.openapi.models.V1ConfigMap;
import io.kubernetes.client.openapi.models.V1ConfigMapList;
import io.micronaut.core.annotation.NonNull;
import io.micronaut.kubernetes.client.informer.Informer;
@Operator(informer = @Informer(apiType = V1ConfigMap.class, apiListType = V1ConfigMapList.class),
onAddFilter = OnAddFilter.class, // (1)
onUpdateFilter = OnUpdateFilter.class, // (2)
onDeleteFilter = OnDeleteFilter.class) // (3)
public class ConfigMapResourceReconcilerWithFilters implements ResourceReconciler<V1ConfigMap> {
@Override
@NonNull
public Result reconcile(@NonNull Request request, @NonNull OperatorResourceLister<V1ConfigMap> lister) {
// .. reconcile
return new Result(false);
}
}
1 | Configuration of onAddFilter . |
2 | Configuration of onUpdateFilter . |
3 | Configuration of onAddFilter . |
12 Logging and debugging
If you need to debug the Micronaut Kubernetes module, you need to set the io.micronaut.kubernetes
logger level
to DEBUG
:
<logger name="io.micronaut.kubernetes" level="DEBUG"/>
By configuring the logger level to TRACE
, the module will produce detailed responses from the Kubernetes API.
<logger name="io.micronaut.kubernetes" level="TRACE"/>
Other package that might produce relevant logging is io.micronaut.discovery
, which belongs to Micronaut Core.
In addition to that, another source of information is
the Environment Endpoint, which outputs all
the resolved PropertySource
s from ConfigMap
s, and their corresponding properties.
13 Breaking Changes
Micronaut Kubernetes 4.x
Micronaut Kubernetes 4.0.0 updates to Kubernetes Client v18 (major). This major upgrade of the Kubernetes Client addresses several security CVEs.
Micronaut Kubernetes 3.x
This section documents breaking changes between Micronaut Kubernetes 2.x and Micronaut Kubernetes 3.x:
In-house Kubernetes client removed
The in-house Kubernetes client io.micronaut.kubernetes.client.v1.*
was deprecated and removed. Instead use the new module micronaut-kubernetes-client
or the reactive alternatives micronaut-client-reactor
or micronaut-client-rxjava2
that extends the client API classes by the reactive support of respective reactive framework.
14 Guides
See the following list of guides to learn more about working with Kubernetes in the Micronaut Framework:
15 Repository
You can find the source code of this project in this repository: