5Using the low-level Kubernetes API HTTP client

6Logging and debugging

Micronaut Kubernetes

Integration between Micronaut and Kubernetes

Version: 1.0.0

1 Introduction

This project eases Kubernetes integration with Micronaut.

It adds support for the following features:

Service Discovery.
Configuration client for config maps and secrets.

To get started, you need to declare the following dependency:

compile 'io.micronaut.kubernetes:micronaut-kubernetes-discovery-client:1.0.0'

<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-discovery-client</artifactId>
    <version>1.0.0</version>
</dependency>

Note that this configuration module requires at least Micronaut 1.2.

To use the BUILD-SNAPSHOT version of this library, check the documentation to use snapshots.

The core library for all features can be configured using the following configuration properties:

🔗

Table 1. Configuration Properties for KubernetesConfiguration
Property	Type	Description
`kubernetes.client.host`	java.lang.String	The Kubernetes API host name
`kubernetes.client.port`	int	The port for the Kubernetes API
`kubernetes.client.secure`	boolean	Set if the Kubernetes API server is exposed over HTTPS
`kubernetes.client.namespace`	java.lang.String	Sets the namespace.
`kubernetes.client.ssl-configuration`	SslConfiguration
`kubernetes.client.logger-name`	java.lang.String
`kubernetes.client.follow-redirects`	boolean
`kubernetes.client.default-charset`	java.nio.charset.Charset
`kubernetes.client.channel-options`	java.util.Map
`kubernetes.client.shutdown-timeout`	java.time.Duration
`kubernetes.client.read-timeout`	java.time.Duration
`kubernetes.client.read-idle-timeout`	java.time.Duration
`kubernetes.client.connect-timeout`	java.time.Duration
`kubernetes.client.num-of-threads`	java.lang.Integer
`kubernetes.client.thread-factory`	java.lang.Class
`kubernetes.client.max-content-length`	int
`kubernetes.client.proxy-type`	java.net.Proxy$Type
`kubernetes.client.proxy-address`	java.net.SocketAddress
`kubernetes.client.proxy-username`	java.lang.String
`kubernetes.client.proxy-password`	java.lang.String

Namespace configuration

When a Micronaut application with this configuration module is running within a Pod in a Kubernetes cluster, it will infer automatically the namespace it’s running from by reading it from the service account secret (which will be provisioned at /var/run/secrets/kubernetes.io/serviceaccount/namespace).

However, the namespace can still be overridden via configuration in bootstrap.yml:

kubernetes:
  client:
    namespace: other-namespace

2 Service Discovery

The Service Discovery module allows Micronaut HTTP clients to discover Kubernetes services.

In any client you can use as Service ID the Kubernetes Endpoints name generated by a Kubernetes Service.

Consider the following Kubernetes service definition:

my-service.yml

kind: Service
apiVersion: v1
metadata:
  name: my-service
spec:
  selector:
    app: MyApp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9376

This specification will create a new Service object named my-service, as well as an Endpoints object also named my-service.

In your HTTP client, you can use my-service as Service ID: @Client("my-service").

Note that service discovery is enabled by default in Micronaut. To disable it, set kubernetes.client.discovery.enabled to false.

Kubernetes API authentication

Micronaut connects to the Kubernetes API using the default service account, which by default only has permissions over the kube-system namespace. The service account used needs some read permissions. Refer to the Kubernetes documentation for more information about Role-based access control (RBAC).

One of the options is to create the following Role and RoleBinding:

auth.yml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: service-discoverer
  namespace: micronaut-kubernetes
rules:
  - apiGroups: [""]
    resources: ["services", "endpoints", "configmaps", "secrets", "pods"]
    verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: default-service-discoverer
  namespace: micronaut-kubernetes
subjects:
  - kind: ServiceAccount
    name: default
    namespace: micronaut-kubernetes
roleRef:
  kind: Role
  name: service-discoverer
  apiGroup: rbac.authorization.k8s.io

In Google Cloud’s Kubernetes Engine, in order to create the above, you must grant your user the ability to create roles in Kubernetes by running the following command:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user yourGoogleAccount@gmail.com

Connecting to services using HTTPS

There are three ways for this library to determine whether a service should be connected to using SSL (the following examples assume there is a Deployment named secure-deployment).

Using `https` as port name

my-service.yml

apiVersion: v1
kind: Service
metadata:
  name: secure-service-port-name
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 1234
      protocol: TCP
      name: https

Using a port ending in 443

Port numbers like 443, 8443, etc. will match.

my-service.yml

apiVersion: v1
kind: Service
metadata:
  name: secure-service-port-number
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 443
      protocol: TCP

Using labels

Set a label named secure with value true to have the client use HTTPS.

my-service.yml

apiVersion: v1
kind: Service
metadata:
  name: secure-service-labels
  labels:
    secure: "true"
spec:
  selector:
    app: secure-deployment
  type: NodePort
  ports:
    - port: 1234
      protocol: TCP

3 Configuration Client

The Configuration client will read Kubernetes' ConfigMaps and Secrets instances and make them available as PropertySources instances in your application.

Then, in any bean you can read the configuration values from the ConfigMap or Secret using @Value or any other way to read configuration values.

Configuration parsing happens in the bootstrap phase. Therefore, to enable distributed configuration clients, define the following in bootstrap.yml (or .properties, .json, etc):

micronaut:
  config-client:
    enabled: true

ConfigMaps

Supported formats for ConfigMaps are:

Java .properties.
YAML.
JSON.
Literal values.

The configuration client by default will read all the ConfigMaps for the configured namespace. You can further filter which config map names are processed by defining kubernetes.client.config-maps.includes or kubernetes.client.config-maps.excludes:

kubernetes:
  client:
    config-maps:
      includes:
        - my-config-map
        - other-config-map

Or:

kubernetes:
  client:
    config-maps:
      excludes: not-this-config-map

In addition to that, Kubernetes labels can be used to better match the config maps that should be available as property sources:

kubernetes:
  client:
    config-maps:
      labels:
        - app: my-app
        - env: prod

Note that on the resulting config maps, you can still further filter them with includes/excludes properties.

Watching for changes in ConfigMaps

By default, this configuration module will watch for ConfigMaps added/modified/deleted, and provided that the changes match with the above filters, they will be propagated to the Environment and refresh it.

This means that those changes will be immediately available in your application without a restart.

Examples

You can create a Kubernetes ConfigMap off an existing file with the following command:

kubectl create configmap my-config --from-file=my-config.properties

Or:

kubectl create configmap my-config --from-file=my-config.yml

Or:

kubectl create configmap my-config --from-file=my-config.json

You can also create a ConfigMap from literal values:

kubectl create configmap my-config --from-literal=special.how=very --from-literal=special.type=charm

Secrets

Secrets read from the Kubernetes API will be base64-decoded and made available as PropertySource s, so that they can be also read with @Value, @ConfigurationProperties, etc.

Only Opaque secrets will be considered.

By default, secrets access is diabled. To enable them, set in bootstrap.yml:

kubernetes:
  client:
    secrets:
      enabled: true

The configuration client, by default, will read all the Secrets for the configured namespace. You can further filter which config map names are processed by defining kubernetes.client.secrets.includes or kubernetes.client.secrets.excludes:

kubernetes:
  client:
    secrets:
      enabled: true
      includes: this-secret

Or:

kubernetes:
  client:
    secrets:
      enabled: true
      excludes: not-this-secret

Similarly to ConfigMaps, labels can also be used to match the desired secrets:

kubernetes:
  client:
    secrets:
      enabled: true
      labels:
        - app: my-app
        - env: prod

Reading `ConfigMap`s and `Secret`s from mounted volumes

In the case of Secrets, reading them from the Kubernetes API requires additional permissions, as stated above. Therefore, you may want to read them from mounted volumes in the pod.

Given the following secret:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

It can be mounted as a volume in a pod or deployment definition:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
  - name: mypod
    image: redis
    volumeMounts:
    - name: foo
      mountPath: "/etc/foo"
      readOnly: true
  volumes:
  - name: foo
    secret:
      secretName: mysecret

This will make Kubernetes to create 2 files:

/etc/foo/username.
/etc/foo/password.

Their content will be the decoded strings from the original base-64 encoded values.

While you could potentially use the java.io or java.nio APIs to read the contents yourself, this configuration module can convert them into a PropertySource so that you can consume the values much more easily. In order to do so, define the following configuration:

kubernetes:
  client:
    secrets:
      enabled: true
      paths:
        - /etc/foo

Each file in the directory will become the property key, and the file contents, the property value.

When kubernetes.client.secrets.paths is defined, the Kubernetes API will not be used to read any other secret. If you still want to read the remaining secrets from the API, set the following configuration:

kubernetes:
  client:
    secrets:
      enabled: true
      use-api: true
      excludes: mysecret  # Because it will be read as a mounted volume
      paths:
        - /etc/foo

In this scenario, if there are property keys defined in both type of secrets, the ones coming from mounted volumes will take precedence over the ones coming from the API.

4 Health Checks

Health Indicators

This configuration module provides a health check that probes communication with the Kubernetes API, and provides some information about the pod where the application is running from.

The service discovery client will also display all the services that were resolved from Kubernetes.

An example output of a /health request would be:

{
  "name": "micronaut-service",
  "status": "UP",
  "details": {
    "kubernetes": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "namespace": "default",
        "podName": "example-service-786cd45b78-bzfw5",
        "podPhase": "Running",
        "podIP": "10.1.3.124",
        "hostIP": "192.168.65.3",
        "containerStatuses": [
          {
            "name": "example-service",
            "image": "registry.hub.docker.com/alvarosanchez/example-service:latest",
            "ready": true
          }
        ]
      }
    },
    "compositeDiscoveryClient(kubernetes)": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "services": {
          "example-service": [
            "http://10.1.3.124:8081",
            "http://10.1.3.126:8081"
          ],
          "non-secure-service": [
            "http://10.1.3.127:1234"
          ],
          "kubernetes": [
            "https://kubernetes:443"
          ],
          "secure-service-port-name": [
            "https://10.1.3.127:1234"
          ],
          "example-client": [
            "http://10.1.3.125:8082"
          ],
          "secure-service-port-number": [
            "https://10.1.3.127:443"
          ],
          "secure-service-labels": [
            "https://10.1.3.127:1234"
          ]
        }
      }
    },
    "diskSpace": {
      "name": "micronaut-service",
      "status": "UP",
      "details": {
        "total": 109702647808,
        "free": 69758287872,
        "threshold": 10485760
      }
    }
  }
}

Health checks require the following dependency:

compile 'io.micronaut:micronaut-management'

<dependency>
    <groupId>io.micronaut</groupId>
    <artifactId>micronaut-management</artifactId>
</dependency>

Also note that in order to see the full details of the health checks you may need additional configuration. Check the documentation of the Health Endpoint for more information about how to configure it.

5 Using the low-level Kubernetes API HTTP client

In an effort to have zero dependencies and a minimal footprint, this configuration module has its own Kubernetes API non-blocking HTTP client. If you need to, you can use it directly in your application:

import io.micronaut.kubernetes.client.v1.KubernetesClient;
import io.reactivex.Flowable;
import io.reactivex.schedulers.Schedulers;
import javax.inject.Singleton;

@Singleton
public class MyService {

    private final KubernetesClient client;

    public MyService(KubernetesClient client) {
        this.client = client;
    }

    public void myMethod() {
        Flowable.fromPublisher(this.client.listServices("default"))
                .subscribeOn(Schedulers.io())
                .subscribe(System.out::println);
    }
}

Note that the client only contains the Kubernetes API endpoints used by either the Discovery Client or the Configuration Client. You can check the API documentation to see the available methods.

6 Logging and debugging

If you need to debug the HTTP calls made to the Kubernetes API, you need to set the io.micronaut.http.client logger level to TRACE:

<logger name="io.micronaut.http.client" level="TRACE"/>

Also, you can set a higher level (DEBUG or TRACE) for this configuration module logging.

<logger name="io.micronaut.kubernetes" level="TRACE"/>

Other package that might produce relevant logging is io.micronaut.discovery, which belongs to Micronaut Core.

In addition to that, another source of information is the Environment Endpoint, which outputs all the resolved PropertySources from ConfigMaps, and their corresponding properties.

Table of Contents

Micronaut Kubernetes

1 Introduction

Namespace configuration

2 Service Discovery

Kubernetes API authentication

Connecting to services using HTTPS

Using https as port name

Using a port ending in 443

Using labels

3 Configuration Client

ConfigMaps

Watching for changes in ConfigMaps

Examples

Secrets

Reading ConfigMaps and Secrets from mounted volumes

4 Health Checks

Health Indicators

5 Using the low-level Kubernetes API HTTP client

6 Logging and debugging

Using `https` as port name

Reading `ConfigMap`s and `Secret`s from mounted volumes