[ + ]

Table of Contents

Micronaut CRaC

Adds support for CRaC (Coordinated Restore at Checkpoint) to the Micronaut Framework.

Version: 2.4.0

1 Introduction

This Micronaut module adds support for CRaC (Coordinated Restore at Checkpoint) within the framework.

2 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-crac/releases

3 Installation

Add the following the dependency to your build:

implementation("io.micronaut.crac:micronaut-crac")
<dependency>
    <groupId>io.micronaut.crac</groupId>
    <artifactId>micronaut-crac</artifactId>
</dependency>

4 CRaC Resources

4.1 DataSources

We currently support Hikari DataSources that are configured to allow suspension with:

datasources.default.allow-pool-suspension=true
datasources:
  default:
    allow-pool-suspension: true
[datasources]
  [datasources.default]
    allow-pool-suspension=true
datasources {
  'default' {
    allowPoolSuspension = true
  }
}
{
  datasources {
    default {
      allow-pool-suspension = true
    }
  }
}
{
  "datasources": {
    "default": {
      "allow-pool-suspension": true
    }
  }
}

When a checkpoint is taken, the pool is suspended and the connections are closed. When the checkpoint is restored, the pool is resumed and the connections are re-established.

4.2 Redis

Since v1.2.1 of this library, we also support Redis connections.

When the checkpoint is taken, we destroy RedisClient, RedisCache and StatefulRedisConnection beans that exist in the application.

These will be re-created once the checkpoint is restored, however you will need to add your beans to the refresh scope so that these new beans are used.

The resources for each of the above can be disabled via configuration:

crac.redis.enabled=false
crac.redis.client-enabled=false
crac.redis.cache-enabled=false
crac.redis.connection-enabled=false
crac:
    redis:
        enabled: false            # disable all redis support
        client-enabled: false     # disable RedisClient support
        cache-enabled: false      # disable RedisCache support
        connection-enabled: false # disable StatefulRedisConnection support
[crac]
  [crac.redis]
    enabled=false
    client-enabled=false
    cache-enabled=false
    connection-enabled=false
crac {
  redis {
    enabled = false
    clientEnabled = false
    cacheEnabled = false
    connectionEnabled = false
  }
}
{
  crac {
    redis {
      enabled = false
      client-enabled = false
      cache-enabled = false
      connection-enabled = false
    }
  }
}
{
  "crac": {
    "redis": {
      "enabled": false,
      "client-enabled": false,
      "cache-enabled": false,
      "connection-enabled": false
    }
  }
}

4.3 Custom CRaC Resources

To provide custom CRaC resources, create beans of type OrderedResource.

Micronaut CRaC registers resources for you into the CRaC Context. You just focus on providing implementations for OrderedResource::beforeCheckpoint and OrderedResource::afterRestore in your resources.

Micronaut CRaC registers resources in order. You can control the order by overriding OrderedResource::getOrder.

5 Refresh scope

Prior to a checkpoint being taken, a RefreshEvent will be published to invalidate all beans in the @Refreshable scope.

This behaviour can be disabled by setting the crac.refresh-beans property to false in the application config.

6 Events

To notify external components when the default Resource handlers execute, there are two events; BeforeCheckpointEvent and AfterRestoreEvent. These events contain the java.time.Instant that the action completed, the length of time it took to execute the action in nanoseconds, and the Resource that was acted upon.

Please see the Micronaut Framework guide for information on how to listen for these events.

7 Context Provider

GlobalCracContextProvider, Micronaut CRaC’s default implementation of CracContextProvider, returns the global context. You can provide a replacement for CracContextProvider and provide your custom Context.

8 Docker Support

Support for building CRaC-enabled Docker images is provided by the Micronaut Gradle or Maven plugin. Either one is capable of generating a docker image containing a CRaC enabled JDK and a pre-warmed, checkpointed application.

9 Testing

When you create a Micronaut Framework application (either via https://launch.micronaut.io or the command line application), it creates a ContextConfigurer with enables eager singleton initialization.

As tests annotated with @MicronautTest are implicitly in the Singleton scope, this can cause problems injecting some beans (for example an HttpClient) into your test class.

To avoid this, you can either disable eager singleton initialization for your tests, or you will need to manually get an instance of the bean you would normally inject. As an example, to get an HttpClient you could do:

Lazily get an HttpClient in a test
@Inject
EmbeddedServer server; // (1)

Supplier<HttpClient> clientSupplier = SupplierUtil.memoizedNonEmpty(() -> // (2)
    server.getApplicationContext().createBean(HttpClient.class, server.getURL())
);

@Test
void testClient() {
    assertEquals("ok", clientSupplier.get().toBlocking().retrieve("/eager")); // (3)
}
Lazily get an HttpClient in a test
@Inject
EmbeddedServer server // (1)

@Memoized // (2)
HttpClient clientSupplier() {
    server.applicationContext.createBean(HttpClient, server.URL)
}

void 'test client'() {
    expect:
    clientSupplier().toBlocking().retrieve("/eager") == "ok" // (3)
}
Lazily get an HttpClient in a test
@field:Inject
lateinit var server: EmbeddedServer // (1)

val client by lazy {
    server.applicationContext.createBean(HttpClient::class.java, server.url) // (2)
}

@Test
fun testClient() {
    assertEquals("ok", client.toBlocking().retrieve("/eager")) // (3)
}
1 Inject the EmbeddedServer as normal
2 Lazily create a HttpClient when it is first called
3 Get the HttpClient and make the request

9.1 Disable Eager Singleton Initialization

When you create a Micronaut Framework application (either via https://launch.micronaut.io or the command line application), it creates a ContextConfigurer with enables eager singleton initialization.

As tests annotated with @MicronautTest are implicitly in the Singleton scope, this can cause problems injecting some beans (for example an HttpClient) into your test class.

To avoid this, you can either disable eager singleton initialization for your tests, or you will need to manually get an instance of the bean you would normally inject. As an example, to get an HttpClient you could do:

Lazily get an HttpClient in a test
@Inject
EmbeddedServer server; // (1)

Supplier<HttpClient> clientSupplier = SupplierUtil.memoizedNonEmpty(() -> // (2)
    server.getApplicationContext().createBean(HttpClient.class, server.getURL())
);

@Test
void testClient() {
    assertEquals("ok", clientSupplier.get().toBlocking().retrieve("/eager")); // (3)
}
Lazily get an HttpClient in a test
@Inject
EmbeddedServer server // (1)

@Memoized // (2)
HttpClient clientSupplier() {
    server.applicationContext.createBean(HttpClient, server.URL)
}

void 'test client'() {
    expect:
    clientSupplier().toBlocking().retrieve("/eager") == "ok" // (3)
}
Lazily get an HttpClient in a test
@field:Inject
lateinit var server: EmbeddedServer // (1)

val client by lazy {
    server.applicationContext.createBean(HttpClient::class.java, server.url) // (2)
}

@Test
fun testClient() {
    assertEquals("ok", client.toBlocking().retrieve("/eager")) // (3)
}
1 Inject the EmbeddedServer as normal
2 Lazily create a HttpClient when it is first called
3 Get the HttpClient and make the request

9.2 Using a CheckpointSimulator

With CRaC, you run your application to a point and then "checkpoint" it. This calls the app to close all its sockets and file handles, and then dumps the memory to disk. When it restarts from this snapshot, it calls the app again to say it’s been restored, and one can re-open files and network connections.

The simulator allows you to synthesise these two calls (before checkpoint and after restore), so that under a test you can check your service works again after it was closed and recreated.

The following example demonstrates how to write and test custom OrderedResource implementations. Here it is doing nothing for sake of the example, but in reality it would have a socket or file that needs to be closed, or something along those lines.

A resource bean
@Singleton
class ResourceBean {

    private boolean running = true; // (1)

    boolean isRunning() {
        return running;
    }

    void stop() {
        this.running = false;
    }

    void start() {
        this.running = true;
    }
}
A resource bean
@Singleton
 class ResourceBean {

    private boolean running = true // (1)

    boolean isRunning() {
        return running
    }

    void stop() {
        this.running = false
    }

    void start() {
        this.running = true
    }
}
A resource bean
@Singleton
class ResourceBean {

    var isRunning = true // (1)
        private set

    fun stop() {
        isRunning = false
    }

    fun start() {
        isRunning = true
    }
}
1 A simulated state for the bean

An OrderedResource which will stop the ResourceBean before a checkpoint and start it again after a restore.

An OrderedResouce for the bean
@Singleton
class ResourceBeanResource implements OrderedResource { // (1)

    private final ResourceBean resourceBean;

    ResourceBeanResource(ResourceBean resourceBean) {
        this.resourceBean = resourceBean;
    }

    @Override
    public void beforeCheckpoint(Context<? extends Resource> context) throws Exception { // (2)
        resourceBean.stop();
    }

    @Override
    public void afterRestore(Context<? extends Resource> context) throws Exception { // (3)
        resourceBean.start();
    }
}
An OrderedResouce for the bean
@Singleton
 class ResourceBeanResource implements OrderedResource { // (1)

    private final ResourceBean resourceBean

    ResourceBeanResource(ResourceBean resourceBean) {
        this.resourceBean = resourceBean
    }

    @Override
    void beforeCheckpoint(Context<? extends Resource> context) throws Exception { // (2)
        resourceBean.stop()
    }

    @Override
    void afterRestore(Context<? extends Resource> context) throws Exception { // (3)
        resourceBean.start()
    }
}
An OrderedResouce for the bean
@Singleton
class ResourceBeanResource(private val resourceBean: ResourceBean) : OrderedResource { // (1)

    @Throws(Exception::class)
    override fun beforeCheckpoint(context: Context<out Resource?>?) { // (2)
        resourceBean.stop()
    }

    @Throws(Exception::class)
    override fun afterRestore(context: Context<out Resource?>?) { // (3)
        resourceBean.start()
    }
}
1 Implement the OrderedResource interface
2 Override to stop the resource before a checkpoint
3 Override to restart it again after a restore
Using CheckpointSimulator in a test
@Inject
BeanContext ctx;

@Test
void testCustomOrderedResourceUsingCheckpointSimulator() {
    ResourceBean myBean = ctx.getBean(ResourceBean.class);
    CheckpointSimulator checkpointSimulator = ctx.getBean(CheckpointSimulator.class); // (1)
    assertTrue(myBean.isRunning());

    checkpointSimulator.runBeforeCheckpoint();  // (2)
    assertFalse(myBean.isRunning());

    checkpointSimulator.runAfterRestore(); // (3)
    assertTrue(myBean.isRunning());
}
Using CheckpointSimulator in a test
@Inject
BeanContext ctx

void "test custom OrderedResource implementation using the CheckpointSimulator"() {
    given:
    ResourceBean myBean = ctx.getBean(ResourceBean)
    CheckpointSimulator checkpointSimulator = ctx.getBean(CheckpointSimulator) // (1)
    expect:
    myBean.running

    when:
    checkpointSimulator.runBeforeCheckpoint()  // (2)
    then:
    !myBean.running

    when:
    checkpointSimulator.runAfterRestore() // (3)

    then:
    myBean.running
}
Using CheckpointSimulator in a test
@field:Inject
lateinit var ctx: BeanContext

@Test
fun testCustomOrderedResourceUsingCheckpointSimulator() {
    val myBean = ctx.getBean(ResourceBean::class.java)
    val checkpointSimulator = ctx.getBean(CheckpointSimulator::class.java) // (1)
    Assertions.assertTrue(myBean.isRunning)

    checkpointSimulator.runBeforeCheckpoint() // (2)
    Assertions.assertFalse(myBean.isRunning)

    checkpointSimulator.runAfterRestore() // (3)
    Assertions.assertTrue(myBean.isRunning)
}
1 Obtain the CheckpointSimulator from the running environment
2 Emulate the checkpoint and assert the bean is no longer running
3 Emulate the restore and assert the bean is running again

10 Info sources

If the info endpoint is enabled, then a crac section will be automatically added which shows the restore time and uptime since restore, both in milliseconds and taken from the CRaCMXBean provided by the CRaC API.

This can be disabled by setting endpoints.info.crac.enabled configuration to false.

11 Guides

See the following list of guides to learn more about working with CRaC in the Micronaut Framework:

https://guides.micronaut.io/latest/tag-crac.html

12 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-crac