[ + ]

Table of Contents

Micronaut Object Storage

Micronaut Object Storage provides a uniform API to create, read and delete objects in the major cloud providers

Version: 2.7.1-SNAPSHOT

1 Introduction

Micronaut Object Storage provides a uniform API to create, read and delete objects in the major cloud providers:

There is also a local storage implementation for testing purposes.

Using this API enables the creation of truly multi-cloud, portable applications.

2 Release History

For this project, you can find a list of releases (with release notes) here:

https://github.com/micronaut-projects/micronaut-object-storage/releases

3 Quick Start

To get started, you need to declare a dependency for the actual cloud provider you are using. See the actual cloud provider documentation for more details:

Then, you can inject in your controllers/services/etc. a bean of type ObjectStorageOperations, the parent interface that allows you to use the API in a generic way for all cloud providers:

@Singleton
public class ProfileService {

    private static final Logger LOG = LoggerFactory.getLogger(ProfileService.class);

    private final ObjectStorageOperations<?, ?, ?> objectStorage;

    public ProfileService(ObjectStorageOperations<?, ?, ?> objectStorage) {
        this.objectStorage = objectStorage;
    }

}
@Singleton
class ProfileService {

    final ObjectStorageOperations<?, ?, ?> objectStorage

    ProfileService(ObjectStorageOperations<?, ?, ?> objectStorage) {
        this.objectStorage = objectStorage
    }

}
@Singleton
open class ProfileService(private val objectStorage: ObjectStorageOperations<*, *, *>) {

}

If your application is not multi-cloud, and/or you need cloud-specific details, you can use a concrete implementation. For example, for AWS S3:

@Controller
public class UploadController {

    private final AwsS3Operations objectStorage;

    public UploadController(AwsS3Operations objectStorage) {
        this.objectStorage = objectStorage;
    }

}
@Controller
class UploadController {

    final AwsS3Operations objectStorage

    UploadController(AwsS3Operations objectStorage) {
        this.objectStorage = objectStorage
    }

}
@Controller
open class UploadController(private val objectStorage: AwsS3Operations) {

}

If you have multiple object storages configured, it is possible to select which one to work with via bean qualifiers.

For example, given the following configuration:

src/main/resources/application-ec2.yml
micronaut.object-storage.aws.pictures.bucket=pictures-bucket
micronaut.object-storage.aws.logos.bucket=logos-bucket
micronaut:
  object-storage:
    aws:
      pictures:
        bucket: pictures-bucket
      logos:
        bucket: logos-bucket
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.aws]
      [micronaut.object-storage.aws.pictures]
        bucket="pictures-bucket"
      [micronaut.object-storage.aws.logos]
        bucket="logos-bucket"
micronaut {
  objectStorage {
    aws {
      pictures {
        bucket = "pictures-bucket"
      }
      logos {
        bucket = "logos-bucket"
      }
    }
  }
}
{
  micronaut {
    object-storage {
      aws {
        pictures {
          bucket = "pictures-bucket"
        }
        logos {
          bucket = "logos-bucket"
        }
      }
    }
  }
}
{
  "micronaut": {
    "object-storage": {
      "aws": {
        "pictures": {
          "bucket": "pictures-bucket"
        },
        "logos": {
          "bucket": "logos-bucket"
        }
      }
    }
  }
}

You then need to use @Named("pictures") or @Named("logos") to specify which of the object storages you want to use.

Uploading files

public String saveProfilePicture(String userId, Path path) {
    UploadRequest request = UploadRequest.fromPath(path, userId); // (1)
    UploadResponse<?> response = objectStorage.upload(request); // (2)
    return response.getKey(); // (3)
}
String saveProfilePicture(String userId, Path path) {
    UploadRequest request = UploadRequest.fromPath(path, userId) // (1)
    UploadResponse response = objectStorage.upload(request) // (2)
    response.key // (3)
}
open fun saveProfilePicture(userId: String, path: Path): String? {
    val request = UploadRequest.fromPath(path, userId) // (1)
    val response = objectStorage.upload(request) // (2)
    return response.key // (3)
}
1 You can use any of the UploadRequest static methods to build an upload request.
2 The upload operation returns an UploadResponse, which wraps the cloud-specific SDK response object.
3 The response object contains some common properties for all cloud vendor, and a getNativeResponse() method that can be used for accessing the vendor-specific response object.

In case you want to have better control of the upload options used, you can use the method upload(UploadRequest, Consumer) of ObjectStorageOperations, which will give you access to the cloud vendor-specific request class or builder.

For example, for AWS S3:

UploadResponse<PutObjectResponse> response = objectStorage.upload(objectStorageUpload, builder -> {
    builder.acl(ObjectCannedACL.PUBLIC_READ);
});
UploadResponse<PutObjectResponse> response = objectStorage.upload(objectStorageUpload, { builder ->
    builder.acl(ObjectCannedACL.PUBLIC_READ)
})
val response = objectStorage.upload(objectStorageUpload) { builder: PutObjectRequest.Builder ->
    builder.acl(ObjectCannedACL.PUBLIC_READ)
}

Retrieving files

public Optional<Path> retrieveProfilePicture(String userId, String fileName) {
    Path destination = null;
    try {
        String key = userId + "/" + fileName;
        Optional<InputStream> stream = objectStorage.retrieve(key) // (1)
            .map(ObjectStorageEntry::getInputStream);
        if (stream.isPresent()) {
            destination = File.createTempFile(userId, "temp").toPath();
            Files.copy(stream.get(), destination, StandardCopyOption.REPLACE_EXISTING);
            return Optional.of(destination);
        } else {
            return Optional.empty();
        }
    } catch (IOException e) {
        LOG.error("Error while trying to save profile picture to the local file [{}]: {}", destination, e.getMessage());
        return Optional.empty();
    }
}
Optional<Path> retrieveProfilePicture(String userId, String fileName) {
    String key = "${userId}/${fileName}"
    Optional<InputStream> stream = objectStorage.retrieve(key) // (1)
            .map(ObjectStorageEntry::getInputStream)
    if (stream.isPresent()) {
        Path destination = File.createTempFile(userId, "temp").toPath()
        Files.copy(stream.get(), destination, StandardCopyOption.REPLACE_EXISTING)
        return Optional.of(destination)
    } else {
        return Optional.empty()
    }
}
open fun retrieveProfilePicture(userId: String, fileName: String): Path? {
    val key = "$userId/$fileName"
    val stream = objectStorage.retrieve<ObjectStorageEntry<*>>(key) // (1)
        .map { obj: ObjectStorageEntry<*> -> obj.inputStream }

    return if (stream.isPresent) {
        val destination = File.createTempFile(userId, "temp").toPath()
        Files.copy(stream.get(), destination, StandardCopyOption.REPLACE_EXISTING)
        destination
    } else {
        null
    }
}
1 The retrieve operation returns an ObjectStorageEntry, from which you can get an InputStream. There is also a getNativeEntry() method that gives you access to the cloud vendor-specific response object.

Deleting files

public void deleteProfilePicture(String userId, String fileName) {
    String key = userId + "/" + fileName;
    objectStorage.delete(key); // (1)
}
void deleteProfilePicture(String userId, String fileName) {
    String key = "${userId}/${fileName}"
    objectStorage.delete(key) // (1)
}
open fun deleteProfilePicture(userId: String, fileName: String) {
    val key = "$userId/$fileName"
    objectStorage.delete(key) // (1)
}
1 The delete operation returns the cloud vendor-specific delete response object in case you need it.

4 Amazon S3

To use Amazon S3, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-aws")
<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-aws</artifactId>
</dependency>

Refer to the Micronaut AWS documentation for more information about credentials and region configuration.

The object storage specific configuration options available are:

🔗
Table 1. Configuration Properties for AwsS3Configuration
Property Type Description

micronaut.object-storage.aws.*.enabled

boolean

Whether to enable or disable this object storage.

micronaut.object-storage.aws.*.bucket

java.lang.String

The name of the AWS S3 bucket.

For example:

src/main/resources/application-ec2.yml
micronaut.object-storage.aws.default.bucket=profile-pictures-bucket
micronaut:
  object-storage:
    aws:
      default:
        bucket: profile-pictures-bucket
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.aws]
      [micronaut.object-storage.aws.default]
        bucket="profile-pictures-bucket"
micronaut {
  objectStorage {
    aws {
      'default' {
        bucket = "profile-pictures-bucket"
      }
    }
  }
}
{
  micronaut {
    object-storage {
      aws {
        default {
          bucket = "profile-pictures-bucket"
        }
      }
    }
  }
}
{
  "micronaut": {
    "object-storage": {
      "aws": {
        "default": {
          "bucket": "profile-pictures-bucket"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is AwsS3Operations.

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

@Singleton
public class S3ClientBuilderCustomizer implements BeanCreatedEventListener<S3ClientBuilder> {

    @Override
    public S3ClientBuilder onCreated(@NonNull BeanCreatedEvent<S3ClientBuilder> event) {
        return event.getBean()
            .overrideConfiguration(c -> c.apiCallTimeout(Duration.of(60, ChronoUnit.SECONDS)));
    }
}
See the guide for Use the Micronaut Object Storage API to Store Files in Amazon S3 to learn more.

5 Azure Blob Storage

To use Azure Blob Storage, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-azure")
<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-azure</artifactId>
</dependency>

Refer to the Micronaut Azure documentation for more information about authentication options.

The object storage specific configuration options available are:

🔗
Table 1. Configuration Properties for AzureBlobStorageConfiguration
Property Type Description

micronaut.object-storage.azure.*.enabled

boolean

Whether to enable or disable this object storage.

micronaut.object-storage.azure.*.container

java.lang.String

The blob container name.

micronaut.object-storage.azure.*.endpoint

java.lang.String

The blob service endpoint, in the format of https://{accountName}.blob.core.windows.net.

For example:

src/main/resources/application-azure.yml
azure.credential.client-secret.client-id=<client-id>
azure.credential.client-secret.tenant-id=<tenant-id>
azure.credential.client-secret.secret=<secret>
micronaut.object-storage.azure.default.container=profile-pictures-container
micronaut.object-storage.azure.default.endpoint=https://my-account.blob.core.windows.net
azure:
  credential:
    client-secret:
      client-id: <client-id>
      tenant-id: <tenant-id>
      secret: <secret>

micronaut:
  object-storage:
    azure:
      default:
        container: profile-pictures-container
        endpoint: https://my-account.blob.core.windows.net
[azure]
  [azure.credential]
    [azure.credential.client-secret]
      client-id="<client-id>"
      tenant-id="<tenant-id>"
      secret="<secret>"
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.azure]
      [micronaut.object-storage.azure.default]
        container="profile-pictures-container"
        endpoint="https://my-account.blob.core.windows.net"
azure {
  credential {
    clientSecret {
      clientId = "<client-id>"
      tenantId = "<tenant-id>"
      secret = "<secret>"
    }
  }
}
micronaut {
  objectStorage {
    azure {
      'default' {
        container = "profile-pictures-container"
        endpoint = "https://my-account.blob.core.windows.net"
      }
    }
  }
}
{
  azure {
    credential {
      client-secret {
        client-id = "<client-id>"
        tenant-id = "<tenant-id>"
        secret = "<secret>"
      }
    }
  }
  micronaut {
    object-storage {
      azure {
        default {
          container = "profile-pictures-container"
          endpoint = "https://my-account.blob.core.windows.net"
        }
      }
    }
  }
}
{
  "azure": {
    "credential": {
      "client-secret": {
        "client-id": "<client-id>",
        "tenant-id": "<tenant-id>",
        "secret": "<secret>"
      }
    }
  },
  "micronaut": {
    "object-storage": {
      "azure": {
        "default": {
          "container": "profile-pictures-container",
          "endpoint": "https://my-account.blob.core.windows.net"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is AzureBlobStorageOperations.

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

@Singleton
public class BlobServiceClientBuilderCustomizer implements BeanCreatedEventListener<BlobServiceClientBuilder> {

    @Override
    public BlobServiceClientBuilder onCreated(@NonNull BeanCreatedEvent<BlobServiceClientBuilder> event) {
        HttpPipelinePolicy noOp = (context, next) -> next.process();
        return event.getBean().addPolicy(noOp);
    }
}

6 Google Cloud Storage

To use Google Cloud Storage, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-gcp")
<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-gcp</artifactId>
</dependency>

Refer to the Micronaut GCP documentation for more information about configuring your GCP project.

The object storage specific configuration options available are:

🔗
Table 1. Configuration Properties for GoogleCloudStorageConfiguration
Property Type Description

micronaut.object-storage.gcp.*.enabled

boolean

Whether to enable or disable this object storage.

micronaut.object-storage.gcp.*.bucket

java.lang.String

The name of the Google Cloud Storage bucket.

For example:

src/main/resources/application-gcp.yml
gcp.project-id=my-gcp-project
micronaut.object-storage.gcp.default.bucket=profile-pictures-bucket
gcp:
  project-id: my-gcp-project

micronaut:
  object-storage:
    gcp:
      default:
        bucket: profile-pictures-bucket
[gcp]
  project-id="my-gcp-project"
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.gcp]
      [micronaut.object-storage.gcp.default]
        bucket="profile-pictures-bucket"
gcp {
  projectId = "my-gcp-project"
}
micronaut {
  objectStorage {
    gcp {
      'default' {
        bucket = "profile-pictures-bucket"
      }
    }
  }
}
{
  gcp {
    project-id = "my-gcp-project"
  }
  micronaut {
    object-storage {
      gcp {
        default {
          bucket = "profile-pictures-bucket"
        }
      }
    }
  }
}
{
  "gcp": {
    "project-id": "my-gcp-project"
  },
  "micronaut": {
    "object-storage": {
      "gcp": {
        "default": {
          "bucket": "profile-pictures-bucket"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is GoogleCloudStorageOperations.

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

@Singleton
public class StorageOptionsBuilderCustomizer implements BeanCreatedEventListener<StorageOptions.Builder> {

    @Override
    public StorageOptions.Builder onCreated(@NonNull BeanCreatedEvent<StorageOptions.Builder> event) {
        return event.getBean()
            .setTransportOptions(HttpTransportOptions.newBuilder().setConnectTimeout(60_000).build());
    }
}
See the guide for Use the Micronaut Object Storage API to Store Files in Google Cloud Storage to learn more.

7 Oracle Cloud Infrastructure (OCI) Object Storage

To use Oracle Cloud Infrastructure (OCI) Object Storage, you need the following dependency:

implementation("io.micronaut.objectstorage:micronaut-object-storage-oracle-cloud")
<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-oracle-cloud</artifactId>
</dependency>

Refer to the Micronaut Oracle Cloud documentation for more information about authentication options.

The object storage specific configuration options available are:

🔗
Table 1. Configuration Properties for OracleCloudStorageConfiguration
Property Type Description

micronaut.object-storage.oracle-cloud.*.enabled

boolean

Whether to enable or disable this object storage.

micronaut.object-storage.oracle-cloud.*.bucket

java.lang.String

The name of the OCI Object Storage bucket.

micronaut.object-storage.oracle-cloud.*.namespace

java.lang.String

The OCI Object Storage namespace used.

For example:

src/main/resources/application-oraclecloud.yml
oci.config.profile=DEFAULT
micronaut.object-storage.oracle-cloud.default.bucket=profile-pictures-bucket
micronaut.object-storage.oracle-cloud.default.namespace=MyNamespace
oci:
  config:
    profile: DEFAULT

micronaut:
  object-storage:
    oracle-cloud:
      default:
        bucket: profile-pictures-bucket
        namespace: MyNamespace
[oci]
  [oci.config]
    profile="DEFAULT"
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.oracle-cloud]
      [micronaut.object-storage.oracle-cloud.default]
        bucket="profile-pictures-bucket"
        namespace="MyNamespace"
oci {
  config {
    profile = "DEFAULT"
  }
}
micronaut {
  objectStorage {
    oracleCloud {
      'default' {
        bucket = "profile-pictures-bucket"
        namespace = "MyNamespace"
      }
    }
  }
}
{
  oci {
    config {
      profile = "DEFAULT"
    }
  }
  micronaut {
    object-storage {
      oracle-cloud {
        default {
          bucket = "profile-pictures-bucket"
          namespace = "MyNamespace"
        }
      }
    }
  }
}
{
  "oci": {
    "config": {
      "profile": "DEFAULT"
    }
  },
  "micronaut": {
    "object-storage": {
      "oracle-cloud": {
        "default": {
          "bucket": "profile-pictures-bucket",
          "namespace": "MyNamespace"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is OracleCloudStorageOperations

Advanced configuration

For configuration properties other than the specified above, you can add bean to your application that implements BeanCreatedEventListener. For example:

//See https://github.com/oracle/oci-java-sdk/blob/master/bmc-examples/src/main/java/ClientConfigurationTimeoutExample.java
@Singleton
public class ObjectStorageClientBuilderCustomizer implements BeanCreatedEventListener<ObjectStorageClient.Builder> {

    public static final int CONNECTION_TIMEOUT_IN_MILLISECONDS = 25000;
    public static final int READ_TIMEOUT_IN_MILLISECONDS = 35000;

    @Override
    public ObjectStorageClient.Builder onCreated(@NonNull BeanCreatedEvent<ObjectStorageClient.Builder> event) {
        ClientConfiguration clientConfiguration =
            ClientConfiguration.builder()
                .connectionTimeoutMillis(CONNECTION_TIMEOUT_IN_MILLISECONDS)
                .readTimeoutMillis(READ_TIMEOUT_IN_MILLISECONDS)
                .build();


        return event.getBean()
            .configuration(clientConfiguration);
    }
}
See the guide for Use the Micronaut Object Storage API to Store Files in Oracle Cloud Infrastructure (OCI) Object Storage to learn more.

8 Local Storage

To use the local storage implementation (useful for tests), you need the following dependency:

testImplementation("io.micronaut.objectstorage:micronaut-object-storage-local")
<dependency>
    <groupId>io.micronaut.objectstorage</groupId>
    <artifactId>micronaut-object-storage-local</artifactId>
    <scope>test</scope>
</dependency>

Then, simply define a local storage:

micronaut.object-storage.local.default.enabled=true
micronaut:
  object-storage:
    local:
      default:
        enabled: true
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.local]
      [micronaut.object-storage.local.default]
        enabled=true
micronaut {
  objectStorage {
    local {
      'default' {
        enabled = true
      }
    }
  }
}
{
  micronaut {
    object-storage {
      local {
        default {
          enabled = true
        }
      }
    }
  }
}
{
  "micronaut": {
    "object-storage": {
      "local": {
        "default": {
          "enabled": true
        }
      }
    }
  }
}
When added to the classpath, LocalStorageOperations becomes the primary implementation of ObjectStorageOperations.

By default, it will create a temporary folder to store the files, but you can configure it to use a specific folder:

🔗
Table 1. Configuration Properties for LocalStorageConfiguration
Property Type Description

micronaut.object-storage.local.*.enabled

boolean

Whether to enable or disable this object storage.

micronaut.object-storage.local.*.path

java.nio.file.Path

The path of the local storage.

For example:

src/main/resources/application-test.yml
micronaut.object-storage.local.default.path=/tmp/my-object-storage
micronaut:
  object-storage:
    local:
      default:
        path: /tmp/my-object-storage
[micronaut]
  [micronaut.object-storage]
    [micronaut.object-storage.local]
      [micronaut.object-storage.local.default]
        path="/tmp/my-object-storage"
micronaut {
  objectStorage {
    local {
      'default' {
        path = "/tmp/my-object-storage"
      }
    }
  }
}
{
  micronaut {
    object-storage {
      local {
        default {
          path = "/tmp/my-object-storage"
        }
      }
    }
  }
}
{
  "micronaut": {
    "object-storage": {
      "local": {
        "default": {
          "path": "/tmp/my-object-storage"
        }
      }
    }
  }
}

The concrete implementation of ObjectStorageOperations is LocalStorageOperations.

9 Guides

See the following list of guides to learn more about working with Object Storage in the Micronaut Framework:

https://guides.micronaut.io/latest/tag-object_storage.html

10 Repository

You can find the source code of this project in this repository:

https://github.com/micronaut-projects/micronaut-object-storage