Using Terraform to Manage an Object Stored in OpenStack Swift

Technologies

OpenStack is a set of software that infrastructure providers can use to give users a user-experience similar to that of AWS or other cloud providers. e.g. Cleura provides a public cloud service which can be managed with OpenStack APIs. – I think the idea for OpenStack was to enable infrastructure providers to serve as a substitute for AWS.. albeit, even amongst OpenStack public clouds, there remain idiosyncracies such that a deployment in one OpenStack cloud might require effort to deploy to another OpenStack cloud.

OpenStack’s Swift is an ‘object storage’ service. More/less the same thing as AWS’ S3 (or Google Cloud’s cloud storage).

Terraform, or its open source fork OpenTofu, is a tool for managing declarations of cloud resources. There’s a Terraform Provider for OpenStack, so that Terraform can be used to work with OpenStack resources.

Task

Here are notes on using Terraform to manage an object in a ‘container’ on OpenStack Swift (‘bucket’ in AWS terms), and to create a (temporary) URL to access that object.

The main advantage of this approach is reyling on Terraform to manage the setup/teardown, as opposed to having to use the web console to manage this, or use the CLI client to manage these. I prefer being able to run terraform apply to ensure things are there; and terraform destroy to ensure things are gone.

Terraform Code for Container and Bucket

Some boilerplate, using the openstack provider for Terraform:

terraform {
  required_version = ">= 0.14.0"
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 1.49.0"
    }
  }
}

As the provider documentation mentions, the Terraform OpenStack provider can pick up authentication credentials from OS_ environment variables. I used the Cleura public OpenStack cloud, and downloaded an ‘rc’ file with the credentials for an OpenStack user to use with a project there.

Then the Terraform code for creating a container rgoulter-storage, and an object named test in that container:

resource "openstack_objectstorage_container_v1" "self" {
  name = "rgoulter-storage"
  metadata = {
    Temp-URL-Key = "some-secret-value"
  }
}

resource "openstack_objectstorage_object_v1" "self" {
  container_name = openstack_objectstorage_container_v1.self.name
  name           = "test"
  content        = "Hello, world!"
}

Resources are named self out of convention. (There aren’t multiple containers, objects in this Terraform codebase).

In this case, the content of the object is set with the content attribute to the object resource. The source attribute is more suitable when you want some file to be uploaded. (c.f. the terraform provider documentation).

Temp-URL-Key is set on the container’s metadata to allow creating temporary URLs to provide access. (c.f. Swift’s temporary url middleware documentation). – This value should be kept secret; this value is used to create temporary URLs to access objects in the container.

Creating Temporary URLs: with Terraform

resource "openstack_objectstorage_tempurl_v1" "self" {
  container  = openstack_objectstorage_container_v1.self.name
  object     = openstack_objectstorage_object_v1.self.name
  method     = "get"
  ttl        = 900
  regenerate = true
}

output "tempurl" {
  value     = openstack_objectstorage_tempurl_v1.self.url
  sensitive = true
}

It’s possible to also use the Terraform OpenStack provider to manage tempurls.

On the one hand, you might as well, if you’re using Terraform for the other parts. On the other hand, I think it smells a bit too much to ‘declare’ the existance of a tempurl with a time-to-live of 900 seconds.

One nice thing about this approach is the output URL value is simple/easy to figure out. i.e. running terraform output -raw tempurl outputs something like:

https://swift-fra1.citycloud.com:8080/swift/v1/AUTH_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/rgoulter-storage/test?temp_url_sig=bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb&temp_url_expires=1999999999

Creating Temporary URLs: with Command Line Client

The documentation about temporary url middleware gives instructions on using the OpenStack Swift client to create a temporary URL.

There are various ways to come up with the OpenStack Storage URL (https://swift-cluster.example.com/v1/my_account/), but here’s what I came up with for the container, object, and temp URL key above:

OS_STORAGE_URL=$(openstack catalog show object-store --format=json | jq -r .endpoints[2].url)
swift tempurl --digest=sha1 GET 900 ${OS_STORAGE_URL}/rgoulter-storage/test some-secret-value

The jq expression isn’t the cleanest. (The [2] is the index the “public” object-store endpoint happened to be in the output of openstack catalog show object-store; albeit, all the endpoints had the same URL).

The URL from swift tempurl can be used for GET requests to access the storage.