The cloud storage new Prefix and suffix lifecycle conditions | C2C Community

The cloud storage new Prefix and suffix lifecycle conditions

  • 24 August 2022
  • 0 replies

Userlevel 7
Badge +32

Recently, Google Cloud Storage launch powerful new object lifecycle rules for GCS to help us control cloud storage costs in new ways. Two new features are included in this launch.

Prefix/suffix lifecycle conditions

Before,  we already had several lifecycle conditions to choose from, such as the age of an object, its version history, a custom timestamp, and more. Now, we can add conditions on the names of objects; specifically, matching a prefix or a suffix.  

Prefix and suffix conditions are helpful for a number of cases. Here are two:

Managing common object prefixes separately. It's quite common to group objects using a common prefix, such as in a dataset. Now, lifecycle conditions can act on those groups using a MatchesPrefix rule.

Managing categories of objects separately. It's also common to use "extensions" on object keys to denote the format of the data; .mp4, .zip, .csv, and so on.  we often have a mix of these within a bucket, and would like to use the extension to manage them separately. A great example is when we have large binary objects that are read seldomly, but metadata about them with a different extension that is read often (think, .mov and .xml). Now, it is easy to keep the metadata "hot" while saving costs with the bulk of data "cold" using a MatchesSuffix rule.

Cleanup of incomplete multipart uploads

In 2021, Google cloud storage launch the Multipart Upload protocol for XML API. This API give us more  smoother migration path by greatly improving with S3 compatibility. One common pain point with multipart uploads is that they sometimes get abandoned. Part storage isn't free, so pruning abandoned uploads is critical housekeeping. Now, we can "set it and forget it" with GCS Lifecycle support for incomplete multipart uploads.

These new lifecycle features are now generally available to all of Google cloud customers. They are supported through the GCS API, GCS GUI, gsutil, gcloud storage, and the client libraries.

0 replies

Be the first to reply!