Version: 2.11.x

# Overview of tiered storage

Pulsar's Tiered Storage feature allows older backlog data to be moved from BookKeeper to long-term and cheaper storage, while still allowing clients to access the backlog as if nothing has changed.

tip

The AWS S3 offloader registers specific AWS metadata, such as regions and service URLs and requests bucket location before performing any operations. If you cannot access the Amazon service, you can use the S3 offloader instead since it is an S3 compatible API without the metadata.

## When to use tiered storage?​

Tiered storage should be used when you have a topic for which you want to keep a very long backlog for a long time.

For example, if you have a topic containing user actions that you use to train your recommendation systems, you may want to keep that data for a long time, so that if you change your recommendation algorithm, you can rerun it against your full user history.

## How to install tiered storage offloaders?​

wget https://archive.apache.org/dist/pulsar/pulsar-2.11.0/apache-pulsar-offloaders-2.11.0-bin.tar.gz

After you download the tarball, untar the offloaders package and copy the offloaders as offloaders in the pulsar directory:

tar xvfz apache-pulsar-offloaders-2.11.0-bin.tar.gzmv apache-pulsar-offloaders-2.11.0/offloaders offloadersls offloaders# tiered-storage-file-system-2.11.0.nar# tiered-storage-jcloud-2.11.0.nar

For more information on how to configure tiered storage, see Tiered storage cookbook.

note
• If you are running Pulsar in a bare metal cluster, make sure that offloaders tarball is unzipped in every broker's pulsar directory.
• If you are running Pulsar in Docker or deploying Pulsar using a docker image (e.g. K8S), you can use the apachepulsar/pulsar-all image instead of the apachepulsar/pulsar image. apachepulsar/pulsar-all image has already bundled tiered storage offloaders.

## How does tiered storage work?​

A topic in Pulsar is backed by a log, known as a managed ledger. This log is composed of an ordered list of segments. Pulsar only writes to the final segment of the log. All previous segments are sealed. The data within the segment is immutable. This is known as a segment-oriented architecture.

The tiered storage offloading mechanism takes advantage of the segment-oriented architecture. When offloading is requested, the segments of the log are copied one by one to tiered storage. All segments of the log (apart from the current segment) written to tiered storage can be offloaded.

Data written to BookKeeper is replicated to 3 physical machines by default. However, once a segment is sealed in BookKeeper, it becomes immutable and can be copied to long-term storage. Long-term storage has the potential to achieve significant cost savings.