Is Tiered Storage currently planned?

[shafqatevo]

Hi,

Couldn't find Tiered Storage in the roadmap (https://nats.io/about/#roadmap). Can anyone pls share what's the current plan for this much-needed feature? What timeframe can it be expected?

[derekcollison]

We have it planned but no schedule yet on when we will do this work as of yet.

[Material-Scientist]

Since Jetstream stores streams as 8MB blocks, maybe a simple interim solution would just be to have a sidecar that watches for completed blocks, copies them to a remote directory, and then replaces the blocks in the original directory with symlinks that point to the files in the remote directory?

Would obviously prefer a native solution by NATS maintainers for prod though.

[hpvd]

yep this is interesting.

Or is there an other way:

would it be possible to do not have tiered storage directly, but similar to KV-Store, a new "type" which consists of two components:

• a connection to an external/S3 storage
• a mechanism which offloads (copy and delete /or mark as delete first) a stream depending on a reason to be specified (e.g. message age / TTL, number of included messages, total size...)

[hpvd]

looks like this may be related to:
TTL support in KV Store: #3251
S3 compatibility for object store #4871

[christhegrand]

We would also love this. We wrote a custom solution to periodically offload data from Jetstream into Google Cloud Storage, but would love to see this functionality built into Jetstream itself.

[hpvd]

please see discussion "S3 next level" for a general approach with further advantage... #6478