Data aggregation and multiple retention periods

[jens-ylja]

Hello,

for my setup I have the following ideas regarding retention and aggregation:

• store data in full granularity (~10s interval) for about 3 months
• store data aggregated to e.g. min, max, avg (~5min interval) for one year
• store data aggregated to e.g. min, max, avg (1d interval) forever

The set of configuration properties lists a single tsdb.retention.threshold. So I assume my desired setup cannot be realized with a single ticktock instance.
Would it be the desired way to run multiple separate ticktock instances - one for each retention period - to achieve the goal?
Aggregation than can be done either while data rushes in or by periodically fetching from the higher granularity and writing to the lower granularity instances. Right?

Thanks

[ytyou]

Thanks, these are excellent ideas. To do what you wanted to do currently, you are right, you will have to run 3 separate TickTockDB instances. And as for aggregation (or downsampling), for now, you have to have client send data in different granularity to different TickTockDB instances, or you can fetch from higher granularity instances and downsample them yourself before writing to lower granularity instances. TickTockDB currently does not do downsampling on incoming data before writing, it can only do it while reading/querying.

[jens-ylja]

Thanks for fast reply.
I think I'll give this a try - will take a while to get it all working.
Once done I'll be back with a field report :)

[ylin30]

It is common in TSDBs that historical data are compacted in different granularity. The older the data the coarse the granularity. It is a great feature. Unfortunately we don't have time to do this yet. But let's keep it in our future feature list.

[ylin30]

#41

Recorded in /Idea. Let's see how many users vote for it.