Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add algorithm attribute to histogram field mapper. #108208

Open
martijnvg opened this issue May 2, 2024 · 1 comment
Open

Add algorithm attribute to histogram field mapper. #108208

martijnvg opened this issue May 2, 2024 · 1 comment

Comments

@martijnvg
Copy link
Member

martijnvg commented May 2, 2024

In Elasticsearch there are two histogram algorithms available: tdigest and hdr. The histogram field mapper is algorithm agnostic. A tdigest or hdr histogram can be stored in a field mapped as histogram. However the features using the histogram field like aggregations don't know what kind of histogram is stored in a histogram field. So it is possible to run a percentile aggregation configured with tdigest on a histogram field that stores a hdr histogram. The result that gets produced will not be useful at all. Ideally this shouldn't be allowed. But today this validation doesn't occur because the field mapper doesn't record what algorithm is used.

Additionally downsampling needs to know what algorithm is used, otherwise downsampling can't merge histograms. This is the reason why today downsampling doesn't support metric fields of type histogram.

The proposal is to add an algorithm attribute to the histogram field type. So that other features can make use of that. I don't think we can make this a required argument for bwc reasons from the start (but maybe many versions from now).

Relates to #74213

@martijnvg martijnvg added >bug >tech debt :StorageEngine/Mapping The storage related side of mappings labels May 2, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants