Fix data corruption in remote write if max_sample_age is applied #14078
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
/fixes #13979
The PR #13002 added the option to drop old samples in remote write.
Unfortunatelly we bumped into a bug causing data to be corrupted (metrics withlabels getting merged with other metrics labels was the most obvious) reported here #13979
after trying to repproduce the issue it showed up it is strictly connected to situations, when retries does happen and the
filter
inbuildTimeSeries
is applied.We did investigate and it appears that the issue is in the newly added
buildTimeSeries
function which does modify thetimeSeries
argument causing the corruption.The suggested change, which avoids modification of the original timeSeries seems to fix the issue, but is naive and not sure how optimal.