Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: writes with incorrect schema should fail #25022

Merged
merged 3 commits into from
May 29, 2024
Merged

Conversation

hiltontj
Copy link
Contributor

@hiltontj hiltontj commented May 24, 2024

Summary of Changes

  • Added a reproducer to replicate the scenario described in backtrace when writing different integer schema #25006
  • Added a step in the write path to validate fields against the existing schema, to ensure that the fields being written have the same types as the existing schema
  • Note: this incurs an extra check against each field in each line written when accept_partial=true; if this degrades performance, we could introduce a parameter that turns this check off, e.g., validate_schema, for users that don't want schema validation.

Example API Response

{
  "error":"partial write of line protocol occurred",
  "data":[
    {
      "original_line":"t0,t0_tag0=initTag t0_f0=0u 1715694000",
      "line_number":1,
      "error_message":"invalid field value in line protocol for field 't0_f0' on line 0: expected type i64, but got u64"
    },
    {
      "original_line":"t0,t0_tag0=initTag t0_f0=1u 1715694001",
      "line_number":2,
      "error_message":"invalid field value in line protocol for field 't0_f0' on line 1: expected type i64, but got u64"
    }
  ]
}

Closes #25006

@hiltontj hiltontj added the v3 label May 24, 2024
@hiltontj hiltontj self-assigned this May 24, 2024
@hiltontj hiltontj force-pushed the hiltontj/25006-writes-bug branch 2 times, most recently from 8709a85 to 119899f Compare May 24, 2024 15:31
@hiltontj hiltontj marked this pull request as ready for review May 24, 2024 17:36
@hiltontj hiltontj merged commit faab7a0 into main May 29, 2024
12 checks passed
@hiltontj hiltontj deleted the hiltontj/25006-writes-bug branch May 29, 2024 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

backtrace when writing different integer schema
2 participants