Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce int32 and int64 types in order to easier transform flow entries into parquet/avro types #771

Open
stloyd opened this issue Nov 8, 2023 · 2 comments
Labels

Comments

@stloyd
Copy link
Member

stloyd commented Nov 8, 2023

we might need to introduce here int32 and int64 in order to easier transform flow entries into parquet/avro types

Originally posted by @norberttech in #770 (comment)

@stloyd stloyd changed the title Introduce here int32 and int64 in order to easier transform flow entries into parquet/avro types Introduce int32 and int64 types in order to easier transform flow entries into parquet/avro types Nov 8, 2023
@stloyd stloyd added this to the 0.6.0 milestone Nov 15, 2023
@stloyd stloyd added the core label Nov 15, 2023
@stloyd stloyd self-assigned this Nov 19, 2023
@stloyd stloyd moved this from Todo to In Progress in Roadmap Nov 19, 2023
@norberttech norberttech moved this from In Progress to On Hold in Roadmap Nov 20, 2023
@norberttech
Copy link
Member

I'm putting this on hold for now. I still think it's necessary to introduce this int32/int64, but I don't want to complicate current developer experience without strong and solid use cases.

@stloyd stloyd removed this from the 0.6.0 milestone Nov 20, 2023
@stloyd stloyd removed their assignment Nov 20, 2023
@norberttech norberttech moved this from On Hold to Todo in Roadmap Jan 27, 2024
@norberttech
Copy link
Member

One quick idea is that for formats like parquet/avro where type length is important, we can use schema metadata.
In most cases, the schema is inferred from the dataset but when passed explicitly it could also carry additional metadata which could help us to choose a better destination type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Todo
Development

Successfully merging a pull request may close this issue.

2 participants