Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-48175][SQL][PYTHON] Store collation information in metadata and not in type for SER/DE #46280

Closed
wants to merge 46 commits into from
Closed

[SPARK-48175][SQL][PYTHON] Store collation information in metadata and not in type for SER/DE #46280

wants to merge 46 commits into from

Commits on Apr 27, 2024

  1. initial impl of new delta schema

    @stefankandic
    stefankandic committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    8edc5ea View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. some improvements

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    2f5a1b8 View commit details
    Browse the repository at this point in the history

  2. formatting improvements

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    8f6856b View commit details
    Browse the repository at this point in the history

  3. minor changes

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    af41b50 View commit details
    Browse the repository at this point in the history

  4. small refactoring

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    001b9e9 View commit details
    Browse the repository at this point in the history

  5. remove bogus comment

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e65285b View commit details
    Browse the repository at this point in the history

  6. remove bogus comment

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    77e4103 View commit details
    Browse the repository at this point in the history

  7. fix scalastyle

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e4a205b View commit details
    Browse the repository at this point in the history

  8. use string as path instead of list of strings

    @stefankandic
    stefankandic committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    bd600da View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. add tests for schema ser/de

    @stefankandic
    stefankandic committed May 7, 2024
    Configuration menu
    Copy the full SHA
    19f5588 View commit details
    Browse the repository at this point in the history

  2. add python impl and tests

    @stefankandic
    stefankandic committed May 7, 2024
    Configuration menu
    Copy the full SHA
    3a7e8f6 View commit details
    Browse the repository at this point in the history

  3. merge with latest master

    @stefankandic
    stefankandic committed May 7, 2024
    Configuration menu
    Copy the full SHA
    79e5aaa View commit details
    Browse the repository at this point in the history

  4. reformat python

    @stefankandic
    stefankandic committed May 7, 2024
    Configuration menu
    Copy the full SHA
    6ebaddf View commit details
    Browse the repository at this point in the history

  5. update method calls

    @stefankandic
    stefankandic committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b488346 View commit details
    Browse the repository at this point in the history

  6. tests passing

    @stefankandic
    stefankandic committed May 7, 2024
    Configuration menu
    Copy the full SHA
    7747947 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. fix equality for struct fields

    @stefankandic
    stefankandic committed May 8, 2024
    Configuration menu
    Copy the full SHA
    0315421 View commit details
    Browse the repository at this point in the history

  2. rename variables

    @stefankandic
    stefankandic committed May 8, 2024
    Configuration menu
    Copy the full SHA
    3b9ad23 View commit details
    Browse the repository at this point in the history

  3. fix python tests

    @stefankandic
    stefankandic committed May 8, 2024
    Configuration menu
    Copy the full SHA
    913095c View commit details
    Browse the repository at this point in the history

  4. revert changes to collationsuite

    @stefankandic
    stefankandic committed May 8, 2024
    Configuration menu
    Copy the full SHA
    f48aa27 View commit details
    Browse the repository at this point in the history

  5. fix pyspark-connect and streaming test

    @stefankandic
    stefankandic committed May 8, 2024
    Configuration menu
    Copy the full SHA
    9283b76 View commit details
    Browse the repository at this point in the history

  6. fix python linter

    @stefankandic
    stefankandic committed May 8, 2024
    Configuration menu
    Copy the full SHA
    72cde10 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. fix python linter

    @stefankandic
    stefankandic committed May 9, 2024
    Configuration menu
    Copy the full SHA
    eb9be6d View commit details
    Browse the repository at this point in the history

  2. add test for map field with .key suffix in the name

    @stefankandic
    stefankandic committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b9021c9 View commit details
    Browse the repository at this point in the history

  3. fix mypy type annotation

    @stefankandic
    stefankandic committed May 9, 2024
    Configuration menu
    Copy the full SHA
    ac0aa1d View commit details
    Browse the repository at this point in the history

  4. improve key in name test

    @stefankandic
    stefankandic committed May 9, 2024
    Configuration menu
    Copy the full SHA
    a8a497f View commit details
    Browse the repository at this point in the history

  5. try to fix mypy errors

    @stefankandic
    stefankandic committed May 9, 2024
    Configuration menu
    Copy the full SHA
    9089f9b View commit details
    Browse the repository at this point in the history

  6. fix mypy

    @stefankandic
    stefankandic committed May 9, 2024
    Configuration menu
    Copy the full SHA
    c34ae9d View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. add provider to schema as well to follow delta protocol

    @stefankandic
    stefankandic committed May 13, 2024
    Configuration menu
    Copy the full SHA
    9cf5b04 View commit details
    Browse the repository at this point in the history

  2. Update python/pyspark/sql/types.py 

    Co-authored-by: Christos Stavrakakis <[email protected]>
    @stefankandic @cstavr
    stefankandic and cstavr committed May 13, 2024
    Configuration menu
    Copy the full SHA
    7cde276 View commit details
    Browse the repository at this point in the history

  3. resolving pr comments

    @stefankandic
    stefankandic committed May 13, 2024
    Configuration menu
    Copy the full SHA
    9e6afe0 View commit details
    Browse the repository at this point in the history

  4. add tests for collation metadata on non string types

    @stefankandic
    stefankandic committed May 13, 2024
    Configuration menu
    Copy the full SHA
    b8626fa View commit details
    Browse the repository at this point in the history

  5. add additional assert for schema check

    @stefankandic
    stefankandic committed May 13, 2024
    Configuration menu
    Copy the full SHA
    10562d4 View commit details
    Browse the repository at this point in the history

  6. remove leftover test

    @stefankandic
    stefankandic committed May 13, 2024
    Configuration menu
    Copy the full SHA
    97631c0 View commit details
    Browse the repository at this point in the history

  7. add checks for collations on wrong type in python

    @stefankandic
    stefankandic committed May 13, 2024
    Configuration menu
    Copy the full SHA
    33ed245 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. add missing docstring and fix mypy error

    @stefankandic
    stefankandic committed May 14, 2024
    Configuration menu
    Copy the full SHA
    2b0a799 View commit details
    Browse the repository at this point in the history

  2. fix condition for valid types

    @stefankandic
    stefankandic committed May 14, 2024
    Configuration menu
    Copy the full SHA
    aff7b58 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. respond to pr comments

    @stefankandic
    stefankandic committed May 15, 2024
    Configuration menu
    Copy the full SHA
    c35d781 View commit details
    Browse the repository at this point in the history

  2. add more tests

    @stefankandic
    stefankandic committed May 15, 2024
    Configuration menu
    Copy the full SHA
    6e379aa View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. make provider lowercase

    @stefankandic
    stefankandic committed May 16, 2024
    Configuration menu
    Copy the full SHA
    0879841 View commit details
    Browse the repository at this point in the history

  2. fix failing test

    @stefankandic
    stefankandic committed May 16, 2024
    Configuration menu
    Copy the full SHA
    1ae29a8 View commit details
    Browse the repository at this point in the history

  3. fix mypy

    @stefankandic
    stefankandic committed May 16, 2024
    Configuration menu
    Copy the full SHA
    fdb0a7e View commit details
    Browse the repository at this point in the history

  4. fix mypy

    @stefankandic
    stefankandic committed May 16, 2024
    Configuration menu
    Copy the full SHA
    c123a4b View commit details
    Browse the repository at this point in the history

  5. minor changes

    @stefankandic
    stefankandic committed May 16, 2024
    Configuration menu
    Copy the full SHA
    eb461b3 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into newDeltaSchema

    @stefankandic
    stefankandic committed May 16, 2024
    Configuration menu
    Copy the full SHA
    1fb404b View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. fix streaming deduplication collation type

    @stefankandic
    stefankandic committed May 17, 2024
    Configuration menu
    Copy the full SHA
    51a9bd6 View commit details
    Browse the repository at this point in the history

  2. make version getter return optional

    @stefankandic
    stefankandic committed May 17, 2024
    Configuration menu
    Copy the full SHA
    379b55a View commit details
    Browse the repository at this point in the history