Commit 4d1c79f
[SPARK-54472][SQL] Add ORC read and write support for TIME type
### What changes were proposed in this pull request?
This PR adds ORC serialization and deserialization support for Spark's TIME type.
### Why are the changes needed?
TIME type currently lacks ORC support, preventing users from:
- Reading/writing ORC files with TIME columns
- Integrating TIME data with existing ORC-based data lakes
- Preserving TIME precision in columnar storage
### Does this PR introduce _any_ user-facing change?
Yes. Users can now:
1. Read ORC with TIME columns
```scala
spark.read.format("orc").load("data.orc")
// Returns DataFrame with TIME columns preserved
```
2. Write DataFrames with TIME to ORC
```scala
val df = spark.sql("SELECT TIME'14:30:45.123456' as shift_start")
df.write.format("orc").save("output.orc")
```
### Technical Details
#### Storage Format
```
ORC Column:
Physical Type: LONG (nanoseconds since midnight)
Custom Attribute: spark.sql.catalyst.type = "time(<precision>)"
Value Range: 0 to 86,399,999,999,999
```
#### Precision Handling
| Precision | Catalyst Type | ORC Attribute | Example Value |
|-----------|---------------|---------------|---------------|
| 0 (seconds) | `TimeType(0)` | `"time(0)"` | `12:34:56` |
| 3 (millis) | `TimeType(3)` | `"time(3)"` | `12:34:56.123` |
| 6 (micros) | `TimeType(6)` | `"time(6)"` | `12:34:56.123456` |
***Future Compatibility***
- Versioned via file metadata: Uses existing `org.apache.spark.version` for compatibility
- Forward-compatible: If ORC adds native TIME type, can migrate based on version
### How was this patch tested?
Added tests in `OrcQuerySuite`
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Generated-by: Claude 3.5 Sonnet
AI assistance was used for:
- Code pattern analysis and design discussions
- Implementation guidance following Spark conventions
- Test case generation and organization
- Documentation and examples
Closes #53185 from vinodkc/br_time_orc_read_write.
Authored-by: vinodkc <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>1 parent 0fa72f7 commit 4d1c79f
File tree
6 files changed
+57
-6
lines changed- sql/core/src
- main/scala/org/apache/spark/sql/execution/datasources/orc
- test/scala/org/apache/spark/sql
- execution/datasources/orc
6 files changed
+57
-6
lines changedLines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
129 | | - | |
130 | | - | |
| 129 | + | |
| 130 | + | |
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
| |||
Lines changed: 0 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
252 | | - | |
253 | 252 | | |
254 | 253 | | |
255 | 254 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
| 101 | + | |
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| |||
Lines changed: 5 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
282 | 282 | | |
283 | 283 | | |
284 | 284 | | |
285 | | - | |
| 285 | + | |
286 | 286 | | |
287 | 287 | | |
288 | 288 | | |
| |||
302 | 302 | | |
303 | 303 | | |
304 | 304 | | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
305 | 309 | | |
306 | 310 | | |
307 | 311 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1247 | 1247 | | |
1248 | 1248 | | |
1249 | 1249 | | |
1250 | | - | |
| 1250 | + | |
1251 | 1251 | | |
1252 | 1252 | | |
1253 | 1253 | | |
| |||
Lines changed: 48 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
899 | 899 | | |
900 | 900 | | |
901 | 901 | | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
902 | 950 | | |
903 | 951 | | |
904 | 952 | | |
| |||
0 commit comments