[SPARK-54446][ML] FPGrowth supports local filesystem #53150
+121
−20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
approved these changes
Nov 21, 2025
| Using.resource( | ||
| new ObjectInputStream(new BufferedInputStream(new FileInputStream(path))) | ||
| ) { ois => | ||
| val schema = ois.readObject().asInstanceOf[StructType] |
Contributor
There was a problem hiding this comment.
This uses Java deserializer which seems unsafe (risk of Remote Code Execution)
Related commit: #50922
Contributor
Author
There was a problem hiding this comment.
resort to arrow format suggested by @cloud-fan
WeichenXu123
requested changes
Nov 21, 2025
Contributor
WeichenXu123
left a comment
WeichenXu123
left a comment
There was a problem hiding this comment.
We need to address the RCE issue :)
zhengruifeng
force-pushed
the
local_fs_fpg
branch
from
b0d6e92 to
7a3db44
Compare
nit
zhengruifeng
force-pushed
the
local_fs_fpg
branch
from
cd9a177 to
8fe0d6e
Compare
cloud-fan
reviewed
Nov 21, 2025
| val spark = df.sparkSession | ||
| val schema = df.schema | ||
| val maxRecordsPerBatch = spark.sessionState.conf.arrowMaxRecordsPerBatch | ||
| df.queryExecution.executedPlan.execute().mapPartitionsInternal { iter => |
Contributor
There was a problem hiding this comment.
Dataset already has def toArrowBatchRdd, shall we reuse it?
cloud-fan
reviewed
Nov 21, 2025
| val schema: StructType = df.schema | ||
| dos.writeUTF(schema.json) | ||
|
|
||
| val iter = DatasetUtils.toArrowBatchRDD(df, "UTC").toLocalIterator |
Contributor
There was a problem hiding this comment.
does the arrow library provide APIs to write to local file?
holdenk
requested changes
Nov 21, 2025
Contributor
holdenk
left a comment
holdenk
left a comment
There was a problem hiding this comment.
Arrow isn't intended for long term storage it's intended as a wire protocol -- I don't love using it for persisting models. I'm -0.9 on this change for now. Parquet seems like a better choice most likely.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
FPGrowth supports local filesystem
Why are the changes needed?
to make FPGrowth work with local filesystem
Does this PR introduce any user-facing change?
yes, FPGrowth will work when local saving mode is one
How was this patch tested?
updated tests
Was this patch authored or co-authored using generative AI tooling?
no