-
Notifications
You must be signed in to change notification settings - Fork 866
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TASK][MEDIUM] Resource isolation in Spark Scala mode #6265
Comments
Intersted in this, Please assign this task to me. |
@beryllw thanks, assgined |
Interesting task |
beryllw
added a commit
to beryllw/incubator-kyuubi
that referenced
this issue
Apr 18, 2024
beryllw
added a commit
to beryllw/incubator-kyuubi
that referenced
this issue
Apr 22, 2024
beryllw
added a commit
to beryllw/incubator-kyuubi
that referenced
this issue
Apr 22, 2024
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What's the level of this task?
MEDIUM
Code of Conduct
Search before creating
Mentor
Skill requirements
Background and Goals
The current Kyuubi Scala mode reuses the code of Spark Shell, while Spark Shell is designed for single session and client deploy mode, which has certain limitations for Kyuubis multi-tenant use cases.
Thanks to Spark Connect, a similar mechanism is required to support isolated sessions (i.e. isolated resources/classloader) so that each REPL user does not conflict with other users on the same cluster.
As of Spark 3.5, the core/sql already has infrastructure to support isolated resources/classloader, see details in
Implementation steps
Investigate how could Kyuubi Spark Scala mode leverage those infrastructures to support session-level isolated resources/classloader in Spark 3.5+
Additional context
Introduction of 2024H1 Kyuubi Code Contribution Program
The text was updated successfully, but these errors were encountered: