[SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4, Jersey to 3.1.11 and Servlet to 6.0 #53116

sarutak · 2025-11-18T17:29:03Z

What changes were proposed in this pull request?

This PR proposes to upgrade Jetty to 12.1.4, Jersey to 3.1.11 and Servlet to 6.0 which need to be upgraded together.
Because Jetty 12 is significantly changed from 11.x, this PR mainly considers following things to upgrade.

Redirect handler for http to https is rewritten using the standard Servlet API because most of the code can be reusable with Filter while it's difficult to do it with Jetty specific Core API.
ProxyRedirectHandler is rewritten using new Core API because HandlerWrapper is no longer available. It's difficult to implement ProxyRedirectHandler with the standard Servlet API so Core API is used.
In several places, the expected status code is changed from 302 to 301 to adapt Jetty's change.
- https://github.com/jetty/jetty.project/blob/4137e00793d583cfc4ce76bf607e482178a85597/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ContextHandler.java#L1205
In several places, consider the case where the value of location header is relative form to adapt Jetty's change.
- https://github.com/jetty/jetty.project/blob/4137e00793d583cfc4ce76bf607e482178a85597/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/Response.java#L415
- https://github.com/jetty/jetty.project/blob/4137e00793d583cfc4ce76bf607e482178a85597/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ContextHandler.java#L1260

Why are the changes needed?

Jetty 11 is already EOF and starting Jan 1, 2026, no more release will be published.
jetty/jetty.project#13918

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests with some tweaks.

Was this patch authored or co-authored using generative AI tooling?

No.

sarutak · 2025-11-18T17:31:39Z

resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/AmIpFilterSuite.scala

    override def setStatus(status: Int): Unit = this.status = status

-    override def setStatus(sc: Int, sm: String): Unit = {}
-


These methods are removed because Servlet 6.0 doesn't support them.
https://jakarta.ee/specifications/servlet/6.0/apidocs/jakarta.servlet/jakarta/servlet/http/httpservletresponse

sarutak · 2025-11-18T17:35:24Z

sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java

    if (queryString == null) {
      return null;
    }
-    Map<String, String[]> params = jakarta.servlet.http.HttpUtils.parseQueryString( queryString );


HttpUtils was deprecated in Servlet 5.0 and removed in 6.0.
https://jakarta.ee/specifications/servlet/5.0/apidocs/jakarta/servlet/http/httputils

Thank you, @sarutak . To minimize the PR, could you spin off this HttpUtils-related change PR?

@dongjoon-hyun
You mean, merge this part first in another PR and then merge this PR right?

Yes, I mean handling HttpUtils-deprecation first in the existing dependencies of master branch. This will help eventually this PR by reducing the complexity.

OK, I will open another one.

…pServlet.java` with `HttpServletRequest.getQueryString` ### What changes were proposed in this pull request? This PR aims to replace `HtmlUtils.parseQueryString` in `ThriftHttpServlet.java` with `HttpServletRequest.getQueryString` ### Why are the changes needed? `HttpUtils` was deprecated in Servlet 5.0 and removed in 6.0. https://jakarta.ee/specifications/servlet/5.0/apidocs/jakarta/servlet/http/httputils This change is necessary for upgrading Jetty to 12 in #53116 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53119 from sarutak/replace-http-utils. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2025-11-19T01:13:52Z

core/src/test/scala/org/apache/spark/ui/UISuite.scala

  }

-  test("SPARK-45522: Jetty 10 and above shouuld return status code 302 with correct redirect url" +
+  test("SPARK-47086: Jetty 12 and above should return status code 301 with correct redirect url" +


Oh, is this changed from 302 (Temporary Redirect) to 301 (Permanent Redirect)?

Yes. When a context root path is accessed without trailing slash, 301 is returned.

https://github.com/jetty/jetty.project/blob/4137e00793d583cfc4ce76bf607e482178a85597/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ContextHandler.java#L1205

https://github.com/jetty/jetty.project/blob/4137e00793d583cfc4ce76bf607e482178a85597/jetty-core/jetty-server/src/main/java/org/eclipse/jetty/server/handler/ContextHandler.java#L1259

While Servlet's HttpServletResponse.sendRedirect still returns 302.

https://github.com/jetty/jetty.project/blob/4137e00793d583cfc4ce76bf607e482178a85597/jetty-ee10/jetty-ee10-servlet/src/main/java/org/eclipse/jetty/ee10/servlet/ServletApiResponse.java#L175

dongjoon-hyun · 2025-11-19T01:16:12Z

dev/deps/spark-deps-hadoop-3-hive-2.3

-jersey-container-servlet/3.0.18//jersey-container-servlet-3.0.18.jar
-jersey-hk2/3.0.18//jersey-hk2-3.0.18.jar
-jersey-server/3.0.18//jersey-server-3.0.18.jar
+jersey-client/3.1.11//jersey-client-3.1.11.jar


Could you mention jersey version in the PR title because this is also important, @sarutak ?

LuciferYang · 2025-11-19T04:36:31Z

As far as I recall, when @HiuKwok previously attempted to migrate to Jetty 12, a problem with the sendRedirect capability was identified, and the records from that time are as follows:

[SPARK-47086][BUILD][CORE][SQL][UI] Migrate from Jetty 11 to Jetty 12 #45500 (comment)

Has this issue been resolved now?

pan3793 · 2025-11-19T05:17:04Z

Jetty 11 is already EOF and starting Jan 1, 2026, no more release will be published.
jetty/jetty.project#13918

It sounds like a shocking news, @dongjoon-hyun, I understand 4.1 has been feature frozen, do you think we can have an exception for this one, otherwise, that means Spark 4.1 has no workaround during its 18 months lifecycle if Jetty 11 exposes new CVEs.

sarutak · 2025-11-19T05:19:14Z

@LuciferYang

Has this issue been resolved now?

Yes. Otherwise, this test should fail because of the following reason:

Access to ~~https://proxy.example.com:443/prefix/ctx1~~ http://$localhost:${serverInfo.boundPort}/ctx1 where ~~https://proxy.example.com:443/prefix/ is a proxy root and~~ ctx1 is a context path root.
Jetty returns status code 301 without rewriting location header. So the location header will be /ctx/ while https://proxy.example.com:443/prefix/ctx/ is expected where https://proxy.example.com:443/prefix is a proxy root.

pan3793 · 2025-11-19T05:56:31Z

core/src/main/scala/org/apache/spark/ui/JettyUtils.scala

+    val redirectServer = if (server.contains(":") && !server.startsWith("[")) {
+      s"[$server]"
+    } else {
+      server
+    }


this can be simplified by using Utils.addBracketsIfNeeded(server)

This part is just brought from the original code. Unless we need to refactor this part within this PR, I'd like to complete upgrading first and then open a followup PR.

was not aware of that, it's fine then.

pan3793 · 2025-11-19T06:10:31Z

core/src/test/scala/org/apache/spark/ui/UISuite.scala

    when(clientRequest.getScheme()).thenReturn("http")
    when(clientRequest.getHeader("host")).thenReturn(s"$localhost:8080")
-    when(clientRequest.getPathInfo()).thenReturn("/proxy/worker-id/jobs")
+    when(clientRequest.getPathInfo()).thenReturn("/worker-id/jobs")


Why is this change needed?

Good question. This change is relevant to this part.

/proxy is a context root so the returned value from HttpServletRequest#getPathInfo should not contain /proxy. In old versions of Jetty, the returned value can strangely contain a context root.
As of Jetty 12, Jetty specific Core API and the standard Servlet API are decoupled and getPathInfo doesn't return a path with a context root (/proxy in this case).

thanks for the explanation, the Jetty 12 behavior sounds more reasonable, do you think it is worth leaving some comment (at def createProxyLocationHeader, not here) to briefly explain such behavior changes?

I'm OK to leave a comment. I'll do it and your another suggestion together in a followup PR.

pan3793

Have manually verified with YARN integration, proxy works as expected.

$ spark-sql
WARNING: Using incubator modules: jdk.incubator.vector
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2025-11-20 06:50:36 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark Web UI available at http://hadoop-master1.orb.local:4040
Spark master: yarn, Application Id: application_1763621405849_0001
spark-sql (default)>

Access http://hadoop-master1.orb.local:4040 and it is correctly redirected to the YARN RM address.

sarutak · 2025-11-20T13:19:34Z

@pan3793 Thank you for verifying!

github-actions bot added SQL STRUCTURED STREAMING WEB UI BUILD SPARK SHELL YARN CORE DSTREAM labels Nov 18, 2025

sarutak commented Nov 18, 2025

View reviewed changes

sarutak mentioned this pull request Nov 18, 2025

[SPARK-54400][SQL] Replace HtmlUtils.parseQueryString in ThriftHttpServlet.java with HttpServletRequest.getQueryString #53119

Closed

sarutak added 2 commits November 19, 2025 09:26

Upgrade Jetty to 12.1.4

9694ad8

Tweak

451d794

sarutak force-pushed the upgrade-jetty branch from ac779f7 to 451d794 Compare November 19, 2025 00:27

dongjoon-hyun reviewed Nov 19, 2025

View reviewed changes

sarutak changed the title ~~[SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4~~ [SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4 and Jersey to 3.1.11 Nov 19, 2025

sarutak changed the title ~~[SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4 and Jersey to 3.1.11~~ [SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4, Jersey to 3.1.11 and Servlet to 6.0 Nov 19, 2025

pan3793 reviewed Nov 19, 2025

View reviewed changes

pan3793 approved these changes Nov 20, 2025

View reviewed changes

		override def setStatus(status: Int): Unit = this.status = status

		override def setStatus(sc: Int, sm: String): Unit = {}

[SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4, Jersey to 3.1.11 and Servlet to 6.0 #53116

Are you sure you want to change the base?

[SPARK-47086][SPARK-48022][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4, Jersey to 3.1.11 and Servlet to 6.0 #53116

Uh oh!

Conversation

sarutak commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Nov 19, 2025

Uh oh!

pan3793 commented Nov 19, 2025

Uh oh!

sarutak commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarutak Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

sarutak commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sarutak commented Nov 18, 2025 •

edited

Loading

sarutak commented Nov 19, 2025 •

edited

Loading

sarutak Nov 19, 2025 •

edited

Loading