Forbidden Exception creating Polaris Rest catalog with Flink 1.20

[David-N-Perkins]

Apache Iceberg version

1.7.1 (latest release)

Query engine

Flink

Please describe the bug 🐞

We attempted to upgrade Iceberg 1.6.1 and Flink 1.18.1 to Iceberg 1.7.1 and Flink 1.20.0. When creating the Iceberg catalog in Flink SQL, we now encounter this exception.

Could not execute SQL statement. Reason:
org.apache.iceberg.exceptions.ForbiddenException: Forbidden:

Here is the create catalog statement.

CREATE CATALOG c_iceberg WITH (
  'type'='iceberg',
  'catalog-type'='rest',
  'uri'='http://polaris:8181/api/catalog',
  'credential'='xxx:xxx',
  'warehouse'='azure-catalog',
  'scope'='PRINCIPAL_ROLE:ALL'
);

I'm not positive, but believe the issue is with Iceberg and not Polaris since we didn't change the Polaris version and it still works fine with the older version.

Here are the logs from Flink.

2024-12-19 21:09:16,010 WARN  org.apache.iceberg.rest.RESTSessionCatalog                   [] - Iceberg REST client is m
issing the OAuth2 server URI configuration and defaults to http://polaris:8181/api/catalogv1/oauth/tokens. This automati
c fallback will be removed in a future Iceberg release.It is recommended to configure the OAuth2 endpoint using the 'oau
th2-server-uri' property to be prepared. This warning will disappear if the OAuth2 endpoint is explicitly configured. Se
e https://github.com/apache/iceberg/issues/10537
2024-12-19 21:09:16,017 WARN  org.apache.iceberg.rest.ErrorHandlers                        [] - Unable to parse error re
sponse
java.io.UncheckedIOException: org.apache.iceberg.shaded.com.fasterxml.jackson.databind.exc.MismatchedInputException: No
content to map due to end-of-input
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1]
        at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:101) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.responses.ErrorResponseParser.fromJson(ErrorResponseParser.java:71) ~[iceberg-flink-r
untime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.parseResponse(ErrorHandlers.java:194) ~[iceberg-fli
nk-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:181) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?
]
        at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.fetchConfig(RESTSessionCatalog.java:1045) ~[iceberg-flink-runtime-
1.20-1.7.1.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:251) ~[iceberg-flink-runtime-1.
20-1.7.1.jar:?]
        at org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:78) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:274) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.flink.CatalogLoader$RESTCatalogLoader.loadCatalog(CatalogLoader.java:163) ~[iceberg-flink-
runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.flink.FlinkCatalog.<init>(FlinkCatalog.java:114) ~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:166) ~[iceberg-flink-runt
ime-1.20-1.7.1.jar:?]
        at org.apache.iceberg.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:139) ~[iceberg-flink-runt
ime-1.20-1.7.1.jar:?]
        at org.apache.flink.table.factories.FactoryUtil.createCatalog(FactoryUtil.java:492) ~[flink-table-api-java-uber-
1.20.0.jar:1.20.0]
        at org.apache.flink.table.catalog.CatalogManager.initCatalog(CatalogManager.java:368) ~[flink-table-api-java-ube
r-1.20.0.jar:1.20.0]
        at org.apache.flink.table.catalog.CatalogManager.createCatalog(CatalogManager.java:322) ~[flink-table-api-java-u
ber-1.20.0.jar:1.20.0]
        at org.apache.flink.table.operations.ddl.CreateCatalogOperation.execute(CreateCatalogOperation.java:88) ~[flink-
table-api-java-uber-1.20.0.jar:1.20.0]

It appears that Flink is able to authenticate and get an access token from Polaris. But when attempting to fetch the Polaris configuration, a 403 response is received, and the body can't be parsed into JSON.

I also checked the Polaris logs.

WARN  [2024-12-19 21:09:16,008 - 171371485] [pool-3-thread-11] [] o.a.p.s.c.DefaultContextResolver: Failed to parse prin
cipal from headers ({x-request-id=a5a35de0-dcb6-40d0-8cc4-a66cba92f077, x-b3-parentspanid=730e3221997844e4, Accept=appli
cation/json, x-client-version=Apache Iceberg 1.7.1 (commit 4a432839233f2343a9eae8255532f911f06358ef), x-b3-sampled=0, Us
er-Agent=Apache-HttpClient/5.4 (Java/11.0.25), X-Forwarded-Proto=http, Host=polaris:8181, Accept-Encoding=gzip, x-gzip,
deflate, x-envoy-attempt-count=1, x-client-git-commit-short=4a43283, x-b3-traceid=43748bd2c2e73bc3730e3221997844e4, x-b3
-spanid=4f6cc905a0d0dbe1, x-forwarded-client-cert=By=spiffe://cluster.local/ns/data-platform/sa/polaris;Hash=55045c2d814
4e5d0718e2706502ac713201f42ec44b799fd508674c475b49465;Subject="";URI=spiffe://cluster.local/ns/data-platform/sa/flink, C
ontent-Length=116, Content-Type=application/x-www-form-urlencoded}); using default-principal
INFO  [2024-12-19 21:09:16,008 - 171371485] [pool-3-thread-11] [] o.a.p.s.tracing.TracingFilter: Started span with paren
t spanId="208f3f42abfc42ee" traceId="170feb2185813eb2f25a5c3306ced564" parentContext="{}"
WARN  [2024-12-19 21:09:16,008 - 171371485] [pool-3-thread-11 - POST /api/catalog/v1/oauth/tokens] [] o.g.j.servlet.WebC
omponent: A servlet request to the URI http://polaris:8181/api/catalog/v1/oauth/tokens contains form parameters in the r
equest body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters.
Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means
 will not work as expected.
[EL Fine]: sql: 2024-12-19 21:09:16.008--ServerSession(214651918)--Connection(166093349)--SELECT PRINCIPALCLIENTID, MAIN
SECRET, MAINSECRETHASH, PRINCIPALID, SECONDARYSECRET, SECONDARYSECRETHASH, SECRETSALT, VERSION FROM PRINCIPAL_SECRETS WH
ERE (PRINCIPALCLIENTID = ?)
        bind => [root]
[EL Fine]: sql: 2024-12-19 21:09:16.009--ServerSession(214651918)--Connection(166093349)--SELECT CATALOGID, ID, CREATETI
MESTAMP, DROPTIMESTAMP, ENTITYVERSION, GRANTRECORDSVERSION, INTERNALPROPERTIES, LASTUPDATETIMESTAMP, NAME, PARENTID, PRO
PERTIES, PURGETIMESTAMP, SUBTYPECODE, TOPURGETIMESTAMP, TYPECODE, VERSION FROM ENTITIES WHERE ((CATALOGID = ?) AND (ID =
 ?))
        bind => [0, 151]
INFO  [2024-12-19 21:09:16,010 - 171371487] [pool-3-thread-11] [] i.o.e.l.LoggingSpanExporter: 'POST /api/catalog/v1/oau
th/tokens' : 170feb2185813eb2f25a5c3306ced564 208f3f42abfc42ee SERVER [tracer: /api/catalog/v1/oauth/tokens:] Attributes
Map{data={url.path=/api/catalog/v1/oauth/tokens, server.address=polaris, http.request.method=POST, realm=default-realm,
url.scheme=http}, capacity=128, totalAddedValues=5}

I don't see anything in the Polaris logs related to fetching the configuration, which makes me think there is something wrong with the request that causes the HTTP server to reject it.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time