Core: Add geometry bounds collector - #17509
Conversation
5ce6fc4 to
f5a4199
Compare
37d325b to
cfb4c00
Compare
2f7b7a1 to
a2e9b21
Compare
- Accumulate two-dimensional bounds across the seven OGC WKB geometry types - Reject unsupported dimensions and malformed structures Generated-by: Codex
a2e9b21 to
12bc49c
Compare
szehon-ho
left a comment
There was a problem hiding this comment.
Thanks for splitting this out — the WKB walk is carefully bounded and the NaN/empty semantics match format/spec.md:772 exactly, including the per-dimension skip and suppressing the box when X or Y is missing.
The main thing I'd like to resolve is that rejecting XYZ/XYM/XYZM fails the write, since those are spec-legal geometries. Beyond that: a placement question on the new public surface in api/, a documentation request on the polygon shell assumption, and some minor notes. Details inline.
One broader question that isn't inline: Parquet 1.17.1 already computes a geospatial bounding box itself for columns annotated GEOMETRY/GEOGRAPHY (ColumnWriterBase threads GeospatialStatistics through writePage, and ColumnChunkMetaData.getGeospatialStatistics() exposes it in the footer, which is where ParquetMetrics already reads stats from). Since TypeToMessageType emits those annotations today, the Parquet path in #17161 would be parsing every WKB value a second time. An Iceberg-side parser is still clearly justified for the Avro path (#17119), so this isn't an argument against the class — but could the PR description say why the Iceberg-side scan is preferred over the footer for Parquet? Worth coordinating with #12347 too, which covers overlapping ground.
Two questions that didn't fit inline:
Should invalid-but-parseable WKB fail the write at all? Rejecting a LINESTRING child inside a MULTIPOINT is defensible, but together with the Z/M rejection there's a pattern of aborting writes over input that other producers accept.
How will geography bounds work? format/spec.md:770 restricts geography bounds to [-180..180]/[-90..90] and permits xmin > xmax for a tighter antimeridian-crossing box. A plain XY min/max is valid but not minimal, and will fall outside the canonical ranges if the input longitudes do. Separate collector, or a mode on this one?
| long typeCode = buffer.getInt() & 0xFFFFFFFFL; | ||
| int geometryType = (int) (typeCode % 1000); | ||
| Preconditions.checkArgument( | ||
| typeCode / 1000 == 0, "Unsupported WKB: only 2D geometries are supported"); |
There was a problem hiding this comment.
Suggest not failing on Z/M. Either read past the extra ordinates and keep the XY bounds — the numDimensions(dimensionGroup, typeCode) handling the earlier version of this code had in #17161 — or, if you'd rather keep the parser strictly 2D, mark the column's bounds unavailable for the file instead of throwing. The first is better since you still get usable bounds.
Z/M geometries are legal Iceberg data: Appendix G specifies the ISO WKB serializations "supporting XY, XYZ, XYM, XYZM", and the bounds section makes Z and M bounds optional rather than constraining input dimensionality. So POINT Z (1 2 3) is valid, but typeCode / 1000 == 0 rejects it.
As written the rejection fails the write rather than dropping bounds. In #17161, GeometryWriter.write() calls metricsBuilder.addValue(buffer) as its first statement with no try/catch, and ParquetWriter.add(T) calls model.write(0, value) unguarded, so the IllegalArgumentException propagates out of FileAppender.add() and aborts the task — a Spark INSERT into a table with 3D geometry would fail the job.
One caveat if you take the second route: it has to invalidate the whole column, not just skip the offending value. Skipping the value would leave bounds accumulated from the other values, which no longer contain every object in the file — format/spec.md:768 requires the box to contain all objects — and under-covering bounds get the file pruned for queries that should match it. Omitting bounds entirely is always safe, since metrics are optional. Parquet's own GeospatialStatistics does exactly this with abort()/isValid(), which may be a useful model.
Minor, on the same statement: this precondition runs before the type is known to be valid, so a garbage code such as 0xFFFFFFFF — for example EWKB with the SRID flag set — is also reported as "only 2D geometries are supported", which points a user at the wrong problem. Including the offending typeCode in the message, or validating the base geometry type first, would help.
There was a problem hiding this comment.
Took the first option — numDimensions() now derives the ordinate count from the type code's dimension group and readCoordinate skips past Z and M, so XYZ/XYM/XYZM values contribute their XY extent instead of failing or discarding the box.
Thanks for pointing at #17161: that handling was lost when this PR was split out, not deliberately dropped. The aborted state the previous revision used is gone with it, which also restores the strict trailing-data check it had to relax.
| // interior rings are contained by the exterior ring and cannot widen the bounds | ||
| for (int i = 1; i < numRings; i += 1) { | ||
| readCoordinateSequence(buffer, false); |
There was a problem hiding this comment.
Suggest promoting this from an inline comment into the class javadoc, since it's a contract callers need to know rather than a local implementation note.
Something like: the box is derived from each polygon's exterior ring only, which assumes OGC-valid polygons whose interior rings lie within the shell. That's the same envelope JTS computes for a Polygon, so Iceberg matching it is reasonable — but Iceberg never validates geometry validity, and for a polygon with a hole extending past the shell the resulting box would not contain the geometry, which means the file gets pruned for queries that should match rather than erroring. Stating it in the javadoc makes that a known, documented assumption instead of something a future reader has to rediscover.
Reading all rings is the alternative if you'd rather not depend on validity — the cost is one pass over bytes already in cache — but I'm fine with the shell-only behavior if it's documented.
There was a problem hiding this comment.
Promoted to the class javadoc.
| * <p>Coordinates are tracked independently for the X and Y dimensions. {@code NaN} values do not | ||
| * contribute to a dimension, and no bounds are produced unless both dimensions are present. | ||
| */ | ||
| public final class GeometryBoundsCollector { |
There was a problem hiding this comment.
Suggest making this package-private in core alongside GeometryFieldMetrics, unless there's a planned api-side caller.
It works mechanically: in #17161 neither ParquetValueWriters.GeometryWriter nor Spark's SparkParquetWriters.GeometryWriter references the parser — both only construct a GeometryFieldMetrics.Builder and call addValue/build. Avro would go through the same public builder. So the parser can stay an implementation detail and never enter the tracked surface, which keeps it free to change when Z/M and geography land. The whole FieldMetrics family already lives in core, and VariantUtil in api is the close precedent for this shape: package-private defensive ByteBuffer parsing reachable only through public types in its package.
To be clear, moving to core alone wouldn't help much — core is in REVAPI_PROJECTS too (build.gradle:112); it's a weaker stability tier per AGENTS.md, not an escape from tracking. The package-private part is what does the work.
The counter-argument for keeping it here is real: BoundingBox and GeospatialBound are in this package, and a package-private class in api would be unusable from core, so those two options are mutually exclusive. Nothing in api needs WKB parsing today — the only BoundingBox users are GeospatialBound and GeospatialPredicateEvaluators, which compares boxes. But if the geospatial literal work in #14101 or the ST_INTERSECTS pushdown in #17175 will need to turn a WKB constant into a box inside api, then this placement is already right. Which is it?
There was a problem hiding this comment.
Neither needs a WKB walk in api.
#14101's BoundingBoxLiteral deserializes through BoundingBox.fromByteBuffer, which reads the serialized box form — two fixed-width corner points — not a WKB geometry; every other construction site in that PR passes two GeospatialBounds directly. #17175 compares a constant box against bounds already read from manifests and touches BoundingBox.java by a single line.
So the parser is now package-private in core, next to where GeometryFieldMetrics will live, and stays off the tracked surface.
| // reusable copies of the accumulated bounds, used to undo a partially parsed value | ||
| private final DimensionBounds xSaved = new DimensionBounds(); | ||
| private final DimensionBounds ySaved = new DimensionBounds(); |
There was a problem hiding this comment.
Suggest dropping the save/restore until a caller needs it.
xSaved/ySaved plus the catch-restore-rethrow at lines 71-75 exist so a caller can keep accumulating after an invalid value, but the planned caller in #17161 lets the exception abort the write, so the restored state is never read. Per "keep the first version of a PR minimal" this could come with the caller that actually swallows the exception. If it stays, the javadoc on add should state the guarantee explicitly, since it's the sort of thing a later edit would silently break.
There was a problem hiding this comment.
Dropped. With Z/M no longer aborting, the only remaining failures are malformed input, which the planned caller lets propagate.
| import org.junit.jupiter.params.provider.Arguments; | ||
| import org.junit.jupiter.params.provider.MethodSource; | ||
|
|
||
| public class TestGeometryBoundsCollector { |
There was a problem hiding this comment.
Suggest dropping the test prefixes from the method names.
AGENTS.md asks for this on new tests and the project is actively removing them — main currently has "Core: Drop test prefix from metadata table test methods" at HEAD. The guidance also asks for package-private test classes and methods, though the existing api tests are overwhelmingly public, so that part is a migration in progress rather than a deviation from local practice; up to you.
There was a problem hiding this comment.
Prefixes were already dropped; the class and its methods are package-private now too.
| Arguments.of( | ||
| "unsupported Z geometry", | ||
| "01e9030000000000000000f03f00000000000000400000000000000840", | ||
| "only 2D geometries are supported"), | ||
| Arguments.of( | ||
| "unsupported M geometry", | ||
| "01d1070000000000000000f03f00000000000000400000000000000840", | ||
| "only 2D geometries are supported"), | ||
| Arguments.of( | ||
| "unsupported ZM geometry", | ||
| "01b90b0000000000000000f03f000000000000004000000000000008400000000000001040", | ||
| "only 2D geometries are supported"), |
There was a problem hiding this comment.
These cases will need to change if the Z/M handling changes per the comment on GeometryBoundsCollector.java:105 — flagging so they aren't missed.
There was a problem hiding this comment.
Updated: extraDimensionsAreIgnored asserts the XY box for POINT Z, POINT M, POINT ZM and LINESTRING Z, plus extraDimensionsNestedInCollectionAreIgnored for a 3D child inside a collection and boundsAcrossValuesWithDifferentDimensions across values of differing dimensionality.
| private static ByteBuffer decode(String hex) { | ||
| byte[] bytes = new byte[hex.length() / 2]; | ||
| for (int i = 0; i < bytes.length; i += 1) { | ||
| int offset = i * 2; | ||
| bytes[i] = (byte) Integer.parseInt(hex.substring(offset, offset + 2), 16); | ||
| } | ||
|
|
||
| return ByteBuffer.wrap(bytes); | ||
| } |
There was a problem hiding this comment.
Suggest adding a case that passes a slice() at a non-zero position, backed by a larger array with a non-zero array offset.
Every case today uses ByteBuffer.wrap(...) at position 0. The implementation is correct for offset buffers — it never touches array()/arrayOffset() and all reads are relative — so this is a guard against future edits rather than a fix. Nice to have alongside the existing position/limit assertions.
Coverage otherwise looks good. I decoded the hex by hand and the MULTIPOINT((1 2),EMPTY,EMPTY,(3 4)) case does exercise a big-endian child inside a little-endian parent, so mixed-endian nesting is covered even though the test plan only claims top-level byte orders.
There was a problem hiding this comment.
Added as boundsFromBufferWithOffset: a slice() at position 11 of a larger array, asserting both the box and that the caller's position is untouched.
- Move the collector to core and make it package-private - Discard all bounds instead of failing when a value carries Z or M - Document the exterior-ring assumption for polygon bounds - Report the offending type code for invalid or unsupported WKB - Drop the unused save and restore of bounds across a failed add Generated-by: Codex
Read past Z and M ordinates instead of discarding the collector's bounds, so XYZ, XYM and XYZM values contribute their XY extent. Those serializations are valid Iceberg data, and dropping the box for a whole file left it unprunable. Generated-by: Codex
Geography needs geodesic edges, periodic longitude, and antimeridian crossing, so state that this collector does not apply to it. Generated-by: Codex
|
Thanks — replies to two of the questions that weren't inline. Parquet already computes a bounding box. Its footer box doesn't follow the Iceberg spec. Iceberg can't recover the dropped dimension from the footer value afterwards. The Avro path in #17119 has no footer statistics to read at all, so a collector is needed regardless. Will coordinate with #12347. Should invalid-but-parseable WKB fail the write? The line I'd draw is spec legality, not parseability. Z/M is legal per |
Summary
Geometry bounds metrics need to inspect WKB coordinates without adding a geometry-library dependency to Iceberg. This adds a package-private collector in
corethat accumulates the lower and upper bounds ofgeometryvalues from their WKB encoding, while leaving each inputByteBufferposition and limit unchanged.This PR only adds the reusable bounds collector. Geometry field metrics and Parquet/Spark writer integration remain in follow-up PRs.
Scope: geometry only
The collector applies to
geometrycolumns, whose edge interpolation is "always linear/planar" (format/spec.md:284) and whose calculations "are always Cartesian" (format/spec.md:301). A box containing every vertex therefore contains the whole geometry, so scanning vertices is both correct and minimal.It is deliberately not used for
geography, which needs a different algorithm: edges are geodesics (format/spec.md:318) that can reach beyond their endpoints in latitude, longitude is periodic, andformat/spec.md:772both permitsxmin > xmaxfor a tighter antimeridian-crossing box and restricts bounds to the canonical[-180..180]/[-90..90]ranges. That belongs in a separate collector, and the numeric semantics are worth agreeing on before implementing — happy to take that discussion to an issue.Why an Iceberg-side WKB scan
Parquet 1.17.1 computes a geospatial bounding box itself and exposes it in the footer via
ColumnChunkMetaData.getGeospatialStatistics(), so for the Parquet path this parser is a second pass over each value. It is still preferred, for three reasons:format/spec.md:774requires NaN to be skipped per dimension, soPOINT (1 NaN)contributes to X while leaving Y unset. Parquet'sBoundingBox.isXValid()/isYValid()instead invalidate a whole dimension once a bound is NaN, which produces different bounds for the same input.org.locationtech.jts.io.WKBReader; Iceberg has no JTS dependency inapiorcore.Reading the Parquet footer instead remains a reasonable optimization for the Parquet writer specifically, and is worth revisiting in #17161 once the semantics above are reconciled. Flagging overlap with #12347.
Test Plan
Verification Commands
AI Disclosure