Cloud Native file formats

[imakihi]

Hi there!
I have been converting all kind of vector and raster data to the cloud native format such as GeoParquet and COG. I also want to convert LAS files to COPC too. So it would be super cool if Terria can support GeoParquet and COPC. COG support looks closet to the final stage so I am super exceited about that. Especially GeoParquet support would be nice.

[nf-s]

Hi @imakihi

I agree, I would also love to add Zarr and FlatGeobuff support.
Currently, we mostly only get demand to add COG support - but I'm sure this will change as more open-data is published in cloud-native file formats.

[imakihi]

@nf-s Thank you for your comment.

Zarr sounds nice too! I am sure you guys are super busy but cloud native formats must be the way for any geospatial web apps in the future.
https://beta.source.coop/repositories/streambatch/worldcereal/description/

Than you for yoru hard work!!

[nf-s]

Thanks @imakihi!

It is great to see more data being published to Source Cooperative - Radiant Earth is an amazing organisation!

We already support pmtiles - so we should try and add one of the pmtiles datasets - https://beta.source.coop/repositories/?tags=pmtiles

[imakihi]

@nf-s
How can I add a pmtile dataset to a TerriaMap? I didn't know that I can use it. That would be super cool. Can you tell me how to do it when you have a chance??
Thanks!!

[techmavengeospatial]

Probably the easiest way to support many new formats is to use QGIS WASM Web Assembly https://github.com/qgis/qgis-js and render them as Raster Tiles for Terriajs and have functions for feature info requests. To do this would need to automate QGIS Project creation.
This would be client-side javascript and not persistent.

There are other WASM Packages like SPL.jS Spatialite, GDAL3.JS, GeoParquet WASM
Most cloud native formats still work better with server-side code than just in browser.

there is also DUCKDB WASM the spatial extension is not supported yet but they are working on supporting spatial extension in future versions of web assembly version

DUCKDB Spatial has access to import and export any OGR format

[nf-s]

Wow qgis-js is very cool - but I don't think we will be able to use it due to GPL license.

I think the bottleneck with these formats for TerriaJS will be rendering - especially with tabular/vector sources. We probably have to look into a more "GPU native" approach - rather than decoding these formats and using TerriaJS existing tabular/vector engine

[TimeScience]

FYI, TERN is publishing all of their point clouds in COPC, so that's another vote for COPC support

[techmavengeospatial]

via DUCKDB WASM with Spatial, httpfs, zipfs extensions you can access other cloud native/optimized formats and even load vector tiles from query result onto the map.
You can also access real-time event driven data feeds via the Radio https://query.farm/duckdb_extension_radio.html and https://query.farm/duckdb_extension_tributary.html
If you are pointing to a Data Lake, Data Lake House and Catalogs then use
https://duckdb.org/docs/stable/core_extensions/iceberg/overview
https://duckdb.org/docs/stable/extensions/arrow https://query.farm/duckdb_extension_airport.html
https://duckdb.org/docs/stable/data/parquet/overview.html
https://duckdb.org/docs/stable/core_extensions/ducklake https://github.com/motherduckdb/duckdb-delta
Azure blob storage https://duckdb.org/docs/stable/extensions/azure
AWS S3 storage https://duckdb.org/docs/stable/extensions/aws
https://duckdb.org/community_extensions/extensions/snowflake
https://github.com/motherduckdb/duckdb-power-query-connector

for Rendering big Geopackage Vector Features you can use duckdb vector tiles output or use NGA GeoPackage-JS
and it's ability to do dynamic raster tiles to map canvas https://ngageoint.github.io/geopackage-js/api/classes/FeatureTileCanvas.html
https://ngageoint.github.io/geopackage-js/api/classes/FeatureTileGenerator.html
Be able to style data to meet it’s requirements
FeatureStyle https://ngageoint.github.io/geopackage-js/api/classes/FeatureStyle.html
FeatureStyleExtension https://ngageoint.github.io/geopackage-js/api/classes/FeatureStyleExtension.html
FeatureStyles https://ngageoint.github.io/geopackage-js/api/classes/FeatureStyles.html

We have a forked version of Terriajs with all of this plus in-browser AI via Transformers.js and local models

[jo-chemla]

Will probably take me some time to parse through this comment, looks really powerful to use duckdb linked to terriamap to allow ingesting larger datasets and provide vector tiles for light client consumption and streaming. Is this TerriaJS fork available anywhere as a demo or repo to build? Thanks!

[Techmaven-geospatial]

ONLY VIDEO we will have these and much more like geospatial full motion video, other web assembly packages like ASSIMPJS for 3D Model conversion, optional python WASM,
We've also tested MapServer WASM and QGIS WASM

We have other versions of terriajs that are available as

• PHP CMS Plugins (WordPress, Drupal, Joomla, ModX, Ghost)
• SparkBoard or Affine Forks for collaborative whiteboard geospatial solutions
• CloudFlare based with cloudflare workers
• Supabase postGIS with deno functions advanced geospatial content management system
• Pocketbase and Trailbase (SQLite) based lightweight Geospatial CMS
• Military C2, COP, SA based Terriajs with TAK Server Integration, advanced Tools, Widgets and Functionality

Also, Earth Explorer 3D Map iOS, Android and Windows app based on Terriajs has been updated to support Gaussian Splats and advanced forms and in-app offline AI capabilities

[jo-chemla]

Great thread, seeing mentions of CNG Cloud Native Geospatial formats & standards to be requested for terriajs rendering.

A few updates for current state-of-things:

• @imakihi pmtiles can be added via mapbox-vector-tiles items, type: mvt see details
• type: COG has been implented in oct-2024 via Cog support cesium only 1 #7209

Regarding the other formats mentioned here and there in the thread,

• I also do think it would be great to add support for Geoparquet (often 10x compression vs geojson), COPC, Zarr, fgb
• I would add geopackage, also binary efficient alternative to ascii geojson
• new OGC APIs are tracked in this thread Support new OGC APIs #4763

Should dedicated issue threads be opened by anyone interested in each format rather than a discussion?

pmtiles via type: mvt

{
  "name": "OSM pmtiles - buildings, roads",
  "type": "mvt",
  "url": "https://demo-bucket.protomaps.com/v4.pmtiles",
  "description": "See [Map](https://pmtiles.io/#url=https%3A%2F%2Fdemo-bucket.protomaps.com%2Fv4.pmtiles&map=0.92/0/0)",
  "style": {
    "layers": [
      {
        "type": "fill",
        "source-layer": "buildings",
        "paint": {
          "fill-color": "#00b",
          "fill-opacity": 0.5,
          "line-color": "#00b",
          "line-width": 1
        }
      },
      {
        "type": "line",
        "source-layer": "roads",
        "paint": {
          "line-color": "#bbb",
          "line-width": 1
        }
      }
    ]
  }
},