-
Notifications
You must be signed in to change notification settings - Fork 58
-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Overturemap parquet files #849
Comments
@sebr72 as discussed, I'm not really satisfied with my current experiment in the overturemap branch. The geoparquet format contains semi structured data which require some changes in the DataTable abstraction. Also, it requires a deep understanding of the geoparquet format. One avenue (probably the best) could be to use the parser available in sedona (the project is written in scala): Another avenue could be to build upon my throw-away overturemaps branch, but I'm not sure about the effort needed to have something robust. In both cases, adding parquet or sedona will result in a lot of new dependencies (hadoop, spark). |
@sebr72 There may also be a third option which is to rely on parquet support in postgresql. I have no experience with this extention. |
@bchapuis I had a look at Sedona and I highlight the following:
I am going to switch to have a look at: |
Yes, I think the suggestion of @Drabble to look into drill is a good idea. We can probably either use it or get inspiration from it for our own implementation. |
@sebr72 @Drabble I will merge the current PR and organize the git history to have three separated commits with our individual contributions. For the following tasks, I suggest we make individual PRs and split the work more clearly.
|
@bchapuis Great job on the pull request! I will look at your new one for nested groups. I would be really interested in making an example to go from Overture data on S3 to serving MVT to a Maputnik frontend. I think this would mean: 1 Fix the code to be able to use a S3 url directly. E.g. s3a://overturemaps-us-west-2/release/2024-05-16-beta.0/theme=admins/type=/ What do you think? |
Yes, the plan sounds good and can probably be addressed with multiple PRs. Maybe we can skip step 4 or use views instead of materialized views. As the daylight distribution with soon be deprecated and replaced by overturemaps, an idea could be to copy the daylight directory and use it as a basis. |
We have a basic support for Overture maps now. Should we consider this issue closed and raise more issues for further improvements to the Overture maps library? |
Congratz guys 🎉 |
https://github.com/OvertureMaps/data
The text was updated successfully, but these errors were encountered: