CSV sources have lat and lon flipped

[iandees]

As reported in openaddresses/openaddresses#5465

It looks like the source is specified correctly because the existing machine generates the correct output, but batch seems to be generating output with lat/lon flipped.

[ingalls]

It does not appear as though this is a widespread issue and is instead specific to this source

The following are CSV sources that work as expected:
https://batch.openaddresses.io/job/79428
https://batch.openaddresses.io/job/73113

[ingalls]

I went through 10 or 15 other CSV sources and am not seeing any issues with the exception of this source. As such I'm going for the decision to simply flip the coords: openaddresses/openaddresses#5594

[iandees]

Thanks for looking. I think we've flipped these back and forth a couple times before if you look at the history of that file. Let's see what the output looks like.

[stefanb]

This seems to happen again, please check openaddresses/openaddresses#5908

[stefanb]

Could you try to define ogr2ogr open parameters X_POSSIBLE_NAMES and Y_POSSIBLE_NAMES according to source's conform in addition to creation parameter GEOMETRY=AS_XY:

batch/task/export.js

Lines 194 to 199 in ba9c2e1

	} else if (exp.format === 'csv') {
	ogr = ogr.format('csv');
	if (job.layer === 'addresses') {
	ogr = ogr.options(['-lco', 'GEOMETRY=AS_XY']);
	} else {

See docs:
https://gdal.org/drivers/vector/csv.html#reading-csv-containing-spatial-information

Starting with GDAL 2.1, it is possible to directly specify the potential names of the columns that can contain X/longitude and Y/latitude with the X_POSSIBLE_NAMES and Y_POSSIBLE_NAMES open option.

[ingalls]

@stefanb This code operated on the output of GeoJSON sources, so the coordinate order is known good.

The code to take the OA sources and convert them to GeoJSON lives here: https://github.com/openaddresses/batch-machine

[stefanb]

Tnx @ingalls for the pointer!

I find it rather strange that test conform definitions changed in openaddresses/batch-machine#15. IMO lat & lon field mappings should not be swapped in there.

[stefanb]

Please check openaddresses/openaddresses#5917 for the timeline.

[stefanb]

For some reason a shapefile source in openaddresses/openaddresses#5907 is also converted to .csv with flipped coordinates.

See openaddresses/openaddresses#5907 (comment)

Is it converting to csv to generate preview image?

[andrewharvey]

Upgrading to gdal 3 may have caused this see https://gdal.org/tutorials/osr_api_tut.html#crs-and-axis-order looks like this was already addressed https://github.com/openaddresses/batch-machine/pull/15/files

[andrewharvey]

Collating all the reported issues, from what I can see this is not an issue to any particular source, so if okay we should close all these tickets and create a new pinned ticket (given the severity and the incoming new tickets) until this is resolved?

openaddresses/openaddresses#5974
openaddresses/openaddresses#5919
openaddresses/openaddresses#5917 (CSV/Addresses fixed - will circle back for buildings)
openaddresses/openaddresses#5912
openaddresses/openaddresses#5911
openaddresses/openaddresses#5906
openaddresses/openaddresses#5465
openaddresses/openaddresses#5450

[stefanb]

Shapefiles have the same problem in some cases, see: openaddresses/openaddresses#5907 (comment)

[ingalls]

@stefanb You were correct, these should not have been flipped in tests. I'm working through a bunch of these issues here: openaddresses/batch-machine#25

[stefanb]

In openaddresses/openaddresses#5909 I have determined, that this is not a problem with CSV anymore.

However, while adding a new shapefile source with building footprints in openaddresses/openaddresses#5907 the lat & lon still appear to be flipped

[stefanb]

Opened openaddresses/batch-machine#26 for shapefiles