autotest: Add a VSIFile wrapper class

[dbaston]

What does this PR do?

This PR adds a gdal.vsi_open() function that returns a file-like object, allowing virtual files to be used as "regular" Python files in some cases. For example, a zipped CSV can be read using the Python csv module:

with gdal.vsi_open(f'/vsizip/prime_meridian.csv') as f:
    records = [x for x in csv.DictReader(f)]

assert records[2]['INFORMATION_SOURCE'] == "Institut Geographique National (IGN), Paris"

Still need to correctly handle different linebreak/OS combinations, ensure errors are raised when accessing a closed file, handle append mode (or raise error), and probably other things.

What are related issues/pull requests?

Tasklist

• Add test case(s)
• Add documentation
• Updated Python API documentation (swig/include/python/docs/)
• Review
• Adjust for comments
• All CI builds and checks have passed

[coveralls]

coverage: 67.069% (-0.002%) from 67.071% when pulling d05da7b on dbaston:python-vsifile-class into 949ac93 on OSGeo:master.

[rouault]

should probably throw an exception is None is returned

[dbaston]

done, also switched to VSIFOpenExL to capture the error message

[rouault]

probably that CPLReadLineL() could be used to avoid this reimplementatoin

[dbaston]

done

[rcoup]

If it's helpful, here's an implementation we have been using at Koordinates for a while, with a set of tests: feel free to crib anything that's useful. I've licensed it as MIT. Any questions most welcome.

https://gist.github.com/rcoup/5f4b49149ca26084eed4dccfd1b12895

• was written quite a while ago — I would guess it could be simpler today incorporating some stdlib updates in recent Python versions and/or other bugfixes. (eg: not having open() essentially C&P)
• Inheriting from io.RawIOBase means it can fit under/into the standard python buffered & text IO wrapper classes
• I've manually removed/replaced/inlined use of a handful of internal library functions & test setup. If there's anything you get stuck on please ask.

[dbaston]

Fantastic, thank you!

[dbaston]

I have not forgotten about this. I'm leaning towards moving it to gdaltest to focus on internal use for now.

[github-actions]

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 28 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

• that all unit tests are passing

• that all comments by reviewers have been addressed

• that there is enough information for reviewers, in particular link
  to any issues which this pull request fixes

• that you have written unit tests where possible
  In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
  If there is no further activity on this pull request, it will be closed in 2 weeks.