Wishlist: example/support for redirection to online (http/https) resources

[yarikoptic]

To a degree it is a reincarnated

• support HTTP Redirect in server and provider #253

It would have been great if there was a sample Provider (and whatever it takes) for a WebDAV server which overall is a simple redirection service to some other http or https website, and assuming some default indexes (e.g. as provided by apache), so it could have config like

          "provider_mapping": {
            "/dav": HTTPRedirectionProvider("https://example.com/someprefix/"),
        },

and have /dav/path/file be redirected to https://example.com/someprefix/path/file.

Alternative possibly immediately useful example which could be sufficient to demonstrate ability to redirect, would be to redirect to S3, assuming that permissions allow for listing of the prefix to discover underlying "directories/" (there are no directories on S3 per se). S3 also has support/semantics for COPY and MOVE so there could be those implemented/supported as well, so making it not only read-only.

This would provide a good example on how to establish any other redirection service where corresponding mapping could be more involved, e.g. relying on obtaining target URLs through some API etc.

[bbockelm]

@yarikoptic - at a proof-of-concept level, I think one could override the do_GET method in wsgidav.request_server.RequestServer to do a targeted redirect based on path. Not a great long-term solution but it'd allow you to get passed the current speedbump of redirects to see if the rest of the idea works.

[jwodder]

@bbockelm How would one do that while using wsgidav as a library (via the normal WsgiDAVApp(config) interface) with a custom provider?

[bbockelm]

@jwodder - I don't think you can do it cleanly like that. That's why I said the suggestion would be a proof-of-concept / not great long-term. E.g., you'd have to reproduce the logic around here:

wsgidav/wsgidav/request_resolver.py

Line 222 in 056831c

app = RequestServer(provider)

and then adding the resolver as middleware as in here:

wsgidav/wsgidav/default_conf.py

Line 61 in 056831c

RequestResolver, # this must be the last middleware item

[jwodder]

@mar10 et alii: I'm interested in submitting a PR that would add support for redirects, and my main question at the moment is what sort of public API you'd like added to dav_provider.py to enable this. I've come up with the following design:

• Add a Redirect class with url: str and status_code: int (default = 302) fields
• Add the following methods to DAVNonCollection:
  • is_redirect(self) -> bool — When this returns True, it signifies that requests to the resource should be redirected to the return value of get_redirect().
    • The default implementation returns self.get_redirect() is not None
  • get_redirect(self) -> Redirect | None
    • The default implementation returns None
  • If is_redirect() returns true, then get_content() can do anything, including erroring or returning None. Other methods remain unaltered.
• I've opted not to add these methods to _DAVResource, as allowing collections to be redirects seems like it would come with a whole bunch of implementation issues.

Thoughts?

[mar10]

How would you implement get_content_length(), get_content_type, get_etag, etc. in these cases?

Maybe it would be better to open a discussion and detail the user story for this.

[yarikoptic]

get_content_length(), get_content_type, get_etag

in our case all 3 instructed by what we have in the metadata provided either from the server (for blobs) or S3 (for files under .zarr/ folders) : get_content_length and get_content_type are known from metadata for all one way or another, and get_etag we return None for some but for some (files under .zarr folders) proxy from metadata/S3.

@jwodder 's sample implementation, without redirects yet, is at https://github.com/dandi/dandi-infrastructure/pull/164/files to describe what we are showing etc... besides our dandi and then fsspec no other fancy dependencies, so should be 'safe-ish' to try ;) There are 2 get_contents which we would really like to avoid and rather handle as redirects since the files could be either too large or too many for our webdav server to handle

1. too large: BlobResource https://github.com/dandi/dandi-infrastructure/pull/164/files#diff-a92786c6cd28a8c2c8a4b01d794af04eee21677dbf8839908a256b44547d60aeR279
2. too many: ZarrEntryResource https://github.com/dandi/dandi-infrastructure/pull/164/files#diff-a92786c6cd28a8c2c8a4b01d794af04eee21677dbf8839908a256b44547d60aeR500 -- we are pretty much proxying some prefix/ on S3

in both cases the target URL is on S3 bucket.

Next portion of the "use-case": private resources

For an authenticated session to webdav -- for "embargoed" resources, if possible to redirect while providing extra header information, URLs would need to be going to be to our API server end-points with Authentication token in the header, and those would redirect to minted S3 URLs upon actual request (so -- double redirection...). If not possible anyhow to provide custom extra headers in redirection, we would need first to issue extra queries to the server, and provide target minted URLs redirects to the actual content on S3.

[jwodder]

Why would we need to implement get_content_length(), get_content_type(), and get_etag()? Looking at the code, it appears that get_content_type() and get_etag() are only used when serving a non-collection resource, but if the resource is a redirect, then wsgidav would just be returning a redirect without any need to call those methods. As for get_content_length(), its only use outside of returning a non-collection resource seems to be for giving file sizes in collection listings, but wsgidav is currently fine with the method returning None.

[yarikoptic]

Happy Holidays (over :-( ). ping @mar10

[mar10]

Hi,
since you already offered to implement a PR, and you seemingly have some running code (and an S3 bucket for testing):
can you share your code as PR, so we can discuss (and test) more concrete?

[jwodder]

We do not have code that implements redirects yet. I was expecting more feedback from you on my proposed design before I implemented anything.

[Wamy-Dev]

Any update on this?