authentication proxy

[JessicaS11]

ORIGINAL POST: #308 by @louis020393 Moved to a new discussion for future findability and categorization.

Hello Jessica

I am Louis and I want to download ICESAT-2 data. I do not know if it is the good place to ask the question. We are using :

region_a = icepyx.Query('ATL03', spatial_extent , date_range)
region_a.earthdata_login()
region_a.download_granules('data_download')

Since the 0.7.0 version of icepyx, the authentification is done with earthacess lib. I do not find on the documentation how to >put proxies to the request session (get_session() line 193 in auth.py) ? Is there any post about this subject ?

Best regards :)

[JessicaS11]

Hello @louis020393! Thanks for getting in touch.

Could you please explain what you are trying to do in a bit more detail? I'm wondering about your use case of submitting a proxy rather than using your earthdata credentials directly.

As you note, from v0.7.0+ authentication is completed using earthaccess. The icepyx auth module you refer to is still in development and has not yet been merged into the development branch (or a released version if you're installing icepyx using pip or conda/mamba). Once it is, the only anticipated change on the user side should be improved automated authentication (i.e. you will no longer need to call region_a.earthdata_login() at all!), with the rest of the changes occurring in the background to facilitate more streamlined authentication handling for working in the cloud.

The get_session() function is actually an earthaccess function, so we would need to look there to see if it's possible to use a proxy to authenticate.

[louis020393]

Thanks a lot Jessica for the answer,

To explain the context, we absolutely have to use our proxy to have access to external resources/data. If not we have a connexion refused. To make the v0.7.0 icepyx lib works, I have to manually modify the get_session() function to add the proxy to the session.
As I wrote before, on the main script we have:

region_a = icepyx.Query('ATL03', spatial_extent , date_range)
region_a.earthdata_login()
region_a.download_granules('data_download')

On the get_session() in auth.py, I manually add :

    def get_session(self, bearer_token: bool = True) -> requests.Session: #Line 193 of auth.py
        session = requests.Session()

        # The line I add
        session.proxies.update({'http': 'http://my.proxy:3128', 'https': 'http://my.proxy:3128'})

#END of the function

Before the v0.7.0 we used environment variables in our main script, example:

import os

os.environ["HTTP_PROXY"] = "my.proxy:3128"
os.environ["HTTPS_PROXY"] = "my.proxy:3128"

Is this clearer ?

Best regards :)

Louis

[JessicaS11]

Hello Louis - yes, thank you for explaining again and providing the examples; I understand now. I thought you were modifying icepyx/core/auth.py rather than earthaccess/utils/auth.py, which led to some of my confusion. I'm really glad you brought this use case up for us, as I suspect you're not the only group trying to use a proxy. It sounds like there are two main pieces that would need addressing to re-streamline your workflow:

1. Allow the use of proxies directly in the earthaccess library (@betolink Any thoughts on this?)
2. Assuming (1) happens, make it possible for the user to pass the proxy info through icepyx to earthaccess. If (1) doesn't happen, consider somehow enabling a proxy session directly in icepyx. (@rwegener2 any thoughts on this?)

@louis020393 Are there any pieces of this effort (either in earthaccess or icepyx) that you would be able or willing to work towards contributing? Thank you!

[louis020393]

@JessicaS11

Thank you for suggesting it 👍. If I have precise proposal, I can work on it. Note that I Have not work on any external project yet, so I do not know how to do it.

Have a nice day/night/afternoon 🙃

[betolink]

Hi @JessicaS11 @louis020393

earthaccess uses requests sessions, it should be possible to use the same environment variables you used before. However, to prevent using mixed authentication methods, earthaccess has trust_env set to False, this means that requests is not using the .netrc or environment variables. I wonder if IcePyx could do something like this:

def get_authenticated_session(proxies):
    session = earthaccess.get_requests_https_session()
    session.proxies.update(proxies)
    return session

Another option would be to override the trust_env and I think NSIDC has a temporary policy to ignore this mixed auth but the correct behavior is to only use bearer tokens. If the trust_env is set to true requests will pick up the HTTP_PROXY env vars and will use the netrc if there is one.

def get_authenticated_session(trust):
    session = earthaccess.get_requests_https_session()
    session.trust_env = trust
    return session

[louis020393]

Hi @betolink and @JessicaS11

I could be a solution but the thing is, IcePyx use earthaccess.login() and 'everything' is done with earthaccess lib. As I understand it, I think that I would be better to have the possibility to pass the proxy to earthaccess.

To add more details, the proxy error appears in the user_profile = session.get(url).json() (l214 of auth.py) :

login, api.py:132
  -> login, auth.py:84
    -> _netrc, auth.py:244
      -> _get_credentials, auth.py:291
        -> get_user_profile, auth.py:214

Louis

[JessicaS11]

Thanks @betolink! This is super helpful.

@rwegener2 just merged #435 which (behind the scenes) turns our authentication into a Mixin that's easily used across other modules. Based on Luis' suggestions, we're exploring the possibility of utilizing the earthdata_login() function (which is otherwise deprecated) as a means to allow users to customize their auth (e.g. add proxies, such as by passing in proxy={'http': 'http://my.proxy:3128', 'https': 'http://my.proxy:3128'}). @louis020393 do you have any thoughts or feedback on such an approach? Would you be interested in working with us on implementing this solution?

[louis020393]

@JessicaS11, I will have a look to @rwegener2 merge when I will have time. As I understand it, thanks to the EarthdataAuthMixin class we have a 'direct' access to the session objet now. It will probably correct my issue.

I need to make some tests. I will let you know how it works.

Louis

[rwegener2]

Thanks @betolink, this is helpful!
@louis020393 with the latest PR you should be able to access and update the session object directly. For example:

region_a = icepyx.Query('ATL03', spatial_extent , date_range)
region_a.session.proxies.update({'http': 'http://my.proxy:3128', 'https': 'http://my.proxy:3128'})
print(region_a.session.proxies)

Let us know if that doesn't work!

[rwegener2]

@betolink I am curious -- what is the difference between the .get_session() method in earthaccess and the .get_requests_https_session() method? Is a regular session not necessarily an https session?

[betolink]

earthaccess.get_requests_https_session() is just a wrapper of auth.get_session(),

When earthaccess started we had to import the classes directly, then I thought that having a unified API like Xarray would be more intuitive. I never deprecated or hid the old methods just in case someone was using them. But because earthaccess uses S3FS sessions for direct access, there is a long name to make a distinction between those and the ones from Requests.

Maybe we should consolidate them to something like get_session(mode="filesystem") for fsspec and a get_session(mode="remote") for Requests

[louis020393]

Hi everyone,
Thanks a lot @betolink and @JessicaS11 , it works perfectly.
Do you plan to create a release with this development version ?
Louis

[rwegener2]

Hi @louis020393! A new version of icepyx (0.8.0) was just released which included the new authentication system. Hope it helps!