sregistry-cli with https

[tejasdixit836]

Hello,

I have trouble using sregistry-cli with sregistry portal, that i have set up with https

I am able to pull/push the images when i am running the sregistry with http but not with https

here is how i have configured https:

DOMAIN_NAME = "https://domainnname"
DOMAIN_NAME_HTTP = "http://domainname"
DOMAIN_NAKED = DOMAIN_NAME_HTTP.replace('http://', '')
(all the ssl certificates in place according to the document and the registry GUI looks fine)

when i am trying to pull/push using sregistry-cli i get this error

ERROR Issue with https://sregistry-domain/api/container/library/alpine:test, try exporting SREGISTRY_REGISTRY_NOHTTPS.

here is my .sregistry file

{
"hub": {
"base": "https://singularity-hub.org/api"
},
"registry": {
"SREGISTRY_REGISTRY_BASE": "https://domainname",
"SREGISTRY_REGISTRY_USERNAME": "user1",
"SREGISTRY_REGISTRY_TOKEN": "fe55533e17c865b33bf583f004fd3token"
},
"SREGISTRY_CLIENT": "registry"
}

if i replace "SREGISTRY_REGISTRY_BASE": "https://domainname" with "http" i get the following error

[client|registry] [database|sqlite:////home/user/.singularity/sregistry.db]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
cnx.do_handshake()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
_raise_current_error()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/connectionpool.py", line 662, in urlopen
self._prepare_proxy(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/connectionpool.py", line 948, in prepare_proxy
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/connection.py", line 394, in connect
ssl_context=context,
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/util/ssl.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='domainname', port=443): Max retries exceeded with url: /containers/2/download/e0f18dea-1bad-441d-ba31-7ce5e851faba (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/sregistry", line 11, in
load_entry_point('sregistry==0.2.32', 'console_scripts', 'sregistry')()
File "/usr/local/lib/python3.6/site-packages/sregistry-0.2.32-py3.6.egg/sregistry/client/init.py", line 323, in main
extra=extra)
File "/usr/local/lib/python3.6/site-packages/sregistry-0.2.32-py3.6.egg/sregistry/client/pull.py", line 37, in main
save=do_save)
File "/usr/local/lib/python3.6/site-packages/sregistry-0.2.32-py3.6.egg/sregistry/main/registry/pull.py", line 116, in pull
show_progress=not self.quiet)
File "/usr/local/lib/python3.6/site-packages/sregistry-0.2.32-py3.6.egg/sregistry/main/base/http.py", line 178, in download
if requests.head(url, verify=verify).status_code in [200, 401]:
File "/usr/local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/api.py", line 101, in head
return request('head', url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='domainname', port=443): Max retries exceeded with url: /containers/2/download/e0f18dea-1bad-441d-ba31-7ce5e851faba (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

can you please help

Thanks in advance

[vsoch]

Hmm, can you please first check the version of open ssl - see the two commands in this issue: https://stackoverflow.com/questions/50246084/django-paypalrestsdk-error-openssl-ssl-error-ssl-routines-tls-process-s

[vsoch]

Another thing to try is pushing/pulling with the Singularity client to see if that also produce an error.

[tejasdixit836]

Thanks for the quick response, singularity pull works fine, i am not able push however.

I add the endpoint as test

singularity remote add --no-login test container.page
singularity remote login test
INFO: Authenticating with remote: test
Generate an API Key at https://containerspage/auth/tokens, and paste here:
API Key:
FATAL: while verifying token: while getting token service uri: error response from server: 404

also an observation that the "usage" commands that gets generated in the portal (for example, if i have an image in my sregistry and i go to usage section to find the relevant commands to pull/push the images) they are not accurate. few of them work and few do not(my observation is that the subdirectory in collection gets skipped in the path) is there a way that i fix this? I am configuring this for users and they might use this as reference

what i get in container page: singularity pull shub://containerpage/milkshakes:banana
actaul pull which works: singularity pull shub://containerpage/milkshakes/pudding:banana

[vsoch]

Why are you using containers.page? Is that your domain? It’s just an example.

[vsoch]

If you pull with Singularity it should be a library:// uri so your testing urls are not correct.

[vsoch]

I’m going back to sleep :) Back in the morning!

[tejasdixit836]

openssl version is OpenSSL 1.0.2k-fips 26 Jan 2017

containerpage=domain
i am not sure about library, but
singularity pull shub://containerpage/milkshakes/pudding:banana is working fine.

sure, have a good sleep :) i will try to figure out something if i can mean while

[vsoch]

Good morning! So the first thing to figure out is why you aren't getting a response, period, with singularity remote login (404 means not found). If "https://container.page/" is really your domain, I don't see anything there, so there's where to start. You should minimally get another error if you have a correctly set up domain. Here is the documentation page for future reference in this issue.

[tejasdixit836]

it is indeed something to do with certificate, but i am not sure what is going wrong. I cannot put my domain name here so i gave container.page

it is interesting that singularity pull works with shub but not with library

singularity pull shub://domainname.com/newt/v1:latest
INFO: Downloading shub image
2.59 MiB / 2.59 MiB [============================================================================] 100.00% 287.67 KiB/s 9s

singularity pull --library https://hpcs-sregistry.shell.com newt/v1:latest
FATAL: While pulling library image: image newt/v1:latest (amd64) does not exist in the library

sregistry pull registry://newt/v1:latest
[client|registry] [database|sqlite:////glb/home/intdr3/.singularity/sregistry.db]
ERROR Issue with https://domainname/api/container/newt/v1:latest, try exporting SREGISTRY_REGISTRY_NOHTTPS.

singularity remote login test
INFO: Authenticating with remote: test
Generate an API Key at https://doaminname/auth/tokens, and paste here:
API Key:
FATAL: while verifying token: while getting token service uri: error response from server: 404

This is how i set up https,

• generated certs and placed it in /etc/ssl/certs and private
  certs has chained.pem and dhparam.pem
  private has domain.key
• then copied the docker-compose.yml from https folder in sregistry to sregistry folder
• started the docker-compose

the portal looks ok with https, push with sregistry works , but not with singularity.

when i in change .sregistry, the sregistry base from https://domainname to http://domainname then i get ssl verify error which i have posted yesterday in the begining

[tejasdixit836]

i could not reply sooner as i was trying to set up everyhting all over again which also dint help !!!

[vsoch]

An issue with certificates wouldn’t be helped by re-creating the server. I don’t know how else to help you without being able to reproduce, you likely haven’t set up your DNS if you can’t see the page in the browser and this is a downstream result of that.

[tejasdixit836]

i can see the page in the browser, i can see all the images that i pushed as well. just not able to use singularity client

[vsoch]

okay this is with https?

[tejasdixit836]

yes with https

[vsoch]

What version of Singularity?

[tejasdixit836]

singularity version 3.4.1-1.2.el7

[vsoch]

But sregistry does or doesn't work with https? You mentioned:

when i in change .sregistry, the sregistry base from https://domainname to http://domainname then i get ssl verify error which i have posted yesterday in the begining

which suggests that it does work.

[vsoch]

Are you using a proxy? Did you generate your certs with let's encrypt or are they self signed? Are the permissions correct? Did you restart containers after?

[tejasdixit836]

works partially ,

cat .sregistry
{
"hub": {
"base": "https://singularity-hub.org/api"
},
"registry": {
"SREGISTRY_REGISTRY_BASE": "https://domainname",
"SREGISTRY_REGISTRY_USERNAME": "intdr3",
"SREGISTRY_REGISTRY_TOKEN": "token"
},
"SREGISTRY_CLIENT": "registry"
}

singularity pull shub://domainname/alpine/test:v1
INFO: Downloading shub image
367.75 KiB / 2.59 MiB [=========>------------------------------------------------------------] 13.84% 229.52 KiB/s 00m09s^C

singularity pull --library https://domainname alpine/test:v1
FATAL: While pulling library image: image alpine/test:v1 (amd64) does not exist in the library

sregistry pull registry://alpine/test:v1
[client|registry] [database|sqlite:////glb/home/inadr3/.singularity/sregistry.db]
ERROR Issue with https://domain/api/container/alpine/test:v1, try exporting SREGISTRY_REGISTRY_NOHTTPS.

sregistry push lolcow_latest.sif --name test/v2:latest
[client|registry] [database|sqlite:////glb/home/inadr3/.singularity/sregistry.db]
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.7-py3.6.egg/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
cnx.do_handshake()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/local/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
_raise_current_error()
File "/usr/local/lib/python3.6/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

in case you are wondering how i managed to push the 2 images, i changed
"SREGISTRY_REGISTRY_BASE": "https://domainname", to "SREGISTRY_REGISTRY_BASE": "http://domainname",

[vsoch]

in case you are wondering how i managed to push the 2 images, i changed
"SREGISTRY_REGISTRY_BASE": "https://domainname", to "SREGISTRY_REGISTRY_BASE": "http://domainname",

Yes, so that suggests your certificates are not created properly, or there is some issue with SSL version conflicts on the host doing the pull and the server. I would next try comparing versions on your host and in the containers, and upgrading where necessary.

[tejasdixit836]

yes proxy is used, i do not generate the certificate, it is generated by another team
i generated dhparam.pem
certs
-rwxrwxrwx. 1 root root 2.8K Jan 2 03:12 chained.pem
-rwxrwxrwx. 1 root root 769 Jan 2 08:52 dhparam.pem

private
-rw-------. 1 root root 1.7K Jan 2 10:49 domain.key

restarted the docker-compose several times

[vsoch]

This is something perhaps your other team can help you with, take the error message verbatim - the attempt to match the domain to the certificate failed - it's clearly related to the certificates. I'm not sure how else to help. Good luck!

[tejasdixit836]

alright, i will contact them, Thank you. when i received the certs , it was with a different name , sregistry.cer and sregistry.pem which i converted into chained.pem and domain.pem. I am assuming if they have got corrupted in the process.

you mentioned to compare the versions of openssl of host with that the containers. I noticed that the openssl is not installed in sregistry_nginx_1,(could not bash into that as even bash is not installed.)

but on other containers, it is installed and it is higher version

OpenSSL 1.0.2k-fips 26 Jan 2017- host

docker exec -it sregistry_uwsgi_1 /bin/bash
root@d7373518460b:/code# openssl version -a
OpenSSL 1.1.1d 10 Sep 2019

so is openssl not installed on sregistry_nginx_1 is this also one of the reason for certificate error?

[vsoch]

You can shell into nginx by changing the entrypoint to sh (bash isn't in the container, but shell is). Something like

docker exec -it sregistry_nginx_1 sh

Before you contact them, either find a host with an updated SSL or update on your host - there are definitely conflicts between older and newer versions that can lead to messages like that.

so is openssl not installed on sregistry_nginx_1 is this also one of the reason for certificate error?

I don't know.

[tejasdixit836]

ok, i will try and match the openssl versions first thing in the morning tomorrow! off to sleep now!

[tejasdixit836]

I tried updating ssl , dint work, still the same error.

I am not sure how singularity pull works with shub and not with library.
singularity pull shub://domainname/alpine2/test:v1
INFO: Downloading shub image
2.59 MiB / 2.59 MiB [==========================================================================] 100.00% 295.36 KiB/s 8s

singularity pull --library https://domainame alpine2/test:v1
WARNING: Unable to get library service URI: error response from server: 404, defaulting to https://keys.sylabs.io
FATAL: While pulling library image: image alpine2/test:v1 (amd64) does not exist in the library

i noticed something today which might explain whats happening.
when i try to add an endpoint , it fails with 404 error

singularity remote login test
INFO: Authenticating with remote: test
Generate an API Key at https://domainname/auth/tokens, and paste here:
API Key:
FATAL: while verifying token: while getting token service uri: error response from server: 404

the reason why i am getting 404 response even when the domain exitsts might be because sregistry is tryinig to search for the tokens in different place!!

the tokens are found in the URL https://domainname/tokens and not https://domainname/auth/tokens(where it is actually being searched).

is there anyway to fix this?

[vsoch]

the tokens are found in the URL https://domainname/tokens and not https://domainname/auth/tokens(where it is actually being searched).

The API is designed so you provide the remote url as the domain you have your singularity registry deployed to, and the Singularity client handles parsing further. It could be that Sylabs has changed their library API, the last version I remember testing (and working) was in the 3.2.x family of versions.

[tejasdixit836]

i installed older version on singularity on one of the machines to see if it works, still the same issue

singularity --version
singularity version 3.2.0-1

singularity remote login test
Generate an API Key at https://domainname/auth/tokens, and paste here:
API Key:
FATAL: while verifying token: while getting token service uri: error response from server: 404

i noticed in your document the drop down looks something little different than that i have . is there anything updated in sregistry that is making difference? please see screen shots

is there any way i can match the path that the singularity is looking for? i.e., https://domainname/auth/tokens.

one more thing is whether i am using http or https, i am not able to delete the repos from sregistry-cli. i get the error message for http as repo doesnt exist for https as ssl error.

i am not sure , what else to try now!!

[vsoch]

Singularity doesn’t support plain http. The screenshot difference isn’t an issue, they are just older pictures. I’m not sure how to help either, I believe this is an issue with your ssl and not related to the registry.