MinIO Signature Mismatch

[namanrawal47]

Expected Behavior

When using the mc command-line utility, running mc ls mybucket/ should display the contents of my bucket.

Current Behavior

mc ls mybucket/ results in "mc: Unable to list folder. The request signature we calculated does not match the signature you provided. Check your key and signing method."

Possible Solution

No clue.

Steps to Reproduce (for bugs)

1. Install the kong-ingress controller on AWS EKS (https://docs.konghq.com/kubernetes-ingress-controller/2.0.x/deployment/eks/)
2. Set up ingress rules to point to Minio service in Kubernetes on port 9000
3. Access Minio using mc after setting up config.json locally

Context

I have several data-sensitive applications that I cannot share here, but they perform calculations and upload results to MinIO. I noticed that all integration tests involving MinIO were failing because of the above error. The files are uploaded successfully based on application logs, but they are inaccessible via mc. I also cannot log into the Minio Console after port-forwarding the container. I am getting a 403 Forbidden when I try to do so.

Regression

No

Your Environment

• Version used (minio --version): RELEASE.2021-11-03T03-36-36Z
• Server setup and configuration: Minio container image comes from Iron Bank (Dept. of Defense hosted container registry). All MinIO data is stored in container under /data. Using MINIO_ROOT_USER and MINIO_ROOT_PASSWORD as opposed to access and secret key environment variables.
• Operating System and version (uname -a): Client machine is OSX. MinIO runtime is Linux s3-minio-898488fcd-ltjtn 5.4.149-73.259.amzn2.x86_64 Full restructure in accordance with #1 SMP Mon Sep 27 12:48:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[harshavardhana]

@namanrawal47 most probably you have a few things wrong like this

• incorrect LB configuration i.e not preserving Host header
• incorrect secret key

This is a simple basic setup issue.

[harshavardhana]

Current version is RELEASE.2021-11-03T03-36-36Z. Previous version is: RELEASE.2021-08-31T05-46-54Z. I can double-check on the secret key being used on Monday, but it's unlikely to be the cause.

Yeah its 100% guaranteed @namanrawal47 these kind of issues cannot happen with the size of our community.

[namanrawal47]

Thanks. Any chance this could be a result of not adding the correct host headers as mentioned above? I'm trying to rule out possibilities. The soonest I'll hear back regarding the secret key will be Monday, so if that's the cause, I'll update this thread.

[harshavardhana]

Thanks. Any chance this could be a result of not adding the correct host headers as mentioned above? I'm trying to rule out possibilities. The soonest I'll hear back regarding the secret key will be Monday, so if that's the cause, I'll update this thread.

 location / {
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
   proxy_set_header X-Forwarded-Proto $scheme;
   proxy_set_header Host $http_host;

These are settings that are needed - most important is

proxy_set_header Host $http_host;

[namanrawal47]

I got around to adding the above nginx directives to nginx.conf located on the kong proxy container at /usr/local/openresty/nginx/conf/nginx.conf, but that didn't seem to change anything. I also edited nginx-kong.conf at /usr/local/kong/nginx-kong.conf, but still getting the same results. Do you have any other suggestions? The credentials have been verified as valid.

One other thing I noticed is that even if I bypass the reverse proxy by port-forwarding the MinIO container on 9001, I can get to the console but can't log in (403 Forbidden, Invalid Credentials). I find that really strange. However, if I change MINIO_ROOT_USER and MINIO_ROOT_PASSWORD back to MINIO_ACCESS_KEY and MINIO_SECRET_KEY, I can log in. This still doesn't help my original problem though of the signature mismatch. Any ideas about what may be going on?

C02Z70K8LVDT:.mc namanrawal$ mc ls mytarget/ --debug
mc: <DEBUG> GET / HTTP/1.1
Host: <scrubbed>
User-Agent: MinIO (darwin; amd64) minio-go/v7.0.16 mc/RELEASE.2021-11-16T20-37-36Z
Authorization: AWS4-HMAC-SHA256 Credential=<scrubbed>/20211208/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20211208T190459Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 403 Forbidden
Content-Length: 334
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Wed, 08 Dec 2021 19:04:59 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
Via: kong/2.5.1
X-Amz-Request-Id: 16BEDCFE51270D2B
X-Content-Type-Options: nosniff
X-Kong-Proxy-Latency: 1
X-Kong-Upstream-Latency: 2
X-Xss-Protection: 1; mode=block

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource>/</Resource><RequestId>16BEDCFE51270D2B</RequestId><HostId><scrubbed></HostId></Error>mc: <DEBUG> Response Time:  86.317151ms

mc: <ERROR> Unable to list folder. The request signature we calculated does not match the signature you provided. Check your key and signing method.
 (1) ls.go:232 cmd.doList(..) Tags: [http://<scrubbed>:80/]
 (0) client-s3.go:2005 cmd.(*S3Client).listInRoutine(..)
 Release-Tag:RELEASE.2021-11-16T20-37-36Z | Commit:d0c62eb584e5 | Host:C02Z70K8LVDT | OS:darwin | Arch:amd64 | Lang:go1.17.2 | Mem:2.8 MB/16 MB | Heap:2.8 MB/7.6 MB

[Haoke98]

Same question and same condition !!!
My qustion is using the nginx over the minio connection in order to achive load balancing between minio clusters actually.
It was work well connecting to the minio directly before, but when I use it over the nginx server not direct to minio , I got the error below:

All my settings refer to the official website example:setup-nginx-proxy-with-minio
After this i configured the MINIO_SERVER_URL and MINIO_BROWSER_REDIRECT_URL, but i also got the error like you, i can bypass the reverse proxy by port-forwarding the MinIO container on 9001, I can get to the console but can't log in (403 Forbidden, Invalid Credentials).

Now I don't know what can I to do, did you resolved this problem?

[harshavardhana]

• Server setup and configuration: Minio container image comes from Iron Bank (Dept. of Defense hosted container registry). All MinIO data is stored in container under /data. Using MINIO_ROOT_USER and MINIO_ROOT_PASSWORD as opposed to access and secret key environment variables.

We do not support custom container images without having a commercial contract with us @namanrawal47 feel free to buy a subscription here https://min.io/pricing

Let me know if you can reproduce this with our published images on quay.io/minio/minio and docker.io/minio/minio

[namanrawal47]

That makes sense. This custom container image was STIG'd by the U.S. Government for potential vulnerabilities, and so it's possible the issue I'm facing is coming from that. I wanted to ask the community to see if anyone faced a similar issue. I will try to reproduce this with a similar image on Docker Hub. Thank you.