Issues with Loki installation

[giscus[bot]]

docs/loki/

How to deploy Grafana Loki in our Raspberry Pi Kuberentes cluster.

https://picluster.ricsanfre.com/docs/loki/

[i5Js]

I guess I will be doing something wrong, because I'm getting this message in the traefik logs:
"GET /k8s-loki/loki_cluster_seed.json HTTP/1.1" 403 430 "-" "-" 53861 "cert-manager-minio-s3-produccion-c7112686c0499b2338ad@kubernetescrd" "http://10.244.7.172:9000" 3ms

[i5Js]

I was checking the doc, and it seems the name of the variables has changed:
accessKeyId -> access_key_id
secretAccessKey -> secret_access_key
s3ForcePathStyle -> s3forcepathstyle

No more errors.

[ricsanfre]

Hi @i5Js , I am not finding any error like the one you are describing. The 403 error you are getting in traefik logs might be related with a wrong configuration of Minio's bucket or loki user access policies.

[i5Js]

Yes, indeed, those values are ok. However, my feeling is because the s3ForcePathStyle remains to false, so the certificate I'm using is not valid. I'm trying to enable it in the values file, but it's not possible, in the config map remains to false.

[i5Js]

New error, this time in the backends and read pods:

level=error ts=2024-03-26T20:09:04.591926831Z caller=ruler.go:571 msg="unable to list rules" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

[ricsanfre]

I am checking my backend pods logs, and I am finding a periodic error which are slightly different:

level=error ts=2024-03-27T09:02:14.612117563Z caller=ruler.go:571 msg="unable to list rules" err="InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: 17C09396DF413057, host id: 9a7271ce6652c656aeda8fce0f72048b32380e8aad06540fee3bb9a4abdfa637"
level=error ts=2024-03-27T09:02:23.002248265Z caller=reporter.go:205 msg="failed to delete corrupted cluster seed file, deleting it" err="InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.\n\tstatus code: 403, request id: 17C09398D34599C0, host id: 9a7271ce6652c656aeda8fce0f72048b32380e8aad06540fee3bb9a4abdfa637"

I think this error is happening because backed pod needs access to S3 bucket, and the current configuration is not providing the proper environment variables.

[i5Js]

I think mine is before that :)

level=error ts=2024-03-27T09:16:38.877704723Z caller=reporter.go:205 msg="failed to delete corrupted cluster seed file, deleting it" err="RequestError: send request failed\ncaused by: Delete \"https://k8s-loki.s3.<endpoint>.com/loki_cluster_seed.json\": dial tcp: lookup k8s-loki.s3.<endpoint>.com on 10.96.0.10:53: no such host"

[i5Js]

And now, I'm getting your error too :)

[ricsanfre]

To solve the backend log error the following configuration need to be added.

  # Configuration for the backend
  backend:
    # Number of replicas for the backend
    replicas: 3
    persistence:
      # -- Size of persistent disk
      size: 10Gi
      # -- Storage class to be used.
      storageClass: longhorn

    # Enable environment variables in config file
    # https://grafana.com/docs/loki/latest/configuration/#use-environment-variables-in-the-configuration
    extraArgs:
      - '-config.expand-env=true'
    extraEnv:
      - name: MINIO_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: loki-minio-secret
            key: MINIO_ACCESS_KEY_ID
      - name: MINIO_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: loki-minio-secret
            key: MINIO_SECRET_ACCESS_KEY

I have updated the installation document accordingly

[i5Js]

It works without any errors. Super!

[i5Js]

More info, the variables are not changed, meaning accessKeyId, secretAccessKey, s3ForcePathStyle are valid, but the cm is not ok, becasue the s3ForcePathStyle remains to false instead true.

s3:
  access_key_id: ${MINIO_ACCESS_KEY_ID}
  bucketnames: k8s-loki
  endpoint: <no encoded>
  http_config:
    idle_conn_timeout: 90s
    insecure_skip_verify: false
    response_header_timeout: 0s
  insecure: false
  region: eu-west-1
  ----> s3forcepathstyle: false <-----

[i5Js]

Then I don't understand.... I will try to uninstall and install it again... look mine:

apiVersion: v1
data:
  config.yaml: |2

    auth_enabled: false
    common:
      compactor_address: 'http://loki-backend:3100'
      path_prefix: /var/loki
      replication_factor: 3
      storage:
        s3:
          access_key_id: ${MINIO_ACCESS_KEY_ID}
          bucketnames: k8s-loki
          endpoint: s3.<endpoint>com
          http_config:
            idle_conn_timeout: 90s
            response_header_timeout: 0s
          insecure: false
          region: eu-west-1
          s3: s3://k8s-loki
          s3forcepathstyle: false
          secret_access_key: ${MINIO_SECRET_ACCESS_KEY}
    frontend:
      scheduler_address: query-scheduler-discovery.logging.svc.cluster.local.:9095
    frontend_worker:
      scheduler_address: query-scheduler-discovery.logging.svc.cluster.local.:9095
    index_gateway:
      mode: ring
    limits_config:
      max_cache_freshness_per_query: 10m
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      split_queries_by_interval: 15m
    memberlist:
      join_members:
      - loki-memberlist
    query_range:
      align_queries_with_step: true
    ruler:
      storage:
        s3:
          access_key_id: ${MINIO_ACCESS_KEY_ID}
          bucketnames: k8s-loki
          endpoint: s3.<endpint>.com
          http_config:
            idle_conn_timeout: 90s
            insecure_skip_verify: false
            response_header_timeout: 0s
          insecure: false
          region: eu-west-1
          s3: s3://k8s-loki
          s3forcepathstyle: false
          secret_access_key: ${MINIO_SECRET_ACCESS_KEY}
        type: s3
    runtime_config:
      file: /etc/loki/runtime-config/runtime-config.yaml
    schema_config:
      configs:
      - from: "2022-01-11"
        index:
          period: 24h
          prefix: loki_index_
        object_store: s3
        schema: v12
        store: boltdb-shipper
    server:
      grpc_listen_port: 9095
      http_listen_port: 3100
    storage_config:
      hedging:
        at: 250ms
        max_per_second: 20
        up_to: 3
    tracing:
      enabled: false
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: loki
    meta.helm.sh/release-namespace: logging
  creationTimestamp: "2024-03-24T13:16:57Z"
  labels:
    app.kubernetes.io/instance: loki
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: loki
    app.kubernetes.io/version: 2.9.6
    helm.sh/chart: loki-5.47.1
   name: loki
  namespace: logging
  resourceVersion: "96336270"
  uid: 5fb76515-afab-42a3-a3f9-b7c97b97cf93

[i5Js]

Same,

  storage:
    s3:
      access_key_id: ${MINIO_ACCESS_KEY_ID}
      bucketnames: k8s-loki
      endpoint: s3.<endpoint>.com
      http_config:
        idle_conn_timeout: 90s
        insecure_skip_verify: false
        response_header_timeout: 0s
      insecure: false
      region: eu-west-1
      s3forcepathstyle: false

To me it's very strange because if you check the hole values of the chart, you will see s3forcepathstyle has the value false by default and the others are null.

[i5Js]

  # -- Storage config. Providing this will automatically populate all necessary storage configs in the templated config.
  storage:
    bucketNames:
      chunks: chunks
      ruler: ruler 
      admin: admin
    type: s3
    s3:
      s3: null
      endpoint: null
      region: null
      secretAccessKey: null
      accessKeyId: null
      signatureVersion: null
      **s3ForcePathStyle: false**
      insecure: false
      http_config: {}
      # -- Check https://grafana.com/docs/loki/latest/configure/#s3_storage_config for more info on how to provide a backoff_config
      backoff_config: {}

[i5Js]

"Yisus" I found the issue. I had a typo, the right value is s3ForcePathStyle and I was using S3ForcePathStyle

Now it's updated.