Hasura Admin UI can be very slow when run inside minikube / Kubernetes

[delaurentis]

Discussed in #10214

^{Originally posted by delaurentis April 23, 2024}
Background

• In production, we run Hasura in Kubernetes / GKE inside Google Cloud.
• For local development, we mirror this setup, by running Hasura inside Kuberntes / minikube.

Problem

We've noticed that using the Hasura Admin UI can become very slow (30 seconds - 1 minute for pages to load). Some operations, like applying permissions can take 10+ minutes. This happens even on high end development machines with lots of RAM / CPU power (16GB, with 8GB reserved to running Kubernetes locally with Minikube)

When we run Hasura outside of Kubernetes, Hasura Admin UI is fast (pages load quickly, and applying permissions happens in <1 minute).

Advice Needed :)

Any tips for improving Hasura's performance with Kubernetes or debugging why it is so slow?

Here's our Kubernetes stateful set definition for Hasura for reference (containers are created for both postgres and hasura)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresdb
spec:
  serviceName: 'postgresdb'
  replicas: 1
  selector:
    matchLabels:
      app: postgresdb
  template:
    metadata:
      labels:
        app: postgresdb
    spec:

      containers:
        - name: postgresdb
          image: <removed for privacy>
          imagePullPolicy: IfNotPresent
          args:
            - -c
            - hba_file=/etc/postgres/conf/pg_hba.conf
            - -c
            - config_file=/etc/postgres/conf/postgresql.conf
            - -c
            - jit=off
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: db-data
              mountPath: /home/postgres/pgdata
            - name: db-certs
              mountPath: /etc/postgres/ssl
            - name: db-config
              mountPath: /etc/postgres/conf
        - name: hasura
          image: hasura/graphql-engine:v2.15.2
          ports:
            - containerPort: 8080
          env:
            - name: ACTION_BASE_URL
              value: http://host.minikube.internal:3000
            - name: HASURA_GRAPHQL_METADATA_DATABASE_URL
              value: postgres://postgres:postgres@localhost:5432/tsdb
            - name: HASURA_GRAPHQL_DATABASE_URL
              value: postgres://postgres:postgres@localhost:5432/tsdb
            - name: HASURA_GRAPHQL_TS_DATABASE_URL
              value: postgres://postgres:postgres@localhost:5432/tsdb
            - name: HASURA_GRAPHQL_ENABLE_CONSOLE
              value: 'true'
            - name: HASURA_GRAPHQL_DEV_MODE
              value: 'true'
            - name: HASURA_GRAPHQL_ENABLED_LOG_TYPES
              value: startup, http-log, webhook-log, websocket-log, query-log
            - name: PGSSLMODE
              value: disable
            - name: HASURA_GRAPHQL_ADMIN_SECRET
              value: <removed for privacy>
            - name: HASURA_GRAPHQL_JWT_SECRET
              value: '{"type":"HS256","key":"<removed for privacy>"}'
      volumes:
        - name: db-readonly-certs
          secret:
            secretName: db-tls-secret
        - name: db-config
          configMap:
            name: postgres-config
        - name: db-certs
          emptyDir: {}
        - name: log
          emptyDir: {}

  volumeClaimTemplates:
    - metadata:
        name: db-data
      spec:
        accessModes: ['ReadWriteOnce']
        storageClassName: 'standard'
        resources:
          requests:
            storage: 10Gi
```</div>