Skip to content

Memphis.dev is an intelligent, frictionless message broker. Made to enable developers to build real-time and streaming features fast.

License

Notifications You must be signed in to change notification settings

superstreamlabs/memphis-k8s

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Github (2)

Discord Code Of Conduct GitHub release (latest by date)

CNCF Silver Member CNCF Silver Member

Memphis.dev is a highly scalable, painless, and effortless data streaming platform.
Made to enable developers and data teams to collaborate and build
real-time and streaming apps fast.

Memphis Kubernetes Deployment

Helm is a k8s package manager that allows users to deploy apps in a single, configurable command.
More information about Helm can be found here.

Memphis is cloud-native and cloud-agnostic to any Kubernetes on any cloud.

Requirements

Minimum Requirements (Without high availability)

ResourceQuantity
Minimum Kubernetes version1.20 and above
K8S Nodes1
CPU2 CPU
Memory4GB RAM
Storage12GB PVC

Recommended Requirements (With high availability)

Resource Minimum Quantity
Minimum Kubernetes version 1.20 and above
K8S Nodes 3
CPU 4 CPU
Memory 8GB RAM
Storage 12GB PVC Per node

Installation

Production

Production-grade Memphis with three memphis brokers configured in cluster-mode

helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && helm install memphis memphis/memphis --set global.cluster.enabled="true" --create-namespace --namespace memphis --wait
Dev

Standard installation of Memphis with a single broker

helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && 
helm install memphis memphis/memphis --create-namespace --namespace memphis --wait

* Optional * Helm deployment options

Option Description Default Value Example
global.cluster.enabled Cluster mode for HA and Performance "false" "false"
exporter.enabled Prometheus exporter "false" "false"
exporter.serviceExposed.enbaled Expose metrics port with memphis service "false" "true"
cluster.enabled Enables Memphis cluster deployment. For fully HA configuration use global.cluster.enabled "false" "true"
cluster.replicas Memphis broker replicas "3" "5"
memphis.image Memphis image name "memphisos/memphis:x.x.x-stable" "memphisos/memphis:latest"
memphis.ui.port Dashboard's (GUI) port 9000 9000
memphis.hosts.uiHostName Which URL should be seen as the "UI hostname" "" "https://memphis.example.com"
memphis.hosts.restgwHostName Which URL should be seen as the "REST Gateway hostname" "" "https://restgw.memphis.example.com"
memphis.hosts.brokerHostName Which URL should be seen as the "broker hostname" "" "memphis.example.com"
memphis.configFile.logsRetentionInDays Amount of days to retain system logs 3 3
memphis.configFile.gcProducerConsumerRetentionInHours Amount of hours to retain producer/consumer in system 3 3
memphis.configFile.tieredStorageUploadIntervalSeconds Interval in seconds between uploads to tiered storage 8 8
memphis.configFile.dlsRetentionHours Amount of hours to retain messages in DLS 3 3
memphis.configFile.userPassBasedAuth Authentication method selector.
true = User + pass
false = User + connection token
"true" "true"
memphis.creds.rootPwd Root password for the dashboard. Randomly generated. "" "superpass"
memphis.creds.connectionToken Token for connecting an app to the Memphis Message Queue. Auto generated.Randomly generated. "" "connectionToken
memphis.creds.jwtSecret For internal traffic. Randomly generated. "" "<JWT_TOKEN>"
memphis.creds.refreshJwtSecret For internal traffic. Randomly generated. "" "<JWT_TOKEN>"
memphis.creds.encryptionSecretKey Encryption secret key for internal encryption. Randomly generated. "" ""
memphis.creds.secretConfig.name *Optional*
Name of the secret
"memphis-creds" "memphis-creds"
memphis.creds.secretConfig.existingSecret *Optional*
Is this secret an existing secret
"false" "false"
memphis.creds.secretConfig.rootPwd_key *Optional*
Name of the key in secret
"ROOT_PASSWORD" "rootPwd"
memphis.creds.secretConfig.connectionToken_key *Optional*
Name of the key in secret
"CONNECTION_TOKEN" "connectionToken"
memphis.creds.secretConfig.jwtSecret_key *Optional*
Name of the key in secret
"JWT_SECRET" "jwtSecret"
memphis.creds.secretConfig.refreshJwtSecret_key *Optional*
Name of the key in secret
"REFRESH_JWT_SECRET" "refreshJwtSecret"
memphis.creds.secretConfig.encryptionSecretKey_key *Optional*
Name of the key in secret
"ENCRYPTION_SECRET_KEY" "encryptionSecretKey"
memphis.creds.secretConfig.refreshJwtSecretRestGW_key *Optional*
Name of the key in secret
"REFRESH_JWT_SECRET_REST_GW" "refreshJwtSecretRestGW"
memphis.creds.secretConfig.jwtSecretRestGW_key *Optional*
Name of the key in secret
"JWT_SECRET_REST_GW" "jwtSecretRestGW"
memphis.extraEnvironmentVars.enabled *Optional*
List of additional environment variables for memphis.
"" vars:
- name: KEY
- valye: value
memphis.tls.verify *Optional*
For encrypted client-memphis communication. Verification for the CA autority. SSL.
"" "true"
memphis.tls.secret.name *Optional*
For encrypted client-memphis communication.
K8S secret name that holds the certs. SSL.
"" "memphis-client-tls-secret"
memphis.tls.cert *Optional*
For encrypted client-memphis communication.
.pem file to use. SSL.
"" "memphis_client.pem"
memphis.tls.key *Optional*
For encrypted client-memphis communication.
Private key file to use. SSL.
"" "memphis-key_client.pem"
memphis.tls.ca *Optional*
For encrypted client-memphis communication.
CA file to use. SSL.
"" "rootCA.pem"
websocket.enabled *Optional* Memphis GUI using websockets for live rendering. "" "false"
websocket.port Memphis GUI using websockets for live rendering. The port can be configured "7770" "77777"
websocket.host Websocket host can be handled on separate LB/DNS. "localhost" "ws.example.com"
websocket.noTLS Websocket can be configured with tls, default is noTLS. "true" "false"
websocket.tls.secret.name *Optional* Memphis GUI using websockets for live rendering.
K8S secret name for the certs
"" "memphis-ws-tls-secret"
websocket.tls.cert *Optional*
Memphis GUI using websockets for live rendering.
.pem file to use
"" "memphis_local.pem"
websocket.tls.key *Optional*
Memphis GUI using websockets for live rendering.
key file
"" "memphis-key_local.pem"
metadata.postgresql.username *Optional*
Username for postgres db
"postgres" "postgres"
metadata.postgresql.existingSecret *Optional*
An ability to provide predefined secret for metadata PostgreSQL credentials
"" "metadata-creds.yaml"
metadata.pgpool.existingSecret *Optional*
An ability to provide predefined secret for metadata PG credentials
"" "metadata-creds.yaml"
metadata.pgpool.tls.enabled *Optional*
Enabling TLS-based communication with PG
"false" "false"
metadata.pgpool.tls.certificatesSecret *Optional*
PG TLS cert secret to be used
"" "tls-secret"
metadata.pgpool.tls.certFilename *Optional*
PG TLS cert file to be used
"" "tls.crt"
metadata.pgpool.tls.certKeyFilename *Optional*
PG TLS key to be used
"" "tls.key"
metadata.pgpool.tls.certCAFilename *Optional*
PG TLS cert CA to be used
"" "ca.crt"
metadata.external.enabled *Optional*
For using external PG instead of deploying dedicated one for Memphis
"false" "true"
metadata.external.dbTlsMutual *Optional*
External PG TLS-basec communication
"true" "true"
metadata.external.dbName *Optional*
External PG db name
"" "memphis"
metadata.external.dbHost *Optional*
External PG db hostname
"" "metadata.example.url"
metadata.external.dbPort *Optional*
External PG db port
"" 5432
metadata.external.dbUser *Optional*
External PG db user
"" "postgres"
metadata.external.dbPass *Optional*
External PG db password
"" "12345678"
metadata.external.secret.enabled *Optional*
Enable an option to use secret for password store
"false" "true"
metadata.external.secret.name *Optional*
Secret name
"" "metadata-secret"
metadata.external.secret.dbPass_key *Optional*
Name of the key in the secret
"" "dbPass"
restGateway.enabled *Optional*
Memphis Rest Gateway can be disabled if not in use
"true" "false"
restGateway.jwtSecret *Optional*
Manual Jwt Token configurtion
"" ""
restGateway.refreshJwtSecret *Optional*
Manual Refresh Jwt Token configurtion
"" ""
auth.enabled *Optional*
Enable using predefined parameters
"false" "true"
auth.enabled.mgmt *Optional*
Management users that will be created at first deployment
"" ""
auth.enabled.client *Optional*
Client users that will be created at first deployment
"" ""
Here is how to run an installation command with additional options -
helm install memphis --set cluster.replicas=3,memphis.creds.rootPwd=rootpassword" memphis/memphis --create-namespace --namespace memphis

Deployed pods

  • memphis. Memphis broker.
  • memphis-rest-gateway. Memphis REST Gateway.
  • memphis-metadata. Metadata store.
  • memphis-metadata-coordinator. Metadata coordinator

For more information on each component, please head to the architecture section.

Deploy Memphis with TLS (encrypted communication via SSL)

0. Optional: Create self-signed certificates

a) Generate a self-signed certificate using mkcert

$ mkcert -client \
-cert-file memphis_client.pem \
-key-file memphis-key_client.pem  \
"127.0.0.1" "localhost" "*.memphis.dev" ::1 \
email@localhost [email protected]

b) Find the rootCA

$ mkcert -CAROOT

c) Create self-signed certificates for client

$ mkcert -client -cert-file client.pem -key-file key-client.pem  localhost ::1 

1. Create namespace + secret for the TLS certs

a) Create a dedicated namespace for memphis

kubectl create namespace memphis

b) Create a k8s secret with the required certs

kubectl create secret generic memphis-client-tls-secret \
--from-file=memphis_client.pem \
--from-file=memphis-key_client.pem \
--from-file=rootCA.pem -n memphis
tls:
  secret:
    name: memphis-client-tls-secret
  ca: "rootCA.pem"
  cert: "memphis_client.pem"
  key: "memphis-key_client.pem"

2. Deploy Memphis with the generated certificate

helm install memphis memphis \
--create-namespace --namespace memphis --wait \
--set \
global.cluster.enabled="true",\
memphis.tls.verify="true",\
memphis.tls.cert="memphis_client.pem",\
memphis.tls.key="memphis-key_client.pem",\
memphis.tls.secret.name="memphis-client-tls-secret",\
memphis.tls.ca="rootCA.pem"

Upgrade existing deployment

For adding TLS support

  1. Create a k8s secret with the provided TLS certs
kubectl create secret generic memphis-client-tls-secret \
--from-file=memphis_client.pem \
--from-file=memphis-key_client.pem \
--from-file=rootCA.pem -n memphis
  1. Upgrade Memphis to use the TLS certs
helm upgrade memphis memphis -n memphis --reuse-values \
--set \
memphis.tls.verify="true",\
memphis.tls.cert="memphis_client.pem",\
memphis.tls.key="memphis-key_client.pem",\
memphis.tls.secret.name="tls-client-secret",\
memphis.tls.ca="rootCA.pem"

Deployment diagram

Memphis Architecture (1)