Skip to content

Commit

Permalink
Rename brokerHost to brokerHosts for Kafka connector
Browse files Browse the repository at this point in the history 
The newest version of the Kafka connector will support multiple
bootstrap servers as a comma-separated list via the brokerHosts
variable, this is a breaking change from previously being
brokerHost, and only supporting a single host.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
  • Loading branch information
@alexellis
alexellis committed Sep 19, 2023
1 parent dcab656 commit 4a53edf
Show file tree
Hide file tree
Showing 9 changed files with 65 additions and 53 deletions.
52 changes: 27 additions & 25 deletions chart/kafka-connector/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,31 +75,33 @@ $ helm upgrade kafka-connector ./chart/kafka-connector \

Additional kafka-connector options in `values.yaml`.

| Parameter | Description | Default |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------ |
| `topics` | Topics to which the connector will bind, provide as a comma-separated list or a single item. | `faas-request` |
| `brokerHost` | location of the Kafka brokers. | `kafka` |
| `asyncInvocation` | For long running or slow functions, offload to asychronous function invocations and carry on processing the stream | `false` |
| `upstreamTimeout` | Maximum timeout for upstream function call, must be a Go formatted duration string. | `2m` |
| `rebuildInterval` | Interval for rebuilding function to topic map, must be a Go formatted duration string. | `30s` |
| `gatewayURL` | The URL for the API gateway. | `http://gateway.openfaas:8080` |
| `printResponse` | Output the response of calling a function in the logs. | `true` |
| `printResponseBody` | Output to the logs the response body when calling a function. | `false` |
| `printRequestBody` | Output to the logs the request body when calling a function. | `false` |
| `fullnameOverride` | Override the name value used for the Connector Deployment object. | `` |
| `sasl` | Enable auth with a SASL username/password | `false` |
| `brokerPasswordSecret` | Name of secret for SASL password | `kafka-broker-password` |
| `brokerUsernameSecret` | Name of secret for SASL username | `kafka-broker-username` |
| `caSecret` | Name secret for TLS CA - leave empty to disable | `kafka-broker-ca` |
| `certSecret` | Name secret for TLS client certificate cert - leave empty to disable | `kafka-broker-cert` |
| `keySecret` | Name secret for TLS client certificate private key - leave empty to disable | `kafka-broker-key` |
| `contentType` | Set a HTTP Content Type during function invocation. | `""` |
| `group` | Set the Kafka consumer group name. | `""` |
| `maxBytes` | Set the maximum size of messages from the Kafka broker. | `1024*1024` |
| `sessionLogging` | Enable detailed logging from the consumer group. | `"false"` |
| `initialOffset` | Either newest or oldest. | `"oldest"` |
| `logs.debug` | Print debug logs | `false` |
| `logs.format` | The log encoding format. Supported values: `json` or `console` | `console` |
| Parameter | Description | Default |
|------------------------|------------------------------------------------------------------------------------------------------------------------------------|--------------------------------|
| `topics` | A single topic or list of comma separated topics to consume. | `faas-request` |
| `replicas` | The number of replicas of this connector, should be set to the size of the partition for the given topic, or a higher lower value. | `1` |
| `brokerHosts` | Host and port for the Kafka bootstrap server, multiple servers can be specified as a comma-separated list. | `kafka:9092` |
| `asyncInvocation` | For long running or slow functions, offload to asychronous function invocations and carry on processing the stream | `false` |
| `upstreamTimeout` | Maximum timeout for upstream function call, must be a Go formatted duration string. | `2m` |
| `rebuildInterval` | Interval for rebuilding function to topic map, must be a Go formatted duration string. | `30s` |
| `gatewayURL` | The URL for the API gateway. | `http://gateway.openfaas:8080` |
| `printResponse` | Output the response of calling a function in the logs. | `true` |
| `printResponseBody` | Output to the logs the response body when calling a function. | `false` |
| `printRequestBody` | Output to the logs the request body when calling a function. | `false` |
| `fullnameOverride` | Override the name value used for the Connector Deployment object. | `` |
| `tls` | Connect to the broker server(s) using TLS encryption | `true` |
| `sasl` | Enable auth with a SASL username/password | `false` |
| `brokerPasswordSecret` | Name of secret for SASL password | `kafka-broker-password` |
| `brokerUsernameSecret` | Name of secret for SASL username | `kafka-broker-username` |
| `caSecret` | Name secret for TLS CA - leave empty to disable | `kafka-broker-ca` |
| `certSecret` | Name secret for TLS client certificate cert - leave empty to disable | `kafka-broker-cert` |
| `keySecret` | Name secret for TLS client certificate private key - leave empty to disable | `kafka-broker-key` |
| `contentType` | Set a HTTP Content Type during function invocation. | `""` |
| `group` | Set the Kafka consumer group name. | `""` |
| `maxBytes` | Set the maximum size of messages from the Kafka broker. | `1024*1024` |
| `sessionLogging` | Enable detailed logging from the consumer group. | `"false"` |
| `initialOffset` | Either newest or oldest. | `"oldest"` |
| `logs.debug` | Print debug logs | `false` |
| `logs.format` | The log encoding format. Supported values: `json` or `console` | `console` |

Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. See `values.yaml` for the default configuration.

Expand Down
6 changes: 3 additions & 3 deletions chart/kafka-connector/quickstart.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@
Example for Kafka helm chart:
```yaml
brokerHost: cp-helm-charts-cp-kafka.default:9092
brokerHosts: cp-helm-charts-cp-kafka.default:9092
tls: false
saslAuth: false
Expand All @@ -75,7 +75,7 @@
Example for Aiven cloud with client certificates:
```yaml
brokerHost: kafka-202504b5-openfaas-910b.aivencloud.com:10905
brokerHosts: kafka-202504b5-openfaas-910b.aivencloud.com:10905
tls: true
saslAuth: false
Expand All @@ -87,7 +87,7 @@
Example for Confluent Cloud with Let's Encrypt and SASL auth:
```yaml
brokerHost: pkc-4r297.europe-west1.gcp.confluent.cloud:9092
brokerHosts: pkc-4r297.europe-west1.gcp.confluent.cloud:9092
tls: true
saslAuth: true
Expand Down
4 changes: 2 additions & 2 deletions chart/kafka-connector/templates/deployment.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,8 @@ spec:
- name: secret_mount_path
value: "/var/secrets"
{{- end }}
- name: broker_host
value: {{ .Values.brokerHost | quote }}
- name: broker_hosts
value: {{ .Values.brokerHosts | quote }}
{{- if .Values.upstreamTimeout }}
- name: upstream_timeout
value: {{ .Values.upstreamTimeout | quote }}
Expand Down
30 changes: 17 additions & 13 deletions chart/kafka-connector/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,15 +1,27 @@
# The Kafka connector is an OpenFaaS Pro feature
#
# Purchase a license at https://openfaas.com/support/
# This component is licensed for OpenFaaS Standard/Enterprise customers only.
# Purchase or obtain a trial license at https://openfaas.com/pricing/

# You will need to create a license named "openfaas-license" - see the
# chart README for detailed instructions.


# Set either a single topic, or multiple with a comma separating each
# topics: payment.requested
# topics: "payment.requested,customer.created,invoice.generated"
topics: faas-request

# The address of the Kafka broker, if multiple are available, include
# them as a comma separated list i.e. "kafka-1:9092,kafka-2:9092,kafka-3:9092"
brokerHosts: kf-kafka:9092

image: ghcr.io/openfaasltd/kafka-connector:0.7.3

# Output detailed logs from the consumer group's lifecycle
sessionLogging: true

# The number of replicas of the connector should usually match the size of the
# partition exactly. So for a topic with 6 partitions, set the replicas to 6.
# This number must not be higher, but can be lower, to save on resources.
replicas: 1

# Max timeout for a function
Expand All @@ -21,14 +33,6 @@ rebuildInterval: 30s
# Use with slow consumers or long running functions
asyncInvocation: false

# Set either a single topic, or multiple with a comma separating each
# topics: payment.requested
# topics: payment.requested,customer.created,invoice.generated
topics: faas-request

# Your Kafka broker
brokerHost: kf-kafka:9092

# 1MB = 1024 bytes * 1024
maxBytes: "1048576"

Expand Down Expand Up @@ -85,8 +89,8 @@ resources:
# limits:
# memory: "256Mi"

# Encryption
tls: false
# Use TLS Encryption, this is usually set to true.
tls: true

# Authentication

Expand Down
5 changes: 2 additions & 3 deletions chart/postgres-connector/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# The Postgres connector is an OpenFaaS Pro feature
#
# Purchase a license at https://openfaas.com/support/
# This component is licensed for OpenFaaS Standard/Enterprise customers only.
# Purchase or obtain a trial license at https://openfaas.com/pricing/

# You will need to create a license named "openfaas-license" - see the
# chart README for detailed instructions.
Expand Down
5 changes: 4 additions & 1 deletion chart/pro-builder/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# You will need to create a license named "openfaas-license" - see the
# This component is licensed for OpenFaaS Standard/Enterprise customers only.
# Purchase or obtain a trial license at https://openfaas.com/pricing/

# You will need to create a license named "openfaas-license" - see the
# chart README for detailed instructions.

# image is for the pro-builder API
Expand Down
6 changes: 6 additions & 0 deletions chart/queue-worker/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
# This component is licensed for OpenFaaS Standard/Enterprise customers only.
# Purchase or obtain a trial license at https://openfaas.com/pricing/

# You will need to create a license named "openfaas-license" - see the
# chart README for detailed instructions.

image: ghcr.io/openfaasltd/jetstream-queue-worker:0.3.13

replicas: 1
Expand Down
5 changes: 2 additions & 3 deletions chart/sns-connector/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# The Kafka connector is an OpenFaaS Pro feature
#
# Purchase a license at https://openfaas.com/support/
# This component is licensed for OpenFaaS Standard/Enterprise customers only.
# Purchase or obtain a trial license at https://openfaas.com/pricing/

# You will need to create a license named "openfaas-license" - see the
# chart README for detailed instructions.
Expand Down
5 changes: 2 additions & 3 deletions chart/sqs-connector/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# The Kafka connector is an OpenFaaS Pro feature
#
# Purchase a license at https://openfaas.com/support/
# This component is licensed for OpenFaaS Standard/Enterprise customers only.
# Purchase or obtain a trial license at https://openfaas.com/pricing/

# You will need to create a license named "openfaas-license" - see the
# chart README for detailed instructions.
Expand Down

0 comments on commit 4a53edf

Please sign in to comment.