Skip to content

Commit

Permalink
fix: Flink operator blueprint deployment errors (#731)
Browse files Browse the repository at this point in the history
  • Loading branch information
@ratnopamc
ratnopamc authored Jan 29, 2025
1 parent 41d6899 commit 13503e1
Show file tree
Hide file tree
Showing 5 changed files with 36 additions and 31 deletions.
6 changes: 3 additions & 3 deletions analytics/terraform/spark-k8s-operator/examples/s3-tables/s3table-spark-operator.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ spec:
imagePullPolicy: IfNotPresent
mainApplicationFile: "s3a://<S3_BUCKET>/s3table-example/scripts/s3table-iceberg-pyspark.py"
arguments:
- "s3a://<S3_BUCKET>/s3table-example/input/"
- "<S3TABLE_ARN>"
- "s3a://<S3_BUCKET>/s3table-example/input/"
- "<S3TABLE_ARN>"
sparkConf:
"spark.app.name": "s3table-example"
"spark.kubernetes.driver.pod.name": "s3table-example"
Expand All @@ -39,7 +39,7 @@ spec:
"spark.hadoop.fs.s3a.readahead.range": "256K"
"spark.hadoop.fs.s3a.input.fadvise": "random"
"spark.hadoop.fs.s3a.aws.credentials.provider.mapping": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider=software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider"
"spark.hadoop.fs.s3a.aws.credentials.provider": "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider" # AWS SDK V2 https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/aws_sdk_upgrade.html
"spark.hadoop.fs.s3a.aws.credentials.provider": "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider" # AWS SDK V2 https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/aws_sdk_upgrade.html

restartPolicy:
type: OnFailure
Expand Down
20 changes: 10 additions & 10 deletions streaming/flink/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,21 @@
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.72 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.34 |
| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |
| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 1.14 |
| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 2.0 |
| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |
| <a name="requirement_random"></a> [random](#requirement\_random) | 3.3.2 |
| <a name="requirement_random"></a> [random](#requirement\_random) | 3.6.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.72 |
| <a name="provider_aws.ecr"></a> [aws.ecr](#provider\_aws.ecr) | >= 3.72 |
| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 1.14 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.34 |
| <a name="provider_aws.ecr"></a> [aws.ecr](#provider\_aws.ecr) | >= 5.34 |
| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 2.0 |
| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | >= 2.10 |
| <a name="provider_random"></a> [random](#provider\_random) | 3.3.2 |
| <a name="provider_random"></a> [random](#provider\_random) | 3.6.0 |

## Modules

Expand All @@ -47,10 +47,10 @@
| [aws_prometheus_workspace.amp](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_workspace) | resource |
| [aws_secretsmanager_secret.grafana](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret) | resource |
| [aws_secretsmanager_secret_version.grafana](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret_version) | resource |
| [kubectl_manifest.karpenter_provisioner](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
| [kubectl_manifest.karpenter_provisioner](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
| [kubernetes_role.flink](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/role) | resource |
| [kubernetes_role_binding.flink](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/role_binding) | resource |
| [random_password.grafana](https://registry.terraform.io/providers/hashicorp/random/3.3.2/docs/resources/password) | resource |
| [random_password.grafana](https://registry.terraform.io/providers/hashicorp/random/3.6.0/docs/resources/password) | resource |
| [aws_ami.x86](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
Expand All @@ -62,7 +62,7 @@
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
| [aws_secretsmanager_secret_version.admin_password_version](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret_version) | data source |
| [kubectl_path_documents.karpenter_provisioners](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/data-sources/path_documents) | data source |
| [kubectl_path_documents.karpenter_provisioners](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/data-sources/path_documents) | data source |

## Inputs

Expand Down
33 changes: 19 additions & 14 deletions streaming/flink/addons.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ module "eks_blueprints_addons" {
}
}
karpenter = {
chart_version = "v0.33.1"
chart_version = "v0.37.1"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
}
Expand All @@ -99,7 +99,7 @@ module "eks_blueprints_addons" {
#---------------------------------------
# AWS for FluentBit - DaemonSet
#---------------------------------------
enable_aws_for_fluentbit = true
enable_aws_for_fluentbit = false
aws_for_fluentbit_cw_log_group = {
create = true
use_name_prefix = false
Expand All @@ -120,19 +120,25 @@ module "eks_blueprints_addons" {
})]
}

#---------------------------------------
# AWS Load Balancer Controller Add-on
#---------------------------------------
enable_aws_load_balancer_controller = true
# turn off the mutating webhook for services because we are using
# service.beta.kubernetes.io/aws-load-balancer-type: external
aws_load_balancer_controller = {
values = [
<<-EOT
clusterName: ${module.eks.cluster_name}
EOT
]
set = [{
name = "enableServiceMutatorWebhook"
value = "false"
}]
}

#---------------------------------------
# Ingress Nginx Add-on
#---------------------------------------
enable_ingress_nginx = true
ingress_nginx = {
timeout = "300"
values = [templatefile("${path.module}/helm-values/nginx-values.yaml", {})]
values = [templatefile("${path.module}/helm-values/nginx-values.yaml", {})]
}

# Required Flink Deployment
Expand Down Expand Up @@ -182,18 +188,17 @@ module "eks_blueprints_addons" {
# Data on EKS Kubernetes Addons
#---------------------------------------------------------------
module "eks_data_addons" {
source = "aws-ia/eks-data-addons/aws"
version = "~> 1.0" # ensure to update this to the latest/desired version

source = "aws-ia/eks-data-addons/aws"
version = "~> 1.0" # ensure to update this to the latest/desired version
oidc_provider_arn = module.eks.oidc_provider_arn

#---------------------------------------------------------------
# Flink Operator Add-on
#---------------------------------------------------------------
enable_flink_operator = true
flink_operator_helm_config = {
version = "1.7.0"
values = [templatefile("${path.module}/helm-values/flink-operator-values.yaml", {})]
version = "1.10.0"
values = [templatefile("${path.module}/helm-values/flink-operator-values.yaml", {})]
}

#---------------------------------------------------------------
Expand Down
8 changes: 4 additions & 4 deletions streaming/flink/versions.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.72"
version = ">= 5.34"
}
kubernetes = {
source = "hashicorp/kubernetes"
Expand All @@ -15,12 +15,12 @@ terraform {
version = ">= 2.4.1"
}
kubectl = {
source = "gavinbunney/kubectl"
version = ">= 1.14"
source = "alekc/kubectl"
version = ">= 2.0"
}
random = {
source = "hashicorp/random"
version = "3.3.2"
version = "3.6.0"
}
}

Expand Down
Binary file removed trino
View file
Binary file not shown.

0 comments on commit 13503e1

Please sign in to comment.