Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Flink operator blueprint deployment errors #731

Merged
merged 4 commits into from
Jan 29, 2025
Merged

fix: Flink operator blueprint deployment errors #731

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8a4b689
fix flink operator blueprint deployment errors
ratnopamc Jan 29, 2025
cab4db0
update flink operator deployment
ratnopamc Jan 29, 2025
16d8a97
update flink operator deployment
ratnopamc Jan 29, 2025
7ac0875
delete trino binary
ratnopamc Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions analytics/terraform/spark-k8s-operator/examples/s3-tables/s3table-spark-operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ spec:
imagePullPolicy: IfNotPresent
mainApplicationFile: "s3a://<S3_BUCKET>/s3table-example/scripts/s3table-iceberg-pyspark.py"
arguments:
- "s3a://<S3_BUCKET>/s3table-example/input/"
- "<S3TABLE_ARN>"
- "s3a://<S3_BUCKET>/s3table-example/input/"
- "<S3TABLE_ARN>"
sparkConf:
"spark.app.name": "s3table-example"
"spark.kubernetes.driver.pod.name": "s3table-example"
Expand All @@ -39,7 +39,7 @@ spec:
"spark.hadoop.fs.s3a.readahead.range": "256K"
"spark.hadoop.fs.s3a.input.fadvise": "random"
"spark.hadoop.fs.s3a.aws.credentials.provider.mapping": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider=software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider"
"spark.hadoop.fs.s3a.aws.credentials.provider": "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider" # AWS SDK V2 https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/aws_sdk_upgrade.html
"spark.hadoop.fs.s3a.aws.credentials.provider": "software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider" # AWS SDK V2 https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/aws_sdk_upgrade.html

restartPolicy:
type: OnFailure
Expand Down
20 changes: 10 additions & 10 deletions streaming/flink/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,21 @@
| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.72 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.34 |
| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.4.1 |
| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 1.14 |
| <a name="requirement_kubectl"></a> [kubectl](#requirement\_kubectl) | >= 2.0 |
| <a name="requirement_kubernetes"></a> [kubernetes](#requirement\_kubernetes) | >= 2.10 |
| <a name="requirement_random"></a> [random](#requirement\_random) | 3.3.2 |
| <a name="requirement_random"></a> [random](#requirement\_random) | 3.6.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 3.72 |
| <a name="provider_aws.ecr"></a> [aws.ecr](#provider\_aws.ecr) | >= 3.72 |
| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 1.14 |
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.34 |
| <a name="provider_aws.ecr"></a> [aws.ecr](#provider\_aws.ecr) | >= 5.34 |
| <a name="provider_kubectl"></a> [kubectl](#provider\_kubectl) | >= 2.0 |
| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | >= 2.10 |
| <a name="provider_random"></a> [random](#provider\_random) | 3.3.2 |
| <a name="provider_random"></a> [random](#provider\_random) | 3.6.0 |

## Modules

Expand All @@ -47,10 +47,10 @@
| [aws_prometheus_workspace.amp](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/prometheus_workspace) | resource |
| [aws_secretsmanager_secret.grafana](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret) | resource |
| [aws_secretsmanager_secret_version.grafana](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/secretsmanager_secret_version) | resource |
| [kubectl_manifest.karpenter_provisioner](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/manifest) | resource |
| [kubectl_manifest.karpenter_provisioner](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/resources/manifest) | resource |
| [kubernetes_role.flink](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/role) | resource |
| [kubernetes_role_binding.flink](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/role_binding) | resource |
| [random_password.grafana](https://registry.terraform.io/providers/hashicorp/random/3.3.2/docs/resources/password) | resource |
| [random_password.grafana](https://registry.terraform.io/providers/hashicorp/random/3.6.0/docs/resources/password) | resource |
| [aws_ami.x86](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
Expand All @@ -62,7 +62,7 @@
| [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
| [aws_secretsmanager_secret_version.admin_password_version](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret_version) | data source |
| [kubectl_path_documents.karpenter_provisioners](https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/data-sources/path_documents) | data source |
| [kubectl_path_documents.karpenter_provisioners](https://registry.terraform.io/providers/alekc/kubectl/latest/docs/data-sources/path_documents) | data source |

## Inputs

Expand Down
33 changes: 19 additions & 14 deletions streaming/flink/addons.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ module "eks_blueprints_addons" {
}
}
karpenter = {
chart_version = "v0.33.1"
chart_version = "v0.37.1"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
}
Expand All @@ -99,7 +99,7 @@ module "eks_blueprints_addons" {
#---------------------------------------
# AWS for FluentBit - DaemonSet
#---------------------------------------
enable_aws_for_fluentbit = true
enable_aws_for_fluentbit = false
aws_for_fluentbit_cw_log_group = {
create = true
use_name_prefix = false
Expand All @@ -120,19 +120,25 @@ module "eks_blueprints_addons" {
})]
}

#---------------------------------------
# AWS Load Balancer Controller Add-on
#---------------------------------------
enable_aws_load_balancer_controller = true
# turn off the mutating webhook for services because we are using
# service.beta.kubernetes.io/aws-load-balancer-type: external
aws_load_balancer_controller = {
values = [
<<-EOT
clusterName: ${module.eks.cluster_name}
EOT
]
set = [{
name = "enableServiceMutatorWebhook"
value = "false"
}]
}

#---------------------------------------
# Ingress Nginx Add-on
#---------------------------------------
enable_ingress_nginx = true
ingress_nginx = {
timeout = "300"
values = [templatefile("${path.module}/helm-values/nginx-values.yaml", {})]
values = [templatefile("${path.module}/helm-values/nginx-values.yaml", {})]
}

# Required Flink Deployment
Expand Down Expand Up @@ -182,18 +188,17 @@ module "eks_blueprints_addons" {
# Data on EKS Kubernetes Addons
#---------------------------------------------------------------
module "eks_data_addons" {
source = "aws-ia/eks-data-addons/aws"
version = "~> 1.0" # ensure to update this to the latest/desired version

source = "aws-ia/eks-data-addons/aws"
version = "~> 1.0" # ensure to update this to the latest/desired version
oidc_provider_arn = module.eks.oidc_provider_arn

#---------------------------------------------------------------
# Flink Operator Add-on
#---------------------------------------------------------------
enable_flink_operator = true
flink_operator_helm_config = {
version = "1.7.0"
values = [templatefile("${path.module}/helm-values/flink-operator-values.yaml", {})]
version = "1.10.0"
values = [templatefile("${path.module}/helm-values/flink-operator-values.yaml", {})]
}

#---------------------------------------------------------------
Expand Down
8 changes: 4 additions & 4 deletions streaming/flink/versions.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 3.72"
version = ">= 5.34"
}
kubernetes = {
source = "hashicorp/kubernetes"
Expand All @@ -15,12 +15,12 @@ terraform {
version = ">= 2.4.1"
}
kubectl = {
source = "gavinbunney/kubectl"
version = ">= 1.14"
source = "alekc/kubectl"
version = ">= 2.0"
}
random = {
source = "hashicorp/random"
version = "3.3.2"
version = "3.6.0"
}
}

Expand Down
Binary file removed trino
View file
Open in desktop
Binary file not shown.
Loading