feat: Adding support for AWS Batch #620
Conversation
* Add the AWS Batch blueprint under `schedulers/terraform` * Adds tutorial for running an example AWS Batch job in the created EKS cluster * Added AWS Batch to Job Schedulers index page.
delagoya
changed the title
There was a problem hiding this comment.
Thanks for the PR @delagoya ! I did first round of review and left some comments.
| module "eks_auth" { | ||
| source = "terraform-aws-modules/eks/aws//modules/aws-auth" | ||
| version = "~> 20.0" | ||
|
|
||
| manage_aws_auth_configmap = true | ||
| aws_auth_roles = [ | ||
| { | ||
| rolearn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/AWSServiceRoleForBatch" | ||
| username = "aws-batch" | ||
| groups = [] | ||
| }, | ||
| { | ||
| rolearn = module.eks.eks_managed_node_groups["default"].iam_role_arn | ||
| username = "system:node:{{EC2PrivateDNSName}}" | ||
| groups = ["system:bootstrappers", "system:nodes"] | ||
| }, | ||
| { | ||
| rolearn = aws_iam_role.batch_eks_instance_role.arn | ||
| username = "system:node:{{EC2PrivateDNSName}}" | ||
| groups = ["system:bootstrappers", "system:nodes"] | ||
| } | ||
| ] | ||
| depends_on = [ | ||
| module.eks, | ||
| kubernetes_namespace.doeks_batch_namespace, | ||
| kubernetes_cluster_role.batch_cluster_role, | ||
| kubernetes_cluster_role_binding.batch_cluster_role_binding, | ||
| kubernetes_role.batch_compute_env_role, | ||
| kubernetes_role_binding.batch_compute_env_role_binding | ||
| ] | ||
| } |
There was a problem hiding this comment.
You dont need this. you can pass the additional auth roles through variables. Check the EKS TF module for input variables.
There was a problem hiding this comment.
Can you point me at an example? I am not seeing the input for config maps from the EKS TF module. I only see how to do this via the aws-auth submodule which I am using in the above code.
| variable "vpc_cidr" { | ||
| description = "VPC CIDR" | ||
| type = string | ||
| default = "10.1.0.0/16" | ||
| } | ||
|
|
||
| # Only two Subnets for with low IP range for internet access | ||
| variable "public_subnets" { | ||
| description = "Public Subnets CIDRs. 62 IPs per Subnet" | ||
| type = list(string) | ||
| default = ["10.1.255.128/26", "10.1.255.192/26"] | ||
| } | ||
|
|
||
| variable "private_subnets" { | ||
| description = "Private Subnets CIDRs. 32766 Subnet1 and 16382 Subnet2 IPs per Subnet" | ||
| type = list(string) | ||
| default = ["10.1.0.0/17", "10.1.128.0/18"] | ||
| } | ||
|
|
||
| variable "eks_public_cluster_endpoint" { | ||
| description = "Whether to have a public cluster endpoint for the EKS cluster. #WARNING: Avoid a public endpoint in preprod or prod accounts. This feature is designed for sandbox accounts, simplifying cluster deployment and testing." | ||
| type = bool | ||
| default = true | ||
| } | ||
|
|
||
| variable "eks_private_cluster_endpoint" { | ||
| description = "Whether to have a private cluster endpoint for the EKS cluster." | ||
| type = bool | ||
| default = true | ||
| } | ||
|
|
There was a problem hiding this comment.
We might need all of this. Let's refer to existing blueprints. You can use this example https://github.com/awslabs/data-on-eks/blob/ff145308263203072229d4db79285919a21af42b/analytics/terraform/spark-k8s-operator/vpc.tf#L7C1-L41C2
There was a problem hiding this comment.
Which part are you referring to? The private vs public endpoint? The CIDR specifications?
|
Any update on review? |
Adding support for AWS Batch, addresses issue #456
schedulers/terraformWhat does this PR do?
🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.
Adds support for AWS Batch.
Motivation
AWS Batch supports running high performance, compute intensive, and batch workloads on EKS. This blueprint provides a template to stand up all required AWS resources and Kubernetes permissions to enable AWS Batch managed services to launch jobs as pods on an EKS cluster.
More
website/docsorwebsite/blogsection for this featurepre-commit run -awith this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes