Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Spark Operator update and IPv6 #609

Closed
wants to merge 31 commits into from

Conversation

alanty
Copy link
Contributor

@alanty alanty commented Aug 12, 2024

What does this PR do?

This branch is what I've been using for testing/benchmarking, opening a PR to discuss and refine:

Big Things

  • Updated the VPC to use database_subnets in the module as EKS subnets for the 100.64.x.x range.
  • Added an enable_ipv6 variable that will create IPv6 resources in the VPC, EKS cluster, and Addons
  • Added a (rough) IRSA role for the Spark Operator so it can leverage PodTemplates stored in S3

Smaller Things/Chores

  • Updated the addons to latest I can find
  • Uses encrypted gp3 volumes by default
  • Updated the examples with docker images for spark 3.5.1 and manifest configs.

Motivation

I made a bunch of changes while getting an environment setup. This version adds IPv6 to the existing VPC architecture rather than starting with a new one, which is kind of overkill compared to #499

More

  • Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
  • Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
  • Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature - WiP
  • Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

@askulkarni2 askulkarni2 changed the title Spark Operator update and IPv6 feat: Spark Operator update and IPv6 Aug 13, 2024
@alanty
Copy link
Contributor Author

alanty commented Aug 14, 2024

Closing this in favor of #499, this version is a bit complex/overkill to maintain.
We'll use a separate blueprint for ipv6, i'll open a new PR to merges the updates/minor stuff from this version on top of the other PR.

@alanty alanty closed this Aug 14, 2024
@alanty alanty deleted the spark-op-update-07-2024 branch October 10, 2024 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant