Manually create a cluster (no terraform)

[GeorgianaElena]

Resources to get us started:

• https://docs.jetstream-cloud.org/getting-started/overview/
• https://www.zonca.dev/posts/2024-12-11-jetstream_kubernetes_magnum

Definition of done:

• 2i2c engineering is able to reliably and consistently deploy a Kubernetes cluster using openstack coe cluster create 3 times in a row over 2 days
• 2i2c engineering is able to reliably and consistently scale the number of nodes the cluster has reliably and repeatably via openscack commands
• This process is documented -> https://hackmd.io/DzgY3PW4TUOMAmpqvgOdZw?view#How-to-create-and-scale-a-cluster-on-Jetstream2

[GeorgianaElena]

@jmunroe, I believe you mentioned that there were some docs about this from the project pythia side 🤔? Could you please share those here so we can have them as a reference when we start working on this on Monday?

[sgibson91]

I interpreted James' comment as "Project Pythia followed JetStream's docs and they got through it fine", just to give a different interpretation. May be wrong though!

[GeorgianaElena]

@sgibson91 makes sense. My understanding was that they experimented with it documented the whole process 🤷‍♀
Putting the jetstream getting started docs in the top comment to have it as a reference

[jmunroe]

My assumption is that these are the correct starting points:

• https://docs.jetstream-cloud.org/ui/cacao/deployment_kubernetes/
• https://docs.jetstream-cloud.org/ui/cacao/deployment_jupyterhub/

There have been previous uses of kubernetes on JetStream2 such as kubespray (docs) but I had understood those to be fairly manual, non-scalable ways of deploying a cluster.

What my ask of the JetStream2 team has been a 'managed kubernetes service' that we can build on top of. I think OpenStack Magnum and ClusterAPI are some of the enabling technologies used by the JetStream2 team but I am not entirely up to speed on the details.

My primary contact at JetStream has been Julian Pistorius (jupist@iu.edu). Julian is already on our 2i2c slack.

[GeorgianaElena]

Current state

@sgibson91 and I started deploying a cluster today to the new allocation that @jmunroe created for us.

The process currently fails with CREATE_FAILED:

status_reason  | Failed to create trustee or trust for Cluster

While investigating this, we realized that we don't have access to run commands such as:

• openstack user show trustee_domain_admin

• openstack service list because of lack of credentials

• ForbiddenException: 403: Client Error for url: https://js2.jetstream-cloud.org:5000/v3/users?name=trustee_domain_admin

• ForbiddenException: 403: Client Error for url: https://js2.jetstream-cloud.org:5000/v3/services, You are not authorized to perform the requested action: identity:list_services.

What we've tried

We tried to create new application credentials that would be more permissive and give it the Unrestricted dangerous option hoping it will give us the right to run the create cluster command, but it didn't work.

The only available roles for us are the ones in the screenshot below:

From the blog post we are following it looks like the loadbalancer role should fix it but that's not available. Also, given that we're not able to list resources either like services, we might need more than that, such as access to the identity API endpoint.

[jmunroe]

I've emailed Julian to seek additional guidance (See https://2i2c.freshdesk.com/a/tickets/2690).

[jmunroe]

Potentially relevant Jetstream2 issues:

• Prove and document use of Kubernetes Cluster API: https://gitlab.com/jetstream-cloud/project-mgt/-/issues/108
• Explore Jupyter interface for Jetstream2: https://gitlab.com/jetstream-cloud/project-mgt/-/issues/100
• Install OpenStack Magnum https://gitlab.com/jetstream-cloud/jetstream2/project-management/-/issues/37
• Problem: There is no documentation for using Magnum https://gitlab.com/jetstream-cloud/jetstream2/project-management/-/issues/217

[GeorgianaElena]

Thank you @jmunroe! I also found some interesting docs about:
- the openstack identity service https://docs.openstack.org/mitaka/install-guide-obs/common/get_started_identity.html that might help us
- magnum specific identity service (keystone): https://docs.openstack.org/magnum/latest/user/#keystone-authn-and-authz

Nvm, we still need permissions to the identity endpoint!

[GeorgianaElena]

Update:

@sgibson91 and I opened a ticket on JetStream2 at https://jetstream-cloud.org/contact/index.html asking for guidance about this permission error. Confirmation email https://2i2c.freshdesk.com/a/tickets/2691

[jmunroe]

I've had success creating a Kubernetes cluster using Magnum following Andrea Zonca's blog post. When I was create the application credentials I did select the 'unrestricted (dangerous)' option. I still can't run all openstack commands (like openstack service list) but it does not seem like those are actual blockers to deploying a scalable kubernetes cluster.

I did need to be patient though. It took 117 minutes for the cluster to be created while in zonca's post he timed it at 9 minutes. Perhaps now that images have been copied over, will it run faster?

I think we should increate the quota on the number of Volume that openstack allows. I'll submit that support ticket to the Jetstream2 team.

@GeorgianaElena Please let me know if you'd like to meet tomorrow so we can make sure you are able to do what appears to work for me. I'll grab an early morning slot on your calendar.