- Introduction
- Bootstrap
- k8s-1
- Install OpenStack (control plane)
- compute-1
- Populate your OpenStack with default values
- For lazy people
- Notes
Main objective is to create a small OpenStack infrastructure within an OVH public cloud project (which is also run by OpenStack by the way :p So we will create an OpenStack over OpenStack).
┌─────────────────────────────────────────────────┐
│ k8s-1 (control plane) │
ssh ├───┐ │
───────────► │ E │ ┌──────────┐ ┌──────────┐ │
you │ N │ │ neutron │ │ mysql │ │
http │ S │ └──────────┘ └──────────┘ │
───────────► │ 3 │ Using: │
├───┘ ┌──────────┐ ┌──────────┐ -kubectl │
│ │ nova │ │ rabbit │ -ansible │
│ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ │
│ │ keystone │ │ horizon │ │
│ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ │
│ │ glance │ │ ... │ │
│ └──────────┘ └──────────┘ │
│ │
└─────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────┐
│ compute-1 (data plane) │
ssh ├───┐ │
you ───────────► │ E │ ┌─────────────────────┐ │
│ N │ │ neutron agents │ Using: │
│ S │ └─────────────────────┘ -ansible │
│ 3 │ │
├───┘ ┌─────────────────────┐ │
│ │ nova compute │ │
│ └─────────────────────┘ │
┌───┐ │ │
│ │ ├───┐ ┌─────────────────────┐ │
│ v │ │ E │ │ openvswitch │ │
inter │ R │ │ N │ └─────────────────────┘ │
net ─────┤ a ├─────┤ S │ │
│ c │ │ 4 │ │
│ k │ ▲ ├───┘ │
│ │ │ │ │
└───┘ │ └─────────────────────────────────────────────────┘
│
└─ Instances public access with /28 network block
routed in vRack (vlan 0)
Both k8s and computes will have a public IP and be accessible from internet.
k8s server will be used to host OpenStack control plane (mostly API, database, queues, schedulers, etc.).
Each OpenStack service will be started in a docker container, orchestrated using kubernetes (k3s).
compute server will be used to host OpenStack data plane (mostly nova compute and neutron agents (L2 and L3)).
compute will also have an extra network interface connected to a vRack (using vlan 0).
In this vRack, a routed network acquired from OVHcloud will give the possibility to create a flat external network.
Instances and routers will be able to use this flat network to reach internet.
ssh [email protected]
# Replace vttx with the username provided to you
# The password is also providedYou need an OpenStack account to continue, source the openrc file now:
source openrcgit clone https://github.com/arnaudmorin/bootstrap-openstack-k8s.git
cd bootstrap-openstack-k8sThe bootstrap.sh script will start 2 instances:
- k8s-1
- compute-1
# Execute this
./bootstrap.sh# List instances you have to retrieve the IPs
openstack server list
# Retry multiple time the previous command until the instances are ready (ACTIVE state)At the end, you should be able to ssh both instances using the zob key:
chmod 600 ansible/files/zob
ssh debian@ip -i ansible/files/zob # replace ip with the real server IPOn the k8s-1 instance, we will install k3s and few other tools to have a full kubernetes cluster.
See https://k3s.io/ for more info.
SSH into k8s-1 and login as root and install k3s:
sudo su -Then
curl -sfL https://get.k3s.io | sh -Create an alias (this will help you saving your keyboard):
alias k='kubectl'Test:
k get allYou should have something like this:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 22sfrep is a tool to generate files from templates. Its re-using the go templating language.
We will use frep to transform k8s templates into manifests.
I did not wanted to use helm for this because it's heavy and more difficult to understand.
frep is much more simpler.
curl -fSL https://github.com/subchen/frep/releases/download/v1.3.12/frep-1.3.12-linux-amd64 -o /usr/local/bin/frep
chmod +x /usr/local/bin/frepMore info here: https://github.com/subchen/frep
We will need ansible and git at some point.
apt-get install -y ansible gitplik is a tool to upload files to a remote URL.
It's useful to easily transfer files a from a system to another.
# nothing to do, it's already done by bootstrap.sh script :)We will need some of the kubernetes templates that are in the repo:
git clone https://github.com/arnaudmorin/bootstrap-openstack-k8s.git
cd bootstrap-openstack-k8sIn the following section, we will use kubectl on k8s-1 to install all OpenStack control plane (so OpenStack will run on top of kubernetes), including the mariadb and rabbitmq.
For steps where you create some kubernetes resources, take your time to check if the resources are well created, eventually slow down your copy paste rate :).
You can also check the result from time to time with:
k get allLet's start!
First, you need to create a config.yaml file:
cp config/config.yaml.sample config/config.yaml
# Change domain - you can use this, it will create a domain automagically:
ip=$(hostname -I | awk '{print $1}')
sed -i -r "s/somewhere.net/${ip}.xip.opensteak.fr/" config/config.yaml
# Review config, eventually amend it if you want
cat config/config.yamlfrep k8s/mysql.yaml.in:- --load config/config.yaml --env db_name=keystone | kubectl apply -f -
frep k8s/mysql.yaml.in:- --load config/config.yaml --env db_name=nova | kubectl apply -f -
frep k8s/mysql.yaml.in:- --load config/config.yaml --env db_name=placement | kubectl apply -f -
frep k8s/mysql.yaml.in:- --load config/config.yaml --env db_name=neutron | kubectl apply -f -
frep k8s/mysql.yaml.in:- --load config/config.yaml --env db_name=glance | kubectl apply -f -
frep k8s/mysql.yaml.in:- --load config/config.yaml --env db_name=skyline | kubectl apply -f -Databases for all OpenStack services are created (empty) during this step.
frep k8s/config.yaml.in:- --load config/config.yaml | kubectl apply -f -All config files for all services are created during this step.
It's time now to sync (create structures / tables) all OpenStack services databases.
frep k8s/mysql-populate.yaml.in:- --load config/config.yaml | kubectl apply -f -Databases for all OpenStack services are now populated with empty tables.
Before going further, it is recommended to wait for the jobs to complete.
You can check the job status with:
k get jobsYou should wait for the jobs to be ready (all jobs must be 1/1)
frep k8s/rabbit.yaml.in:- --load config/config.yaml | kubectl apply -f -frep k8s/keystone.yaml.in:- --load config/config.yaml | kubectl apply -f -Wait for the deployment to be ready (wait for READY 3/3):
k get deployment keystoneKeystone need also some bootstraping, which can be done using the following playbook.
ansible-playbook ansible/bootstrap-keystone.yamlThis will create the endpoints for all OpenStack services and also two users (admin and demo).
This will also install the openstack command line tool (openstack-client) and create two openrc file (one for each user). With this, you will then be able to manipulate your OpenStack cluster from k8s-1 server.
frep k8s/glance.yaml.in:- --load config/config.yaml | kubectl apply -f -frep k8s/placement.yaml.in:- --load config/config.yaml | kubectl apply -f -frep k8s/neutron.yaml.in:- --load config/config.yaml | kubectl apply -f -frep k8s/nova.yaml.in:- --load config/config.yaml | kubectl apply -f -Nova will take longer than others, this is the more complex.
You can wait for all pods to be ready (STATUS running or completed for all of them)
k get pods | grep novafrep k8s/skyline.yaml.in:- --load config/config.yaml | kubectl apply -f -Skyline is not mandatory, it's a web based UI for OpenStack (like horizon dashboard, but newer).
When you executed the bootstrap-keystone playbook, you also installed an openrc_admin file to access your OpenStack cluster as an administrator.
Source this file:
source /root/openrc_adminNow try each element of your OpenStack
openstack token issueWhich should give you something like:
# This command is doing an API call against `keystone` and should give you something like:
+------------+---------------------------------------
| Field | Value
+------------+---------------------------------------
| expires | 2021-11-05T21:35:40+0000
| id | gAAAAABhhZWc-f-TGVV2NNqN03KXLpgIDmBD2f
| project_id | 966629a3e2f34ad996d0ec8d57f6a1bd
| user_id | fd6a41322bdc4941ae74354697fcc2db
+------------+---------------------------------------If you have your token, it means keystone is good!
openstack image list
# This will do an API call against `glance` APIIf this is answering an empty line, you're good! (you don't have any image yet)
openstack network list
# This will do an API call against `neutron` APIIf this is answering an empty line, you're good! (you don't have any network yet)
openstack server list
# This will do an API call against `nova` APIIf this is answering an empty line, you're good! (you don't have any instance yet)
Skyline is a web interface, so you should browse the page.
http://skyline.${ip}.xip.opensteak.fr
If something failed, this is perhaps due to a race condition during deployment.
So you can rollout restart some services.
Most of the issue will appear on nova side:
k rollout restart deployment/nova-api
k rollout restart deployment/nova-metadata-apiYou may also have to re-execute some jobs, such as nova db sync.
You can either do it from nova-api pod, or delete and recreate the related job.
To make sure that this is the root cause of your issue, you can connect on the DB:
k exec -it mysql-89d76b8-qtx2j -- mysql -u root -pYou can also check the logs:
k logs nova-api-b6995b597-xmvfm # replace nova-api-b6995b597-xmvfm with your pod name, you can get it with: k get pods | grep novaOne of the most common race condition is the failure on nova db sync.
You can restart it by deleting the jobs and do it again
k get jobs | grep nova
k delete job nova-db-sync
# And then apply again the mysql-populate
frep k8s/mysql-populate.yaml.in:- --load config/config.yaml | kubectl apply -f -You will need the config.yaml file on your compute, so plik it and copy the URL:
plik -s config/config.yaml
# Copy the URL, you will need it in few secsNow that the OpenStack control plane is ready, you can install your compute.
Like you did for k8s-1, now SSH in compute-1 and login as root.
sudo su -We will need some of the ansible playbooks that are in the repo:
apt-get install -y git ansible
git clone https://github.com/arnaudmorin/bootstrap-openstack-k8s.git
cd bootstrap-openstack-k8scd config
# Paste the cURL (result of `plik -s` command from k8s-1)
# You should now have config.yaml along side with config.yaml.sample
# Get back to previous folder
cd ..All OpenStack services running on the compute are going to be executed outside of kubernetes (kubernetes is installed only on k8s-1 node, not on the compute-1).
To install them, we rely on a playbook:
ansible-playbook ansible/bootstrap-compute.yamlBack on your k8s-1 node, as root:
# Be root if not already done
sudo su -
# Source helper functions
source /root/helper
# Following actions are done as admin
source /root/openrc_admin
create_flavors
create_image_cirros
create_image_debian
# Before running this one, adjust the parameters with your network settings
# If you need to buy an IPFO block, check the tool in order-ipfo/ folder
# For VeryTechTrip event, you should have received a START and END values
create_network_public START END
# Following actions are done as demo
source /root/openrc_demo
create_network_private
create_rules
create_key
create_server_public
# After creating your first public server, you can grab it's ip with:
openstack server list
# To login:
ssh cirros@ip_of_server # password is gocubsgoIf you are lazy doing all the steps manually, you can use the the run-kitty-run.sh script just after the bootstrap.sh and everything should be done automatically:
./run-kitty-run.shIf you decide to add more k8s-x and compute-x nodes, this is very easy, just edit the bootstrap.sh script and start again.
To let k3s on other nodes join the first one, just use something like this:
export K3S_TOKEN='xxxyyyy' # retrieve from master node with: cat /var/lib/rancher/k3s/server/node-token
export K3S_URL='https://ip_first_node:6443'
curl -sfL https://get.k3s.io | sh -