Skip to content

loodse/kubectl-hacking

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

title author patat
$ kubectl hacking - my journey with kate
Tobias Schneck, Loodse GmbH
wrap pandocExtensions margins theme
true
patat_extensions
autolink_bare_uris
left right
5
5
syntaxHighlighting code codeBlock
decVal
bold
onDullRed
dullBlack
onDullWhite
dullBlack
onDullWhite

My journey to Kubernetes

Java programmer -> Testautomation -> Docker -> OpenShift -> Kubernetes

                              ..              
                              ..              
                           ........           
                   ...  ..++..++..++..  ...   
                    ..+++..  .++.  ..+++..    
                     .+.++.. .++. ..++.+.     
                    .+.  .++++++++++.  .+.    
                    ++.  ..++....++..  .++    
                  ..++.+++++++..+++++++.++..  
                .....++..  .++++++.  ..++.....
                     .++.  ++.  .++  .++.     
                      ..+..+.    .+..+..      
                         .+.++++++.+.         
                        .+.        .+.        
                        +.          .+                          

📧 mailto:[email protected] | 🐦 https://twitter.com/toschneck | 🐙 https://github.com/toschneck


kubectl - what's this about?


kubectl - what's this about?


basics - bash config

  • Enable bash auto completion (update compatible)

    # add to ~/.bashrc
    source <(kubectl completion bash)
    alias k=kubectl
    complete -F __start_kubectl k
  • beautify your bash prompt -> powerline go

    1. Install binary go get -v -u github.com/justjanne/powerline-go
    2. ensure powerline-go is available in the path: export PATH=$PATH:$GOPATH/bin
    3. config ~/.bashrc -> source ./powerline-go/.bashrc

basics - kubectl help?

  • kubectl [command] --help is very helpful!
  • kubectl explain [object][.field][...] information about spec fields!
  • kubectl api-resources shows available objects - with CRDs!

Examples

kubectl get --help
#options for all commands
kubectl options

kubectl explain pod.spec.containers.ports
kubectl explain svc.spec.type

kubectl api-resources --api-group=apps
kubectl api-resources -o wide

basics - How to manage multiple cluster?

  1. $KUBECONFIG environment variable or --kubeconfig flag

  2. Merge multiple config files and use kubectl set-context

# list and change context
kubectl config get-contexts
kubectl config use-context CONTEXT_NAME

# set the used default namespace
kubectl config set-context --current --namespace=default

# merge multiple config files
KUBECONFIG=conf1:conf2 kubectl config view --flatten > merged.conf

Attention: set-context modifies context values! use-context change the current context!


Tooling fast context switching with kubectx and kubens

Special thx to Ahmet Alp Balkan

Installation

cd $HOME/bin
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectx
wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubens
chmod 755 kubectx kubens
wget https://github.com/junegunn/fzf-bin/releases/download/0.18.0/fzf-0.18.0-linux_amd64.tgz
tar xf fzf-0.18.0-linux_amd64.tgz
# → kubectx, kubens, fzf

...


Tooling - Usage kubectx, kubens

#### CONTEXT switching ####
# fuzzy search list
kubectx
# direct select e.g. `default` context 
kubectx default
# select last context
kubectx -

#### NAMESPACE switching ####
# fuzzy search list
kubens
# direct select e.g. `kube-system` namespace 
kubens kube-system
# select last namespace
kubens -

Tooling - fuzzy search fzf

  • Search a file the fuzzy way

    fzf
    fzf --preview 'cat {}'  # with preview 
  • Key binding → add ./fzf/.fzf.bash to your ~/.bashrc

    [ -f ~/.fzf.bash ] && source ~/.fzf.bash
  • Pipe kubectl output, e.g. logs, config

    kubectl logs POD_NAME | fzf
    kubectl get pod POD_NAME -o yaml | fzf

kubectl output parameter

  • --v=9 Debug verbosity 0-10

  • -o wide, -o yaml shows more important information about an object

  • --show-labels and --label-columns=k8s-app structure your output

  • -l k8s-app=my-app, --field-selector=status.phase=Running select objects

  • -A (new) using all namespaces

  • -o json | jq 'expresion' combine JSON and jq to get more details (useful for scripting)

  • jsonpath=JSONPATH_EXP powerful one line helper to get multiple valuesx of a json output

  • kubectl describe OBJECT shows information and events


kubectl output parameter

Examples

# all runnings pods
kubectl get pod --field-selector=status.phase=Running -A
 
# node kernel version
kubectl get nodes -o json | jq '.items[].status.nodeInfo.kernelVersion' -r

# all used images
kubectl get pods -A \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.containers[*].image}{"\n"}{end}'

# Check which nodes are ready
JSONPATH='{range .items[*]}{"\n---\n"}{@.metadata.name}: 
{"\n"}{range @.status.conditions[*]}{@.type}={@.status}; {"\n"}{end}{end}' \
 && kubectl get nodes -o jsonpath="$JSONPATH"

Quick wins - let kubectl help you!

use run for resource creation

  • inline paramaters: --image=image, --env="key=value", --port=port, --replicas=replicas, --label="myapp=app1"
  • inline manifest overrides: --overrides JSON
  • --generatortrigger different kind of object creation:
  • (alternative) --restart Never/OnFailure or --schedule=CRON
kubectl run --generator=run-pod/v1 --image nginx:1.13 web
kubectl run --generator=run-pod/v1 --image nginx:1.13 --overrides='{"spec": {"hostNetwork": true}} web-host'

kubectl run # without flag creates a deployment
kubectl run --restart=Never  # creates a Pod
kubectl run --restart=OnFailure # creates a job
kubectl run --restart=OnFailure -schedule="* * * * *" # creates a cronjob

K8s doc ref: https://kubernetes.io/docs/reference/kubectl/conventions/#generators


use run for quick debugging

  • run ... -- argument pass the arguments directly to the container
    # start a simple web image
    kubectl run --image=loodse/demo-www --port 80 web-deployment
    kubectl get pods -o wide
    
    # verify usage by a debug container
    kubectl run --image=nicolaka/netshoot --generator=run-pod/v1 --rm -it -- bash
    # ... inside the container: wget WEB_DEPLOYMENT_POD_IP -O -

NOTE: most --generators deprecated, but replacement kubectl create don't support -it 😢


Quick wins - let kubectl help you!

create templates

  • run --restart / create combined with --dry-run -o yaml and creates a template for common resources

    # create a deployment yaml file
    kubectl create deployment web-template --image=loodse/demo-www --dry-run -o yaml > dep.create.yaml
    kubectl run --image=loodse/demo-www --port 80 --dry-run -o yaml web-template > dep.run.yaml
    
    # job with 10 sleep 
    kubectl run --image=busybox --restart=OnFailure --dry-run -o yaml job -- /bin/sleep 10 > job.run.yaml
  • --export get a pod's YAML without cluster specific information

#deployment
kubectl get deployment web-template -o yaml --export > dep.export.yaml
vim dep.export.yaml 
kubectl apply -f dep.export.yaml   

#service
kubectl get service web-template --export -o yaml > svc.export.yaml

Quick wins - let kubectl help you!

Modify resources

  • Use inplace editor functionality

    • KUBE_EDITOR sets the local editor
    • kubectl edit TYP OBJECT open in cluster resource
  • Use apply for mutable objects, replace for immutable objects. Note: You can use -f FOLDER for using multi manifests!

    kubectl apply -f dep.yaml
    # delete resource and recreates it
    kubectl replace --force -f pod.yaml
  • Use scaling functions

    • kubectl autoscale deployment foo --min=2 --max=10 add HPA
    • kubectl scale deployment --replicas=10 web-deployment scales up

Quick wins - let kubectl help you!

use expose for service creation

Can reference pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs).

  • --port listing port to match at referenced resource
  • --type type of Service: ClusterIP (default), NodePort, LoadBalancer, ExternalName
  • --target-port port at the service
  • --selector specify label selector
kubectl expose deployment web-deployment --type=LoadBalancer --port=80
kubectl get nodes -o wide
kubectl get nodes --selector=kubernetes.io/role!=master \
  -o jsonpath={.items[0].status.addresses[?\(@.type==\"ExternalIP\"\)].address}
  • Combine with port-forward for quick testing or debugging
    • can target pod, deployment, service + localport:remoteport for port mapping
kubectl port-forward svc/web-deployment 8080:80 &
curl localhost:8080

Quick wins - let kubectl help you!

Modify resources

  • Manipulate current objects, e.g. the image value
    # use set for common modification
    kubectl set image deployment/web-deployment web-deployment=loodse/demo-www
    kubectl set env deployment/web-deployment TEST=val
    
    # use patch for all other, e.g. service type
    kubectl patch svc/web-deployment -p '{"spec":{"type":"NodePort"}}' 
    
    # Update a container's image; spec.containers[*].name is required because it's a merge key
    kubectl patch pod/podname -p \
     '{"spec":{"containers":[{"name":"web-deployment","image":"loodse/demo-www"}]}}'

Extend kubectl with plugins

  • Enable kubectl plugin manager krew

    # add to ~/.bashrc
    # export KREW_ROOT=/path/to/krew-folder
    export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
  • Plugin management

kubectl-krew search
kubectl-krew install view-secret
kubectl-krew upgrade

Extend kubectl with plugins

  • Example: decode base64 secrets
kubectl get secret
kubectl view-secret default-token-976rc namespace
  • Example: tree view
kubectl tree deployment web-template
NAMESPACE  NAME                                   READY  REASON  AGE
default    Deployment/web-template                -              44m
default    └─ReplicaSet/web-template-655c987cb8   -              44m
default      └─Pod/web-template-655c987cb8-5dzrh  True           44m

Use kubectl the fuzzy way with fubectl

  • bash/zsh wrapper based on kubectl, jq and fzf

  • Interactive search and interaction with Kubernetes objects

    • support's also CRD's
    • No in-cluster installation needed
  • Installation

    curl -LO https://rawgit.com/kubermatic/fubectl/master/fubectl.source
    # add to `~/.bashrc`
    [ -f <path-to>/fubectl.source ] && source <path-to>/fubectl.source  

khelp
# [k] like g for git but 233% as effective!
# [ka] get all pods in namespace
# [kall] get all pods in cluster
# [kwa] watch all pods in the current namespace
# [kwall] watch all pods in cluster
# [kp] open kubernetes dashboard with proxy
# [kwatch] watch resource
# [kdebug] start debugging container in the current namespace
# [kube_ctx_name] get the current context
# [kube_ctx_namespace] get current namespace
# [kget] get a resource by its YAML
# [ked] edit a resource by its YAML
# [kdes] describe resource
# [kdel] delete resource
# [klog] fetch log from container
# [kex] execute command in container
# [kfor] port-forward a container port to your local machine
# [ksearch] search for string in resources
# [kcl] context list
# [kcs] context set
# [kcns] context set default namespace
# [kwns] watch pods in a namespace
# [ktree] prints a tree of k8s objects (kubectl tree plugin needs to be installed)
# [kbash] create a pod (ubuntu) with a bash
# [konsole] create root shell (alpine) on a node

Troubleshooting

  • Take a look for objects in state Pending, Error, CrashLoopBackOff and inspect: kubectl describe

  • Use port-forward to test different connections, e.g. service or pod

  • Use prepared debug container for e.g. network debugging

    kubectl run --image=nicolaka/netshoot --restart=Never --rm -it -- bash
    # with fubectl
    kdebug     #start netshoot container
    konsole    #start shell on HOST of node!
  • top for resource usage, requires metrics-server

    kubectl top node
    kubectl top pod   

Troubleshooting

  • Reproduce the event and stream all matching logs, e.g. with label name=myLabel

    • kubectl logs -f -l name=myLabel --all-containers
    • kubectl get events -A
  • exec into running container

    • kubectl exec my-pod -- ls -la /
    • kubectl exec my-pod -it -- sh
  • Inspect the whole cluster state

    # download the state
    kubectl cluster-info dump --output-directory=./output/cluster-state
    
    # diagnose it
    tree ./output/cluster-state
    
    grep -r Error output/cluster-state
    grep -C 5 -r Error output/cluster-state

Cluster Inspection Tools

++ need no running components in the cluster

  • provides a curses based terminal UI
  • interactive view similar to htop
  • Kubernetes Cluster Sanitizer
  • Find errors and warnings

Cluster Management by Cluster API

  • Manage Cluster's by CRDs to abstract of the provider (cloud/on-prem)
  • Currently mostly used for machine creation, see as e.g. machine-controller implementations
    • Used by e.g. HA cluster management tool kubeOne
  • Immutable machine objects handle cluster nodes similar to pods
    Deployment -> ReplicaSet -> Pod -> Container
    MachineDeployment -> MachineSet -> Machine -> Node
  • Independent management Layer between the clouds

Cluster Management by Cluster API

Basic API objects for automation for the patch, resource and access management

# see the machine definition
kubectl describe machine -n kube-system MACHINE_NAME
kubectl get machinedeployment,machineset,machine,node -n kube-system
kubectl tree machinedeployment pool-1 -n kube-system

# scale machines
kubectl scale md --replicas=3 pool-1
# update e.g. kubernetes version, machine size, ...
kubectl edit machinedeployment

# machine to node reference:
kubectl get machine -n kube-system \
  -o jsonpath='{range .items[*]}{@.metadata.name}{" >> "}{@.status.nodeRef.name}{"\n"}{end}}'

Manage VMs with kubevirt


Questions?

I'm happy to answer!

Slides?

Take a look at https://github.com/loodse/kubectl-hacking

Something to add?

Open a pull request 😉

Thx for your attention!


References:


About

$ kubectl hacking - my journey with kate

Resources

Stars

Watchers

Forks

Packages

No packages published