Invalid SSH identity files

[DjVinnii]

Description

The SSH identity files generated by modules/host/main.tf seems to be incorrect. Resulting in a failed deployment.

│ Error: local-exec provisioner error
│
│   with module.kube-hetzner.module.agents["0-0-agent-fsn1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 63, in resource "hcloud_server" "server":
│   63:   provisioner "local-exec" {
│
│ Error running command 'until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o
│ PubkeyAuthentication=yes -i /tmp/badvf73ve9dg0swlwxmr -o ConnectTimeout=2 -p 22 root@<redacted> true 2> /dev/null
│ do
│   echo "Waiting for MicroOS to become available..."
│   sleep 3
│ done
│ ': exit status 2. Output: /bin/sh: 6: Syntax error: end of file unexpected (expecting "do")
│
╵
╷
│ Error: local-exec provisioner error
│
│   with module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 63, in resource "hcloud_server" "server":
│   63:   provisioner "local-exec" {
│
│ Error running command 'until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o
│ PubkeyAuthentication=yes -i /tmp/t5jhlscdhq5m0jrczfzj -o ConnectTimeout=2 -p 22 root@<redacted> true 2> /dev/null
│ do
│   echo "Waiting for MicroOS to become available..."
│   sleep 3
│ done
│ ': exit status 2. Output: /bin/sh: 6: Syntax error: end of file unexpected (expecting "do")
│
╵
╷
│ Error: local-exec provisioner error
│
│   with module.kube-hetzner.module.control_planes["2-0-control-plane-hel1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 63, in resource "hcloud_server" "server":
│   63:   provisioner "local-exec" {
│
│ Error running command 'until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o
│ PubkeyAuthentication=yes -i /tmp/67s0b0m7rduy1iojdjyb -o ConnectTimeout=2 -p 22 root@<redacted> true 2> /dev/null
│ do
│   echo "Waiting for MicroOS to become available..."
│   sleep 3
│ done
│ ': exit status 2. Output: /bin/sh: 6: Syntax error: end of file unexpected (expecting "do")
│
╵
╷
│ Error: local-exec provisioner error
│
│   with module.kube-hetzner.module.control_planes["0-0-control-plane-fsn1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 63, in resource "hcloud_server" "server":
│   63:   provisioner "local-exec" {
│
│ Error running command 'until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o
│ PubkeyAuthentication=yes -i /tmp/25q4h1s2r2kn7yxgpqro -o ConnectTimeout=2 -p 22 root@<redacted> true 2> /dev/null
│ do
│   echo "Waiting for MicroOS to become available..."
│   sleep 3
│ done
│ ': exit status 2. Output: /bin/sh: 6: Syntax error: end of file unexpected (expecting "do")
│

Kube.tf file

locals {
  # You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env
  # within your shell, such as such: export TF_VAR_hcloud_token=xxxxxxxxxxx
  # If you choose to define it in the shell, this can be left as is.

  # Your Hetzner token can be found in your Project > Security > API Token (Read & Write is required).
  # hcloud_token = "xxxxxxxxxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token

  source = "kube-hetzner/kube-hetzner/hcloud"

  ssh_public_key = file("~/.ssh/kube_hetzner.pub")
  ssh_private_key = file("~/.ssh/kube_hetzner")

  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cax11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1",
      server_type = "cax11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-fsn1",
      server_type = "cax21",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  initial_k3s_channel = "v1.28"

  cni_plugin = "cilium"

  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]

  use_control_plane_lb = true

  create_kubeconfig = false
}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

I noticed /tmp contains some weird generated filenames:

One of the used files is /tmp/67s0b0m7rduy1iojdjyb, but that one doesn't have any content. The weird part is that the incorrect files do contain my private key.

Platform

Linux (WSL)

[mysticaltech]

@DjVinnii Make sure you update WSL to the latest version and generate your key using the same method as described in our SSH doc here please. https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/docs/ssh.md

[DjVinnii]

The added \r looked like something with line endings, so I did some more investigation. It turns out that the module that Terraform downloads has CRLF line endings. When I manually changed .terraform/modules/kube-hetzner/modules/host/main.tf to LF it started working (other errors now pops up, but at least it is progress). This makes sense as the multiline commands in this file will have the same line endings as the file itself.

According to Terraform, the idiomatic style is to use Unix-style line endings (LF). So it might need to be changed in this repo.

[mysticaltech]

@DjVinnii Super interesting, thanks for this, I will investigate that more.

[morinow]

Had exactly the same issue. I started using Terraform on a mounted directory from the Windows host located at '/mnt/c/...' and experienced this issue. What fixed it for me in the end was freshly cloning my working repository into a 'real' WSL directory like '/home/...' and start using Terraform from there. That worked out fine. Maybe my WSL Git handles line-endings differently?

[mysticaltech]

@morinow Most probably something along these lines. Thanks for sharing!

[DjVinnii]

What fixed it for me in the end was freshly cloning my working repository into a 'real' WSL directory like '/home/...' and start using Terraform from there.

The weird thing in my case is that I already used /home/.... Maybe my issue is caused by using an IDE on Windows and opening the folder via the WSL share (\\wsl.localhost\Ubuntu-22-04\home\...) from there

[Crease29]

I ran into the exact same issue on Windows with WSL2.
What fixed it for me was to run this command in my project directory to convert the line endings to LF only:

find .terraform -type f -print0 | xargs -0 dos2unix

Make sure dos2unix is installed.

[mysticaltech]

Thanks @Crease29, great tip!