expose kubeconfig as terraform output

[phaer]

This makes the output of terraform output a bit more verbose,
but allows to use kubeconfig when including kube-hetzner into
other terraform projects without having the path depend on the
modules source. It's local in this example but could just as well
be a remote github repository.

Example terraform code:

module "kube-hetzner" {
  source = "./kube-hetzner"

hcloud_token = "hahaha-not-really"
public_key   = "/home/me/.ssh/my-first-key.pub"
private_key  = null
location                  = "fsn1"
agent_server_type         = "cpx21"
control_plane_server_type = "cpx11"
lb_server_type            = "lb11"
servers_num               = 2
agents_num                = 2
hetzner_ccm_containers_latest = true
hetzner_csi_containers_latest = true
}

resource "local_file" "kubeconfig" {
    content  = module.kube-hetzner.kubeconfig
    filename = "kubeconfig.yaml"
}

output "controlplanes_public_ip" {
  value       = module.kube-hetzner.controlplanes_public_ip
}

[phaer]

Ah, that didn't work as expected. terraform validate ran successfully locally because kubeconfig.yaml already existed locally. Going to look at it again later on

[mysticaltech]

Seems interesting! Curious to see where this leads.

[phaer]

@mysticaltech This got a bit bigger than expected:

• It would be great to have kubeconfig be a part of terraforms dependency graph & state. This would enable multi-user setups (because it does not depend on local disk files anymore) and to use kubeconfig to authenticate hashicorp/kubernetes while still passing terraform validate - I think thats not possible with the current solution.
• A drawback is that we can't deploy CCM, CSI & Traefik from our local host, because we don't have kubeconfig.yaml locally while provisioning the control plane, only afterwards.
• so we just run kubectl apply on the control plane itself, after k3s is ready
• but to do so, we need to deploy all manifests. I've merged the patches into a single kustomization.yaml file, because that makes the deployment of those files to the control-plane server easier.
• we could also put the traefik config into the same kustomization file, which would save us one of the file provisioner blocks.

What do you think? Is this a feasible approach?

This is still a draft atm, because it does not work with SSH agents yet, because the remote provider used does not support it yet, waiting for tenstad/terraform-provider-remote#31 :D

[mysticaltech]

Definitely could work! It also seems like a cleaner way of doing things, not dispersed between local and remote. Keep up the good work man!

[mnencia]

tenstad/terraform-provider-remote#31 has been merged :-)

[mysticaltech]

Well done @phaer! So that makes the data 'remote_file' workable in all of our use cases correct? :) This PR is huge but seems awesome.

About the new OS, hopefully very soon! I will keep you posted in #35.

[phaer]

@mysticaltech Hi, I think we should wait until terraform-provider-remote cuts a new release, because I don't think binaries are available before that, so users would need to build the provider themselves.

Regarding the new OS: I did some experiments with Ranchers cOS-toolkit as well as Fedora CoreOS, but nothing to release yet. OpenSuSE MicroOS looks interesting as well, but I wont have time too look into it properly for at least a week

[mysticaltech]

Good to hear, thanks @phaer!

[phaer]

A new release has been published: https://github.com/tenstad/terraform-provider-remote/releases/tag/v0.0.23

Should we merge that as is? I am going to try to rebase this onto the staging branch this week.

EDIT: also started work on the same thing + staging changes. https://github.com/phaer/kube-hetzner/tree/expose-kubeconfig-staging. Does not work for me yet, because of some strange ssh issue where authentication via terraform works before first reboot, but does not after while it's seemingly trying the same public key. And interactive ssh authentication works both before and after the reboot.

[mysticaltech]

@phaer When staging merges into master, we will have a k3os branch for those that want to use it, and I prefer it stays the way it is now because we have all tested it and we know that all works.

What I propose is that you rebase for staging, make sure everything works, and I will go through it again. I need to test it and understand it because a lot of changes are going on. Please 🙏

[phaer]

superseeded by #46