Example Kubeconfig for int128/kubelogin

This is a Kubeconfig that you can use if you are in the home-workers group.

You may contact KingdonB, the admin of the Kingdon-CI group, if you are not in the group. Hello CNCF Livestream!

Our Dex config adds GitHub connectivity to any self-managed Kubernetes cluster, making possible to connect to Kubernetes. We need Kubernetes for Flux.

(Solution in Golang)

tl;dr: Skip straight to the solution (if you're up for compiling some Golang.)

Getting Started

Copy the Kubeconfig below to ~/.kube/config and install kubelogin to login.

Prerequisites

You need flux, kubectl, and kubelogin.

With Homebrew, Krew, or Chocolatey:

# Homebrew (macOS and Linux)
brew install int128/kubelogin/kubelogin

# Krew (macOS, Linux, Windows and ARM)
kubectl krew install oidc-login

# Chocolatey (Windows)
choco install kubelogin

Visit int128/kubelogin for more information. You need kubelogin so you can login to Kube. (No duh!)

Kubeconfig

The Kubeconfig:

apiVersion: v1
clusters:
  - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTnpjd01EWTBPRFV3SGhjTk1qTXdNakl4TVRrd09EQTFXaGNOTXpNd01qRTRNVGt3T0RBMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTnpjd01EWTBPRFV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTbU9Id3VMTTBIY2pJQlZZLzB4SEhQbDRnQ0pCazBiUHF1N2hqeGxpdVUKckVRTERqU1ZySVNqWE1WN3RTODNVaHVsaUE4QTlwZC9mSkdEQi9mVk13Z2NvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVU9xc1l5d0xjOHU4b29VTnFUUDhPCjdhWEFwbll3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSXhTY3p6WXdKT25BMlEvc3FJRVdGbDBBZjJzN2c0ZHYKNk1kWkl5UmZKeE1DSVFDYUFWT1VBK0VsM2htbGZVdU9tU2g5elR4djlIKzJvZWtneDRBOFNFdyswQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
        server: https://howard.moomboo.stage
    name: howard-stage
contexts:
  - context:
        cluster: newexample
        user: kubelogin
    name: newexample
kind: Config
preferences: {}
current-context: howard-space
users:
  - name: kubelogin
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        args:
        - oidc-login
        - get-token
        - --oidc-issuer-url=https://dex.howard.moomboo.space
        - --oidc-client-id=wg-kubelogin
        - --oidc-client-secret=AiAImuXKhoI5ApvKWF988txjZ+6rG3S7o6X5En
        - --oidc-extra-scope=groups
        - --oidc-extra-scope=offline_access
        command: kubectl
        env: null
        provideClusterInfo: false

In the example kubeconfig above, some of the details are significant.

  • oidc-login is the name of the kubectl or krew plugin for kubelogin.

  • The client-id and client-secret are significant as is issuer-url You may likely change these values (if you are not the Home Workers team.)

Refresh Tokens

  • Requesting the offline_access scope

This will result in Dex generating a refresh token, which will only be expired according to Dex's own expiration and rotation settings.

Refresh tokens are usually much longer lived than id tokens, and in fact can be configured so that they never expire. Take care with this configuration!

GitHub Groups

The extra-scopes: groups is also significant, in that GitHub as a baseline provides only unstable identifiers: email and username, and this makes GitHub generally unusable for managing authn for RBAC on a set of individuals.

We can still use GitHub though, since we have adopted groups as our source of authority in the example org; there are no surprises for us as there is no individual in our RBAC config, only groups in the format: kingdon-ci:group.

Self-Service Cluster Management

  • Each cluster gets a certificate-authority-data that must authenticate the cluster's Kubernetes API service's own (usually self-signed) cert data.

This certificate-authority-data would not usually change very often, except at certificate renewal time, or for clusters that are likely to come and go.

Since Cluster-API is a self-service cluster management tool for devs, we will absolutely need to cope with clusters coming and going! Perhaps all the time.

How exactly to refresh the CA data when it has gone stale, is left as an exercise for the reader. (Edit: this has been solved in Golang.)

tl;dr: Run kubelogin

The Kubeconfig can also be set up for what the kubelogin documentation calls Standalone Mode.

The old tl;dr was based on Standalone Mode: just run kubelogin to authorize kubectl.

With the exec mode configuration above, you don't need to run kubelogin anymore. But you might try kubelogin --setup to learn how to configure this from scratch!

Risks and mitigation

This method uses the exec plugin at runtime, whereas in standalone mode the id-token gets embedded directly in your kubeconfig; with this method it is still stored on disk, as is the refresh token, but you can still freely copy the kubeconfig and share it without compromising your identity.

One can harvest an id-token or a refresh-token, which is a security concern that might warrant further mitigation. But the fact that id-token expires quickly helps a bit, and Dex has other mitigations that can make refresh-token expire as soon as it's used, which could make it harder to copy and re-use without detection.

Take care that your home directory is not shared access; this is generally standard operating procedure. A home directory should be chmod -rwx for all entities other than the owner.

The id token should be stored on a local disk, never on any shared filesystem.

Kubernetes RBAC

When you have configured Dex and RBAC, according to the Weave GitOps Docs:

the users on cluster clients should not be configured for the same groups as Weave GitOps.

For the purposes of our Developer clusters, where authenticated users are all authorized as cluster-admin, we do not care about this kind of finer point.

We can prepare RBAC roles and an OIDC-enabled kubeconfig, to avoid a need to copy any permissioned access tokens around.

We should not usually need to break the glass!

The RBAC permissions are handled with Kustomize bases in a management cluster.

(Here is the rbac Kustomize base module that creates this specific RBAC binding!)

This configuration uses your GitHub OIDC through a Dex issuer that has been set up according to Weave GitOps docs. You can use it to get direct access to a Kubernetes cluster, or you may shut direct access and use only GitOps.

Kconf

I found this great tool to help manage multiple kubeconfigs easily, called kconf:

Our kubeconfig may get some stale certificate-authority-data or need tuning up.

ISA there will be a way to easily get fresh kubeconfig with all clusters that you have access to, based on your OIDC. In the mean time, use kconf and your text editor to make this kubeconfig your own.

You can find a permalink to this example kubeconfig for kubelogin here:

Kubeconfig-ca-fetch

I built a partial solution to some of the issues described above. This helps when the certificate-authority-data has gone stale, though it may be insecure.

Try cloning this repo and run make. Just copy the super-tl;dr blob below, (or click the links and read what you're doing)

There is a list of clusters in main.go which the user may edit. The heavy lifting is handled through a package!

It is my first Go code in a while (so please be gentle!)

Code

The package kubeconfigcafetch:

link

Clone that repo and take a look at the code to see the business logic I used to derive CA data from a connection to the clusters. This is totally unsafe for you, (but probably OK for me, since near all of these links are on my local network!)

Assuming make clean && make all kube.config is fine, you'll find the clusters ready in kube.config.

or run

make tldr

and simply watch what happens.

Errors in turkey.local

Some clusters in the turkey.local domain will be unreachable by default, unless you have Tailscale. Like I said, this is my local network. If you are reading this, welcome to literally inside my home.

NB: Do remember to chmod go-r as below, an id-token is your identity and it must not be divulged.

kubelogin in standalone mode writes the secret id-token directly into kube.config. So take care!

Please don't let any bad hackers in (but also, hello friendly neighborhood hackers!)

Errors with M1 Mac and Rancher Desktop

If you get this error in a loop when you first try Kubelogin, and you've installed Rancher Desktop:

...transport.go:243] Unable to cancel request for *exec.roundTripper...
...transport.go:243] Unable to cancel request for *exec.roundTripper...
...transport.go:243] Unable to cancel request for *exec.roundTripper...

You may be experiencing this issue which can be solved by avoiding the use of ~/.rd/bin/kubectl with the kubelogin plugin.

Rancher Desktop is great, but I have no idea why it distributes an apparently broken version of kubectl that doesn't work with kubelogin, so just don't use that and this problem goes away.

Remove ~/.rd/bin from your PATH and install kubectl from homebrew.

(I use Rancher Desktop without this, and it works perfectly fine for me.)

super-tl;dr
git clone https://github.com/kingdon-ci/kubeconfig-ca-fetch && cd kubeconfig-ca-fetch
make build kube.config
chmod 600 kube.config
export KUBECONFIG=`pwd`/kube.config
kubelogin
kubectl get nodes

You can also put that in ~/.kube/config for a more permanent setting.

(WARNING: the next version will overwrite your Kubeconfig
but you have 3 seconds to press Ctrl+C and cancel if it's a mistake.)

If you run

make supertldr

then it will overwrite the ~/.kube/config with one generated by kubeconfig-ca-fetch, instead of simply generating the kube.config in the current working directory, which is make tldr.

Read more

Read about kubeconfig-ca-fetch, and understand: it's really not safe to do this, but at the same time if aware of the risks, it can be plenty safe yet.

I tried to convert this to Wasm and it literally wasn't possible to do what I've done in this code, because this is a technically unsafe operation. We're reaching out to the wire and asking it who the authority is. This is asking to be MITM'ed if you used it every day, without any mitigation for those attacks.

If you are in the habit of making your KUBECONFIG in this way, then you must take care not to over-YOLO. Be sure you have a way to verify the certificates out-of-band, (like comparing them against the ones you see on this web site!)

Source for this web page

So it is important that we can publish a web site in order to close the loop,

Surely, now that you can visit the web page here and confirm the CA data that was downloaded matches what's on this web page, we can strengthen the trust in the found CA data's actual verity by some measure.

Find the GitHub sources for the current page at the link above.