The OG Kubernetes PaaS platform (v2).
The Package manager for Kubernetes.
The Kubernetes-native CD toolkit.
This is a Kubeconfig that you can use if you are in the home-workers group.
You may contact KingdonB, the admin of the Kingdon-CI group, if you are not in the group. Hello CNCF Livestream!
Our Dex config adds GitHub connectivity to any self-managed Kubernetes cluster, making possible to connect to Kubernetes. We need Kubernetes for Flux.
tl;dr: Skip straight to the solution (if you're up for compiling some Golang.)
Copy the Kubeconfig below to ~/.kube/config
and install kubelogin
to login.
You need flux
, kubectl
, and kubelogin
.
With Homebrew, Krew, or Chocolatey:
# Homebrew (macOS and Linux)
brew install int128/kubelogin/kubelogin
# Krew (macOS, Linux, Windows and ARM)
kubectl krew install oidc-login
# Chocolatey (Windows)
choco install kubelogin
Visit int128/kubelogin for more information.
You need kubelogin
so you can login to Kube. (No duh!)
The Kubeconfig:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTnpjd01EWTBPRFV3SGhjTk1qTXdNakl4TVRrd09EQTFXaGNOTXpNd01qRTRNVGt3T0RBMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTnpjd01EWTBPRFV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTbU9Id3VMTTBIY2pJQlZZLzB4SEhQbDRnQ0pCazBiUHF1N2hqeGxpdVUKckVRTERqU1ZySVNqWE1WN3RTODNVaHVsaUE4QTlwZC9mSkdEQi9mVk13Z2NvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVU9xc1l5d0xjOHU4b29VTnFUUDhPCjdhWEFwbll3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSXhTY3p6WXdKT25BMlEvc3FJRVdGbDBBZjJzN2c0ZHYKNk1kWkl5UmZKeE1DSVFDYUFWT1VBK0VsM2htbGZVdU9tU2g5elR4djlIKzJvZWtneDRBOFNFdyswQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://howard.moomboo.stage
name: howard-stage
contexts:
- context:
cluster: newexample
user: kubelogin
name: newexample
kind: Config
preferences: {}
current-context: howard-space
users:
- name: kubelogin
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- oidc-login
- get-token
- --oidc-issuer-url=https://dex.howard.moomboo.space
- --oidc-client-id=wg-kubelogin
- --oidc-client-secret=AiAImuXKhoI5ApvKWF988txjZ+6rG3S7o6X5En
- --oidc-extra-scope=groups
- --oidc-extra-scope=offline_access
command: kubectl
env: null
provideClusterInfo: false
In the example kubeconfig above, some of the details are significant.
oidc-login
is the name of the kubectl
or krew
plugin for kubelogin
.
The client-id
and client-secret
are significant as is issuer-url
You may likely change these values (if you are not the Home Workers team.)
offline_access
scopeThis will result in Dex generating a refresh token, which will only be expired according to Dex's own expiration and rotation settings.
Refresh tokens are usually much longer lived than id tokens, and in fact can be configured so that they never expire. Take care with this configuration!
The extra-scopes: groups
is also significant, in that GitHub as a baseline
provides only unstable identifiers: email
and username
, and this makes
GitHub generally unusable for managing authn for RBAC on a set of individuals.
We can still use GitHub though, since we have adopted groups as our source
of authority in the example org; there are no surprises for us as there is no
individual in our RBAC config, only groups in the format: kingdon-ci:group
.
certificate-authority-data
that must authenticate the
cluster's Kubernetes API service's own (usually self-signed) cert data.This certificate-authority-data
would not usually change very often, except
at certificate renewal time, or for clusters that are likely to come and go.
Since Cluster-API is a self-service cluster management tool for devs, we will absolutely need to cope with clusters coming and going! Perhaps all the time.
How exactly to refresh the CA data when it has gone stale, is left as an exercise for the reader. (Edit: this has been solved in Golang.)
kubelogin
¶The Kubeconfig can also be set up for what the kubelogin
documentation calls
Standalone Mode.
The old tl;dr was based on Standalone Mode: just run kubelogin
to authorize kubectl
.
With the exec mode configuration above, you don't need to run kubelogin
anymore.
But you might try kubelogin --setup
to learn how to configure this from scratch!
This method uses the exec plugin at runtime, whereas in standalone mode the
id-token
gets embedded directly in your kubeconfig
; with this method it is
still stored on disk, as is the refresh token, but you can still freely copy
the kubeconfig and share it without compromising your identity.
One can harvest an id-token
or a refresh-token
, which is a security concern
that might warrant further mitigation. But the fact that id-token
expires quickly
helps a bit, and Dex has other mitigations that can make refresh-token
expire as
soon as it's used, which could make it harder to copy and re-use without detection.
Take care that your home directory is not shared access; this is generally
standard operating procedure. A home directory should be chmod -rwx
for all
entities other than the owner.
The id token should be stored on a local disk, never on any shared filesystem.
When you have configured Dex and RBAC, according to the Weave GitOps Docs:
the users on cluster clients should not be configured for the same groups as Weave GitOps.
For the purposes of our Developer clusters, where authenticated users are all
authorized as cluster-admin
, we do not care about this kind of finer point.
We can prepare RBAC roles and an OIDC-enabled kubeconfig, to avoid a need to copy any permissioned access tokens around.
We should not usually need to break the glass!
The RBAC permissions are handled with Kustomize bases in a management cluster.
(Here
is the rbac
Kustomize base module that creates this specific RBAC binding!)
This configuration uses your GitHub OIDC through a Dex issuer that has been set up according to Weave GitOps docs. You can use it to get direct access to a Kubernetes cluster, or you may shut direct access and use only GitOps.
I found this great tool to help manage multiple kubeconfigs easily, called kconf
:
Our kubeconfig may get some stale certificate-authority-data
or need tuning up.
ISA there will be a way to easily get fresh kubeconfig with all clusters that
you have access to, based on your OIDC. In the mean time, use kconf
and your
text editor to make this kubeconfig your own.
You can find a permalink to this example kubeconfig for kubelogin
here:
I built a partial solution to some of the issues described above. This helps when the certificate-authority-data has gone stale, though it may be insecure.
Try cloning this repo and
run make
. Just copy the super-tl;dr blob below, (or click the links and read
what you're doing)
There is a list of clusters in main.go which the user may edit. The heavy lifting is handled through a package!
It is my first Go code in a while (so please be gentle!)
The package kubeconfigcafetch
:
Clone that repo and take a look at the code to see the business logic I used to derive CA data from a connection to the clusters. This is totally unsafe for you, (but probably OK for me, since near all of these links are on my local network!)
Assuming make clean && make all kube.config
is fine, you'll find the clusters
ready in kube.config
.
or run
make tldr
and simply watch what happens.
turkey.local
¶Some clusters in the turkey.local
domain will be unreachable by default, unless you have Tailscale.
Like I said, this is my local network. If you are reading this, welcome to literally inside my home.
NB: Do remember to chmod go-r
as below, an id-token
is your identity and it must not be divulged.
kubelogin
in standalone mode writes the secret id-token
directly into kube.config
. So take care!
Please don't let any bad hackers in (but also, hello friendly neighborhood hackers!)
If you get this error in a loop when you first try Kubelogin, and you've installed Rancher Desktop:
...transport.go:243] Unable to cancel request for *exec.roundTripper...
...transport.go:243] Unable to cancel request for *exec.roundTripper...
...transport.go:243] Unable to cancel request for *exec.roundTripper...
You may be experiencing this issue which can be solved
by avoiding the use of ~/.rd/bin/kubectl
with the kubelogin
plugin.
Rancher Desktop is great, but I have no idea why it distributes an apparently
broken version of kubectl that doesn't work with kubelogin
, so just don't use
that and this problem goes away.
Remove ~/.rd/bin
from your PATH
and install kubectl
from homebrew.
(I use Rancher Desktop without this, and it works perfectly fine for me.)
git clone https://github.com/kingdon-ci/kubeconfig-ca-fetch && cd kubeconfig-ca-fetch
make build kube.config
chmod 600 kube.config
export KUBECONFIG=`pwd`/kube.config
kubelogin
kubectl get nodes
You can also put that in ~/.kube/config
for a more permanent setting.
(WARNING: the next version will overwrite your Kubeconfig
but you have 3 seconds to press Ctrl+C and cancel if it's a mistake.)
If you run
make supertldr
then it will overwrite the ~/.kube/config
with one generated by
kubeconfig-ca-fetch
, instead of simply generating the kube.config
in the
current working directory, which is make tldr
.
Read about kubeconfig-ca-fetch
, and understand: it's really not safe to do
this, but at the same time if aware of the risks, it can be plenty safe yet.
I tried to convert this to Wasm and it literally wasn't possible to do what I've done in this code, because this is a technically unsafe operation. We're reaching out to the wire and asking it who the authority is. This is asking to be MITM'ed if you used it every day, without any mitigation for those attacks.
If you are in the habit of making your KUBECONFIG
in this way, then you must
take care not to over-YOLO. Be sure you have a way to verify the certificates
out-of-band, (like comparing them against the ones you see on this web site!)
So it is important that we can publish a web site in order to close the loop,
Surely, now that you can visit the web page here and confirm the CA data that was downloaded matches what's on this web page, we can strengthen the trust in the found CA data's actual verity by some measure.
Find the GitHub sources for the current page at the link above.