Kubernetes
This section contains tips and advice for interacting with our Kubernetes deployments (most notably sourcegraph.com and k8s.sgdev.org).
How to set up access to Kubernetes
-
Make sure that you have been granted access to our Google Cloud project: https://console.developers.google.com/project/sourcegraph-dev?authuser=0. You may need to change
authuser
to the index of your sourcegraph.com Google account. -
Install the
gcloud
command (CLI for interacting with the Google Cloud):curl https://sdk.cloud.google.com | bash
-
Get authorization for your
gcloud
command:gcloud auth login
-
Install the
kubectl
command (CLI for interacting with Kubernetes):gcloud components install kubectl
-
Configure
kubectl
to point to the desired cluster using the appropriategcloud container clusters get-credentials
command listed in “Instances”. -
Verify that you have access to kubernetes:
kubectl get pods --all-namespaces
Reauthenticate kubectl
If you see the following when running kubectl
commands:
Unable to connect to the server: x509: certificate signed by unknown authority
Just run the appropriate gcloud container clusters get-credentials
command listed at the top of this document again to reauthenticate.
Scaling Kubernetes clusters
Cluster scale should be managed via terraform. Please reference google_container_node_pool.primary_containerd_nodes.node_count
this line in cloud’s terraform configuration to see where the number of nodes is configured for the cluster, and gke_num_nodes
in the tfvars file to see the current number of nodes. For more details, see the terraform provider documentation.
Any changes to the cluster scale made via kubectl will eventually be overwritten by the values set in terraform.
Kubernetes backups
Snapshots of all Kubernetes resources are taken periodically and pushed to kube-backup.
These example commands are for the dot-com
cluster where the Sourcegraph application is deployed to the prod
namespace.
kubectl cheatsheet
List all pods | kubectl get pods --namespace=prod -o=wide |
Describe the properties of a pod. | kubectl --namespace=prod describe pod $POD_NAME |
Pull logs | kubectl --namespace=prod logs $POD_NAME |
Get an interactive shell in a running pod container | kubectl exec --namespace=prod -ti $POD_NAME -- /bin/sh |
Edit a “deployment” (such as changing environment variables). | kubectl edit deployment --namespace=prod DEPLOYMENT_NAME Note that the deployment name is not the pod name, and affects all pods running that deployment. |
SSH into the VM running a pod. | Find the node ID from the NODE column of kubectl get pods --namespace=prod -o=wide . Go to the Google Compute Engine dashboard and click the “SSH” button in the top left to get the gcloud command to SSH into the node.kubectl -n prod exec -it POD_NAME /bin/sh |
Kill a pod. All of our pods are part of a deployment, so the deployment will spin up a replacement pod automatically. | kubectl delete --namespace=prod pod $POD_NAME |
Get a PostgreSQL client on the prod database. | kubectl exec --namespace=prod -ti $PSQL_POD_ID -- psql -U sg |
List versions in production. |
kubectl -n prod get deploy -o jsonpath='{.items[*].spec.template.spec.containers[0].image} ' | tr ' ' '\n' | sort -u
|
Get access to Jaeger locally. |
kubectl port-forward --namespace=prod svc/jaeger-query 16686
|
Get access to debug server locally. |
kubectl port-forward $(kubectl get po --no-headers -l app=repo-updater | cut -d ' ' -f 1) 6060
|