Suspending a managed instance

How to request

1. Create a new issue using this template
2. Notify Delivery in #cloud-devops channel, linking to the issue

SLAs for this can be found with our other managed instance SLAs.

Engineering instructions

This page documents how suspension of managed instances are done for managed instances. This is useful when Sales team would like to re-engage a customer at a later date and they already have a managed instance provisioned. We will tear down most resources while still keeping the persistent data (disk snapshot) around.

Managed instances configuration is tracked in deploy-sourcegraph-managed-instances.

For basic operations like accessing an instance for these steps, see managed instances operations.

• General setup
• Sourcegraph suspend
  • 0) Sourcegraph suspend setup
  • 1) Mark the database as ready-only
  • 2) Create a snapshot of the current deployment
  • 3) Remove some resources
  • 4) Wrapping up

General setup

Managed instances configuration is tracked in deploy-sourcegraph-managed - make sure you have the latest revision of this repository checked out. For basic operations like accessing an instance for these steps, see managed instances operations.

First, ensure you have the prerequisites installed and up-to-date:

• Terraform CLI
• Deno CLI (used for scripting upgrades)
• Comby CLI (used for rewriting configuration)
• jq

Many of the following commands in this guide, as well as the commands operations, use environment variables. Export the appropriate values for the upgrade so you don’t lose track:

# name of customer deployment (should match folder)
export CUSTOMER=<customer_or_instance_name>
# should match GCP project prefix - typically the default is correct
export PROJECT_PREFIX=sourcegraph-managed
export CURRENT_DEPLOYMENT=$(gcloud compute instances list --project=${PROJECT_PREFIX}-${CUSTOMER} | grep -v "executors" | awk 'NR>1 { if ($1 ~ "-red-") print "red"; else print "black"; }')
# The value can be found in the Managed Instances vault
# https://my.1password.com/vaults/nwbckdjmg4p7y4ntestrtopkuu/allitems/d64bhllfw4wyybqnd4c3wvca2m
export TF_VAR_opsgenie_webhook=<OpsGenie Webhook value>

export TF_VAR_cf_origin_cert_base64=$(gcloud secrets versions access latest --project=sourcegraph-dev --secret="SOURCEGRAPH_WILDCARD_CERT" | base64)
export TF_VAR_cf_origin_private_key_base64=$(gcloud secrets versions access latest --project=sourcegraph-dev --secret="SOURCEGRAPH_WILDCARD_KEY" | base64)

Make sure your copy of the deploy-sourcegraph-managed repository is up to date:

git checkout main
git pull origin main

Sourcegraph suspend

git checkout -b $CUSTOMER/suspend-instance
# all the below steps are documented assuming you are in the customer deployment directory
cd $CUSTOMER

../util/set-db-readonly.sh $CURRENT_DEPLOYMENT true

../util/create-snapshot.ts $CURRENT_DEPLOYMENT
git add . && git commit -m "$CUSTOMER: snapshot deployment"

terraform apply

deployments = []
disks       = {}

terraform plan && terraform apply

git add . && git commit -m "$CUSTOMER: suspend instance" && git push origin HEAD