Deleting a managed instance
SOC2/CI-41
How to request
- Create a new issue using this template
- Notify Cloud in [#cloud] channel, linking to the issue
SLAs for this can be found with our other managed instance SLAs.
Deletion guidance for engineering
Deleting a managed instance is initiated by a CE team member by creating an teardown request. There are a few logical steps to follow.
-
Confirm the request with the CE Team. Data is not able to be recovered after this process is performed.
-
Destroy the customer’s managed instance infrastructure with Terraform.
-
Delete GCP Disk Snapshots
-
Delete GCP Project for the managed instance.
-
Remove audit monitoring from GCP project.
-
Update documentation to remove references of the managed instance.
Confirm teardown request with the CE Team:
Send a message to the #ce team’s Slack channel. An example could be:
Hello, @ce! The <customer name> managed instance is scheduled for teardown in 1 hour. If this is in error, please notify me immediately.
Teardown request issue: <issue link>
Destroy the managed instance infrastructure
Important! Always start with a fresh terminal session to ensure you do not have any conflicting environment variables.
Setup
Clone or navigate to the sourcegraph/deploy-sourcegraph-managed
repository
-
Navigate into the repository
cd deploy-sourcegraph-managed
-
Pull in the latest changes
git checkout main
git pull
-
Setup the environment:
export TF_VAR_opsgenie_webhook=$(gcloud secrets versions access latest --project=sourcegraph-secrets --secret=OPSGENIE_WEBHOOK)
export CUSTOMER=<customer>
-
Create a branch for the teardown
git checkout -b $CUSTOMER/destroy-managed-instance
Navigate to the customer’s managed instance directory
cd $CUSTOMER
Allow the GCP KMS Crypto Key and CloudSQL instance to be deleted
By default, the KMS Crypto Key and CloudS QL instance is prevented from being deleted. This must be changed in order for Terraform to remove all resources.
-
Disable Cloud SQL delete protection by editing the
infrastructure.tf
:module "managed_instance" { + cloud_sql_deletion_protection = false source = "../modules/terraform-managed-instance-new" }
-
Disable KMS key delete protection:
sed -i '' 's/ prevent_destroy = true/ prevent_destroy = false/g' ../modules/terraform-managed-instance-new/infrastructure.tf
-
Apply changes:
terraform init # switch to local module terraform apply
Destroy the infrastructure
This will remove all GCP infrastructure except the Terraform remote state and GCP project.
terraform destroy
# Restore module after destroying instance
git restore ../modules/terraform-managed-instance-new/infrastructure.tf
Delete snapshots
Scheduled snapshots are not managed by Terraform. In order to remove the GCP project, remaining snapshots must be deleted.
Please double-check the value of the $CUSTOMER environment variable in your current session.
gcloud compute snapshots list --project=sourcegraph-managed-$CUSTOMER | grep "data" | awk '{print $1}' | xargs gcloud compute snapshots delete --project=sourcegraph-managed-$CUSTOMER --quiet
Remove the GCP project
# Under $CUSTOMER directory
cd project
terraform destroy -var-file=../terraform.tfvars
Remove customer files
# Back to the repository root
cd ../../
rm -rf $CUSTOMER
Commit the change
To prevent an initial state circular dependency, the Terraform state for GCP projects is committed as a file into the infrastructure repository. Make sure to include this in the pull request.
git add $CUSTOMER
git commit -m "managed-instance-${CUSTOMER}: Remove GCP infrastructure and project"
git push origin HEAD
Review the proposed changes carefully.
Create the pull request
Title: managed-instance-$CUSTOMER: Teardown Managed Instance
Link tear-down request issue in the description
Wait for checks to pass, approval and then merge pull request.
Remove infrastructure monitoring
In the repository root of the sourcegraph/deploy-sourcegraph-managed:
cd monitoring
terraform apply
Remove audit monitoring from removed GCP project
In the repository root of the sourcegraph/infrastructure:
cd security/auto-discovery
terraform apply
Update documentation to remove references of the managed instance
Updating outstanding tracking issues
Search for any open upgrade tracking issues, edit the description to remove the customer’s entry. No need to upgrade if there isn’t anything to upgrade!
This GitHub Query may be helpful.
Close teardown request issue
- Validate that the teardown request issue has references to the pull requests showing the necessary changes. If any are missing, update the PR descriptions with a link to the teardown request issue.
- Close the teardown request issue.