MI 1.1 Upgrade process
Prereq
Environment
You have go
installed
You have gcloud-cli
(and you’re authenticated!)
Configure required env var
export MG_DEPLOY_SOURCEGRAPH_MANAGED_PATH=/path/to/deploy-sourcegraph-managed/repo
Install mg
(or run go run ./util/cmd/.
for all subsequent mg
commands)
make install
# if you don't have $GOBIN
# add $HOME/.bin to your path
GOBIN=~/.bin make install
Upgrade managed_instance
terraform module
- Retrieve the latest executors module release version from https://github.com/sourcegraph/terraform-google-executors/tags
git checkout -b upgrade-executors-$version
- Open modules/executors/main.tf and bump referenced upstream module version if it is outdated
- Determine the next tag of
mi-module-vx.y.z-va
, e.g.mi-module-v3.40.1-v1
.vx.y.z
should match the sourcegraph release versionva
is used to track revision to the module in between the same sourcegraph release
- Do a global string replacement of the referenced module
source
to the next tag for every instances- the reference exists in each
$CUSTOMER/infrastructure.tf
in themanaged_instance
module
- the reference exists in each
- Open a Pull Request, tag the latest
main
with the above tag
Steps
Create the instance upgrade tracking issue
Open the output link in your browser to create the tracking issue
mi create-tracking-issue -target $VERSION // e.g. 4.0.0
Ensure new version of docker-compose.yaml
file is in the golden directory
If they are not, download the file and open a PR to commit the file prior to upgrade
mi update-golden -target $VERSION // e.g. 4.0.0
Ensure config.yaml
file in customer directory is up-to-date
Check $CUSTOMER/config.yaml
and make sure the file is present
Upgrade
Configure facts
export CUSTOMER=demo
export VERSION=3.40.0
Automated Upgrade (recommended)
You should not upgrade more than 5 instances at a time.
gh workflow run mi_upgrade.yml -f customer=$CUSTOMER -f version=$VERSION
Follow the notification in #cloud-notifications.
If the upgrade succeed, followed the instruction in the generated PR, follow the step below to trigger CI check in order to merge the PR
- close PR
- delete branch
- restore the branch
- re-open the PR
Finally, update the tracking issue
If the upgrade fail, follow the logs and restore the instance.
Manual Upgrade (deprecated)
Create branch
git checkout -b $CUSTOMER/upgrade-v$VERSION
Upgrade the deployment. At a high level, this will perform the following steps
- update the symbolic link of
$CUSTOMER/$CURRENT_DEPLOYMENT/docker-compose/docker-compose.yaml
to the target golden filegolden/docker-compose.X.Y.Z.yaml
- sync the updated deployment artifact (e.g. docker-compose.yaml, prometheus_targets.yaml) to the VM
- backup the Cloud SQL instance
- run
docker-compose up -d
on the VM
mi --customer $CUSTOMER upgrade --target $VERSION
(Optional) If the instance has executors enabled (search for enable_executors = true
in $CUSTOMER/terraform.tfvars
), make sure the terraform module is up-to-date, then apply the terraform module
You should be expecting some
replacement
on the executors docker-mirror compute instance and the instance group
# Found in the [Managed Instances vault](https://my.1password.com/vaults/nwbckdjmg4p7y4ntestrtopkuu/allitems/d64bhllfw4wyybqnd4c3wvca2m)
export TF_VAR_opsgenie_webhook=<OpsGenie Webhook value>
terraform apply
Confirm instance health
SOC2/CI-108
Follow these steps
mi --customer $CUSTOMER check
Wrapping up
Commit your change:
git add . && git commit -m "$CUSTOMER: update docker-compose.yaml"
Create a pull request.
SOC2/CI-108
You MUST link the pull request to the Github issue that caused the upgrade.
gh pr create --title "$CUSTOMER: upgrade to $NEW_VERSION" --body "Part of <link to release tracking Github ticket>\n## Test plan No review required: normal upgrade\n"
Fallback plan
Follow restore process
Automated upgrades
For patch release, which contains only images version upgrade, without executor upgrades, Upgrade Managed Instance Github Action can be used. It will perform all mandatory steps from Upgrade section and open Pull Request. After Pull Request is open, please add link to upgrade issue and Github Action performing the upgrade to have history of performed steps.