Manually deploy a config change#

While deploys generally go through our GitHub Actions workflow, sometimes you need to deploy from your laptop - primarily when testing changes on staging or actively fixing an outage. After doing a manual deploy, you must make a PR and get it deployed as soon as possible - otherwise your changes might get reverted the next time a different PR is merged and deploys to the hub you deployed to!

Our deployment scripts live in the deployer/ of this repository, and can deploy one or more hubs to our clusters.

Deploy a single hub#

  1. Setup your local environment to be able to do deploys

  2. Make the config change you want to deploy.

  3. Deploy just a single hub:

    deployer deploy <cluster-name> <hub-name>
    

    The script will look for a hub named <hub-name> in the cluster config defined at config/clusters/<cluster-name>/cluster.yaml and read in the *.values.yaml files associated with that hub.

  4. You can deploy to all hubs on a given cluster by omitting the hub name.

    deployer deploy <cluster-name>
    

Note

You should mostly use the staging hub in the 2i2c cluster for testing.

Run a health check on the hub#

After doing a deploy, you should run a health check on the hub you just deployed to make sure it continues to function. You can do this manually, but there’s also some automation that runs through some basic tests.

The automated test, creates a new hub user called deployment-service-check, starts a server for them, and runs the test notebooks. It checks that the notebook runs to completion and the output cells have the expected value.

  • If the health check passes, then the user’s server is stopped and deleted.

  • If the health check fails, then their user server will be left running, but

  • it will get deleted together with the user in the next iteration.

To run the automated health check on a hub, run

deployer run-hub-health-check <cluster-name> <hub-name>

This will test a simple notebook, but not any dask functionality. To test dask functionality as well, run

deployer run-hub-health-check --check-dask-scaling <cluster-name> <hub-name>

These tests are automatically run when you deploy via CI.