Skip to main content
Ctrl+K

docs

  • Twitter
  • Mastodon
  • Contact
  • Blog
  • Twitter
  • Mastodon
  • Contact
  • Blog

Infrastructure Guide

Get started

  • Documentation structure
  • Setting up your local environment to work on this repo

SRE guide

  • Support tasks
    • Access home directories of users of a hub
    • Decrypt encrypted information sent to support@2i2c.org
    • Build a Docker Image remotely
    • Apply Cloud Credits
    • Give Grafana access to a community representative
  • Manage cloud and Kubernetes infrastructure
    • Authenticating with cloud providers for 2i2c engineers
    • Manual node administration
    • Cull resources
  • Scaling nodepools
    • AWS
    • Azure
  • Common problems and their solutions

Hub deployment guide

  • Create new cloud accounts
    • Create a new GCP project
    • Create a new AWS account
  • Add Kubernetes clusters
    • New Kubernetes cluster on GCP or Azure
    • New Kubernetes cluster on AWS
    • AWS with NASA SMCE
  • Deploy support components
    • Configure and deploy the support chart
    • Setup grafana dashboards
    • Register the cluster’s Prometheus server with the central Grafana
  • Deploy a new hub
  • Enable authentication
    • GitHub Orgs and Teams
    • CILogon
  • Other hub-related operations
    • Manually deploy a config change
    • Moving Hubs
      • Move a Hub to a new URL
      • Move a Hub across clusters
    • Delete a hub

How-to guides

  • Enable specific hub features
    • 1.1. New Kubernetes cluster on GCP or Azure
    • 1.2. New Kubernetes cluster on AWS
    • 2. Dedicated nodepool on a shared cluster (recommended for events)
    • 1.1. Authenticate using CILogonOAuthenticator
    • 1.2. Authenticate using GitHubOAuthenticator
    • 1.3. Authenticate using TmpAuthenticator for an Ephemeral hub
    • 2. Anonymize usernames
    • 1. Community specific hub domain
    • 2. Configure the hub login page
    • 1. Shared data directories
    • 2. Object storage buckets (persistent/scratch)
    • 2.1. Cloud Permissions
    • 3. Setup a database server per user
    • 4. Setup a shared database for all users on the hub
    • 1. Allow users to push to GitHub from the Hub (gh-scoped-creds)
    • 2. Enable `nbgitpuller` for private repos (git-credential-helper)
    • 3. Authenticated static websites
    • 4. (default enabled) Configurator
    • 5. (dedicated clusters only) Grafana
    • 6. Using JupyterHub as an identity provider
    • 1. GPU
    • 2. Cull resources (jupyterhub-idle-culler)
    • 2. Allow users to setup custom, free-form user profile choices
    • Enable user image features
    • Use Rocker based R images
  • Calculate how much cloud infrastructure costs
  • Modify our custom JupyterHub image
  • Manage domains of hubs
    • Set a community-specific domain using a CNAME
    • Setup Domain Redirects
    • Override a hub’s domain name
  • Enable GitHub Organisation authentication for Grafana
  • Manage a hub’s user environment
  • Upgrade Kubernetes clusters
    • Upgrade Kubernetes cluster on AWS
  • Troubleshooting
    • Look at logs to troubleshoot issues
      • Cloud specific logging
      • Kubectl logging
      • Common errors and what logs to check
    • SSH into nodes
    • Troubleshoot prometheus issues
    • CILogon: switch Identity Providers or user accounts
  • Regenerate a password for a user in a NASA SMCE account

Topic guides

  • Access and credentials
    • Authenticating with cloud providers for 2i2c engineers
    • Secrets and private keys
  • Infrastructure
    • Configuration structure
    • Cluster design considerations
    • Terraform
    • Hub helm charts
    • Network policy
    • User home directory storage
  • Monitoring and alerting
    • Grafana dashboards
    • Prometheus
    • Simple HTTPS uptime checks
  • Features available on the hubs
  • Resource Allocation on Profile Lists

Reference

  • List of running hubs
  • List of hub features
  • Our CI/CD system
    • Automatic hub deployment
    • Automatically bumping image tags and helm sub-chart versions
  • Tools used in this repo

Contributing

  • Review and merge guidelines
  • Repository
  • Open issue
  • .md

Troubleshooting

Troubleshooting#

We have many different tools at our disposal to troubleshoot issues that may arise.

  • Look at logs to troubleshoot issues
    • Cloud specific logging
    • Kubectl logging
    • Common errors and what logs to check
  • SSH into nodes
    • Kubernetes ephemeral containers
    • GCP
    • AWS
  • Troubleshoot prometheus issues
    • Running out of disk space
  • CILogon: switch Identity Providers or user accounts
    • Switch Identity Providers
    • Switch user accounts
  • 403 - Unauthorized errors

previous

Upgrade Kubernetes cluster on AWS

next

Look at logs to troubleshoot issues

By 2i2c.org

© Copyright 2020, 2i2c.org.

By the International Interactive Computing Collaboration (2i2c)