Cost Monitoring System

Cost Monitoring System#

The Cost Monitoring System is designed to monitor and attribute cloud infrastructure costs to 2i2c hub deployments. This system integrates with the AWS Cost Explorer API to provide detailed cost insights from a hub’s Grafana dashboard.

For details on how to enable the cost monitoring system, see the how-to guide.

Note

See Cost Monitoring Availability for a description of which clusters the Cost Monitoring System is available.

Components#

1. AWS IAM Role Configuration#

A dedicated IAM role called jupyterhub_cost_monitoring_iam_role is created using terraform in order to grant the necessary permissions for accessing the Cost Explorer API.

2. JupyterHub Cost Monitoring#

A Python-based web server is deployed to interact with the Cost Explorer API. It retrieves cost data from the AWS Cost Explorer API and serves it as JSON for Grafana to consume.

The helm deployment is unconditionally enabled, unless explicitly overridden in the config/clusters/<cluster_name>/support.values.yaml file, and the configuration is automatically defined in the helm-charts/support/values.jsonnet file.

3. Grafana Dashboard#

A custom dashboard is presently defined in the infrastructure repository (to be upstreamed to jupyterhub/grafana-dashboards).

This enables Grafana to query the web server for cost data, allowing users to visualize and analyze cloud expenses directly within the Grafana interface.

It uses the Infinity Grafana plugin to serve JSON from AWS Cost Explorer API, for use by Grafana dashboard panels.

Technical implementation#

The system relies on at least one of these tags activated to track resource cost allocations:

  • 2i2c:hub-name

  • 2i2c:node-purpose

  • 2i2c.org/cluster-name

  • alpha.eksctl.io/cluster-name

  • kubernetes.io/cluster/<cluster_name>

  • kubernetes.io/created-for/pvc/name

  • kubernetes.io/created-for/pvc/namespace

Important

Currently, on clusters that have a k8s version greater or equal with 1.30, terraform managed resources already have the 2i2c.org/cluster-name tag configured via the default_tags variable, and eksctl managed resources already have the tag configured for node groups via nodegroup.libsonnet.

On clusters that have a k8s version less than 1.30, eksctl managed resources, the alpha.eksctl.io/cluster-name and kubernetes.io/cluster/<cluster name> tags are present and used instead.

New clusters have all eksctl managed resources configured to be tagged, not just the node groups. This isn’t important to ensure for existing clusters’ cost monitoring though.

The system also relies on the tag 2i2c:hub-name to be specified in addition to the tags above for any cloud infra tied to specific hubs.

We only need to ensure the 2i2c.org/cluster-name and 2i2c:hub-name tags are declared, the others are applied by eksctl and Kubernetes controllers that can create cloud resources to represent k8s resources (block storage volumes for k8s PV resources referencing certain storage classes, and load balancers for k8s Service’s of type LoadBalancer).

The following resources are known to be hub specific in some cases and known to incur costs.

  • S3 buckets in terraform

  • EFS storage in terraform

  • EBS volumes in terraform

  • Node groups in eksctl