Calculate how much cloud infrastructure costs#
We have several community contracts where we first pay their cloud bills and
then invoice them for it. Any community paying their cloud bills through 2i2c
must have their hub(s) under a cluster.yaml file declaring
[gcp|aws].billing.paid_by_us=true.
Practically to invoice these communities, we update a spreadsheet for billing
summarizing monthly costs for communities, and then request invoicing help from
CS&S via the #billing slack channel.
Important
If you do not have access to the #billing slack channel, ask in the
#team-updates channel and someone will add you.
This documentation helps you update that spreadsheet for communities on shared clusters and generate CSV files for dedicated clusters.
Important
Make sure you have access to the cloud costs folder and spreadsheet for billing before following the steps below. Your 2i2c.org email should give you access.
Communities with dedicated cloud accounts#
Get a community dedicated AWS accounts’s costs#
Note
As of 2024-04-17, we only manage the cost for AWS accounts associated with our
AWS management account. If a future cluster deviates from this, you can tell by
aws.billing.paid_by_us being set true in its cluster.yaml.
Login to shared AWS SSO at https://2i2c.awsapps.com/start#/
Select the
2i2c-sandboxaccount, as it is the primary billing accountSelect ‘AdministratorAccess’ to open the AWS Console for this account
Visit the “Monthly Costs By Linked Account” report (direct link) via “Billing and Cost Management” -> “Cost Explorer Saved Reports”
On the right sidebar under “Time -> Date Range”, select all the completed months we want to get data for
On the right sidebar under “Time -> Granularity”, ensure its selected as “Monthly”
On the right sidebar under “Group by -> Dimension”, select “Linked account”
Click the ‘Download as CSV’ button
AWS billing UI#
Run the following deployer command to convert the generated CSV file into the format required for the invoicing process.
deployer transform cost-table aws pathto/downloaded/csvfile
This will output a new CSV file to your local filesystem called
AWS_{START_MONTH}_{END_MONTH}.csv.Upload this CSV file to the cloud costs folder
As of July 2024, ping James and Harold in the
#billingslack channel to let them know the info for dedicated clusters is now available and provide a link to the file you have just uploaded
Get a community dedicated GCP projects’ costs#
Important
Currently this is the recommended way of retrieving the costs from GCP.
Go to the 2i2c billing account on GCP
Select ‘Reports’ on the left sidebar
Under time range on the right sidebar, select ‘Invoice Month’
Select the time range you are interested in. Note that this has to be at least two months right now, or the next step does not work
Under ‘Group by’, select ‘Month -> Project’ or just ‘Project’ if the time range is only one month
Under the chart, click the ‘Download CSV’ button.
Important
It is important to not change the name of the file downloaded from GCP as it is used by the deployer to incur the month we are billing for.
GCP billing UI#
Run the following deployer command to convert the generated CSV file into the format required for the invoicing process.
deployer transform cost-table gcp pathto/downloaded/csvfile
This will output a new CSV file to your local filesystem called
GCP_{START_MONTH}_{END_MONTH}.csv.Upload this CSV file to the cloud costs folder
As of July 2024, ping James and Harold in the
#billingslack channel to let them know the info for dedicated clusters is now available and provide a link to the file just uploaded
Experimental#
We have an unfinished attempt to automate collection of community monthly costs. This heading retains documentation about that, but we are for 2024 not driving development here and instead relying on the manual approaches.
Get costs automatically via the deployer (not ready to be used yet!)#
Warning
This deployer command requires more development work and is not yet recommended to be used.
The generate cost-table subcommand of the deployer will go through all our
clusters set up on Google Cloud, and tell you how much they cost.
Pre-requisites for running it:
You have all tools required required to work on this repo setup.
Run
gcloud application-default auth login, and authenticate with your2i2c.orggoogle account. This account must have permissions to all GCP projects. This requirement will be relaxed at some point in the future.
There is a private Google Sheet that has monthly costs for all the clusters that are configured to have bigquery export.
This sheet is currently manually updated. You can update it by running
deployer generate cost-table --output 'google-sheet'. It will by default
update the sheet to provide information for the last 12 months. You can control
the period by passing in the start_month and end_month parameters.
If you just want to take a look at the costs in the terminal, you can also run
deployer generate cost-table --output 'terminal' instead.
Warning
If the script is run before the end of the month, the total costs would not be accurate. Run the script once the month has finished to get an accurate amount of the previous month’s costs.
Caveats#
The data comes from bigquery costs export, so is only available and accurate after that has been enabled. For billing data before this was enabled, you need to manually go look in the cloud console.