Event infrastructure preparation checklist

Specifically for events, the node sharing benefits via profile lists vs. setting a minimum node count are:

• no terraform/eks infrastructure changes

  they shouldn’t require modifying terraform/eks code in order to change the underlying cluster architecture thanks to Our instance type choice that should cover most usage needs

• more cost flexibility

  we can setup the infrastructure a few days before the event by opening a PR, and then just merge it as close to the event as possible. Deploying an infrastructure change for an event a few days before isn’t as costly as starting “x” nodes before, which required an engineer to be available to make terraform changes as close to the event as possible due to costs

• less engineering intervention needed

  the instructors are empowered to “pre-warm” the hub by starting notebook servers on nodes they wish to have ready.

Make sure the information in the event GitHub issue was filled in, especially the hub that is going to be used, the number of expected users before an event and their expected resource needs (if that can be known by the community beforehand).

Check that the current number of users/node respects the following general event wishlist.

• have at least 3-4 people on a node as few users per node cause longer startup times, but no more than ~100

• don’t have more than 30% of the users waiting for a node to come up

  If the current number of users per node doesn’t respect the rules above, you should adjust the instance type so that it does. Note that if you are changing the instance type, you should also consider re-writing the allocation options, especially if you are going with a smaller machine than the original one.

  deployer generate resource-allocation choices <instance type>

The oversubscription factor is how much larger a limit is than the actual request (aka, the minimum guaranteed amount of a resource that is reserved for a container). When this factor is greater, then a more efficient node packing can be achieved because usually most users don’t use resources up to their limit, and more users can fit on a node.

However, a bigger oversubscription factor also means that the users that use more resources than they are guaranteed can get their kernels killed or CPU throttled at some other times, based on what other users are doing. This inconsistent behavior is confusing to end users and the hub, so we should try and avoid this during events.

For an event, you should consider an oversubscription factor of 1.

• if the instance type remains unchanged, then just adjust the limit to match the memory guarantee if not already the case

• if the instance type also changes, then you can use the deployer generate resource-allocation command, passing it the new instance type and optionally the number of choices.

  You can then use its output to:

  • either replace all allocation options with the ones for the new node type

  • or pick the choice(s) that will be used during the event based on expected usage and just don’t show the others