Exoscale’s New Karpenter Integration: Simplifying Kubernetes Node Management

Exoscale Scalable Kubernetes Service (SKS) has long time support for cluster-autoscaler to automatically adjust the number of nodes in a Kubernetes cluster based on the resource requests of the workloads running in the cluster. This integration has been a key feature for our users, allowing them to optimize their resource usage and reduce costs. This implementation relies on SKS Nodepool implementation. With the introduction of Karpenter, a modern open-source framework, we are excited to enhance our SKS Pro offering even further. Karpenter is a flexible, high-performance Kubernetes cluster autoscaler that improves upon the existing cluster-autoscaler by providing faster scaling, better resource utilization, and support for a wider range of workloads (like GPUs).

In this blog post, we will explore the benefits of using Karpenter with Exoscale SKS, how to get started with the integration, and some best practices for optimizing your Kubernetes workloads.

Key differences between Karpenter and Cluster Autoscaler

While Cluster Autoscaler scales predefined node groups (such as SKS Nodepool) by adding or removing whole nodes when it detects unschedulable pods, Karpenter provisions nodes directly based on each pod’s requirements. This lets Karpenter choose from a wide range of instance types at provisioning time, react faster to demand, and aggressively consolidate or right‑size underutilized capacity (depending on given policies).

Whereas Cluster Autoscaler works best with stable, homogeneous pools and conservative scale-down timings, Karpenter excels with heterogeneous, bursty, or specialized workloads (for example GPUs or spot/ephemeral capacity) and uses CRDs to express scheduling and cost constraints for more precise bin‑packing and cost optimization.

Karpenter’s ability to rapidly provision and scale nodes based on real-time workload demands makes it an ideal fit for Exoscale’s infrastructure. By leveraging Karpenter, Exoscale users can benefit from:

1. Improved Resource Utilization: Karpenter’s fine-grained scheduling allows for better packing of workloads onto nodes, reducing waste and improving overall resource efficiency.
2. Faster Scaling: With Karpenter, new nodes can be provisioned in response to demand spikes much more quickly, ensuring that workloads have the resources they need when they need them.
3. Support for Diverse Workloads: Karpenter’s flexibility means it can accommodate a wider variety of workloads, including those that require specialized hardware like GPUs.
4. Cost Optimization: By using Karpenter’s advanced scheduling capabilities, Exoscale users can optimize their infrastructure costs, ensuring they only pay for the resources they actually need.

In the following sections, we will guide you through the process of integrating Karpenter with your Exoscale Kubernetes clusters and share best practices for maximizing its benefits.

Enhancing Kubernetes Workloads with Karpenter at Exoscale

In order to use Karpenter with SKS, you need to enable it as an additional add-on. We provide Karpenter as an optional managed add-on in SKS Pro clusters with no additional costs. All prerequisites are setup for you by our platform, including the necessary IAM roles and rbac configuration.

First, create a cluster enabling Karpenter. Beware to not forget setting security groups properly as explained in our quick start guide.

If you create the cluster with the CLI, you can use the –enable-karpenter flag:

❯ exo compute sks create --help
This command creates an SKS cluster.

[...]
      --enable-karpenter                       deploy Karpenter node provisioner

In the portal, you have access to the Karpenter option in the “Add-ons” section when creating a new SKS Pro cluster or updating it (unchecked by default):

The option is also available in the Terraform provider starting from v0.67.0, in the exoscale_sks_cluster resource (enable_karpenter boolean field).

Once you created the cluster, you can see all pods are in pending state:

❯ kubectl get pod -A                
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-59556d9b4c-5tkfb   0/1     Pending   0          22h
kube-system   coredns-7d78fc9666-t6f4v                   0/1     Pending   0          22h
kube-system   coredns-7d78fc9666-zghws                   0/1     Pending   0          22h
kube-system   konnectivity-agent-6cd7d54dc6-9v52j        0/1     Pending   0          22h
kube-system   konnectivity-agent-6cd7d54dc6-vd76t        0/1     Pending   0          22h
kube-system   metrics-server-6654fc5ff6-bthqm            0/1     Pending   0          22h

To fix this, you need to create two additional resources.

First you need an ExoscaleNodeClass. This resource defines the template and configuration used to create instances in Exoscale. You can create multiple ExoscaleNodeClass resources to define different node templates. Here is an example of an ExoscaleNodeClass:

apiVersion: karpenter.exoscale.com/v1
kind: ExoscaleNodeClass
metadata:
  name: standard
spec:
  imageTemplateSelector:
    # version: Kubernetes version (semver format like "1.34.1")
    # If omitted (or if you use imageTemplateSelector: {}), the control plane's
    # current Kubernetes version will be auto-detected at runtime
    version: "1.34.1"

    # variant: Template variant (optional, defaults to "standard")
    # Options: "standard" for regular workloads, "nvidia" for GPU-enabled nodes
    variant: "standard"
  
  # Disk size in GB (default: 50)
  diskSize: 100
  
  # Security groups (optional)
  # List the security group IDs to attach to instances
  # Ensure to set a correct security group allowing necessary traffic
  # like CNI or ingress
  securityGroups:
    - "<setme>"
  
  antiAffinityGroups: []
  privateNetworks: []

Now you can create a NodePool using this ExoscaleNodeClass. The NodePool resource defines the scaling policies and constraints for the nodes created by Karpenter. Here is an example of a NodePool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: standard
spec:
  template:
    metadata:
      labels:
        nodepool: standard
    spec:
      nodeClassRef:
        group: karpenter.exoscale.com
        kind: ExoscaleNodeClass
        name: standard
      
      startupTaints:
        - key: karpenter.sh/unregistered
          effect: NoExecute
      
      requirements:
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values:
            # - "standard.small"
            - "standard.medium"
            - "standard.large"
            - "standard.extra-large"

      expireAfter: 30m
  
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    
    consolidateAfter: 30m
    
    budgets:
    - nodes: "10%"  # Disrupt at most 10% of nodes at once

  weight: 50

  limits:
    cpu: 1000
    memory: 4000Gi

This NodePool allows Karpenter to create nodes using the standard ExoscaleNodeClass template. It specifies that nodes can only be of type standard.medium, standard.large, or standard.extra-large. The expireAfter field indicates that nodes should be terminated after 30 minutes of inactivity. The disruption policy is set to consolidate nodes when they are empty or underutilized, with a maximum of 10% of nodes being disrupted at once. This NodePool can scale up to a maximum of 1000 CPUs and 4000Gi of memory.

If you need more information about NodePools, please refer to the official documentation.

When you have created these two resources, Karpenter will be able to provision nodes in your Exoscale SKS cluster, depending on the nodeSelectors and tolerations of your workloads.

In our example, after creating the ExoscaleNodeClass and NodePool, Karpenter will start provisioning nodes to accommodate the pending pods. You can monitor the status of the nodes and pods using the following commands:

# Karpenter NodeClaims, holding Exoscale real resources
kubectl get nodeclaim
NAME             TYPE              CAPACITY    ZONE       NODE                            READY   AGE
standard-w5kwn   standard.medium   on-demand   ch-gva-2   karpenter-test-standard-w5kwn   True    13m
standard-whvjs   standard.medium   on-demand   ch-gva-2   karpenter-test-standard-whvjs   True    16m

# Kubernetes Nodes created by Karpenter
kubectl get node        
NAME                            STATUS   ROLES    AGE   VERSION
karpenter-test-standard-w5kwn   Ready    <none>   14m   v1.33.2
karpenter-test-standard-whvjs   Ready    <none>   17m   v1.33.2

# Now all pods are running
❯ kubectl get pod -A 
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-59556d9b4c-9knmm   1/1     Running   0          15m
kube-system   calico-node-2b6s5                          1/1     Running   0          17m
kube-system   calico-node-99gl7                          1/1     Running   0          15m
kube-system   coredns-7d78fc9666-6h5x6                   1/1     Running   0          17m
kube-system   coredns-7d78fc9666-jfnb8                   1/1     Running   0          14m
kube-system   konnectivity-agent-6cd7d54dc6-dxz87        1/1     Running   0          17m
kube-system   konnectivity-agent-6cd7d54dc6-xs2tv        1/1     Running   0          14m
kube-system   kube-proxy-cxpdp                           1/1     Running   0          17m
kube-system   kube-proxy-xp6mf                           1/1     Running   0          15m
kube-system   metrics-server-6654fc5ff6-2gl7j            1/1     Running   0          17m

As you see in the above output, Karpenter has successfully provisioned two standard.medium nodes to accommodate the pending pods. It’s the most cost-effective instance type matching the requests of the pods and our NodePool. It also ensures that pod anti-affinity rules are respected by placing pods on different nodes.

Best Practices for Using Karpenter with Exoscale SKS

To get the most out of Karpenter with Exoscale SKS, consider the following best practices:

1. Define Multiple ExoscaleNodeClass Resources: Create different ExoscaleNodeClass resources for various workload types (e.g., CPU-intensive, memory-intensive, GPU workloads). This allows Karpenter to select the most appropriate instance type for each workload.
2. Set Appropriate Resource Requests and Limits: Ensure that your pods have accurate resource requests and limits defined. This helps Karpenter make informed decisions about node provisioning and scaling.

Conclusion

Karpenter represents a significant advancement in Kubernetes autoscaling, offering improved resource utilization, faster scaling, and support for diverse workloads. By integrating Karpenter with Exoscale SKS, users can take advantage of these benefits to optimize their Kubernetes deployments.

Our goal at Exoscale is to provide our users with the best tools and technologies to manage their cloud infrastructure effectively. We believe that Karpenter is a powerful addition to our SKS offering, and we encourage our users to explore its capabilities. We are committed to supporting Karpenter and will continue to enhance our integration based on user feedback and evolving best practices. We will integrate it natively in our SKS service in the future to offer a fully managed experience.