Autoscaling with the CG Operator

Before you begin

1. Ensure that you have a Kubernetes cluster with two or more NR1™ nodes.

2. Ensure that you have a Prometheus database, configured to scrape node metrics on port 9100. In this walkthrough, it is assumed that you can access the Prometheus UI and the Prometheus query URL. We designate the Prometheus host by the PROMETHEUS_HOST environment variable.

3. You need Prometheus exporters configured and running in your cluster.

4. You need a client implementation that sends multiple inference requests in parallel.

Deploying the server and the client, observing metrics

1. Create the CG custom resource as in the previous section:

   $ kubectl apply -f resnet_cg.yaml
   NAME            STATUS   ROLES    AGE     VERSION         LABELS
   cg.k8s.neureality.ai/resnet50 created
   

2. Reduce the number of replicas to 1:

   $ kubectl scale cg resnet50 --replicas 1
   

3. Verify that only a single pod is running:

   $ kubectl get pod -l cg-id=7
   NAME                                      READY   STATUS    RESTARTS   AGE
   cg-deployment-resnet50-5669f8b674-2sl2p   2/2     Running   0          18s
   

4. Deploy your client that sends multiple inference requests in parallel. Use cg-service-resnet50 as the server host URL.

5. Observe jobs_in_progress metric in the Prometheus UI.

6. Query jobs_in_progress using the Prometheus Query URL:

   $ curl -s -X POST "${PROMETHEUS_HOST}/api/v1/query" --data-urlencode "query=jobs_in_progress" | jq .
   {
       "status": "success",
       "data": {
           "resultType": "vector",
           "result": [
           {
               "metric": {
               "__name__": "jobs_in_progress",
               "cg": "7",
               "container": "node-exporter",
               "endpoint": "http-metrics",
               "instance": "172.23.5.31:9100",
               "job": "node-exporter",
               "namespace": "monitoring",
               "pod": "prometheus-community-prometheus-node-exporter-8fv9w",
               "service": "prometheus-community-prometheus-node-exporter"
               },
               "value": [
               1718719124.793,
               "0"
               ]
           },
           {
               "metric": {
               "__name__": "jobs_in_progress",
               "cg": "7",
               "container": "node-exporter",
               "endpoint": "http-metrics",
               "instance": "172.23.5.77:9100",
               "job": "node-exporter",
               "namespace": "monitoring",
               "pod": "prometheus-community-prometheus-node-exporter-z7h9z",
               "service": "prometheus-community-prometheus-node-exporter"
               },
               "value": [
               1718719124.793,
               "16"
               ]
           }
           ]
       }
   }
   

Deploying the Prometheus Adapter

Follow the instructions in Deploying the Prometheus Adapter section, on Horizontal autoscaling page.

Using the scaling section in CG custom resource

Check the CG custom resource with scaling section.

apiVersion: k8s.neureality.ai/v1alpha1
kind: CG
metadata:
  name: resnet50
spec:
  id: 7
  uri:  "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/resnet50_8b_ai100.tar.gz"
  replicas: 1
  scaling:
    minReplicas: 1
    maxReplicas: 3
    metrics:
    - name: jobs_per_cg_pod
      desiredAverage: 8

Important points to note:

• minReplicas: minimal number of replicas of the CG that the operator can create

• maxReplicas: maximal number of replicas of the CG that the operator can create

• metrics section

  • name: the name of the metric, in this case jobs_in_progress

  • desiredAverage: the desired value of the metric, average over all the pods of the same CG.

This metric specification instructs the HorizontalPodAutoscaler to operate towards the goal of having on average 8 jobs per CG pod. You may want to change the target value according to your client and your environment.

Watch an increase in server pods

1. Apply the CG custom resource with scaling section:

   $ kubectl apply -f resnet_cg_scaling.yaml
   

2. Deploy your client pods. In the tests we ran, we managed to send enough inference requests to achive metric value of 21 and to trigger an increase of the number of pods by the Horizontal Pod Autoscaler.

3. Wait for more CG pods to be created:

   $ kubectl get pod -l cg-id=7
   NAME                             READY   STATUS    RESTARTS   AGE
   resnet-server-6bbd7554d8-gp775   2/2     Running   0          4h11m
   resnet-server-6bbd7554d8-spjqh   0/2     Running   0          2m37s
   

Cleanup

1. After finishing your work, delete the clients and the CG:

   $ kubectl delete -f resnet_client_job.yaml
   $ kubectl delete -f resnet_cg_scaling.yaml