Load Balancing

In this section, we run multiple replicas of the same CG and perform load balancing of client connections between the replicas. We use Kubernetes services to perform load balancing. Kubernetes services also allow us to use hostnames to address CGs, instead of using pod IPs.

Before you begin

To follow the example in this section, you need a Kubernetes cluster with at least two NR1™ nodes.

Walkthrough

Since a CG pod is a regular Kubernetes pod, you can use it in a higher-level constructs like Deployment and Service.

See the following example of a Deployment for CGs:

resnet_deployment.yaml
  1apiVersion: apps/v1
  2kind: Deployment
  3metadata:
  4  name: resnet50-server
  5  annotations:
  6    neureality.ai/cg-id: "7"
  7  labels:
  8    app.kubernetes.io/name: resnet50
  9spec:
 10  replicas: 2
 11  selector:
 12    matchLabels:
 13      cg-id: "7"
 14  template:
 15    metadata:
 16      annotations:
 17        neureality.ai/cg-id: "7" # required for CNI
 18      labels:
 19        cg-id: "7" # required for pod anti affinity, to prevent allocation of the same CG to the same NR1M node
 20    spec:
 21      nodeSelector:
 22        neureality.ai/arch: nr1 # required to select NR1M to run this pod
 23      affinity:
 24        podAntiAffinity: # required to prevent allocation of the same CG to the same NR1M node
 25          requiredDuringSchedulingIgnoredDuringExecution:
 26          - labelSelector:
 27              matchExpressions:
 28              - key: cg-id
 29                operator: In
 30                values:
 31                - "7"
 32            topologyKey: "kubernetes.io/hostname"
 33      initContainers:
 34      - name: provisioner-client
 35        image: customer-registry.neureality.ai/k8s/cg-pod-provisioner-client:v1.0.0-rc1
 36        imagePullPolicy: IfNotPresent
 37        env:
 38          - name: CG_ID
 39            value: "7" # ID of the CG, must be unique for all the NR1M nodes in the cluster
 40          - name: CG_NAME
 41            value: resnet50 # name of the CG, must be unique for all the NR1M nodes in the cluster
 42          - name: CG_URI
 43            value: "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/resnet50_8b_ai100.tar.gz" # URL of the CG in the CG store
 44          - name: CG_NAMESPACE # namespace of the CG, here the same as the Kubernetes namespace of the CG pod
 45            valueFrom:
 46              fieldRef:
 47                fieldPath: metadata.namespace
 48          - name: PROVISIONER_HOST
 49            valueFrom:
 50              fieldRef:
 51                fieldPath: status.hostIP
 52          - name: PROVISIONER_PORT
 53            value: "5200"
 54      containers:
 55      - name: health-handler
 56        image: customer-registry.neureality.ai/k8s/cg-pod-health-handler:v1.0.0-rc1
 57        imagePullPolicy: IfNotPresent
 58        env: # the same environment variables as for provisioner-client container
 59          - name: CG_ID 
 60            value: "7"
 61          - name: CG_NAME
 62            value: resnet50
 63          - name: CG_URI
 64            value: "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/resnet50_8b_ai100.tar.gz"
 65          - name: CG_NAMESPACE
 66            valueFrom:
 67              fieldRef:
 68                fieldPath: metadata.namespace
 69          - name: PROVISIONER_HOST
 70            valueFrom:
 71              fieldRef:
 72                fieldPath: status.hostIP
 73          - name: PROVISIONER_PORT
 74            value: "5200"
 75          - name: CG_STORE_CREDENTIALS_SCHEME
 76            value: "s3"
 77          - name: CG_STORE_ACCESS_KEY_ID
 78            valueFrom:
 79              secretKeyRef:
 80                name: cg-store-credentials
 81                key: accessKeyID
 82                optional: true
 83          - name: CG_STORE_SECRET_ACCESS_KEY
 84            valueFrom:
 85              secretKeyRef:
 86                name: cg-store-credentials
 87                key: secretAccessKey
 88                optional: true
 89          - name: CG_STORE_SESSION_TOKEN
 90            valueFrom:
 91              secretKeyRef:
 92                name: cg-store-credentials
 93                key: sessionToken
 94                optional: true
 95          - name: CONTAINER_REGISTRY_SERVER
 96            valueFrom:
 97              secretKeyRef:
 98                name: compute-graph-container-registry-credentials
 99                key: server
100                optional: true
101          - name: CONTAINER_REGISTRY_USERNAME
102            valueFrom:
103              secretKeyRef:
104                name: compute-graph-container-registry-credentials
105                key: username
106                optional: true
107          - name: CONTAINER_REGISTRY_PASSWORD
108            valueFrom:
109              secretKeyRef:
110                name: compute-graph-container-registry-credentials
111                key: password
112                optional: true
113        ports:
114          - name: liveness
115            containerPort: 8080
116            protocol: TCP
117        livenessProbe:
118          httpGet:
119            path: /healthy
120            port: liveness
121          failureThreshold: 3
122          periodSeconds: 10
123        startupProbe:
124          httpGet:
125            path: /provisioned
126            port: liveness
127          failureThreshold: 60
128          periodSeconds: 10
129          initialDelaySeconds: 10
130        lifecycle:
131          preStop:
132            httpGet:
133              port: 5200
134              path: /v1/cgs/default/resnet50/delete # the path of the form /v1/cgs/<namespace>/<cg name>/delete
135              host: localhost
136      - name: aiserver-proxy # proxy to the AI server that runs on the host
137        image: customer-registry.neureality.ai/k8s/aiserver-proxy:v1.0.1-rc1
138        imagePullPolicy: IfNotPresent
139        env:
140        - name: PROVISIONER_HOST
141          valueFrom:
142            fieldRef:
143              fieldPath: status.hostIP
144        - name: PROVISIONER_PORT
145          value: "5200"
146        - name: CG_STORE_CREDENTIALS_SCHEME
147          value: "s3"
148        - name: CG_STORE_ACCESS_KEY_ID
149          valueFrom:
150            secretKeyRef:
151              name: cg-store-credentials
152              key: accessKeyID
153              optional: true
154        - name: CG_STORE_SECRET_ACCESS_KEY
155          valueFrom:
156            secretKeyRef:
157              name: cg-store-credentials
158              key: secretAccessKey
159              optional: true
160        - name: CG_STORE_SESSION_TOKEN
161          valueFrom:
162            secretKeyRef:
163              name: cg-store-credentials
164              key: sessionToken
165              optional: true
166        - name: CONTAINER_REGISTRY_SERVER
167          valueFrom:
168            secretKeyRef:
169              name: compute-graph-container-registry-credentials
170              key: server
171              optional: true
172        - name: CONTAINER_REGISTRY_USERNAME
173          valueFrom:
174            secretKeyRef:
175              name: compute-graph-container-registry-credentials
176              key: username
177              optional: true
178        - name: CONTAINER_REGISTRY_PASSWORD
179          valueFrom:
180            secretKeyRef:
181              name: compute-graph-container-registry-credentials
182              key: password
183              optional: true
184        ports:
185          - name: aiserver
186            containerPort: 5200
187            protocol: TCP
188        command: ["/bin/sh"]
189        args: ["-c", "AISERVER_HOST=$(PROVISIONER_HOST) AISERVER_PORT=$(PROVISIONER_PORT) envsubst < /etc/envoy/envoy_template.yaml > /etc/envoy/envoy.yaml && /usr/local/bin/envoy -c /etc/envoy/envoy.yaml"]
190      tolerations: # toleration for neureality-server
191      - key: neureality-server
192        operator: Equal
193        value: "true"
194        effect: NoExecute
195      imagePullSecrets:
196      - name: neureality-container-registry-credentials

Note the following fields:

  • replicas: specifies the number of replicas of the same CG. Each replica (CG pod) will be provisioned on a different NR1™ node.

  • selector: specifies how pods of this Deployment are referenced. In this case, by the label cg-id being equal to 7.

  1. Deploy the Deployment (no pun intended):

    $ kubectl apply -f resnet_deployment.yaml
NAME            STATUS   ROLES    AGE     VERSION         LABELS
deployment.apps/resnet50-server created

  2. Check the pods and their nodes:

    $ kubectl get pod -l cg-id=7 -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP               NODE            NOMINATED NODE   READINESS GATES
resnet50-server-7d7dd74558-mv7sj   2/2     Running   0          22m   172.16.64.67     aspen-nr1s7-7   <none>           <none>
resnet50-server-7d7dd74558-n7ww6   2/2     Running   0          22m   172.16.102.223   aspen-nr1s3-1   <none>           <none>

    Note that the pods are deployed to different NR1™ servers. This is because the Deploymnent’s template contains the podAntiAffinity construct referencing pods with cg-id label. Thanks to the podAntiAffinity construct, Kubernetes never schedules CG pods with the same CG ID to the same server.

  3. See an example of a service for the deployment above:

    resnet_service.yaml
     1apiVersion: v1
 2kind: Service
 3metadata:
 4  name: resnet50
 5spec:
 6  selector:
 7    cg-id: "7"
 8  ports:
 9    - name: inference
10      protocol: TCP
11      port: 1234
12    - name: aiserver
13      protocol: TCP
14      port: 5200

    Note the following fields:

    • selector: specifies how target pods for this service are selected. In this case, by the label cg-id being equal to 7.

    • ports: specifies the inference and AI server ports of NR1™ servers.

  4. Deploy the service:

    $ kubectl apply -f resnet_service.yaml
NAME            STATUS   ROLES    AGE     VERSION         LABELS
service/resnet50 created

  5. Check the service:

    $ kubectl get service resnet50
NAME            STATUS   ROLES    AGE     VERSION         LABELS
kubectl get service resnet50
NAME       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
resnet50   ClusterIP   10.110.227.136   <none>        1234/TCP   6s

    Notice the Cluster IP of the service, you will see it later in the logs.

  6. Observe the YAML of the inference client:

    resnet_client_job_service.yaml
     1apiVersion: batch/v1
 2kind: Job
 3metadata:
 4  name: resnet-client
 5  labels:
 6    app.kubernetes.io/name: resnet-client
 7spec:
 8  completions: 8
 9  parallelism: 4
10  completionMode: Indexed
11  backoffLimit: 1
12  template:
13    metadata:
14      labels:
15        app.kubernetes.io/name: resnet-client
16    spec:
17      restartPolicy: Never
18      containers:
19      - name: client
20        image: customer-registry.neureality.ai/k8s/resnet50-client:v1.0.7-rc1
21        imagePullPolicy: IfNotPresent
22        env:
23        - name: AI_SERVER_HOST
24          value: resnet50
25        - name: HTTP_PORT
26          value: '5200'
27        - name: CG_NAMESPACE
28          value: default
29        - name: CG_NAME
30          value: resnet50
31      imagePullSecrets:
32      - name: neureality-container-registry-credentials

    Note that this time you specify the hostname of the service as the value of the AI_SERVER_HOST variable. (For more information, read Kubernetes DNS). Another change in the YAML: completions is set to 8, parallelism is set to 4, which means we want to run eight client pods, four pods in parallel. (For more information, read Kubernetes Jobs parallel execution).

  7. Apply the job YAML to your cluster:

    $ kubectl apply -f resnet_client_job.yaml

  8. Wait until eight client pods are completed:

    $ kubectl get pod -l app.kubernetes.io/name=resnet-client
NAME                    READY   STATUS      RESTARTS   AGE
resnet-client-0-22lw6   0/1     Completed   0          2m9s
resnet-client-1-gffp6   0/1     Completed   0          2m9s
resnet-client-2-csgn9   0/1     Completed   0          2m9s
resnet-client-3-bjt6z   0/1     Completed   0          2m9s
resnet-client-4-cgw77   0/1     Completed   0          111s
resnet-client-5-ht85s   0/1     Completed   0          111s
resnet-client-6-msrff   0/1     Completed   0          111s
resnet-client-7-hzrnl   0/1     Completed   0          110s

  9. Check the client logs and verify that all the clients were able to recieve all the inference results successfully:

    $ kubectl logs -l app.kubernetes.io/name=resnet-client --tail 100 | grep success | wc -l
16

  10. Check the success metrics of your NR1™ nodes:

    $ for NODE_IP in $(kubectl get node $(kubectl get pod -l cg-id=7 -o jsonpath={.items[*].spec.nodeName}) -o jsonpath='{.items[*].status.addresses[?(@.type == "InternalIP")].address}'); do echo -n "${NODE_IP}: "; curl -s $NODE_IP:9100/metrics | grep 'success_count{cg="7"}'; done
172.23.5.77: success_count{cg="7"} 6
172.23.5.31: success_count{cg="7"} 10

    The metrics show that one NR1™ server served 6 inference requests, while the other one served 10 inference requests.

    Imperfect load balancing

    Note that the distribution of inference requests between NR1™ servers is not perfectly even. Kubernetes services implementation may differ from cluster to cluster. Some implementations perform load balancing probabalisitically - on a large number of requests the distribution will strive to be even.

  11. After you have finished your work, delete the client pods, CG and the service:

    $ kubectl delete -f resnet_client_job.yaml
$ kubectl delete -f resnet_service.yaml
$ kubectl delete -f resnet_deployment.yaml