Using the CG Operator

An example YAML of a CG custom resource.

resnet_cg.yaml

apiVersion: k8s.neureality.ai/v1alpha1
kind: CG
metadata:
  name: resnet50
spec:
  id: 7
  uri:  "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/resnet50_8b_ai100.tar.gz"
  replicas: 2

You need to specify the following fields:

• In metadata:

  • name: the CG name

• In spec:

  • id: you must allocate a CG ID, a unique integer per Kubernetes cluster.

  • uri: the URI of the CG in a CG store.

  • replicas: the number of replicas of the CG.

1. Create the CG custom resource:

   $ kubectl apply -f resnet_cg.yaml
   NAME            STATUS   ROLES    AGE     VERSION         LABELS
   cg.k8s.neureality.ai/resnet50 created
   

2. Check the CG pods created by the operator. Their number should be equal to the number of replicas you specified in the CG custom resource.

   $ kubectl get pod -l cg-id=7
   NAME                                      READY   STATUS    RESTARTS   AGE
   cg-deployment-resnet50-5669f8b674-2sl2p   1/2     Running   0          18s
   cg-deployment-resnet50-5669f8b674-5vjsw   1/2     Running   0          18s
   

3. Check the CG service created by the operator:

   $ kubectl get svc cg-service-resnet50
   NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
   cg-service-resnet50   ClusterIP   10.100.228.126   <none>        1234/TCP,8080/TCP,5200/TCP   58s
   

4. Observe the YAML of the inference client for the CG:

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: resnet-client
     labels:
       app.kubernetes.io/name: resnet-client
   spec:
     completions: 8
     parallelism: 4
     completionMode: Indexed
     backoffLimit: 1
     template:
       metadata:
         labels:
           app.kubernetes.io/name: resnet-client
       spec:
         restartPolicy: Never
         containers:
         - name: client
           image: customer-registry.neureality.ai/k8s/resnet50-client:v1.0.7-rc1
           imagePullPolicy: IfNotPresent
           env:
           - name: AI_SERVER_HOST
             value: cg-service-resnet50
           - name: HTTP_PORT
             value: '5200'
           - name: CG_NAMESPACE
             value: default
           - name: CG_NAME
             value: resnet50
         imagePullSecrets:
         - name: neureality-container-registry-credentials
   

   This is the same client as in Load balancing section, but with the value of the AI_SERVER_HOST environment variable set to cg-service-resnet50 (the name of the service created by the CG operator).

5. Apply the job YAML to your cluster:

   $ kubectl apply -f resnet_client_job.yaml
   

6. Wait until eight client pods are completed:

   $ kubectl get pod -l app.kubernetes.io/name=resnet-client
   NAME                    READY   STATUS      RESTARTS   AGE
   resnet-client-0-22lw6   0/1     Completed   0          2m9s
   resnet-client-1-gffp6   0/1     Completed   0          2m9s
   resnet-client-2-csgn9   0/1     Completed   0          2m9s
   resnet-client-3-bjt6z   0/1     Completed   0          2m9s
   resnet-client-4-cgw77   0/1     Completed   0          111s
   resnet-client-5-ht85s   0/1     Completed   0          111s
   resnet-client-6-msrff   0/1     Completed   0          111s
   resnet-client-7-hzrnl   0/1     Completed   0          110s
   

7. Check the client logs and verify that all clients were able to recieve all the inference results successfully:

   $ kubectl logs -l app.kubernetes.io/name=resnet-client --tail 100 | grep success | wc -l
   16
   

8. After finishing your work, delete the clients and the CG:

   $ kubectl delete -f resnet_client_job.yaml
   $ kubectl delete -f resnet_cg.yaml