Providing OpenAI API

By default, NeuReality Compute Graphs serve AI over the fabric (AIoF) protocol. For LLMs, list models, create chat completion, and create completion OpenAI API endpoints are also supported. In this tutorial you will learn how deploy a Compute Graph that provides OpenAI API endpoints.

Here is an example YAML of a CG pod, that represents a Meta Llama-3 8B Compute Graph and provides OpenAI APIs.

It is identical to the YAML of the CG Pod from the CG Pod section, except for the MODEL_NAME and DEFAULT_WORKING_POINT environment variables. The value of MODEL_NAME is "meta-llama/Meta-Llama-3-8B". OpenAI API requests will specify this model name as a parameter. DEFAULT_WORKING_POINT equals "true", which specifies that this CG is the default CG for this model. Remember to replace the value of CG_URI with your CG artifact location.

llm_pod_openai_api.yaml
  1apiVersion: v1
  2kind: Pod
  3metadata:
  4  name: llama-3-8b
  5  annotations:
  6    neureality.ai/cg-id: "7" # required for CNI
  7  labels:
  8    cg-id: "7" # required for pod anti affinity, to prevent allocation of the same CG to the same NR1 node
  9spec:
 10  nodeSelector:
 11    neureality.ai/arch: nr1 # required to select NR1 to run this pod
 12  affinity:
 13    podAntiAffinity: # required to prevent allocation of the same CG to the same NR1 node
 14      requiredDuringSchedulingIgnoredDuringExecution:
 15      - labelSelector:
 16          matchExpressions:
 17          - key: cg-id
 18            operator: In
 19            values:
 20            - "7"
 21        topologyKey: "kubernetes.io/hostname"
 22  initContainers:
 23  - name: provisioner-client
 24    image: customer-registry.neureality.ai/k8s/cg-pod-provisioner-client:v1.2.2
 25    imagePullPolicy: IfNotPresent
 26    env:
 27      - name: CG_ID
 28        value: "7" # ID of the CG, must be unique for all the NR1 nodes in the cluster
 29      - name: CG_NAME
 30        value: llama-3-8b # name of the CG, must be unique for all the NR1 nodes in the cluster
 31      - name: CG_URI
 32        value: "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/llama_3_8b.tar.gz" # URL of the CG in the CG store
 33      - name: CG_NAMESPACE # namespace of the CG, here the same as the Kubernetes namespace of the CG pod
 34        valueFrom:
 35          fieldRef:
 36            fieldPath: metadata.namespace
 37      - name: PROVISIONER_HOST
 38        valueFrom:
 39          fieldRef:
 40            fieldPath: status.hostIP
 41      - name: PROVISIONER_PORT
 42        value: "5200"
 43      - name: MODEL_NAME
 44        value: meta-llama/Meta-Llama-3-8B
 45      - name: DEFAULT_WORKING_POINT
 46        value: "true"
 47      - name: CG_STORE_CREDENTIALS_SCHEME
 48        value: "s3" # the URI scheme of the cg store
 49      - name: CG_STORE_ACCESS_KEY_ID # AWS access key ID
 50        valueFrom:
 51          secretKeyRef:
 52            name: cg-store-credentials
 53            key: accessKeyID
 54            optional: true
 55      - name: CG_STORE_SECRET_ACCESS_KEY # AWS secret access key
 56        valueFrom:
 57          secretKeyRef:
 58            name: cg-store-credentials
 59            key: secretAccessKey
 60            optional: true
 61      - name: CG_STORE_SESSION_TOKEN # AWS session token
 62        valueFrom:
 63          secretKeyRef:
 64            name: cg-store-credentials
 65            key: sessionToken
 66            optional: true
 67      - name: CONTAINER_REGISTRY_SERVER # server of container registry of ARM runtime images
 68        valueFrom:
 69          secretKeyRef:
 70            name: compute-graph-container-registry-credentials
 71            key: server
 72            optional: true
 73      - name: CONTAINER_REGISTRY_USERNAME # username of container registry of ARM runtime images
 74        valueFrom:
 75          secretKeyRef:
 76            name: compute-graph-container-registry-credentials
 77            key: username
 78            optional: true
 79      - name: CONTAINER_REGISTRY_PASSWORD # password for container registry of ARM runtime images
 80        valueFrom:
 81          secretKeyRef:
 82            name: compute-graph-container-registry-credentials
 83            key: password
 84            optional: true
 85  containers:
 86  - name: health-handler
 87    image: customer-registry.neureality.ai/k8s/cg-pod-health-handler:v1.2.2
 88    imagePullPolicy: IfNotPresent
 89    env: # the same environment variables as for provisioner-client container
 90      - name: CG_ID 
 91        value: "7"
 92      - name: CG_NAME
 93        value: llama-3-8b
 94      - name: CG_URI
 95        value: "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/llama_3_8b.tar.gz"
 96      - name: MODEL_NAME
 97        value: meta-llama/Meta-Llama-3-8B
 98      - name: DEFAULT_WORKING_POINT
 99        value: "true"
100      - name: CG_NAMESPACE
101        valueFrom:
102          fieldRef:
103            fieldPath: metadata.namespace
104      - name: PROVISIONER_HOST
105        valueFrom:
106          fieldRef:
107            fieldPath: status.hostIP
108      - name: PROVISIONER_PORT
109        value: "5200"
110      - name: CG_STORE_CREDENTIALS_SCHEME
111        value: "s3"
112      - name: CG_STORE_ACCESS_KEY_ID
113        valueFrom:
114          secretKeyRef:
115            name: cg-store-credentials # the name of the secret for CG store credentials
116            key: accessKeyID
117            optional: true # the container can start even if the secret does not exist
118      - name: CG_STORE_SECRET_ACCESS_KEY
119        valueFrom:
120          secretKeyRef:
121            name: cg-store-credentials
122            key: secretAccessKey
123            optional: true
124      - name: CG_STORE_SESSION_TOKEN
125        valueFrom:
126          secretKeyRef:
127            name: cg-store-credentials
128            key: sessionToken
129            optional: true
130      - name: CONTAINER_REGISTRY_SERVER
131        valueFrom:
132          secretKeyRef:
133            name: compute-graph-container-registry-credentials # the name of the secret for CG container registry credentials
134            key: server
135            optional: true
136      - name: CONTAINER_REGISTRY_USERNAME
137        valueFrom:
138          secretKeyRef:
139            name: compute-graph-container-registry-credentials
140            key: username
141            optional: true
142      - name: CONTAINER_REGISTRY_PASSWORD
143        valueFrom:
144          secretKeyRef:
145            name: compute-graph-container-registry-credentials
146            key: password
147            optional: true
148    ports:
149      - name: liveness
150        containerPort: 8080
151        protocol: TCP
152    livenessProbe:
153      httpGet:
154        path: /healthy
155        port: liveness
156      failureThreshold: 3
157      periodSeconds: 10
158    startupProbe:
159      httpGet:
160        path: /provisioned
161        port: liveness
162      failureThreshold: 60
163      periodSeconds: 10
164      initialDelaySeconds: 10
165    lifecycle:
166      preStop:
167        httpGet:
168          port: 5200
169          path: /v1/cgs/default/llama-3-8b/delete # the path of the form /v1/cgs/<namespace>/<cg name>/delete
170          host: localhost
171  - name: aiserver-proxy # proxy to the AI server that runs on the host
172    image: customer-registry.neureality.ai/k8s/aiserver-proxy:v1.2.2
173    imagePullPolicy: IfNotPresent
174    env:
175    - name: PROVISIONER_HOST
176      valueFrom:
177        fieldRef:
178          fieldPath: status.hostIP
179    - name: PROVISIONER_PORT
180      value: "5200"
181    ports:
182      - name: aiserver
183        containerPort: 5200
184        protocol: TCP
185    command: ["/bin/sh"]
186    args: ["-c", "AISERVER_HOST=$(PROVISIONER_HOST) AISERVER_PORT=$(PROVISIONER_PORT) envsubst < /etc/envoy/envoy_template.yaml > /etc/envoy/envoy.yaml && /usr/local/bin/envoy -c /etc/envoy/envoy.yaml"]
187  tolerations: # toleration for neureality-server
188  - key: neureality-server
189    operator: Equal
190    value: "true"
191    effect: NoExecute
192  imagePullSecrets:
193  - name: neureality-container-registry-credentials

Let’s apply the CG Pod YAML and see the CG provisioned on an NR1™ node:

  1. Deploy the CG pod:

    $ kubectl apply -f llm_pod_openai_api.yaml
NAME            STATUS   ROLES    AGE     VERSION         LABELS
pod/llama-3-8b created

  2. Check the pod and its node:

    $ kubectl get pod llama-3-8b -o wide
NAME       READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
llama-3-8b 1/1     Running   0          2m49s   172.23.5.77   aspen-nr1s1-1   <none>           <none>

  3. Use kubectl wait --for=condition=Ready pod <pod name> to wait for the pod to become ready.

  4. Let’s use the provisioner API and verify that the CG was indeed provisioned and its protocol is OpenAI API. Let’s get the IP of the node the pod runs on:

    $ NODE_IP=$(kubectl get node $(kubectl get pod llama-3-8b -o jsonpath={.spec.nodeName}) -o jsonpath='{.status.addresses[?(@.type == "InternalIP")].address}')

  5. Check the status of the provisioned CG. The URI path is /v1/cgs/<namespace>/<cg name>. Note that your output may vary according to the model of the CG.

    $ curl http://${NODE_IP}:5200/v1/cgs/default/llama-3-8b
{
    "requestMethod": "GET",
    "requestURI": "/v1/cgs/default/llama-3-8b",
    "result": {
        "cgID": 7,
        "cgName": "llama-3-8b",
        "cgURI": "s3://my-cg-store.s3.eu-central-1.amazonaws.com/mycgs/llama_3_8b.tar.gz",
        "default_working_point": true,
        "inputTensors": [
            {
                "dtype": "UINT8",
                "shape": [
                    1,
                    512
                ]
            },
            {
                "dtype": "UINT8",
                "shape": [
                    1,
                    4096
                ]
            }
        ],
        "instance_count": 1,
        "model_name": "meta-llama/Meta-Llama-3-8B",
        "namespace": "default",
        "outputTensors": [
            {
                "dtype": "UINT8",
                "shape": [
                    1,
                    4096
                ]
            }
        ],
        "persist": true,
        "profilerLogSize": 0,
        "provisionPercent": 100,
        "provisionStatus": "Provision completed",
        "replicas": 1,
        "status": "provisioned"
    }
}

    Notice the model_name and default_working_point fields in the returned result.

  6. Use curl to send inference requests to the deployed Compute Graph. You will use the Node IP of the CG pod as the inference server’s IP. (Note that for Open AI we use the IP of the node on which the CG pod runs, and not the IP of the CG pod).

    You will run curl pod from Istio samples to send inference requests on the Kubernetes pod network.

    $ kubectl apply -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/curl/curl.yaml

  7. Let’s check that the CG provides OpenAI models API. Note that NeuReality inference modules provide OpenAI APIs on port 5500.

    In this example you will use the jq command, to display JSON in a visually nice presentation. You will pipe the output of curl into jq.

    $ kubectl exec -it $(kubectl get pod -l app=curl -o jsonpath={.items..metadata.name}) -c curl -- curl $NODE_IP:5500/v1/models | jq
{
    "object": "list",
    "data": [
        {
        "id": "meta-llama/Meta-Llama-3-8B",
        "object": "model",
        "created": 1750254844,
        "owned_by": "organization-owner"
        }
    ]
}

    Notice meta-llama/Meta-Llama-3-8B in the list of the served models.

  8. Send a completion request, for example with prompt “What is the capital of France?”.

    $ kubectl exec -it $(kubectl get pod -l app=curl -o jsonpath={.items..metadata.name}) -c curl -- curl $NODE_IP:5500/v1/completions -H "Content-Type: application/json" -d'{ "model": "meta-llama/Meta-Llama-3-8B", "prompt": "What is the capital of France?", "max_tokens": 20, "temperature": 0 }' | jq .
{
    "id": "cmpl-91f5fedf8a1e4a9d9fc29697c46cfc44",
    "object": "text_completion",
    "created": 1750256850,
    "model": "meta-llama/Meta-Llama-3-8B",
    "choices": [
        {
        "index": 0,
        "text": " Paris, of course! But what is the capital of France in French? Paris, bien sûr!",
        "logprobs": null,
        "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 8,
        "total_tokens": 28,
        "completion_tokens": 20
    }
}

    Notice the answer: Paris, of course! But what is the capital of France in French? Paris, bien sûr!

  9. After you have finished using your CG and want to unprovision it, delete the pod:

    $ kubectl delete -f llm_pod_openai_api.yaml
pod "llama-3-8b" deleted