Orchestration

This section presents the NR1™ AI Inference Appliance orchestration solution based on Kubernetes, demonstrating how it facilitates scaling AI pipelines, meeting strict latency requirements, and optimizing operational costs.

NR1 AI Inference Appliance is comprised of NR1™ modules. NR1 modules execute Compute Graphs (CGs) – graph representations of inference pipelines. When a data center solution comprises multiple NR1 modules — whether as part of a single NR1 AI Inference Appliance or across several — the need for orchestrating compute graphs (CGs) becomes essential.

The orchestration of CGs means deciding which NR1 module to use for CG provisioning, how many replicas of each CG to provision, how to detect which CG and which NR1 module are “healthy” and operate correctly. Additional orchestration tasks involve creating and updating network load balancers to target the provisioned CGs and specifying the access map between client applications and CGs.

A modern data center may contain thousands of machines of different kinds. Therefore, the orchestration of multiple CGs on multiple NR1 systems cannot be performed manually, and automatic orchestration is required. Moreover, the utilization of inference services may change drastically, depending on various parameters, for example, on the surge and decline in the number of users who are interested in applications that consume inference.

Kubernetes is used as an orchestration solution for NR1 inference appliances. Kubernetes is a popular open-source framework for orchestrating containers, with pluggable architecture that can be easily extended without changing the code. Since Kubernetes is used for orchestration, NR1 inference appliance benefits from a rich ecosystem of tools integrated with Kubernetes, in particular Argo CD, Prometheus, Elasticsearch and many others.

Since Kubernetes infrastructure is not aware of CGs, these graphs cannot be orchestrated directly. To enable CG orchestration in Kubernetes, each CG in the system is represented as a Kubernetes pod, which we call a CG pod. A CG pod software runs on the CPU of NR1 module in tandem with the accelerated CG and presents this CG to Kubernetes. This way, Kubernetes orchestrates CGs indirectly, by orchestrating the corresponding CG pods. This document describes a set of Kubernetes plugins, reference deployments and integrations implemented to allow CG orchestration in an effective manner.

• Orchestrating CGs by Kubernetes
• Before you begin
• CG pod
• Providing OpenAI API
• Resource Management for CGs
• Load Balancing
  • Before you begin
  • Walkthrough
• Horizontal autoscaling
  • Before you begin
  • Deploying the server and the client, observing metrics
  • Deploying the Prometheus Adapter
  • Deploying the Horizontal Pod Autoscaler
  • Watch an increase in server pods
  • Cleanup
• CG operator
  • Installing CG Operator
  • Using the CG Operator
  • Autoscaling with the CG Operator
• Node Problem Detector
  • Installing the NeuReality Node Problem Detector
  • Using the NeuReality Node Problem Detector
• Adding NeuReality AI Inference Modules to Kubernetes clusters
  • Joining a node
  • Removing a node
• CNI setup
• Prometheus Exporters setup
• Network Policies
  • Before you begin
  • The story
  • Initial setup
  • Defining a Network Policy
  • Validate the policy
  • Cleanup