Resource Management for CGs

Since NeuReality’s compute graphs (CGs) are represented by Kubernetes pods (CG pods), you can use Kubernetes resource management for pods to manage CG resources. CG resources are exposed by NeuReality device plugins. A CG pod can specify requirements for resources of NR1™.

When you define a Pod, you can optionally specify how many resources of a Node the Pod requires, for example CPU or memory, of the number of required GPUs. The resources are specified as requests and/or as limits. Requests specify how many resources should be reserved for the pod, limits specify how many resources the pod is allowed to consume. Kubernetes performs bookkeeping of the resources that pods requested until now and uses this information to decide where a new pod should be placed. Kubernetes may enforce the limits by killing the pod (for example in the case of memory) or by throttling resource consumption (for example in the case of CPU).

Some resources, like CPU and memory, are managed by Kubernetes by default. Kubernetes can be extended using device plugins or Dynamic Resource Allocation to manage additional resources, for example RDMA devices.

To enable resource management of NR1™ nodes, follow the instructions at https://github.com/neureality-sdk/device-plugins to install NeuReality device plugins.

The following resources are managed for NR1™ nodes. Unless you are interested in the architecture details of NR1™, you do not need to understand the resources since the requests are generated automatically by the NeuReality toolchain.

  • neureality.ai/compute_dma_memory_in_mb – a 1MB unit of Compute DMA memory

  • neureality.ai/cpu_dma_memory_in_mb – a 1MB unit of CPU DMA memory

  • neureality.ai/docker_partition_in_mb – a 1MB unit of ARM-as-CE docker image size

  • neureality.ai/ai_hypervisor_queue – number of queues

When you perform kubectl describe node on an NR1™ node, you will see the resources above appear in the Capacity and Allocatable sections, for example:

Capacity:
  cpu:                                     8
  ephemeral-storage:                       1945080Ki
  hugepages-1Gi:                           0
  hugepages-2Mi:                           2Gi
  hugepages-32Mi:                          0
  hugepages-64Ki:                          0
  memory:                                  32731152Ki
  neureality.ai/cpu_dma_memory_in_mb:      512
  neureality.ai/compute_dma_memory_in_mb:  116244
  neureality.ai/docker_partition_in_mb:    13417
  neureality.ai/ai_hypervisor_queue:       65536
  pods:                                    110

When you compile a pipeline, resources.yaml is generated automatically in the same directory as the CG descriptor.

Let’s take a look at resources.yaml generated for a Resnet 50 pipeline:

neureality.ai/compute_dma_memory_in_mb: 3
neureality.ai/cpu_dma_memory_in_mb: 0
neureality.ai/docker_partition_in_mb: 2392
neureality.ai/ai_hypervisor_queue: 7

resources.yaml specifies how many resources a CG requires. You can copy the contents of resources.yaml to the requests and limits sections of the resources field of the health-handler container:

  containers:
  - name: health-handler
    resources:
      requests:
        neureality.ai/compute_dma_memory_in_mb: 3
        neureality.ai/cpu_dma_memory_in_mb: 0
        neureality.ai/docker_partition_in_mb: 2392
        neureality.ai/ai_hypervisor_queue: 7
      limits:
        neureality.ai/compute_dma_memory_in_mb: 3
        neureality.ai/cpu_dma_memory_in_mb: 0
        neureality.ai/docker_partition_in_mb: 2392
        neureality.ai/ai_hypervisor_queue: 7

Attention

Note that NeuReality resources limits must be equal to requests.

Attention

Note that currently NR1™ does not enforce the resource limits, so neither Kubernetes nor NR1™ will prevent your CG from consuming more resources.

Once you deploy a CG pod, Kubernetes will take into consideration the availability of the requested resources on NR1™ nodes. This way Kubernetes will ensure that the CG will get the resources it requires and provisioning of the CG will not fail due to lack of resources.