NeuReality Software SDK V1.0 Release Notes

Release Date

August 15, 2024

Target Audience

This is a NeuReality Software SDK v1.0 release for selected early adopters. The use of the SDK is subject to the Early Access Program Agreement (or similar written instrument) executed by and between your organization and NeuReality (‘License Agreement’) and is provided solely for the purpose of Proof of Concept (PoC) programs.

What’s new in this release?

The NeuReality NR1-S™ AI Inference solution is a holistic system comprised of hardware appliance, software platform, and software SDK - the APIs that significantly simplify the deployment, management, and usage of AI application. It enables you to deliver AI workloads with high performance at scale and is suitable for use in both enterprise data centers and public clouds.

NeuReality NR1-S AI Inference Solution

NR1-S Inference Appliance

The NR1-S AI Inference Appliance holds multiple NR1-M™ AI Inference modules, each equipped with an NR1 AI-CPU card, designed to pair with an AI Accelerator chip. This allows for optimal performance and scalability, while reducing costs and power consumption. In addition, the NR1-S Appliance provides centralized BMC, power, and thermal control. NR1-S Appliance may host up to 10 NR1-M cards in configuration 1:1 with AI Accelerator chip or up to 4 NR1-M cards in configuration 1:4 with AI Accelerator chip. Qualcomm Cloud AI 100 Standard and Professional deep learning accelerators are used in this release.

NR1™ AI-CPU

The NR1™ AI SoC stands out as the innovative AI-CPU. Functioning as an AI inference server-on-a-chip, it distinguishes itself as a hardware device meticulously tailored for workflow optimization. Equipped with specialized processing units and native networking and virtualization features, the NR1 AI-CPU is designed with comprehensive AI pipeline offload capabilities. Its NR1 AI-Hypervisor™ hardware takes charge of data-path processing, encompassing pre- and post-processing engines, and an embedded networking interface. The engines include:

• Media and Data Compute

  • Kernel libraries for processing of vision, audio, math, language, recommendation

  • 4x video/JPEG decoders

  • 16x audio/speech DSPs

  • 16x general purpose vector DSPs

• 6.NR1 AI-over-Fabric™ Networking Technology

  • 2x 25/50 GbE ports

  • AI-over-fabric (over TCP / RoCEv2) for high efficiency and reduced latency

  • Client-server & server-server links support

  • Line rate cryptography supporting IPsec/TLS

  • 2 tiers of isolated network functions (CSP networking)

  • Secured boot support + measured Boot for NR1-M-TPM

• NR1 AI-Hypervisor™ - hardware-based sequencing, QOS, and data movement

• PCIe GEN5 x16 (RC, EP, multi-device up to 256)

• 10 channels of LPDDR5 32bit 6400Mb/s (up to 160GB, 256GB/s)

NR1-M™ AI Inference Module

The NR1-M AI Inference module is a full-height double-wide PCIe card containing one NR1 AI-CPU that can connect to any external AI Accelerator chip - GPU, XPU, FPGA, ASIC - to significantly optimize its utilization rate. The NR1-M is a Full-Height Full-Length (FHFL) dual-slot PCIe Gen5 x16 module that includes a single 7 nm NR1™ SoC with LPDDR5 160 GB on-module memory. NR1-M meets the PCIe CEM 5.0 specifications and supports passive cooling for closed-loop thermal control in a PCIe server.

NR1-S AI Inference Cluster

NR1-S AI Inference clusters are available for use in enterprise data centers or via Cirrascale public cloud.

The following plans are supported:

• Demo - access to Performance Demo application running on NR1-S AI Inference cluster. Includes full deployment setup for selected AI pipelines.

• PoC - full access to PoC NR1-S AI Inference cluster. Enables hands-on experience with the development and deployment of AI pipelines on NR1-S AI Inference appliance. End to end demonstration pipelines, pre/post-processing functions, models, and tutorials are included. Data Center integration and BMC are supported. Scalable infrastructure, from a single acceleration card to the entire cluster.

NeuReality SDK

• NeuReality SDK enables the optimization, deployment, validation, and management of AI pipelines on a NR1-S AI Inference appliance for a variety of applications, including natural language processing, computer vision, and more.

• Deployment API (package neureality) for building the AI pipeline, validating them on x86 functional simulator equipped with Qualcomm Cloud AI100 accelerators1, and profiling them on NR1-S AI inference appliance.

  • AI pipelines can include PyTorch, ONNX, Python models compiled with Qualcomm Cloud AI100 SDK v1.12, Python models, model ensembles, and end to end workloads in Python programming language.

  • The collection of supported AI models is aligned with Qualcomm Cloud AI100 SDK v1.12.

  • Operators library (package neureality.nrlib) is enriched with the following pre/post-processing functions: Resize, Normalize, Crop, Transpose, NMS, Unravel, Trilu, Argmax, Rescale_boxes, Decode_boxes. See Operators library user manual for additional information.

  • Models compiled with Qualcomm Cloud AI100 SDK and validated in this release:

    • Albert-base v2

    • BERT-base

    • BERT-large

    • Roberta-base

    • Roberta-large

    • Deberta-v3-xsmall

    • All-MiniLM-L6-v2

    • SDXL

    • ViT-base

    • Yolo5s

    • Resnet50

• Management API for model/pipeline provisioning and monitoring server health.

  • Pipeline loading/unloading on the server

  • Configuring server parameters

  • Server monitoring via server/pipeline health queries

  • REST., Python programming language

• Serving API (package neureality.client) for producing inference requests from client-side application to NR1 AI inference server.

  • Based on a lightweight client runtime library that can be linked with your client apps, to offload inference requests to the NR1-S appliance

  • Client connection management

  • Synchronous/asynchronous serving

  • Available in Python and C programming languages

• Orchestration and Monitoring of inference clusters including NR1-S AI inference appliances.

  • Telemetry tools for server monitoring

  • Server monitoring via metrics with Prometheus

  • Server logging with FluentBit

  • Kubernetes cluster as an orchestration solution.

  • Kubernetes plugins for solution orchestration over cluster with NR1-S AI Inference Appliances

  • Plugins for specialized NeuReality resources

    • CG Pod

    • CG Operator

  • Liveness checks - NR1 node, CG pod

  • Network offload to AIoF for inference requests passing CNI plugin

  • Reference implementation for Calico plugin (IPinIP)

  • Reference cluster implementation

  • Reference configuration of metrics collection with Prometheus

  • Horizontal Pod Autoscaling configuration based on the collected metrics

  • Reference configuration of Fluentbit for logs collection

• BMC Interfaces support NR1-S AI Inference Appliance management in the data center.

  • 2x 1GbE ports on rear panel

  • IPMI interface

  • Power/temperature management/events

  • System reset

  • Networking configuration

  • FRU

  • Firmware upgrade via BMC GUI

• Documentation - samples, tutorials and API reference for all APIs.

• Performance Demo Application is supplied to provide reference implementation and empirical benchmark. Selected pipelines are supported. For additional information about the demo application, refer to the NR1-S Performance Demo User Guide.

Delivery Packages and Upgrades

NeuReality Software SDK is delivered as follows:

Software Package	Delivery
NR1 AI-CPU Platform	NeuReality Software Artifactory
NR1-M AI Inference Module Platform	NeuReality Software Artifactory
NR1-S AI Inference Appliance BMC	NeuReality Software Artifactory
Toolchain library (Deployment API)	NeuReality Docker Registry
Client library (Serving API)	NeuReality GitHub
Kubernetes plugins	NeuReality GitHub

Documentation

Please visit NeuReality Developer Portal for more information about NeuReality Software SDK, tutorials, API reference, videos, and events.

Known Limitations

#	Description	Workaround	Software Package
AV-4373	Auto-negotiation is frequently used in Ethernet networks to find the most efficient mode of operation for two linked devices. NR1-M module auto negotiation feature is not functional in the present release.	The settings controlled by auto-negotiation (such as port speed and flow control configurations) should be manually configured to the desired values using the remote management interface (IPMI).	Platform
ASAI-2303	NR1-M module network port does not support IPv6	NA	Platform
ASAI-2304	Priority flow control (PFC) is not supported on NR1M card network port.	Keep PFC feature disabled.	Platform
ASAI-6845	In some cases, client traffic may be interrupted returning an error message stating “unknown hdr_type”.	NA	Platform
AV-4293	When LFC feature is enabled in NR1-M module, RX occasional packet drops may occur in the RX path.	NA	Platform
AV-4183	Dual port is not supported in NR1M card. Single port mode is supported.	NA	Platform
AV-4181	Once the NR1-M module network interface is up, it cannot be brought down without an NR1-M module reboot.	Changing the network MTU which requires bringing down the network interface requires rebooting NR1-M module to finalize the MTU change.	Platform
ASAI-5574	In rare instances, the Qualcomm Cloud AI100 accelerator may encounter an error state that cannot be resolved with a standard reboot.	Additional reboot resolves the issue.	Platform
ASAI-6667	CNI offloading does not support CG pod on Host network. Sending connection requests via AIoF protocol to CG pods with settings hostNetwork field set to True will produce failure.	Do not set hostNetwork field of CG pods to true.	APIs
ASAI-5172	Cannot deploy more than one compute graph on a single NR1-M module.	NA	APIs
ASAI-6677	In very rare instances, supplying input tensors to Qualcomm Cloud AI100 accelerator may cause misalignment errors “Requested DMA Write/Read operation will result in out of bounds write/read”.	NA	APIs
ASAI-5955	Resize operator does not support images with dimensions smaller than 64x64.	NA	APIs
ASAI-7177	In PoC systems, a single NR1-M module can support up to 255 concurrent client connections.	NA	APIs

How to Get Help

Need a Hand? We’ve Got You Covered. Create a ticket with us, and let’s tackle those challenges together. Use one of the following channels to raise the issue:

• CRM tool

• Slack

• Email