NeuReality Software Development Kit V1.1 Release Notes

Release Date

November 14, 2024

Target Audience

This is a NeuReality SDK v1.1 release for selected early adopters. The use of the SDK is subject to the Early Access Program Agreement (or similar written instrument) executed by and between your organization and NeuReality (“License Agreement”) and is provided solely for the purpose of Proof of Concept (PoC) programs.

What’s new in this release?

Deployment API (package neureality) for building the AI pipeline, validating them on x86 functional simulator equipped with Qualcomm Cloud AI100 accelerators1, and profiling them on NR1-S AI inference appliance.
- New Technologies: LLM
- Models compiled with Qualcomm Cloud AI100 SDK and validated in this release:
1. LLaMa 3 7B
Management API for model/pipeline provisioning and monitoring server health.
- Logging and metrics configuration
- Configuration end point:

Field	Description
Log aggregation mode	Enables Fluentbit daemon to send logs to an external logging aggregator such as Fluent, Logstash, Logz.io or Datadog. 3 modes are supported: · Disabled · Local · Remote, will be set by sending FQDN remote.
Log aggregation remote FQDN	IP / hostname of log aggregator remote server
Log aggregation remote port
K8S Connect	Connect/disconnect to/from K8S cluster.
Monitoring collector	Configure node exporter to be enabled natively.

New Metrics - see a table below for a list of metrics:

Metric	Type	Description
Dynamic Memory CAP	Gauge	This metric shows the amount of used dynamic memory
Input Queue time	Counter	Queue time of inference request. Measured from AIoF request received to AIoF data read request sent.
Network Rx Time	Counter	Time network stack spends while fetching the request’s data. Counted from the first AIoF data read to the last AIoF data RX. Request Time = Processing Time + Queue Time + Network Rx time.
Total Processing Time	Counter	Cumulative time the job spends in all Compute Engines (CEs). For each CE, the time from receiving the job to the CE and until releasing the job from that CE is measured. The total time spent in all CEs is measured for the job. The time of AI hypervisor processing and reading the job information is excluded. Note: When working with DLA, this value would include the DLA’s queue time.
CE Processing Time	Counter	The time from receiving the job to the CE and until releasing the job from that CE is measured. Reported per CE inside the CG Does not include NIF (which is a CE).
Input Queue Time	Summary	Queue time before network RX.
Request Time	Summary	Bucket definitions: Buckets: less than 5usec, 5-20usec, 20-100usec, 100usec-1msec, 1-5msec, 5-20msec, 20-100msec, 100-500msec, 500msec-1sec, 1-1.5sec, 1.5-2sec, 2-2.5sec, 2.5-5sec, 5-7.5sec, 7.5-10sec, more than 10sec.
Network Rx Time	Summary	Same as above.
Processing Time	Summary	Same as above.
Batch size	Summary	Bucket size for various batch buckets are reported. 32 buckets are uniformly divided according to maximum batch size. For example, for batch size = 256, then bucket-0 is for batch size 0-7 etc.
Number of AIOF RX Pkts	Counter	Total number of packets received by AIoF software stack.
Number of AIOF TX Pkts	Counter	Total number of packets sent by AIoF software stack.
Drop Count	Counter	Total number of dropped RX packets.

Serving API (package neureality.client) for producing inference requests from client-side application to NR1 AI inference server.
- LLM interface - see more details in NeuReality SDK User Guide.
- LLM sample - text summarization.
Kubernetes cluster as an orchestration solution.
- Auto-scaling support for CG operator
- Resource scheduling aware of specific NR1 resources:

Resource	Description
Docker Size [GB]	ARM docker partition size.
CPU (ARM) Memory Size [GB]	ARM runtime memory required for CG.
Compute Memory Size [GB]	Compute memory required for CG e.g., for NN-model and other static programs.
Queue Number	Number of QSS queues required for managing the hardware AI-Hypervisor.

BMC interfaces support NR1-S AI Inference Appliance management in the data center.
- Extended IPMI interface
- Power/temperature management/events/alerts
- User management
- BMC GUI
Documentation - GenAI samples and tutorials
Performance Demo application is supplied to provide reference implementation and empirical benchmark. Selected pipelines are supported. For additional information about the demo application, refer to the NR1-S Performance Demo User Guide.

Delivery Packages and Upgrades

NeuReality Software SDK is delivered as follows:

Software Package	Delivery
NR1 AI-CPU Platform	NeuReality Software Artifactory
NR1-M AI Inference Module Platform	NeuReality Software Artifactory
NR1-S AI Inference Appliance BMC	NeuReality Software Artifactory
Toolchain library (Deployment API)	NeuReality Docker Registry
Client library (Serving API)	NeuReality GitHub
Kubernetes plugins	NeuReality GitHub

Documentation

Please visit NeuReality Developer Portal for more information about NeuReality Software SDK, tutorials, API reference, videos, and events.

Known Limitations

#	Description	Workaround	Temporary Limitation
AV-4373	Auto-negotiation is frequently used in Ethernet networks to find the most efficient mode of operation for two linked devices. NR1-M module auto negotiation feature is not functional in the present release.	The settings controlled by auto-negotiation (such as port speed and flow control configurations) should be manually configured to the desired values using the remote management interface (IPMI).	Yes
ASAI-2303	NR1-M module network port does not support IPv6	NA	Yes
AV-4123	Dual port is not supported in NR1M card. Single port mode is supported.	NA	NA
AV-4183	NR1-M module supports an Ethernet port (port0) with speeds of 25Gbps and 50Gbps.	NA	NA
AV-4181	Once the NR1-M module network interface is up, it cannot be brought down without an NR1-M module reboot.	Changing the network MTU which requires bringing down the network interface requires rebooting NR1-M module to finalize the MTU change.	NA
ASAI-5172	Cannot deploy more than one compute graph on a single NR1-M module.	NA	Yes
ASAI-6677	In very rare instances, supplying input tensors in FP32 format into dsp vector unit of Qualcomm Cloud AI100 accelerator may cause misalignment errors, as follows: • Requested DMA Write operation will result in out of bounds write  • Requested DMA Read operation will result in out of bounds Read	Use tensors aligned to 128B.	Yes
AV-3829	Large receive offload (LRO) is enabled for connections where the total header length, including the TCP constant header, exceeds 63 bytes.	NA	Yes
ASAI-7553	Non-contiguous arrays are not supported in NeuReality programming model and Deployment API. Using non-contiguous arrays will not be flagged as an error by the compiler, leading to run time errors during simulation or device execution.	User code must avoid creating non-contiguous arrays. Input and output arrays submitted to the Deployment API should be contiguous in memory. For details on generating a contiguous array, please refer to the following: numpy.ascontiguousarray — NumPy v2.0 Manual	Yes
ASAI-5172	Cannot deploy more than one compute graph on a single NR1-M module.	NA	Yes
ASAI-7896	After NR1-S power cycle some of the AI-100 cards can not detected by the NR1-M module	Run the following sequence after the NR1-S power cycle ends: fpgatool power 17 off fpgatool power 18 off fpgatool power 7 off fpgatool power 8 off fpgatool switch 4 off fpgatool power 17 on fpgatool power 18 on fpgatool switch 4 on fpgatool power 7 on fpgatool power 8 on This sequence brings up the switch and some of the non-detected cards.	Yes
ASAI-7637	Network interface instability. In rare cases, NR1-M network interface may become nonfunctional, exposing an NR_STATUS_ENET error to the client.	Error detection necessitates rebooting the NR1-M card.	Yes
ASAI-7492	The turbo_mode_off compilation option is not supported.	NA	Yes

How to Get Help

Need a Hand? We’ve Got You Covered. Create a ticket with us, and let’s tackle those challenges together. Use one of the following channels to raise the issue: