AI Server

NR1™ AI Inference solution is tailored for teams aiming to deploy machine learning (ML) workloads into production reliably, scalably, and cost-effectively. It enables AI application developers to integrate cutting-edge pre-trained models and pipelines into their applications effortlessly. By facilitating the transition from workload creation to production deployment, NeuReality AI inference solution fosters collaboration between developers and in-house data science teams.

NeuReality provides a cloud and data center inferencing solution that is optimized for NR1 AI Inference Appliances. It supports a specialized high performing AI over the fabric (AIoF) protocol that allows remote clients to request inferencing for any model/pipeline that is being managed by the appliance. In addtion, for LLM, NeuReality supports a reduced set of OpenAI API. This document describes how to operate the inference solution on NR1 inference appliance.

Setup

Please refer to the NR1 AI Inference Appliance User Guide document for more information.

By default, after plugging the NR1 AI Inference Appliance to at least one power supply, the BMC will boot and will turn all NR1™ modules and deep learning accelerators connected to them.

It is necessary to configure BMC and NR1 module network interfaces to operate the NR1 AI Inference Appliance over the network.

BMC Configuration

The NR1 management network is pre-configured to use DHCP.

The NR1 BMC IP settings can be configured via the Web GUI that is accessible using the address https:://<BMC_IP>. The default user/password is root/0penBmc (notice 0 is zero).

In case the DHCP server is not available, the initial IP needs to be configured manually via console.

Please refer to the NR1 AI Inference Appliance BMC User Guide document for more information.

Connecting to BMC console

Prerequisites

1. USB to RJ45 adapter (such as the following one).

1. Terminal application (Teraterm or equivalent).

BMC Serial Connection

1. Plug the USB adapter cable into your computer and RJ45 of the cable to NR1 BMC console port (rightmost RJ-45 socket) as appears in the next drawing:

   

2. Launch Teraterm and configure the port to 1152008N1.

   

3. Login to the console using default credentials (user: root, password: 0penBmc).

4. Type: “ifconfig eth0” to show assigned IP address of management interface.

   

   Note: If the DHCP server is unavailable and IP needs to be set statically, temporarily set the IP manually so you can connect to the WEB GUI to set it permanently.

   ifconfig eth0 <IP_ADDRESS> netmask

   IE. Ifconfig eth0 192.168.1.2 netmask 255.255.255.0

BMC Network Configuration

To configure the BMC network:

1. Login to Web GUI

2. Select Settings > Network

   

3. Edit the network configuration for the eth0 port.

   

NR1 Module Configuration

Network Configuration

NR1 module configuration is supported via IPMI commands to the NR1 AI Inference Appliance. Before NR1 modules can be used, the network interfaces must be configured. By default, both the management port and port 1 are configured to use DHCP. Port1 is preconfigured as:

• Static link speed of 50Gb (auto negotiation disabled)

• FEC disabled

• Flow control disabled

Port2 is currently not in use.

Note: The connection between NR1 to the network switch must be done using standard copper QSFP cabling. The network switch must be configured to the same parameters as configured in NR1.

NR1 Network IPMI Commands

General IPMI command to communicate with NR1 AI Inference Appliance:

ipmitool -I lanplus -C 17 -H <host name> -U root -P 0penBmc <ipmi command and args>

Hostname

Set hostname for slot

Syntax

Set:

ipmitool raw 0x30 0x21 <slot> <string length> <hostname>

Get:

ipmitool raw 0x30 0xA1 <slot>

Returns the hostname currently set for the given slot.

Arguments

slot the slot number

string length length of hosname, including null terminator

hostname hostname string in hex ascii byte values, including null terminator

Example

Set the hostname of slot 3 to “dude”:

ipmitool raw 0x30 0x21 0x3 0x5 0x64 0x75 0x64 0x65 0x0

Note that the length with NULL terminator is 5 = 0x5.

NTP Server

Set NTP server for slot

Syntax

Set:

ipmitool raw 0x30 0x22 <slot> <string length> <NTP server name>

Get:

ipmitool raw 0x30 0xA2 <slot>

Returns the NTP server currently set for the given slot.

Arguments

slot the slot number

string length length of NTP server name, including null terminator

NTP server name NTP server name string in hex ascii byte values, including null terminator

Example

Set NTP server name of slot 4 to “time.google.com”:

ipmitool raw 0x30 0x22 0x4 0x10 0x74 0x69 0x6D 0x65 0x2E 0x67 0x6F 0x6F 0x67 0x6C 0x65 0x2E 0x63 0x6F 0x6D 0x00

Note that the length with NULL terminator is 16 = 0x10.

DNS Server

Set DNS server for slot

Syntax

Set:

ipmitool raw 0x30 0x23 <slot> <string length> <DNS server name>

Get:

ipmitool raw 0x30 0xA3 <slot>

Returns the DNS server currently set for the given slot.

Arguments

slot the slot number

string length length of DNS server name, including null terminator

DNS server name DNS server name string in hex ascii byte values, including null terminator

Example

Set DNS server name of slot 5 to “8.8.8.8”:

ipmitool raw 0x30 0x23 0x5 0x08 0x38 0x2E 0x38 0x2E 0x38 0x2E 0x38 0x00

Note that the length with NULL terminator is 8 = 0x8.

Management or Ethernet Port DHCP

Enables DHCP for interface.

Configuring the IP of the port will disable DHCP - see relevant command.

The ‘get’ command below will return 0 (disabled) if the port IP has been set.

Syntax

Set (enable dhcp):

ipmitool raw 0x30 <0x31|0x41> <slot>

Get:

ipmitool raw 0x30 <0xB1|0xC1> <slot>

Returns the current state of DHCP for interface on the given slot (0=disabled, 1-enabled).

Arguments

cmd one of the following:

• 0x31 - for management port set

• 0x41 - for ethernet port set

• 0xB1 - for management port get

• 0xC1 - for ethernet port get

slot the slot number

Example

Set DHCP for interface of slot 6 to ‘enabled’:

ipmitool raw 0x30 0x31 0x6

Management or Ethernet Port IP, subnet, gateway, DNS server

Customer can provide manual IP configuration for Management or Ethernet interface.

This will also disable DHCP mode — refer to the relevant command.

Syntax

Set:

ipmitool raw 0x30 <0x32|0x42> <slot> <string length> <configuration string>

Get:

ipmitool raw 0x30 <0xB2|0xC2> <slot>

Returns the relevant interface’s configuration currently set for the given slot.

Arguments

cmd one of the following:

• 0x32 - for management port set

• 0x42 - for ethernet port set

• 0xB2 - for management port get

• 0xC2 - for ethernet port get

slot the slot number

string length length of the interface configuration, including the null terminator

configuration string Interface configuration string in hex ascii byte values, including the null terminator

The string is received in the following format:

ip:subnet_mask:[gateway]:[dns]

which is manifested in one of the following input formats:

ip:subnet_mask

ip:subnet_mask:gateway

ip:subnet_mask::dns

ip:subnet_mask:gateway:dns

Example

Set management port configuration of slot 7 to “192.168.6.77:255.255.255.1:192.168.6.1:1.1.1.1”:

ipmitool raw 0x30 0x32 0x7 0x2F 0x31 0x39 0x32 0x2e 0x31 0x36 0x38 0x2e 0x36 0x2e 0x37 0x37 0x3a 0x32 0x35 0x35 0x2e 0x32 0x35 0x35 0x2e 0x32 0x35 0x35 0x2e 0x31 0x3a 0x31 0x39 0x32 0x2e 0x31 0x36 0x38 0x2e 0x36 0x2e 0x31 0x3a 0x31 0x2e 0x31 0x2e 0x31 0x2e 0x31 0x00

Note that the length with NULL terminator is 47 = 0x2F.

Ethernet Port Speed

Set ethernet port speed.

2 speeds are supported:

• 25000

• 50000

Syntax

Set:

ipmitool raw 0x30 0x43 <slot> <speed>

Get:

ipmitool raw 0x30 0xC3 <slot>

Returns the management port speed currently set for the given slot.

Arguments

slot the slot number

speed one of the following:

• 0xA8 0x61 - 25000

• 0x50 0xc3 - 50000

Example

Set ethernet port speed of slot 8 to 25000:

ipmitool raw 0x30 0x43 0x8 0xA8 0x61

Ethernet FEC mode

Enable or disable Forward Error Correction mechanism in PCS.

Syntax

Set:

ipmitool raw 0x30 0x44 <slot> <op>

Get:

ipmitool raw 0x30 0xC4 <slot>

Returns the FEC mode state currently set for the given slot.

Arguments

slot the slot number

op one of the following:

• 0 - disabled

• 1 - enabled

Example

Get FEC mode state currently set for slot 9:

ipmitool raw 0x30 0xC4 0x9

Ethernet Port Flow Control

Enable or disable flow control mechanism in PCS.

Syntax

Set:

ipmitool raw 0x30 0x45 <slot> <op>

Get:

ipmitool raw 0x30 0xC5 <slot>

Returns the port flow control state currently set for the given slot.

Arguments

slot the slot number

op one of the following:

• 0 - disabled

• 1 - enabled

Example

Enable port flow control for port 10:

ipmitool raw 0x30 0x45 0xA 0x1

Ethernet Port MTU

Set MTU.

Syntax

Set:

ipmitool raw 0x30 0x46 <slot> <mtu>

Notice: setting the MTU will not apply until reboot is done.

Reboot is not done as part of this command, it’s up to the user to perform it.

Get:

ipmitool raw 0x30 0xC6 <slot>

Returns the port MTU.

Notice: after setting MTU, it will be returned by get command, even though it will not apply until next reboot.

Arguments

slot the slot number

mtu onw of the following:

• 0xdc 0x05 - 1500

• 0x28 0x23 - 9000

Example

Set ethernet port speed of slot 2 to 9000:

ipmitool raw 0x30 0x46 0x02 0x28 0x23

NR1 Management Interface

All management operations are done via the management API or the BMC commands

Firmware Upgrade

NR1 eMMC has dual partitions keeping the primary and backup firmware.

During firmware upgrade, the secondary partition is updated and after completion the boot partition selection is switched. The firmware is placed in read-only partition and is replaced completely with the new firmware.

User data is stored on a separate read/write partition and is persistent across upgrades.

Upgrading the firmware of the NR1 is done by the NR1 module by downloading the firmware and automatically installing it.

Factory Reset

Factory reset is a procedure that deletes all user configuration and files from R/W partitions. It doesn’t change the current installed firmware.

NR1 Factory Reset Procedure

$ trigger-factory-reset.sh
$ sync && reboot

Metrics

Overview

Metrics are a specific kind of telemetry data essential for data center management. They are collected from all endpoints for health and performance monitoring and are used to implement decisions for resource scaling and runtime capacity management. Metrics represent a snapshot of the state of a device and are used by the collection agent to build state overtime and identify change to a device’s health.

An endpoint typically reports metrics for:

• Performance

• Capacity

• Utilization

• Errors

Types

All metrics must be 64-bit data type.

Type	Description
Gauge	A value sampled and reported when query arrives
Counter	A value collected over time and reported per query. Value can only increase wrapping around when it reaches its MAX 64-bit value.
Aggregation	A value collected over time and reported per query. Represents counter aggregation function - avg, max, min etc.
Summary a.k.a Histogram	A predefined set of counters reported in bins. Value can only increase wrapping around when it reaches its MAX 64-bit value.

Format

Metrics must comply with the OpenMetrics standard as implemented by Prometheus for exposing metrics by containers and collecting them by the metrics system. This standard is the same as Prometheus metrics standard.

Aggregation

Support metrics reporting and collecting using Prometheus.

The following system diagram illustrates the deployment.

Metric aggregation using Prometheus

Definition

NR1 Metrics

Metric	Type	Description	Name in Prometheus
ARM CPU UTIL	Gauge	ARM CPU utilization in percentage (0-100)	100 - (avg(irate(node_cpu_seconds_total{mode=”idle”}[30s])) * 100)
ARM MEM CAP	Gauge	ARM Memory usage in MB (from OS)	100 * (1 - ((avg_over_time(node_memory_MemFree_bytes{job=~”prod”}[30m]) + avg_over_time(node_memory_Cached_bytes{job=~”prod”}[30m]) + avg_over_time(node_memory_Buffers_bytes{job=~”prod”}[30m])) / avg_over_time(node_memory_MemTotal_bytes{job=~”prod”}[30m])))
ARM VOLUME	Gauge	Storage usage in MB	“100 - ((node_filesystem_avail_bytes{mountpoint=”/”,fstype!=”rootfs”} * 100) / node_filesystem_size_bytes{mountpoint=”/”,fstype!=”rootfs”})””
ARM STORAGE READ	Counter	Storage read (kb)	node_disk_read_bytes_total
ARM STORAGE WRITE	Counter	Storage write (kb)	node_disk_written_bytes_total
DEVICE TEMP [Celsius]	Gauge	Max temperature read from all NR1 on-die temperature sensors	node_hwmon_temp_celsius{chip=<chip_name>,sensor=”temp1”}, where chip_name is one of the following: “platform_2f800180_sencon_tr”, “platform_37800180_sencon_tl”, “platform_39800180_sencon_br”, “platform_3b800180_sencon_bl”, “platform_3f095c_sencon_bot”
ARM CLK [KHz]	Gauge	CPU Clock frequency.	node_cpu_frequency_max_hertz
STATIC MEM CAP	Gauge	How much static memory is being used (taken from Provisioner) in MB	static_mem_cap
PROVISIONED CGS	Gauge	Number of current active CGs	provisioned_cgs

CG Metrics - reported per provisioned CG

Metric	Type	Description	Name in Prometheus
Jobs In Progress	Gauge	Indication of the number of jobs pending in the device.This gauge increments with each new AIoF request received and decrements when an AIoF result is completely sent.	jobs_in_progress{cg=”X”}
Success Count	Counter	Number of successful inference requests received (each request is counted as 1, even if the request contains a batch)	success_count{cg=”X”}
Failure Count	Counter	Number of failed inference requests received (each request is counted as 1, even if the request contains a batch)	failure_count{cg=”X”}
Execution Count	Counter	Sum of Failure and Success counters	execution_count{cg=”X”}
Inference Count	Counter	Number of inferences performed	inference_count{cg=”X”}
Request Time	Counter	Cumulative end-to-end inference request handling time, from the moment the request entered NR1 to the moment the response is sent to the client. Counted by NIF in ARM CPU.	request_time{cg=”X”}

Enable/Disable

The feature is on by default.

To turn it on/off you can use:

ipmitool raw 0x30 0x13 <slot> <op>

where

slot the slot number

op one of the following

• 0 - disabled

• 1 - enabled

To see the current status, you can use:

ipmitool raw 0x30 0x93 <slot>

Getting metrics

A simple way to get the current metrics is to run:

curl <node ip address>:9100/metrics

This command will print out the raw metrics.

Configuring Prometheus

After installing Prometheus, you can edit the corresponding yml file and add your node as a target.

For example:

$ cat prometheus.yml
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
alerting:
alertmanagers:
    - static_configs:
        - targets:
scrape_configs:
- job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]
- job_name:       'node'
    scrape_interval: 1s
    static_configs:
    - targets: ['172.23.8.82:9100']
        labels:
        group: 'production'

In this case the node ip is 172.23.8.82.

Once the configuration is done - run Prometheus (./prometheus).

Then you can go over in the browser to <your Prometheus server IP>:9090.

And you will get an interactive website in which you can insert queries about your node and see nice graphs. For example:

NIF resnet50_demo test jobs_in_progress

Logging

General IPMI command to configure the NR1 log aggregator service.

Overview

To perform logs aggregation the NR1 uses fluent-bit: super-fast, lightweight, and highly scalable logging and metrics processor and forwarder (https://www.fluentbit.io). On our systems we are using a unified configuration file that will collect and filter events from various sources (system logs, docker daemon and containers).

Modes and Parameters

The user may set the Fluent Bit service to one of 3 modes:

Disabled: The service is disabled and will not collect any event.

ipmitool raw 0x30 0x11 <slot> 0x00

Local: The service will collect events and store them in local files.

ipmitool raw 0x30 0x11 <slot> 0x01

Remote: The service will collect events and forward them to a remote Fluent Bit server.

ipmitool raw 0x30 0x12 <slot> <strlen> <string>

For example, if you wish to set the hostname as ‘10.10.10.1’ use the following command: ipmitool raw 0x30 0x12 0x0a 0x31 0x30 0x2e 0x31 0x30 0x2e 0x31 0x30 0x2e 0x31 The string ‘0.10.10.1’ is converted to 10 hex bytes 0x31 0x30 0x2e 0x31 0x30 0x2e 0x31 0x30 0x2e 0x31.

Use the following commands to verify that your settings are correct:

Get the current mode of operation:

ipmitool raw 0x30 0x91 <slot>

The return value is one of the following:

0 - disabled.

1 - local.

2 - remote.

Get the current remote fluent-bit server:

ipmitool raw 0x30 0x92 <slot>

The return value format is as follows:

0c XX XX XX XX XX XX XX XX XX XX XX 00

The 1st byte is the length of the retrieved string (including the NULL termination), In the example above 12 bytes (0c), 11 bytes in HEX and NULL (00).