Serving API

AIOF

To simplify communication with the NR1™ AI Inference Appliance, the Serving API provides several functions for offloading inference requests, together with examples of how to use them.

It is based on a lightweight client runtime library that can be linked with your application. Client connection management and synchronous/asynchronous serving are supported. The API is available in Python and C programming languages.

The library has been tested with Ubuntu operating system, versions 22.04.4 and 24.04.

For more information and examples refer to the library project in GitHub.

For the API reference, visit the nr-client documentation for CPP and Python programming languages.

OpenAI-compatible API

Supported Endpoints

1. Completions (/completions)

Description:
Generates text completions based on the provided prompt.

Request Parameters

  • model (string): Model identifier.

  • prompt (string | list<string> | list<int> | list<list<int>>): Prompt text(s) or token IDs.

  • max_tokens (int, optional): Maximum tokens to generate.

  • temperature (float, optional, range: 0–2): Sampling randomness.

  • frequency_penalty (float, optional, range -2 - 2): Penalize new tokens based on their existing frequency in the text so far.

  • presence_penalty (float, optional, range -2 - 2): Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics.

  • logits_bias (map<token_id, score>, optional): Modify the likelihood of specified tokens appearing in the completion.

  • logprobs (int, optional): Include the log probabilities on the logprobs most likely output tokens, as well the chosen tokens.

  • top_p (float, optional, range: 0–1): Nucleus sampling parameter.

  • stop (string | list, optional): Tokens that terminate completion.

  • suffix (string, optional): Text appended after the generated completion.

  • stream (bool, optional): Whether to stream back partial progress.

  • stream_options (obj, optional): Options for streaming response. Only set this when you set stream: true.

    • include_usage (bool, optional): If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array.

  • Additional vLLM parameters: top_k, min_p, ignore_eos, min_tokens, skip_special_tokens, stop_token_ids. These parameters are an extension to OpenAI API parameters. For information about these parameters: https://docs.vllm.ai/en/v0.6.5/dev/sampling_params.html

Not Supported

  • echo, best_of, n > 1, user

Response

  • id: Unique completion ID.

  • model: Model used.

  • choices: Completion results containing text, index, finish_reason, and optionally logprobs.

  • usage: Token information (prompt_tokens, completion_tokens, total_tokens).

2. Chat Completions (/chat/completions)

Description:
Creates a model response for the given chat conversation.

Request Parameters

  • model (string): Model identifier.

  • messages (list): Conversation history; each message has a role (system/user/assistant) and content.

  • max_tokens (int, optional): Maximum tokens to generate.

  • temperature (float, optional, range: 0–2): Sampling randomness.

  • frequency_penalty (float, optional, range -2 - 2): Penalize new tokens based on their existing frequency in the text so far

  • presence_penalty (float, optional, range -2 - 2): Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics.

  • logits_bias (map<token_id, score>, optional): Modify the likelihood of specified tokens appearing in the completion.

  • logprobs (bool, optional): Whether to return log probabilities of the output tokens or not.

  • top_logprobs (int, optional): An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.

  • top_p (float, optional, range: 0–1): Nucleus sampling parameter.

  • stop (string | list, optional): Tokens that terminate completion.

  • stream (bool, optional): Whether to stream back partial progress.

  • stream_options (obj, optional): Options for streaming response. Only set this when you set stream: true.

    • include_usage (bool, optional): If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array.

  • Additional vLLM parameters: top_k, min_p, ignore_eos, min_tokens, skip_special_tokens, stop_token_ids. This parameters are an extension to OpenAI Api parameters. For information of these parameters: https://docs.vllm.ai/en/v0.6.5/dev/sampling_params.html

Not Supported

  • Tool calling or function calling

  • Structured outputs (response_format)

  • LoRA

  • Multi-modal input

  • Other parameters:

    • store, service_tier, seed, prediction, metadata, max_completion_tokens - please use max_tokens.

    • n > 1

    • reasoning_effort, user and web_search_options.

Response

  • id: Unique chat completion ID.

  • model: Model used.

  • choices: Chat completion results containing message, finish_reason, and optionally logprobs.

  • usage: Token usage information.

3. Embeddings (/embeddings)

Description:
Creates an embedding vector(s) representing the input text.

Request Parameters

  • input (string | list<string> | list<int> | list<list<int>>): Text(s) or token IDs.

  • model (string, optional): Embedding model identifier.

  • encoding_format (string, optional): float or base64.

Not Supported

  • dimensions and user

Response

  • data: Embedding results with fields index, object, and embedding.

  • usage: Token information.

4. Moderations (/moderations)

Description:
Evaluates text input for potentially harmful content.

Supported Models:

  • Llama Guard 3 Model Family.

Request Parameters

  • input (string | list<string>): Text(s) to moderate.

Not Supported

  • Multi-modal input

Response

  • results: Moderation results containing flagged status and categories indicating specific moderation issues.

Common Error Structure

In case of any error, a message of this response will be sent:

  • object: error

  • message: Description of the error.

  • type: Error category.

Notes

  • Parameter validations are enforced (e.g., temperature range: 0–2, top_p range: 0–1).

  • For more information: https://platform.openai.com/docs/api-reference

Ensure adherence to documented constraints when integrating with this API.

API Reference

Quick Start with the NeuReality AI Playground API

Let’s learn how to use the AI Playground API to build applications with NeuReality. You will need:

  1. A NeuReality AI Playground account

  2. An AI Playground API key

To create an account and obtain the API key:

  1. Log into the NeuReality AI Playground at https://playground.neureality.ai/.

  2. From the menu on the left side of the screen, select Platform > API Integration:

    API Integration Menu

  3. Go to the API key panel on the top-right corner of the screen and click the Copy button to copy the key to the clipboard.

  4. Configure your application to work with the Endpoint URL and API key specified on this screen, instead of the standard OpenAI API URL and key.

    API Key Panel

  5. You are ready for inference. Use the OpenAI compatible Completions API to produce inference requests.