Inference¶

HTTP client for deployed Arena inference endpoints. Deployment metadata is fetched on construction (unless probe_on_init=False) and exposed as metadata (StatusResponse). Use get_action() (RL), predict() (supervised), or generate() / generate_stream() (LLM). Errors raise ArenaInferenceError.

RL and supervised requests use base64 .npy serialization via agilerl.arena.inference.serde. LLM requests use plain JSON with LLMParams.

Every request carries your own Arena credential, a profile personal access token (arena_pat_...) or an access token from login(). A deployment that keeps memory per user works out who is calling from it, so there is nothing extra to pass. open_inference_agent() fills the credential in from the client. Building an Agent directly reads ARENA_API_KEY when you pass no api_key. The credential is never logged, shown in repr(), or written to disk.

Pass session_id to generate() or generate_stream() to continue an earlier chat, and use list_sessions() and get_session() to find one. Sessions come back most recently updated first. Leaving session_id out sends no session field, so the deployment starts a new conversation; no id is minted here. The CLI does mint one, which is how it continues a conversation across commands.

delete_session() removes a conversation and its messages from the deployment. It reports whether there was anything to delete rather than raising, since a session that is already gone and one that belongs to someone else answer the same way.

Which sessions you can see, and which you can delete, depends on the deployment’s memory scope, set with memory_scope on deploy_agent().

CLI: arena agent run <deployment> then arena agent generate --prompt '...' (uses generate_stream; pass a deployment name to generate to override). Add --session-id to continue a chat. arena agent sessions list and arena agent sessions get <session-id> show what is stored. Prompts continue one conversation by default, so the id need not be repeated; arena agent sessions resume switches to an older one, arena agent sessions clear starts a new one on the next prompt, and arena agent sessions delete <session-id> removes one from the deployment.

class agilerl.arena.inference.LLMParams(*, max_new_tokens: int = 100, temperature: float = 0.7, top_p: float = 0.9, do_sample: bool = True)¶

LLM sampling parameters model.

Parameters:

max_new_tokens (int) – The maximum number of new tokens to generate.
temperature (float) – The temperature of the LLM.
top_p (float) – The top-p value of the LLM.
do_sample (bool) – Whether to sample the LLM.

model_config: ClassVar[ConfigDict] = {'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.StatusResponse(*, success: bool = True, deployment_id: str = '', instance_id: str = '', agent: AgentInfo | None = None)¶

Status response model.

GET /status is public, so it reports little beyond the deployment id. Fields a deployment leaves out stay at their defaults, and agent is None when it does not describe its shape at all.

Parameters:

success (bool) – Whether the status request was successful.
deployment_id (str) – The deployment ID.
instance_id (str) – The instance ID.
agent (AgentInfo | None) – The agent info, or None if the deployment did not report it.

model_config: ClassVar[ConfigDict] = {'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.PredictResult(*, batch_size: int, inference_time_ms: float, success: bool = True)¶

Metadata from POST /predict (excluding deserialized tensors).

Parameters:

batch_size (int) – The batch size of the prediction.
inference_time_ms (float) – The inference time in milliseconds.
success (bool) – Whether the prediction was successful.

model_config: ClassVar[ConfigDict] = {'extra': 'ignore', 'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.LLMCompletionResult(*, prompt: str = '', completion: str = '')¶

Single prompt/completion pair from POST /generate.

Parameters:

prompt (str) – The prompt for the LLM completion.
completion (str) – The completion for the LLM completion.

model_config: ClassVar[ConfigDict] = {'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.LLMResults(*, results: list[LLMCompletionResult], batch_size: int, inference_time_ms: float, tokens_per_second: float, success: bool = True)¶

LLM completion response model.

Parameters:

results (list[LLMCompletionResult]) – The results of the LLM completions.
batch_size (int) – The batch size of the LLM completions.
inference_time_ms (float) – The inference time in milliseconds.
tokens_per_second (float) – The tokens per second of the LLM completions.
success (bool) – Whether the LLM completions were successful.

model_config: ClassVar[ConfigDict] = {'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.SessionInfo(*, session_id: str = '', created_at: str | None = None, last_updated: str | None = None)¶

One chat session summary from GET /sessions.

Parameters:

session_id (str) – The ID of the session.
created_at (str | None) – When the session was opened.
last_updated (str | None) – When the session was last written to.

model_config: ClassVar[ConfigDict] = {'extra': 'ignore', 'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.SessionMessage(*, role: str = '', content: str = '')¶

One message in a chat session.

Parameters:

role (str) – The role of the message author.
content (str) – The message text.

model_config: ClassVar[ConfigDict] = {'extra': 'ignore', 'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.SessionDetail(*, session_id: str = '', created_at: str | None = None, last_updated: str | None = None, messages: list[~agilerl.arena.inference.agent.SessionMessage] = <factory>)¶

A SessionInfo plus its messages, from GET /sessions/{session_id}.

Parameters:: messages (list[SessionMessage]) – The messages in the session.

model_config: ClassVar[ConfigDict] = {'extra': 'ignore', 'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.arena.inference.Agent(endpoint: str, *, api_key: str | None = None, org_key: str | None = None, external_user_id: str | None = None, timeout: int = 30, generate_params: LLMParams | None = None, probe_on_init: bool = True)¶

HTTP client for a deployed Arena inference endpoint.

When probe_on_init is True (default), status() runs once at construction and stores metadata. Wrong endpoint for the deployment type returns HTTP 400 from the server.

Every request carries your own Arena credential as Authorization: Bearer. A deployment that keeps memory per user works out who is calling from it, so there is nothing else to pass. Organisation-key clients also send X-External-User-Id. The credential is never logged, shown in repr(), or written to disk.

Parameters:

endpoint (str) – Base URL of the Arena inference deployment.
api_key (str | None) – Profile PAT (arena_pat_<uuid>_<secret>) or an access token from login(). Falls back to the ARENA_API_KEY environment variable.
org_key (str | None) – Organisation API key (arena_org_…). Requires external_user_id. Falls back to ARENA_ORG_KEY.
external_user_id (str | None) – Partner user id sent as X-External-User-Id. Requires org_key. Falls back to ARENA_EXTERNAL_USER_ID.
timeout (int) – Request timeout in seconds.
generate_params (LLMParams | None) – Default LLM sampling parameters; defaults to LLMParams when None.
probe_on_init (bool) – If True, fetch GET /status during __init__.

close() → None¶: Close the underlying HTTP connection pool.

delete_session(session_id: str) → bool¶

Delete a chat session and every turn the deployment stored for it.

Deleting is idempotent: a session that is already gone, or that belongs to someone else under the deployment’s memory scope, answers the same way, so this reports whether there was anything to delete rather than raising.

Parameters:: session_id (str) – The ID of the session to delete.
Returns:: True if the session was deleted, False if the deployment has no such session visible to you.
Return type:: bool

generate(prompts: str | list[str], *, params: LLMParams | dict[str, Any] | None = None, session_id: str | None = None) → LLMResults¶

Generate LLM completions.

Parameters:

prompts (str | list[str]) – The prompts to the agent.
params (LLMParams | dict[str, Any] | None) – The parameters to the agent.
session_id (str | None) – Chat session to continue, as listed by list_sessions(). When omitted the field is left out of the request and the deployment starts a new conversation; no session id is minted here.

Returns:

The LLM results.

Return type:

LLMResults

generate_stream(prompt: str, *, params: LLMParams | dict[str, Any] | None = None, session_id: str | None = None) → Iterator[str]¶

Stream generated tokens for a single prompt.

Parameters:

prompt (str) – The prompt to the agent.
params (LLMParams | dict[str, Any] | None) – The parameters to the agent.
session_id (str | None) – Chat session to continue, as listed by list_sessions(). When omitted the field is left out of the request and the deployment starts a new conversation; no session id is minted here.

Returns:

An iterator of strings.

Return type:

Iterator[str]

Get actions from a deployed RL agent.

For multi-agent deployments, pass observation as dict[agent_id, obs]. Per-agent action_mask in info must be a dict keyed by agent id.

Parameters:

observation (RLData) – The observation to the agent.
batched (bool) – Whether to batch the observation.
hidden_state (RLData | None) – The hidden state of the agent.
info (dict[str, RLData] | None) – The info of the agent.
env_defined_actions (RLData | None) – The env defined actions of the agent.

Returns:

A tuple containing the action and the hidden state.

Return type:

tuple[RLData, RLData | None]

get_session(session_id: str) → SessionDetail¶

Fetch one chat session and its messages.

Parameters:: session_id (str) – The ID of the session to fetch.
Returns:: The session and its messages.
Return type:: SessionDetail

list_sessions() → list[SessionInfo]¶

List the chat sessions the calling user has with this deployment.

Returns:: The sessions, most recently updated first as ordered by the deployment. Empty when the user has no conversations yet.
Return type:: list[SessionInfo]

Return a prediction from a supervised agent.

Parameters:

inputs (RLData) – The inputs to the agent.
batched (bool) – Whether to batch the inputs.

Returns:

A tuple containing the prediction and the metadata.

Return type:

tuple[RLData, PredictResult]

Raises:

ArenaInferenceError – If the agent is not supervised.

status(*, refresh: bool = False) → StatusResponse¶

Return deployment metadata.

Parameters:: refresh (bool) – Whether to refresh the metadata.
Returns:: The deployment metadata.
Return type:: StatusResponse