Arena Client

Arena is an RLOps platform that streamlines the reinforcement learning development process and accelerates iteration through distributed training and evolutionary hyperparameter optimization. It provides managed cloud infrastructure purpose-built for RL workloads, so researchers and engineers can focus on algorithms and environments rather than cluster management.

The Arena client, available as both a CLI (arena) and a Python SDK (ArenaClient), allows registered users to do everything from their own development environment: validate custom environments, submit training experiments, monitor progress, and deploy trained agents for inference.

Tip

Sign up to Arena for free now and get 110 free training credits (~20 hours) to get started!

Installation

To use the Arena client, install the standalone package directly with lightweight dependencies, or use the AgileRL extra:

pip install agilerl-arena
# or
pip install "agilerl[arena]"

Authentication

Authentication is resolved automatically in priority order:

1. api_key argument passed directly to the client or CLI.

2. ARENA_API_KEY environment variable: if set, no login is needed.

3. Stored OAuth credentials from ~/.arena/credentials.json (persisted after a successful arena login).

4. Interactive device login: opens a browser for OAuth authorization.

The simplest approach for scripting and CI is to set the environment variable:

export ARENA_API_KEY="arena_pat_..."

Note

Personal access tokens can be found in the Arena account profile, under Profile management -> CLI API Key.

Once set, all CLI commands and SDK calls authenticate automatically without requiring arena login.

For interactive use, arena login opens a browser-based OAuth flow and persists credentials locally so you only need to log in once per machine:

Python

from agilerl.arena.client import ArenaClient

# Option 1: Set ARENA_API_KEY env var, no login needed
client = ArenaClient()

# Option 2: Pass the key explicitly
client = ArenaClient(api_key="arena_pat_...")

# Option 3: Interactive device login (one-time)
client = ArenaClient()
client.login()  # opens browser, persists credentials

CLI

# One-time interactive login (persists to ~/.arena/credentials.json)
arena login

# Or skip login entirely with an env var or flag
export ARENA_API_KEY="arena_pat_..."

Custom Environments

Before launching a training run a custom environment or dataset, Arena requires these to be validated, ensuring that the environment is importable, has the correct interface (e.g. Gymnasium or PettingZoo), and can be stepped without errors. After validation, environments are automatically profiled to determine their compute and memory footprint. This information is then used to estimate a ceiling on the resources available for training, depending on the selected cluster tier.

Users can register two types of environments on Arena:

• Custom Gymnasium/PettingZoo: upload a source directory, file, or .tar.gz archive containing your environment code.

• LLM Datasets: upload a dataset file or reference a HuggingFace dataset ID to create a language-model fine-tuning environment.

Uploading and Validating

In order to create and validate a custom Gymnasium/PettingZoo environment from scratch, users need to provide a source file or directory containing the environment code. Multi-agent environments must be signaled through the multi_agent flag. Additionally, an entrypoint can be provided to the environment class to use, which is useful when multiple entrypoints exist in the same path. Please refer to the ArenaClient.validate_environment method documentation for more details.

When you point source at a directory, Arena packages the whole folder and uploads it. A requirements.txt in the folder is installed before your environment runs, and an env_config.yaml is used to set the arguments passed to your environment’s constructor. You can also pass these files explicitly with requirements and env_config (--requirements and --env-config on the CLI).

Once your environment / dataset has been validated successfully, you will be able to view it in the Environments / Datasets section in Arena.

RL Environments

The following commands use the merge-env/ directory, which contains a MergeEnv Gymnasium environment. See Training PPO on a Custom Environment in Arena for a full walkthrough.

Python

# Upload, create, and validate in one step
result = client.validate_environment(
    source="merge-env/",
    name="merge-env",
)

# Re-validate an already-registered environment
result = client.validate_environment(name="merge-env", version="v1")

CLI

# Upload and validate in one step
arena env validate merge-env \
    --source merge-env/

# Re-validate an already-registered environment
arena env validate merge-env --version v1

LLM Datasets

Python

result = client.create_dataset(
    name="my-dataset",
    category="reasoning",
    column_mapping={"question": "prompt", "answer": "completion"},
    file="./my_dataset/data.csv",
)

CLI

# Upload a local CSV and create the dataset
arena datasets create my-dataset \
    --category reasoning \
    --column-mapping '{"question": "prompt", "answer": "completion"}' \
    --file ./my_dataset/data.csv

Additional Tools

Here is a list of additional methods provided by the ArenaClient to help you navigate the custom environments workflow in Arena. You can find the analogous CLI commands by running arena env --help.

• list_environments(): List all registered environments and their versions.

• list_environment_entrypoints(): List available entrypoints for a specific environment version.

• environment_exists(): Check whether an environment (and optionally a version) is registered.

• profile_environment(): Profile a validated environment to determine its resource requirements.

• delete_environment(): Delete one or all versions of a registered environment.

Project Management

Projects are the top-level organisational unit in Arena. Every experiment belongs to a project, and you must specify a project when submitting training jobs.

Some useful command to manage your projects:

Python

# List all projects
projects = client.list_projects()

# Create a new project
client.create_project(name="CartPole-HPO", description="HPO on CartPole")

# Delete a project
client.delete_project("CartPole-HPO")

# Show the current default
client.get_default_project()

CLI

# List all projects
arena projects list

# Create a new project
arena projects create CartPole-HPO --description "HPO on CartPole"

# Delete a project
arena projects delete CartPole-HPO --yes

# Show the current default
arena projects get-default

Tip

You can set a default project to work on by doing the following. This will be stored in the ~/.arena/config.json file.

arena projects set-default <project-name>

Submitting Experiments

Once your environment or dataset has been validated, you can submit training jobs for it to Arena. AgileRL has its own managed cloud infrastructure with automatic checkpointing, metric logging, and real-time monitoring accessible directly from the Arena dashboard. Training runs in a distributed manner across multiple nodes (this can be customized by the user) to achieve lightning-fast training for arbitrarily large populations of agents.

Available Clusters on Arena

We currently provide a wide variety of compute resources on Arena, including GPU and CPU clusters. These might change over time, so it is recommended to use the ArenaClient.list_resources method to get the latest list of available resources before setting off an experiment.

Python

resources = client.list_resources()

CLI

arena resources list

Example output from CLI command:

Resource ID	GPUs	CPUs	RAM (GB)	Credits/node-hour
arena-medium	1x nvidia-l4	15	55.0	2.41
arena-large	1x nvidia-l4	30	110.0	3.65
arena-large-4gpu	4x nvidia-l4	47	165.0	8.41
arena-large-compute-96	N/A	95	180.0	8.57
arena-a100-40gb-2gpu	2x nvidia-tesla-a100	23	156.0	15.74
arena-extra-large	8x nvidia-l4	94	370.0	16.82
arena-large-compute-192	N/A	191	370.0	17.15
arena-a100-40gb-4gpu	4x nvidia-tesla-a100	47	312.0	31.47

Job Submission

A training configuration is defined via a manifest (YAML or JSON file) describing the algorithm, environment, training parameters, and evolutionary HPO settings. The formulation of the manifest is described in the Manifest Formulation section, and is mostly analogous for both the LocalTrainer and training jobs in Arena.

Here is an example manifest for training DQN on LunarLander-v3:

dqn.yaml

algorithm:
  name: DQN

environment:
  name: LunarLander-v3
  num_envs: 16

training:
  max_steps: 1_000_000
  target_score: 200.0
  pop_size: 4
  evo_steps: 10_000

network:
  latent_dim: 128
  encoder_config:
    hidden_size: [128]
    activation: ReLU
  head_config:
    hidden_size: [128]
    activation: ReLU

replay_buffer:
  max_size: 100_000

mutation:
  probabilities:
    no_mut: 0.4
    arch_mut: 0.2
    new_layer: 0.2
    params_mut: 0.2
    act_mut: 0.2
    rl_hp_mut: 0.2
  rl_hp_selection:
    lr:
      min: 0.0000625
      max: 0.01
    batch_size:
      min: 8
      max: 512
  mutation_sd: 0.1
  rand_seed: 42

tournament_selection:
  tournament_size: 2
  elitism: true

Python

result = client.submit_experiment(
    manifest="dqn.yaml",
    resource_id="arena-medium",
    num_nodes=2,
    project="my-project",
    experiment_name="lunar-lander-dqn",
)

CLI

# Submit an experiment
arena experiments submit dqn.yaml \
    --resource-id arena-medium \
    --num-nodes 2 \
    --project my-project \
    --experiment-name lunar-lander-dqn

You can also submit experiments via the ArenaTrainer class, which provides a higher-level interface:

from agilerl.training.trainer import ArenaTrainer

# Don't need to provide a client if we have already authenticated through `arena login`
trainer = ArenaTrainer.from_manifest(
    manifest="dqn.yaml",
)
result = trainer.train()

See also

• Manifest Formulation section for an overview of the training manifest and its options.

• Trainers section for more information on the ArenaTrainer class and its usage.

Additional Tools

Here is a list of additional methods provided by the ArenaClient for managing experiments. You can find the analogous CLI commands by running arena experiments --help.

• list_experiments(): List all experiments in a project.

• list_checkpoints(): List saved checkpoints for an experiment.

• resume_experiment(): Resume a stopped experiment with a new step budget.

• stop_experiment(): Stop a running experiment.

• download_experiment_metrics(): Download training metrics as CSV to a local file.

• deploy_agent(): Deploy a trained checkpoint to an inference endpoint.

Agent Deployment

Once training is complete, you can deploy a checkpoint from an experiment to an Arena inference endpoint and interact with it in real time. Deployed agents expose an HTTP API that accepts observations and returns actions, making the integration of trained RL policies into applications seamless.

To deploy an agent, you can use the deploy_agent() method by specifying the experiment name and optionally the checkpoint you wish to deploy. If the latter is not provided, the checkpoint with the largest fitness value will be deployed by default.

Python

# Deploy an agent from an experiment
client.deploy_agent(
   experiment_name="<experiment-name>",
   checkpoint="<checkpoint-name>",  # optional, defaults to best fitness
)

CLI

# Deploy the best checkpoint from an experiment
arena agent deploy lunar-lander-dqn

# Deploy a specific checkpoint
arena agent deploy lunar-lander-dqn --checkpoint step_500000

Interacting with a Deployed Agent

Each deployment runs one agent type. The simplest way to interact with one is to open it by name with open_inference_agent(), which returns an Agent you can make requests to:

from agilerl.arena import ArenaClient

with ArenaClient() as client:
    with client.open_inference_agent("<deployment-name>") as agent:
        print(agent.metadata.agent.algo, agent.metadata.agent.llm)

Call the matching agent method; the server returns HTTP 400 if the route does not match the deployment:

• RL (single- or multi-agent, recurrent): get_action()

• Supervised (SFT): predict()

• LLM: generate() or generate_stream()

Tip

For LLM deployments, set an active agent with arena agent run <deployment-name> and you can drop the deployment name from later CLI commands. arena agent generate --prompt "..." then makes inference requests to the active agent.

RL Inference

get_action() serializes NumPy observations (base64 .npy wire format), supports batched inference, Dict / Tuple observation spaces, and recurrent hidden states.

import numpy as np
import gymnasium as gym

from agilerl.arena import ArenaClient

env = gym.make("LunarLander-v3")

with ArenaClient() as client:
    with client.open_inference_agent("lunar-lander-dqn") as agent:
      # Single request
      obs, _ = env.reset()
      action, hidden_state = agent.get_action(obs)

      # Batched inference
      batch_size = 8
      obs_batch = np.stack([env.observation_space.sample() for _ in range(batch_size)])
      actions, _ = agent.get_action(obs_batch, batched=True)

Multi-agent RL passes per-agent observations and returns a dict of actions:

obs = {"agent_0": obs_a, "agent_1": obs_b}
actions, _ = agent.get_action(obs)
# actions["agent_0"], actions["agent_1"]

Tutorial

Training PPO on a Custom Environment in Arena

Complete walkthrough of validating, training, and deploying a custom environment on Arena.

Training GRPO on GSM8K in Arena

Complete walkthrough of fine-tuning an LLM with GRPO on the GSM8K dataset in Arena.