Training¶

Trainers¶

The Trainer classes provide a high-level, manifest-driven interface for running AgileRL evolutionary training. See the Trainers guide for usage examples and manifest reference.

Abstract base trainer for AgileRL evolutionary training.

Parameters:

algorithm (AlgoSpecT | str) – An algorithm spec or a string algorithm name.
environment (EnvSpecT | str) – A gymnasium.Env instance, a PettingZoo ParallelEnv instance, or an env-name string.
training (TrainingSpec) – Training loop parameters (max steps, population size, etc.).
mutation (MutationSpec | None) – Mutation probabilities and RL-HP ranges.
tournament (TournamentSelectionSpec | None) – Tournament selection configuration.
replay_buffer (ReplayBufferT | None) – Replay buffer configuration. Off-policy algorithms auto-create a default buffer when this is None.
resume_from_checkpoint (str | None) – Path to resume from checkpoint.
device (str | torch.device) – Torch device (e.g. "cpu", "cuda").
accelerator (Accelerator | None) – Accelerator instance.

classmethod from_manifest(manifest: str | Path | dict[str, Any] | TrainingManifest, **kwargs: Any) → Self¶

Instantiate a Trainer from a JSON-style manifest or a TrainingManifest instance.

The manifest supplies the algorithm, environment, and training configuration; any trainer-specific construction arguments are passed through as keyword arguments (e.g. device, accelerator, and resume_from_checkpoint for LocalTrainer, or client and api_key for ArenaTrainer).

Parameters:

manifest (str | Path | dict[str, Any] | TrainingManifest) – Path to a YAML/JSON file, or a raw dict, or a TrainingManifest instance.
kwargs – Trainer-specific construction arguments forwarded to the subclass constructor.

Returns:

A fully configured Trainer instance.

Return type:

SelfTrainerT

abstract train() → tuple[list[RLAlgorithm | MultiAgentRLAlgorithm | LLMAlgorithm], list[float]] | dict[str, Any]¶

Run the training loop.

LocalTrainer runs training locally and returns a tuple of (population, fitnesses) where population is the final evolved population and fitnesses contains each agent’s fitness from the final evaluation round.
ArenaTrainer submits a job to Arena and returns the API response as a dict.

Returns:: The training result, whose type depends on the trainer.
Return type:: tuple[PopulationT, list[float]] | dict[str, Any]

Local trainer that streamlines the AgileRL evolutionary training process.

Automatically builds the components necessary for RL training with evolutionary HPO from a series of Pydantic models that validate the specified training configuration, and dispatches to the algorithm-specific training loop through LocalTrainer.train(). Handles all of the RL training paradigms available in AgileRL.

Parameters:

algorithm (AlgorithmSpec | str) – An :class:`AlgorithmSpec instance or a string algorithm name.
environment (gym.Env | ParallelEnv) – An RL environment following Gymnasium or PettingZoo API.
training (TrainingSpec | None) – Training parameters. Defaults to TrainingSpec() (1M steps, single agent, no HPO).
mutation (MutationSpec | Mutations | None) – Mutation probabilities and RL hyperparameter ranges. When an RLAlgorithmSpec is used and hp_config is not set on it, hyperparameter ranges are derived from mutation.rl_hp_selection.
tournament (TournamentSelectionSpec | TournamentSelection | None) – Tournament selection configuration.
replay_buffer (ReplayBufferSpec | ReplayBuffer | None) – Replay buffer configuration.
hpo (bool) – Whether to enable evolutionary HPO using default mutation probabilities, tournament selection, and RL hyperparameters to mutate. Defaults to False.
resume_from_checkpoint (str | None) – Path to resume from checkpoint.
device (str) – Torch device string (e.g. "cpu", "cuda").
accelerator (Accelerator | None) – Accelerator instance.

train(verbose: bool = True, save_elite: bool = False, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, checkpoint_steps: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None) → tuple[list[RLAlgorithm | MultiAgentRLAlgorithm | LLMAlgorithm], list[float]]¶

Run a local training job given the passed configuration.

Parameters:

verbose (bool) – If True, print verbose output. Defaults to True.
save_elite (bool) – If True, save the elite agent. Defaults to False.
elite_path (str | None) – The path to save the elite agent. Defaults to None.
wb (bool) – If True, enable Weights & Biases logging. Defaults to False.
tensorboard (bool) – If True, enable TensorBoard logging. Defaults to False.
tensorboard_log_dir (str | None) – The path to save the TensorBoard logs. Defaults to None, which will use the default TensorBoard log directory tensorboard_logs.
checkpoint_steps (int | None) – The number of steps between checkpoints. Defaults to None.
checkpoint_path (str | None) – The path to save the checkpoints. Defaults to None.
overwrite_checkpoints (bool) – If True, overwrite the checkpoint. Defaults to False.
wandb_api_key (str | None) – The Weights & Biases API key. Defaults to None.
wandb_kwargs (dict[str, Any] | None) – The Weights & Biases keyword arguments. Defaults to None.

Returns:

A tuple of (population, fitnesses) where population is the final evolved population and fitnesses contains each agent’s fitness from the final evaluation round.

Return type:

tuple[PopulationT, list[float]]

Submits AgileRL training jobs to the Arena RLOps platform.

Parameters:

algorithm (AlgoSpecT | str) – An :class:`AlgorithmSpec instance or a string algorithm name.
environment (ArenaEnvSpec | str) – An :class:`ArenaEnvSpec instance or a string env name.
training (TrainingSpec) – Training loop parameters.
client (ArenaClient | None) – An authenticated ArenaClient. One is created automatically using the provided API key. Defaults to None.
api_key (str | None) – The Arena API key. Defaults to None.
mutation (MutationSpec | None) – Mutation probabilities and RL-HP ranges. Defaults to None.
tournament (TournamentSelectionSpec | None) – Tournament selection configuration. Defaults to None.
replay_buffer (ReplayBufferT | None) – Replay buffer configuration. Defaults to None.

Build the manifest and submit the training job to Arena.

Parameters:

resource_id (str | int | None) – Arena cluster type or resource id for the job.
num_nodes (int | None) – The number of nodes to use for training.
project (str | None) – The project to submit the experiment to.
experiment_name (str | None) – The name of the experiment to submit.
reward_file (str | Path | bytes | None) – Python reward module for reasoning dataset jobs.
completion (str | None) – Optional model completion for reward validation.

Returns:

Arena API response.

Return type:

dict[str, Any]

Training Functions¶

If you are using a Gym-style environment, our on- and off-policy training functions return a population of trained agents and logged training metrics.

agilerl.training.train_off_policy.train_off_policy(env: str | Env | VectorEnv | AsyncVectorEnv, env_name: str, algo: str, pop: list[DQN | RainbowDQN | DDPG | TD3], memory: ReplayBuffer | PrioritizedReplayBuffer | MultiStepReplayBuffer, init_hp: dict[str, Any] | None = None, mut_p: dict[str, Any] | None = None, max_steps: int = 1000000, evo_steps: int = 10000, eval_steps: int | None = None, eval_loop: int = 1, learning_delay: int = 0, eps_start: float = 1.0, eps_end: float = 0.01, eps_decay: float = 0.999, target: float | None = None, n_step_memory: MultiStepReplayBuffer | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, checkpoint: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, save_elite: bool = False, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, verbose: bool = True, accelerator: Accelerator | None = None, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None) → tuple[list[DQN | RainbowDQN | DDPG | TD3], list[float]]¶

Run the general online off-policy RL training; returns trained population of agents and their fitnesses.

Parameters:

env (Gym-style environment) – The environment to train in. Can be vectorized.
env_name (str) – Environment name
algo (str) – RL algorithm name
pop (list[RLAlgorithm]) – Population of agents
memory (object) – Experience Replay Buffer
init_hp (dict, optional) – Dictionary containing initial hyperparameters, defaults to None
mut_p (dict, optional) – Dictionary containing mutation parameters, defaults to None
max_steps (int, optional) – Maximum number of steps in environment, defaults to 1000000
evo_steps (int, optional) – Evolution frequency (steps), defaults to 10000
eval_steps (int, optional) – Number of evaluation steps per episode. If None, will evaluate until environment terminates or truncates. Defaults to None
eval_loop (int, optional) – Number of evaluation episodes, defaults to 1
learning_delay (int, optional) – Steps in environment before starting learning, defaults to 0
eps_start (float, optional) – Maximum exploration - initial epsilon value, defaults to 1.0
eps_end (float, optional) – Minimum exploration - final epsilon value, defaults to 0.1
eps_decay (float, optional) – Epsilon decay per episode, defaults to 0.995
target (float, optional) – Target score for early stopping, defaults to None
n_step_memory (object, optional) – Multi-step Experience Replay Buffer to be used alongside Prioritized ERB, defaults to None
tournament (object, optional) – Tournament selection object, defaults to None
mutation (object, optional) – Mutation object, defaults to None
checkpoint (int, optional) – Checkpoint frequency (steps), defaults to None
checkpoint_path (str, optional) – Location to save checkpoint, defaults to None
overwrite_checkpoints (bool, optional) – Overwrite previous checkpoints during training, defaults to False
save_elite (bool, optional) – Boolean flag indicating whether to save elite member at the end of training, defaults to False
elite_path (str, optional) – Location to save elite agent, defaults to None
wb (bool, optional) – Weights & Biases tracking, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
verbose (bool, optional) – Display training stats, defaults to True
accelerator (accelerate.Accelerator(), optional) – Accelerator for distributed computing, defaults to None
wandb_api_key (str, optional) – API key for Weights & Biases, defaults to None
wandb_kwargs (dict, optional) – Additional kwargs to pass to wandb.init()

Returns:

Trained population of agents and their fitnesses

Return type:

tuple[list[RLAlgorithm], list[float]]

agilerl.training.train_on_policy.train_on_policy(env: str | Env | VectorEnv | AsyncVectorEnv, env_name: str, algo: str, pop: list[PPO], init_hp: dict[str, Any] | None = None, mut_p: dict[str, Any] | None = None, max_steps: int = 1000000, evo_steps: int = 10000, eval_steps: int | None = None, eval_loop: int = 1, target: float | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, checkpoint: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, save_elite: bool = False, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, verbose: bool = True, accelerator: Accelerator | None = None, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None, collect_rollouts_fn: Callable[[PPO, str | Env | VectorEnv | AsyncVectorEnv, int], None] | None = None) → tuple[list[PPO], list[float]]¶

Run the general on-policy RL training; returns trained population of agents and their fitnesses.

Parameters:

env (Gym-style environment) – The environment to train in. Can be vectorized.
env_name (str) – Environment name
algo (str) – RL algorithm name
pop (list[RLAlgorithm]) – Population of agents
init_hp (dict, optional) – Dictionary containing initial hyperparameters, defaults to None
mut_p (dict, optional) – Dictionary containing mutation parameters, defaults to None
max_steps (int, optional) – Maximum number of steps in environment, defaults to 1000000
evo_steps (int, optional) – Evolution frequency (steps), defaults to 10000
eval_steps (int, optional) – Number of evaluation steps per episode. If None, will evaluate until environment terminates or truncates. Defaults to None
eval_loop (int, optional) – Number of evaluation episodes, defaults to 1
target (float, optional) – Target score for early stopping, defaults to None
tournament (object, optional) – Tournament selection object, defaults to None
mutation (object, optional) – Mutation object, defaults to None
checkpoint (int, optional) – Checkpoint frequency (steps), defaults to None
checkpoint_path (str, optional) – Location to save checkpoint, defaults to None
overwrite_checkpoints (bool, optional) – Overwrite previous checkpoints during training, defaults to False
save_elite (bool, optional) – Boolean flag indicating whether to save elite member at the end of training, defaults to False
elite_path (str, optional) – Location to save elite agent, defaults to None
wb (bool, optional) – Weights & Biases tracking, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
verbose (bool, optional) – Display training stats, defaults to True
accelerator (accelerate.Accelerator(), optional) – Accelerator for distributed computing, defaults to None
wandb_api_key (str, optional) – API key for Weights & Biases, defaults to None
wandb_kwargs – Additional kwargs to pass to wandb.init()
collect_rollouts_fn (Callable or None, optional) – Optional function used to collect rollouts. If None and agents use a rollout buffer, a default function will be selected based on whether the agent is recurrent.

Returns:

Trained population of agents and their fitnesses

Return type:

list[RLAlgorithm], list[float]

If you are training on static, offline data, you can use our offline RL training function.

agilerl.training.train_offline.train_offline(env: str | Env | VectorEnv | AsyncVectorEnv, env_name: str, algo: str, pop: list[CQN], memory: ReplayBuffer, init_hp: dict[str, Any] | None = None, mut_p: dict[str, Any] | None = None, max_steps: int = 1000000, evo_steps: int = 10000, eval_steps: int | None = None, eval_loop: int = 1, target: float | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, checkpoint: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, save_elite: bool = False, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, verbose: bool = True, accelerator: Accelerator | None = None, dataset: ReplayDataset | None = None, minari_dataset_id: str | None = None, remote: bool = False, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None) → tuple[list[CQN], list[float]]¶

Run the general offline RL training; returns trained population of agents and their fitnesses.

Parameters:

env (Gym-style environment) – The environment to train in
env_name (str) – Environment name
algo (str) – RL algorithm name
pop (list[CQN]) – Population of agents
memory (ReplayBuffer) – Experience Replay Buffer
init_hp (dict, optional) – Dictionary containing initial hyperparameters, defaults to None
mut_p (dict, optional) – Dictionary containing mutation parameters, defaults to None
max_steps (int, optional) – Maximum number of steps in environment, defaults to 1000000
evo_steps (int, optional) – Evolution frequency (steps), defaults to 10000
eval_steps (int, optional) – Number of evaluation steps per episode. If None, will evaluate until environment terminates or truncates. Defaults to None
eval_loop (int, optional) – Number of evaluation episodes, defaults to 1
target (float, optional) – Target score for early stopping, defaults to None
tournament (object, optional) – Tournament selection object, defaults to None
mutation (object, optional) – Mutation object, defaults to None
checkpoint (int, optional) – Checkpoint frequency (steps), defaults to None
checkpoint_path (str, optional) – Location to save checkpoint, defaults to None
overwrite_checkpoints (bool, optional) – Overwrite previous checkpoints during training, defaults to False
save_elite (bool, optional) – Boolean flag indicating whether to save elite member at the end of training, defaults to False
elite_path (str, optional) – Location to save elite agent, defaults to None
wb (bool, optional) – Weights & Biases tracking, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
verbose (bool, optional) – Display training stats, defaults to True
accelerator (accelerate.Accelerator(), optional) – Accelerator for distributed computing, defaults to None
dataset (ReplayDataset | None, optional) – Offline RL dataset (h5py file). Required when minari_dataset_id is not provided, defaults to None
minari_dataset_id (str, optional) – Minari dataset ID for loading data, defaults to None
remote (bool, optional) – Load Minari dataset from remote, defaults to False
wandb_api_key (str, optional) – API key for Weights & Biases, defaults to None
wandb_kwargs (dict, optional) – Additional kwargs to pass to wandb.init()

Returns:

Trained population of agents and their fitnesses

Return type:

tuple[list[CQN], list[float]]

The multi-agent off-policy and on-policy training functions handle PettingZoo-style environments and multi-agent algorithms.

agilerl.training.train_multi_agent_off_policy.train_multi_agent_off_policy(env: ParallelEnv | AsyncPettingZooVecEnv, env_name: str, algo: str, pop: list[MADDPG | MATD3], memory: ReplayBuffer, sum_scores: bool = True, init_hp: dict[str, Any] | None = None, mut_p: dict[str, Any] | None = None, max_steps: int = 50000, evo_steps: int = 25, eval_steps: int | None = None, eval_loop: int = 1, learning_delay: int = 0, target: float | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, checkpoint: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, save_elite: bool = False, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, verbose: bool = True, accelerator: Accelerator | None = None, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None) → tuple[list[MADDPG | MATD3], list[float]]¶

Run the general off-policy multi-agent RL training; returns trained population of agents and their fitnesses.

Parameters:

env (Gym-style environment) – The environment to train in. Can be vectorized.
env_name (str) – Environment name
algo (str) – RL algorithm name
pop (list[MADDPG | MATD3]) – Population of agents
memory (ReplayBuffer) – Experience Replay Buffer
sum_scores (bool, optional) – Boolean flag indicating whether to sum sub-agents scores, typically True for co-operative environments, defaults to True
init_hp (dict) – Dictionary containing initial hyperparameters.
mut_p (dict, optional) – Dictionary containing mutation parameters, defaults to None
max_steps (int, optional) – Maximum number of steps in environment, defaults to 50000
evo_steps (int, optional) – Evolution frequency (steps), defaults to 25
eval_steps (int, optional) – Number of evaluation steps per episode. If None, will evaluate until environment terminates or truncates. Defaults to None
eval_loop (int, optional) – Number of evaluation episodes, defaults to 1
learning_delay (int, optional) – Steps in environment before starting learning, defaults to 0
target (float, optional) – Target score for early stopping, defaults to None
tournament (object, optional) – Tournament selection object, defaults to None
mutation (object, optional) – Mutation object, defaults to None
checkpoint (int, optional) – Checkpoint frequency (steps), defaults to None
checkpoint_path (str, optional) – Location to save checkpoint, defaults to None
overwrite_checkpoints (bool, optional) – Overwrite previous checkpoints during training, defaults to False
save_elite (bool, optional) – Boolean flag indicating whether to save elite member at the end of training, defaults to False
elite_path (str, optional) – Location to save elite agent, defaults to None
wb (bool, optional) – Weights & Biases tracking, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
verbose (bool, optional) – Display training stats, defaults to True
accelerator (accelerate.Accelerator(), optional) – Accelerator for distributed computing, defaults to None
wandb_api_key (str, optional) – API key for Weights & Biases, defaults to None
wandb_kwargs (dict, optional) – Additional kwargs to pass to wandb.init()

Returns:

Trained population of agents and their fitnesses

Return type:

tuple[list[MADDPG | MATD3], list[float]]

agilerl.training.train_multi_agent_on_policy.train_multi_agent_on_policy(env: ParallelEnv | AsyncPettingZooVecEnv, env_name: str, algo: str, pop: list[IPPO], sum_scores: bool = True, init_hp: dict[str, Any] | None = None, mut_p: dict[str, Any] | None = None, max_steps: int = 50000, evo_steps: int = 25, eval_steps: int | None = None, eval_loop: int = 1, target: float | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, checkpoint: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, save_elite: bool = False, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, verbose: bool = True, accelerator: Accelerator | None = None, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None) → tuple[list[IPPO], list[float]]¶

Run the general on-policy multi-agent RL training; returns trained population of agents and their fitnesses.

Parameters:

env (Gym-style environment) – The environment to train in. Can be vectorized.
env_name (str) – Environment name
algo (str) – RL algorithm name
pop (list[object]) – Population of agents
sum_scores (bool, optional) – Boolean flag indicating whether to sum sub-agents scores, typically True for co-operative environments, defaults to True
init_hp (dict) – Dictionary containing initial hyperparameters.
mut_p (dict, optional) – Dictionary containing mutation parameters, defaults to None
max_steps (int, optional) – Maximum number of steps in environment across the entire population, defaults to 50000
evo_steps (int, optional) – Evolution frequency (steps), defaults to 25
eval_steps (int, optional) – Number of evaluation steps per episode. If None, will evaluate until environment terminates or truncates. Defaults to None
eval_loop (int, optional) – Number of evaluation episodes, defaults to 1
target (float, optional) – Target score for early stopping, defaults to None
tournament (object, optional) – Tournament selection object, defaults to None
mutation (object, optional) – Mutation object, defaults to None
checkpoint (int, optional) – Checkpoint frequency (steps), defaults to None
checkpoint_path (str, optional) – Location to save checkpoint, defaults to None
overwrite_checkpoints (bool, optional) – Overwrite previous checkpoints during training, defaults to False
save_elite (bool, optional) – Boolean flag indicating whether to save elite member at the end of training, defaults to False
elite_path (str, optional) – Location to save elite agent, defaults to None
wb (bool, optional) – Weights & Biases tracking, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
verbose (bool, optional) – Display training stats, defaults to True
accelerator (accelerate.Accelerator(), optional) – Accelerator for distributed computing, defaults to None
wandb_api_key (str, optional) – API key for Weights & Biases, defaults to None
wandb_kwargs (dict, optional) – Additional kwargs to pass to wandb.init()

Finally, if you are training a LLM, you can use our LLM training functions. We have one for preference-based reinforcement learning (finetune_llm_preference) which should be used with DPO, and one for reinforcement learning with verifiable rewards (finetune_llm_reasoning) which should be used with GRPO.

agilerl.training.llm.reasoning.finetune_llm_reasoning(pop: list[SupportedReasoning], env: ReasoningGym | None = None, env_fn: Callable[[], ReasoningGym] | None = None, init_hp: dict[str, Any] | None = None, save_elite: bool | None = None, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, evo_steps: int | None = None, checkpoint_steps: int | None = None, checkpoint_path: str | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None, evaluation_interval: int = 10, max_reward: int | None = None, verbose: bool = True, accelerator: Accelerator | None = None, max_steps: int | None = None, num_epochs: int | None = None) → list[SupportedReasoning]¶

Finetunes a population of GRPO/LLMPPO/LLMREINFORCE agents on a ReasoningGym.

Parameters:

pop (list[GRPO | LLMPPO | LLMREINFORCE]) – Population of reasoning RL agents to finetune.
env (ReasoningGym | None) – Shared ReasoningGym environment. Mutually exclusive with env_fn.
env_fn (Callable[[], ReasoningGym] | None) – Factory that creates one ReasoningGym per agent. Mutually exclusive with env.
init_hp (dict, optional) – Initial hyperparameters for the population
save_elite (bool, optional) – Whether to save the elite model, defaults to None
elite_path (str, optional) – Path to save the elite model, defaults to None
wb (bool, optional) – Whether to use Weights and Biases, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
evo_steps (int, optional) – Number of steps between evolution, defaults to None
checkpoint_steps (int, optional) – Number of steps between checkpoints, defaults to None
checkpoint_path (str | None, optional) – Directory for periodic checkpoints; falls back to elite_path, defaults to None
tournament (TournamentSelection, optional) – Tournament selection object, defaults to None
mutation (Mutations, optional) – Mutation object, defaults to None
wandb_api_key (str, optional) – Wandb API key, defaults to None
wandb_kwargs (dict, optional) – Additional kwargs to pass to wandb.init()
evaluation_interval (int, optional) – Number of steps between evaluation, defaults to 10
max_reward (int, optional) – Maximum reward to aim for, defaults to None
verbose (bool, optional) – Whether to print verbose output, defaults to True
accelerator (Accelerator, optional) – Accelerator object, defaults to None
max_steps (int, optional) – Maximum number of steps to run, defaults to None
num_epochs (int, optional) – Number of epochs to run, if set, takes precedence over max_steps, defaults to None

Returns:

The finetuned population.

Return type:

PopulationType

agilerl.training.llm.preference.finetune_llm_preference(pop: list[DPO], env: PreferenceGym | None = None, env_fn: Callable[[], PreferenceGym] | None = None, init_hp: dict[str, Any] | None = None, save_elite: bool | None = None, elite_path: str | None = None, wb: bool = False, tensorboard: bool = False, tensorboard_log_dir: str | None = None, evo_steps: int | None = None, checkpoint_steps: int | None = None, checkpoint_path: str | None = None, tournament: TournamentSelection | None = None, mutation: Mutations | None = None, wandb_api_key: str | None = None, wandb_kwargs: dict[str, Any] | None = None, evaluation_interval: int = 10, verbose: bool = True, accelerator: Accelerator | None = None, max_steps: int | None = None, num_epochs: int | None = None) → list[DPO]¶

Finetune a population of DPO agents on pairwise preference data.

Parameters:

pop (list[DPO]) – Population of DPO agents to finetune.
env (PreferenceGym | None) – Shared PreferenceGym environment. Mutually exclusive with env_fn.
env_fn (Callable[[], PreferenceGym] | None) – Factory that creates one PreferenceGym per agent. Mutually exclusive with env.
init_hp (dict, optional) – Initial hyperparameters for the population, defaults to None
save_elite (bool, optional) – Whether to save the elite model, defaults to None
elite_path (str, optional) – Directory for checkpoints, defaults to None
wb (bool, optional) – Whether to use Weights and Biases, defaults to False
tensorboard (bool, optional) – TensorBoard tracking, defaults to False
tensorboard_log_dir (str, optional) – Directory for TensorBoard logs, defaults to None
evo_steps (int, optional) – Number of steps between evolution, defaults to None
checkpoint_steps (int, optional) – Number of steps between checkpoints, defaults to None
checkpoint_path (str | None, optional) – Directory for periodic checkpoints; falls back to elite_path, defaults to None
tournament (TournamentSelection, optional) – Tournament selection object, defaults to None
mutation (Mutations, optional) – Mutation object, defaults to None
wandb_api_key (str, optional) – Wandb API key, defaults to None
wandb_kwargs (dict, optional) – Additional kwargs to pass to wandb.init()
evaluation_interval (int, optional) – Number of steps between evaluation, defaults to 10
verbose (bool, optional) – Whether to print verbose output, defaults to True
accelerator (Accelerator, optional) – Accelerator object, defaults to None
max_steps (int, optional) – Maximum number of steps to run, defaults to None
num_epochs (int, optional) – Number of epochs to run, if set, takes precedence over max_steps, defaults to None

Returns:

The finetuned population.

Return type:

PopulationType