Training & Replay Buffer Specs

class agilerl.models.training.TrainingSpec(*, max_steps: Annotated[int, Ge(ge=1)] = 1000000, evo_steps: Annotated[int | None, Ge(ge=1)] = None, pop_size: Annotated[int, Ge(ge=1)] = 1, eval_steps: int | None = None, eval_loop: Annotated[int, Ge(ge=1)] = 1, replay_buffer: ReplayBufferSpec | None = None, hpo: bool = True, target_score: float | None = None, learning_delay: int = 0, eps_start: float | None = None, eps_end: float | None = None, eps_decay: float | None = None, checkpoint_steps: int | None = None, checkpoint_path: str | None = None, overwrite_checkpoints: bool = False, evaluation_interval: Annotated[int, Ge(ge=1)] = 10, num_epochs: Annotated[int | None, Ge(ge=1)] = None, episode_steps: Annotated[int, Ge(ge=1)] = 500, sum_scores: bool = True, reporting_interval: Annotated[int, Ge(ge=1)] = 1024, experience_sharing: bool = False)

Pydantic model for AgileRL training arguments.

Parameters:

  • max_steps (int) – Maximum number of steps to train for. Defaults to 1,000,000.

  • evo_steps (int | None) – Number of steps to train between evolutions.

  • pop_size (int) – Number of agents in the population. Defaults to 1.

  • eval_steps (int | None) – Number of steps to train for evaluation. Defaults to None.

  • eval_loop (int) – Number of evaluation episodes. Defaults to 1.

  • replay_buffer (ReplayBufferSpec | None) – Replay buffer specification.

  • hpo (bool) – Whether to use hyperparameter optimization.

  • target_score (float | None) – Target score for early stopping.

  • learning_delay (int) – Number of steps before starting learning.

  • eps_start (float | None) – Initial exploration probability.

  • eps_end (float | None) – Final exploration probability.

  • eps_decay (float | None) – Rate of decay of the exploration probability.

  • checkpoint_steps (int | None) – The number of steps between checkpoints.

  • checkpoint_path (str | None) – The path to save the checkpoints.

  • overwrite_checkpoints (bool) – If True, overwrite the checkpoints in the checkpoint directory.

  • evaluation_interval (int) – Number of steps between evaluations.

  • num_epochs (int | None) – Number of epochs to train for.

  • episode_steps (int) – Number of steps to train for each episode (only applicable for bandits).

  • sum_scores (bool) – Whether to sum sub-agent scores (only applicable for multi-agent). Typically True for cooperative environments. Defaults to True.

  • reporting_interval (int) – Number of steps between reporting.

  • experience_sharing (bool) – Whether to share experiences between agents.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class agilerl.models.training.ReplayBufferSpec(*, max_size: ~typing.Annotated[int, ~annotated_types.Ge(ge=1)] = 100000, standard_buffer: bool = True, n_step_buffer: bool = False, n_step_buffer_args: ~agilerl.models.training.NStepBufferArgs = <factory>, per_buffer: bool = False, per_buffer_args: ~agilerl.models.training.PerBufferArgs = <factory>)

Pydantic model for AgileRL replay buffers.

Parameters:

  • memory_size (int) – The memory size of the replay buffer. Defaults to 100,000.

  • standard_buffer (bool) – Whether to use the standard replay buffer. Defaults to True.

  • n_step_buffer (bool) – Whether to use the n-step replay buffer. Defaults to False.

  • n_step_buffer_args (NStepBufferArgs) – The arguments for the n-step replay buffer. Defaults to NStepBufferArgs.

  • per_buffer (bool) – Whether to use the prioritized experience replay buffer. Defaults to False.

  • per_buffer_args (PerBufferArgs) – The arguments for the prioritized experience replay buffer. Defaults to PerBufferArgs.

  • n_step (int | None) – The number of steps to use for the n-step replay buffer. Defaults to None.

init_buffer(algo_spec: AlgoSpecT, device: str | torch.device = 'cpu') → BufferT

Initialize the replay buffer.

Parameters:

  • algo_spec (AlgoSpecT) – Algorithm specification

  • device (str | torch.device) – Device

Returns:

Replay buffer

Return type:

BufferT

init_n_step_buffer(algo_spec: AlgoSpecT, device: str | torch.device = 'cpu') → BufferT | None

Initialize the n-step replay buffer for combined PER + n-step setups.

Returns None unless both per_buffer and n_step_buffer are True.

Parameters:

  • algo_spec (AlgoSpecT) – Algorithm specification.

  • device (str | torch.device) – Device.

Returns:

A MultiStepReplayBuffer or None.

Return type:

BufferT | None

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].