Mutation¶
Mutation is periodically used to explore the hyperparameter space, allowing different hyperparameter combinations to be trialled during training. If certain hyperparameters prove relatively beneficial to training, then that agent is more likely to be preserved in the next generation, and so those characteristics are more likely to remain in the population.
- The
Mutations()
class is used to mutate agents with pre-set probabilities. The available mutations currently implemented are: No mutation
Network architecture mutation - adding layers or nodes. Trained weights are reused and new weights are initialized randomly.
Network parameters mutation - mutating weights with Gaussian noise.
Network activation layer mutation - change of activation layer.
RL algorithm mutation - mutation of learning hyperparameter, such as learning rate or batch size.
Mutations.mutation()
returns a mutated population.
Tournament selection and mutation should be applied sequentially to fully evolve a population between evaluation and learning cycles.
from agilerl.hpo.mutation import Mutations
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
mutations = Mutations(algo='DQN', # Algorithm
no_mutation=0.4, # No mutation
architecture=0.2, # Architecture mutation
new_layer_prob=0.2, # New layer mutation
parameters=0.2, # Network parameters mutation
activation=0, # Activation layer mutation
rl_hp=0.2, # Learning HP mutation
rl_hp_selection=['lr', 'batch_size'], # Learning HPs to choose from
mutation_sd=0.1, # Mutation strength
arch=NET_CONFIG['arch'], # Network architecture
rand_seed=1, # Random seed
device=device)
Parameters¶
- class agilerl.hpo.mutation.Mutations(algo, no_mutation, architecture, new_layer_prob, parameters, activation, rl_hp, rl_hp_selection=['lr', 'batch_size', 'learn_step'], mutation_sd=0.1, activation_selection=['ReLU', 'ELU', 'GELU'], min_lr=0.0001, max_lr=0.01, min_learn_step=1, max_learn_step=120, min_batch_size=8, max_batch_size=1024, agent_ids=None, arch='mlp', mutate_elite=True, rand_seed=None, device='cpu', accelerator=None)¶
The Mutations class for evolutionary hyperparameter optimization.
- Parameters:
algo (str or dict) – RL algorithm. Use str e.g. ‘DQN’ if using AgileRL algos, or provide a dict with names of agent networks
no_mutation (float) – Relative probability of no mutation
architecture (float) – Relative probability of architecture mutation
new_layer_prob (float) – Relative probability of new layer mutation (type of architecture mutation)
parameters (float) – Relative probability of network parameters mutation
activation (float) – Relative probability of activation layer mutation
rl_hp (float) – Relative probability of learning hyperparameter mutation
rl_hp_selection (list[str]) – Learning hyperparameter mutations to choose from
mutation_sd (float) – Mutation strength
activation_selection (list[str], optional) – Activation functions to choose from, defaults to [“ReLU”, “ELU”, “GELU”]
min_lr (float, optional) – Minimum learning rate in the hyperparameter search space
max_lr (float, optional) – Maximum learning rate in the hyperparameter search space
min_learn_step (int, optional) – Minimum learn step in the hyperparameter search space
max_learn_step (int, optional) – Maximum learn step in the hyperparameter search space
min_batch_size (int, optional) – Minimum batch size in the hyperparameter search space
max_batch_size (int, optional) – Maximum batch size in the hyperparameter search space
agents_id (list[str]) – List of agent ID’s for multi-agent algorithms
arch (str, optional) – Network architecture type. ‘mlp’ or ‘cnn’, defaults to ‘mlp’
mutate_elite (bool, optional) – Mutate elite member of population, defaults to True
rand_seed (int, optional) – Random seed for repeatability, defaults to None
device (str, optional) – Device for accelerated computing, ‘cpu’ or ‘cuda’, defaults to ‘cpu’
accelerator (accelerate.Accelerator(), optional) – Accelerator for distributed computing, defaults to None
- activation_mutation(individual)¶
Returns individual from population with activation layer mutation.
- Parameters:
individual (object) – Individual agent from population
- architecture_mutate(individual)¶
Returns individual from population with network architecture mutation.
- Parameters:
individual (object) – Individual agent from population
- classic_parameter_mutation(network)¶
Returns network with mutated weights.
- Parameters:
network – Neural network to mutate
- get_algo_nets(algo)¶
Returns dictionary with agent network names.
- Parameters:
algo (str) – RL algorithm
- mutation(population, pre_training_mut=False)¶
Returns mutated population.
- no_mutation(individual)¶
Returns individual from population without mutation.
- Parameters:
individual (object) – Individual agent from population