EvolvableNetwork

Parameters

class agilerl.networks.base.EvolvableNetwork(*args: Any, **kwargs: Any)

Base class for evolvable networks, i.e., evolvable modules that are configured in a specific way for a reinforcement learning algorithm, similar to how CNNs are used as building blocks in ResNet, VGG, etc. An evolvable network automatically inspects the passed observation space to determine the appropriate encoder to build through the AgileRL evolvable modules, inheriting the mutation methods of any nested evolvable modules.

Note

Currently, evolvable networks should only have the encoder (which, if not specified by the user, is automatically built from the observation space) and a ‘head_net’ attribute that processes the latent encodings into the desired number of outputs as evolvable components. For example, in RainbowQNetwork, we disable mutations for the advantage net and apply the same mutations to it as the ‘value’ net, which is the network head in this case. Users should follow the same philosophy.

Parameters:

  • observation_space (spaces.Space) – Observation space of the environment.

  • encoder_cls (str | type[EvolvableModule] | None) – Encoder class to use for the network. Defaults to None, whereby it is automatically built using an AgileRL module according the observation space.

  • encoder_config (ConfigType | None) – Configuration of the encoder. Defaults to None.

  • action_space (spaces.Space | None) – Action space of the environment. Defaults to None.

  • min_latent_dim (int) – Minimum dimension of the latent space representation. Defaults to 8.

  • max_latent_dim (int) – Maximum dimension of the latent space representation. Defaults to 128.

  • latent_dim (int) – Dimension of the latent space representation. Defaults to 32.

  • simba (bool) – If True, use a SimBa network for the encoder for vector spaces. Defaults to False.

  • recurrent (bool) – If True, use a recurrent network for 2D observations. Defaults to False, whereby the encoder is a nn.Flatten() followed by an EvolvableMLP.

  • device (DeviceType) – Device to use for the network. Defaults to “cpu”.

  • random_seed (int | None) – Random seed to use for the network. Defaults to None.

property activation: str | None

Activation function of the network.

Returns:

Activation function.

Return type:

str | None

add_latent_node(numb_new_nodes: int | None = None) → MutationApplyDict

Add a latent node to the network.

Parameters:

numb_new_nodes (int, optional) – Number of new nodes to add, defaults to None

Returns:

Configuration for adding a latent node.

Return type:

dict[str, Any]

build_network_head(net_config: NetConfig | dict[str, Any]) → None

Build the head of the network.

Parameters:

net_config (NetConfigType) – Configuration of the network head.

change_activation(activation: str, output: bool = False) → None

Change the activation function for the network.

Parameters:

  • activation (str) – Activation function to use.

  • output (bool, optional) – If True, change the output activation function, defaults to False

create_mlp(num_inputs: int, num_outputs: int, name: str, net_config: NetConfig | dict[str, Any]) → EvolvableMLP

Build the head of the network based on the passed configuration.

Parameters:

  • num_inputs (int) – Number of inputs to the network head.

  • num_outputs (int) – Number of outputs of the network head.

  • name (str) – Name of the network head.

  • net_config (dict[str, Any]) – Configuration of the network head.

Returns:

Network head.

Return type:

EvolvableMLP

property encoder_config: dict[str, Any]

Net configuration for encoder.

Returns:

Initial dictionary for the network.

Return type:

dict[str, Any]

extract_features(x: Tensor | TensorDict | tuple[Tensor, ...] | dict[str, Tensor], hidden_state: None = None) → Tensor
extract_features(x: Tensor | TensorDict | tuple[Tensor, ...] | dict[str, Tensor], hidden_state: dict[str, Tensor]) → tuple[Tensor, dict[str, Tensor]]

Extract features from the encoder part of the network.

Parameters:

  • x (TorchObsType) – Input observation to extract features from

  • hidden_state (dict[str, torch.Tensor], optional) – Hidden states for recurrent networks (unused in non-recurrent networks)

Returns:

The encoded features, and (for recurrent networks) the next hidden-state dict if a hidden state was passed

Return type:

torch.Tensor | tuple[torch.Tensor, dict[str, torch.Tensor]]

forward_head(latent: Tensor, *args: Any, **kwargs: Any) → Tensor

Forward pass of the network head using pre-computed latent encodings.

Parameters:

latent (torch.Tensor) – Latent encodings from the encoder.

Returns:

Output of the network head.

Return type:

torch.Tensor

get_init_dict() → dict[str, Any]

Constructor arguments for the network.

Returns:

The dictionary of constructor arguments.

Return type:

dict[str, Any]

property head_config: dict[str, Any]

Net configuration for head.

Returns:

Initial dictionary for the network.

Return type:

dict[str, Any]

init_weights_gaussian(std_coeff: float = 4.0, output_coeff: float = 2.0) → None

Initialize the weights of the network with a Gaussian distribution.

Parameters:

  • std_coeff (float, optional) – Coefficient for the standard deviation of the Gaussian distribution, defaults to 4.0

  • output_coeff (float, optional) – Coefficient for the standard deviation of the Gaussian distribution for the output layer, defaults to 2.0

initialize_hidden_state(batch_size: int = 1) → dict[str, Tensor]

Initialize the hidden state for the network.

Parameters:

env (GymEnvType) – The environment to initialize the hidden state for

recreate_encoder() → None

Recreate the encoder of the network.

recreate_network() → None

Recreate the network after a mutation. Implemented by subclasses.

remove_latent_node(numb_new_nodes: int | None = None) → MutationApplyDict

Remove a latent node from the network.

Parameters:

numb_new_nodes (int, optional) – Number of nodes to remove, defaults to None

Returns:

Configuration for removing a latent node.

Return type:

dict[str, Any]