Saving and Loading LLM Checkpoints

LLM checkpoints in AgileRL can persist just LoRA adapters, the full model, and optionally the optimizer/LR-scheduler state. The same on-disk layout is used for single-process, DDP, and FSDP2 runs: sharded parameters are gathered to full tensors before save, so a checkpoint is not tied to the rank count that wrote it. The defaults are lora_only=True and save_optimizer=True.

Checkpoint layout on disk

A typical checkpoint directory written by save_checkpoint() looks like:

checkpoint_dir/
├── attributes.pt              # algorithm hyperparameters; may also
│                              # contain the actor state_dict and/or
│                              # optimizer state depending on flags
├── actor/
│   ├── adapter_model.safetensors
│   └── adapter_config.json
├── reference/                 # only if use_separate_reference_adapter=True
│   ├── adapter_model.safetensors
│   └── adapter_config.json
└── critic/                    # only for algorithms with a value head
    ├── adapter_model.safetensors
    └── adapter_config.json

Which adapter subdirectories appear depends on the algorithm:

  • SFT: actor only.

  • DPO, GRPO: actor + reference.

  • PPO-LLM (with value head): actor + reference + critic.

Saving

agent.save_checkpoint(
    path,
    lora_only=True,        # default: adapters only, no base weights
    save_optimizer=True,   # default: persist optimizer + LR scheduler
)

The four combinations:

lora_only

save_optimizer

Produces

True

True

Adapter dirs on disk; optimizer state inside attributes.pt.

True

False

Adapter dirs only. No optimizer state.

False

True

Full actor state_dict + optimizer state, both inside attributes.pt.

False

False

Full actor state_dict inside attributes.pt (no optimizer state).

Under FSDP2, adapter weights and optimizer state are gathered to full tensors before they are written into this same layout. There is no separate sharded checkpoint directory.

Common scenarios:

# Periodic snapshot during training (adapters + optimizer, so training
# can resume where it left off):
agent.save_checkpoint(path)

# Release a deployable artefact (adapters only, no training state):
agent.save_checkpoint(path, save_optimizer=False)

# Persist the full model, base weights included, not just the adapters
# (e.g. for hand-off to a consumer that can't re-download the base):
agent.save_checkpoint(path, lora_only=False, save_optimizer=False)

Loading

agent.load_checkpoint(
    path,
    load_optimizer=True,   # restore optimizer + LR scheduler
)

save_optimizer and load_optimizer are independent flags: you can load a checkpoint that contains optimizer state while passing load_optimizer=False to keep the live optimizer, or load a weights-only checkpoint with load_optimizer=True (in which case a UserWarning is emitted and the existing optimizer is kept as-is).

load_checkpoint() expects the live algorithm to already be configured against the same base model. It restores adapter weights on top of that base and, by default, copies the just-loaded actor adapter onto reference so that SFT → DPO → GRPO pipelines work out of the box: the actor trained in stage N becomes the reference for stage N+1.

The checkpoint’s LoRA config must match the live algorithm’s (rank, target modules, etc.); a mismatch raises ValueError. Re-create the agent with the checkpoint’s LoRA config to load it.

Note

Checkpoints written by older AgileRL versions that used DeepSpeed engine directories (a save_checkpoint/ shard folder, no actor weights in attributes.pt) cannot be loaded.

Common scenarios:

# Resume training:
agent.load_checkpoint(path, load_optimizer=True)

# Inference / evaluation with a checkpoint that may or may not contain
# optimizer state, which we don't need:
agent.load_checkpoint(path, load_optimizer=False)