Saving and Loading LLM Checkpoints¶
LLM checkpoints in AgileRL can persist just LoRA adapters, the full model, and
optionally the optimizer/LR-scheduler state. The same on-disk layout is used
for single-process, DDP, and FSDP2 runs: sharded parameters are gathered to
full tensors before save, so a checkpoint is not tied to the rank count that
wrote it. The defaults are lora_only=True and save_optimizer=True.
Checkpoint layout on disk¶
A typical checkpoint directory written by save_checkpoint() looks like:
checkpoint_dir/
├── attributes.pt # algorithm hyperparameters; may also
│ # contain the actor state_dict and/or
│ # optimizer state depending on flags
├── actor/
│ ├── adapter_model.safetensors
│ └── adapter_config.json
├── reference/ # only if use_separate_reference_adapter=True
│ ├── adapter_model.safetensors
│ └── adapter_config.json
└── critic/ # only for algorithms with a value head
├── adapter_model.safetensors
└── adapter_config.json
Which adapter subdirectories appear depends on the algorithm:
SFT:
actoronly.DPO, GRPO:
actor+reference.PPO-LLM (with value head):
actor+reference+critic.
Saving¶
agent.save_checkpoint(
path,
lora_only=True, # default: adapters only, no base weights
save_optimizer=True, # default: persist optimizer + LR scheduler
)
The four combinations:
|
|
Produces |
|---|---|---|
|
|
Adapter dirs on disk; optimizer state inside
|
|
|
Adapter dirs only. No optimizer state. |
|
|
Full actor |
|
|
Full actor |
Under FSDP2, adapter weights and optimizer state are gathered to full tensors before they are written into this same layout. There is no separate sharded checkpoint directory.
Common scenarios:
# Periodic snapshot during training (adapters + optimizer, so training
# can resume where it left off):
agent.save_checkpoint(path)
# Release a deployable artefact (adapters only, no training state):
agent.save_checkpoint(path, save_optimizer=False)
# Persist the full model, base weights included, not just the adapters
# (e.g. for hand-off to a consumer that can't re-download the base):
agent.save_checkpoint(path, lora_only=False, save_optimizer=False)
Loading¶
agent.load_checkpoint(
path,
load_optimizer=True, # restore optimizer + LR scheduler
)
save_optimizer and load_optimizer are independent flags: you can
load a checkpoint that contains optimizer state while passing
load_optimizer=False to keep the live optimizer, or load a
weights-only checkpoint with load_optimizer=True (in which case a
UserWarning is emitted and the existing optimizer is kept as-is).
load_checkpoint() expects the live algorithm to already be configured
against the same base model. It restores adapter weights on top of that base
and, by default, copies the just-loaded actor adapter onto reference
so that SFT → DPO → GRPO pipelines work out of the box: the actor trained
in stage N becomes the reference for stage N+1.
The checkpoint’s LoRA config must match the live algorithm’s (rank,
target modules, etc.); a mismatch raises ValueError. Re-create the
agent with the checkpoint’s LoRA config to load it.
Note
Checkpoints written by older AgileRL versions that used DeepSpeed engine
directories (a save_checkpoint/ shard folder, no actor weights in
attributes.pt) cannot be loaded.
Common scenarios:
# Resume training:
agent.load_checkpoint(path, load_optimizer=True)
# Inference / evaluation with a checkpoint that may or may not contain
# optimizer state, which we don't need:
agent.load_checkpoint(path, load_optimizer=False)