NVIDIA-NeMo · pstjohn · Aug 26, 2026 · Aug 26, 2026 · Aug 26, 2026 · Aug 26, 2026
@@ -63,6 +63,34 @@ Only load configuration files and `_target_` values from trusted sources. A YAML
 
 The `distributed:` section is **not** instantiated using `_target_`. Recipes parse it with a fixed schema. Use `strategy: fsdp2`, `strategy: ddp`, or `strategy: megatron_fsdp`. You can also configure parallelism sizes, such as `dp_size`, `tp_size`, and `pp_size`, and strategy-specific options. When pipeline parallelism is enabled (`pp_size > 1`), add a `pipeline:` subsection with options such as `pp_schedule`, `pp_microbatch_size`, and `layers_per_stage`. For examples, see the [Pipeline Parallelism with AutoPipeline](/development/pipeline-parallelism) guide and the recipe configs.
 
+## Select Trainable Modules
+
+The LLM and VLM fine-tuning recipes accept a `freeze_config:` section. Use `freeze_modules` and `unfreeze_modules` with typed selectors to control which modules are trainable:
+
+```yaml
+freeze_config:
+  freeze_modules:
+  - path: vision_tower        # exact canonical module path
+  - path: audio_tower         # repeat the key to select more exact module paths
+  - glob: "*.speech_encoder"  # case-sensitive shell-style glob on the full module path
+  unfreeze_modules:
+  - path: multi_modal_projector
+```
+
+Each selector is a mapping with exactly one key — `path` matches one module by its exact fully qualified name, while `glob` matches module names with `fnmatch`-style wildcards (`*` crosses `.` separators, so `*_proj` matches projection modules at any depth). List entries are additive: repeat `path` or `glob` entries to select several modules, and a single entry combining both keys is rejected. Matching is recursive: a selected module's entire subtree is frozen or unfrozen. Bare strings are rejected; unknown options and selectors that match no parameters raise an error before training starts.
+
+Trainability is resolved in a fixed order. Full fine-tuning preserves the model's existing `requires_grad` state, including intentionally frozen model-owned parameters; PEFT establishes a LoRA-trainable, base-frozen baseline. `freeze_modules` selectors then freeze their modules, and `unfreeze_modules` selectors unfreeze theirs, winning on overlap — this makes combinations such as LoRA plus a fully trainable multimodal projector a two-line configuration. Freezing only controls `requires_grad`; it never changes a parameter's storage dtype. The policy is validated on the complete model, re-resolved after tensor/expert parallel and activation-checkpointing surgery immediately before DDP/FSDP construction, and resolved once more after checkpoint loading before optimizer construction.
+
+The modality-specific booleans `freeze_vision_tower`, `freeze_audio_tower`, `freeze_language_model`, and `freeze_video_embedder` remain supported and keep their established attribute and substring matching, applied before `unfreeze_modules`:
+
+```yaml
+freeze_config:
+  freeze_vision_tower: true
+  freeze_audio_tower: true
+```
+
+Legacy-only configurations retain the implicit `freeze_vision_tower: true` default. A configuration that declares `freeze_modules` or `unfreeze_modules`, even as an empty list, uses explicit-selector semantics and does not implicitly freeze vision modules; add a vision selector (for example, `glob: "*vision*"`) when such a configuration should also freeze the vision tower. Omit both selector fields to retain legacy behavior; `null`, `false`, and numeric values are rejected.
+
 ## Prewarm One-Time CUDA Initialization
 
 {/* docs-review-start: mamba-ssd-prewarm */}

@@ -1,17 +1,17 @@
 ---
 title: "Sequence Classification (SFT/PEFT) with NeMo AutoModel"
-description: ""
+description: "Train a sequence classification model with NeMo AutoModel using GLUE MRPC, RoBERTa, and optional LoRA."
 position: 10
 ---
 ## Introduction
 
-Sequence classification tasks (e.g., sentiment analysis, topic classification, GLUE tasks) map input text to a discrete label. NeMo AutoModel provides a lightweight recipe specialized for this setting that integrates with popular pretrained model formats and dataset sources. Integration with Hugging Face is supported.
+Sequence classification tasks (for example, sentiment analysis, topic classification, and GLUE tasks) map input text to a discrete label. NeMo AutoModel provides a lightweight recipe specialized for this setting that integrates with popular pretrained model formats and dataset sources. Integration with Hugging Face is supported.
 
-This guide shows how to train a sequence classification model using the `TrainFinetuneRecipeForSequenceClassification` recipe, including optional Parameter-Efficient Fine-Tuning (LoRA).
+This guide shows how to train a sequence classification model using the `TrainFinetuneRecipeForSequenceClassification` recipe, including optional Parameter-Efficient Fine-Tuning (PEFT) with LoRA.
 
 ## Quickstart
 
-Use the example config for GLUE MRPC with RoBERTa-large + LoRA:
+Use the example config for GLUE MRPC with RoBERTa-large and LoRA:
 
 ```bash
 uv run automodel examples/llm_seq_cls/glue/mrpc_roberta_lora.yaml
@@ -66,6 +66,7 @@ distributed:
   tp_size: 1
   cp_size: 1
   sequence_parallel: false
+  autocast_dtype: bfloat16
 
 peft:
   _target_: nemo_automodel.components._peft.lora.PeftConfig
@@ -76,6 +77,10 @@ peft:
   alpha: 16
   dropout: 0.1
 
+freeze_config:
+  unfreeze_modules:
+  - glob: "*classifier"
+
 dataset:
   _target_: nemo_automodel.components.datasets.llm.seq_cls.GLUE_MRPC
   split: train
@@ -108,8 +113,10 @@ optimizer:
 
 ## LoRA (PEFT) Settings
 
-- `target_modules`: glob to select linear layers (e.g., `"*.proj"`).
-- `dim` (rank), `alpha`, `dropout`: tune per model/compute budget. Values `dim=8, alpha=16, dropout=0.1` are a good starting point for RoBERTa.
+- `target_modules`: Glob to select linear layers (for example, `"*.proj"`).
+- `dim` (rank), `alpha`, `dropout`: Tune per model and compute budget. Values `dim=8, alpha=16, dropout=0.1` are a good starting point for RoBERTa.
+- `freeze_config.unfreeze_modules`: Keeps the classification head fully trainable while PEFT freezes other non-LoRA parameters.
+- `distributed.autocast_dtype`: Runs the forward pass in the selected compute dtype while trainable parameters retain their configured storage dtype, including when single-rank FSDP is skipped.
 - The recipe automatically applies the adapters; no additional code changes are required.
 
 ## Running on Multiple GPUs

@@ -328,7 +328,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -329,7 +329,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -152,7 +152,6 @@ distributed:
   ep_size: 8            # 128 MoE experts across 8 GPUs
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -156,7 +156,6 @@ distributed:
   ep_size: 8            # 128 MoE experts across 8 GPUs
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -59,7 +59,6 @@ distributed:
 # Full fine-tune: audio tower + language model trainable; vision tower frozen
 # (we never feed images on AMI).
 freeze_config:
-  freeze_embeddings: false
   freeze_vision_tower: true
   freeze_audio_tower: false
   freeze_language_model: false

@@ -62,7 +62,6 @@ distributed:
 # Full fine-tune: audio tower + language model trainable; vision tower frozen
 # (we never feed images on AMI).
 freeze_config:
-  freeze_embeddings: false
   freeze_vision_tower: true
   freeze_audio_tower: false
   freeze_language_model: false

@@ -72,7 +72,6 @@ distributed:
 # Full fine-tune: audio tower + language model trainable; vision tower frozen
 # (we never feed images on these English ASR corpora).
 freeze_config:
-  freeze_embeddings: false
   freeze_vision_tower: true
   freeze_audio_tower: false
   freeze_language_model: false

@@ -88,7 +88,6 @@ distributed:
 # Full fine-tune: audio tower + language model trainable; only the vision tower
 # stays frozen (we never feed images). No PEFT adapters.
 freeze_config:
-  freeze_embeddings: false
   freeze_vision_tower: true
   freeze_audio_tower: false
   freeze_language_model: false

@@ -82,7 +82,6 @@ distributed:
 # Full fine-tune: audio tower + language model trainable; only the vision tower
 # stays frozen (we never feed images on these English ASR corpora). No PEFT.
 freeze_config:
-  freeze_embeddings: false
   freeze_vision_tower: true
   freeze_audio_tower: false
   freeze_language_model: false

@@ -100,7 +100,6 @@ lr_scheduler:
   min_lr: 1.0e-6
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -38,17 +38,21 @@ distributed:
   cp_size: 1
 
   sequence_parallel: false
+  autocast_dtype: bfloat16
 
 peft:
   _target_: nemo_automodel.components._peft.lora.PeftConfig
   target_modules:
   - "*.query"
   - "*.value"
-  # Note: classifier head is fully trained (not LoRA), unfrozen automatically in train_seq_cls.py
   dim: 8
   alpha: 16
   dropout: 0.1
 
+freeze_config:
+  unfreeze_modules:
+  - glob: "*classifier"
+
 dataset:
   _target_: nemo_automodel.components.datasets.llm.seq_cls.GLUE_MRPC
   split: train
@@ -71,4 +75,3 @@ optimizer:
   eps: 1e-8
   lr: 2.0e-5  # Standard learning rate for BERT/RoBERTa fine-tuning
   weight_decay: 0.01  # Crucial for stable training on small datasets
-
@@ -108,7 +108,6 @@ lr_scheduler:
   min_lr: 5.0e-7
 
 freeze_config:
-  freeze_embeddings: true      # NO-OP in this path (apply_parameter_freezing ignores it); kept for parity
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false # embeddings + tied LM head trainable — required to learn tokens 48/49/52

@@ -88,7 +88,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_language_model: false
 

@@ -86,7 +86,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_language_model: false
 

@@ -99,7 +99,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_language_model: false
 

@@ -89,7 +89,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_language_model: false
 

@@ -100,7 +100,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_language_model: false
 

@@ -90,7 +90,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -103,7 +103,6 @@ peft:
   use_triton: True
 
 freeze_config:
-  freeze_embeddings: false
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -106,7 +106,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -96,7 +96,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -92,7 +92,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -96,7 +96,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -118,7 +118,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -92,7 +92,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -106,7 +106,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -99,7 +99,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -125,7 +125,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -115,7 +115,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -109,7 +109,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -104,7 +104,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -97,7 +97,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -106,7 +106,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -111,7 +111,6 @@ lr_scheduler:
   lr_decay_style: cosine
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -97,7 +97,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false

@@ -93,7 +93,6 @@ optimizer:
   betas: [0.9, 0.95]
 
 freeze_config:
-  freeze_embeddings: true
   freeze_vision_tower: true
   freeze_audio_tower: true
   freeze_language_model: false