Skip to content

Drop the documented grad_hooks ZeRO option, which does not exist #8242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vineethsaivs wants to merge 1 commit into
base: master
Choose a base branch
from
+0 −9
Open

Drop the documented grad_hooks ZeRO option, which does not exist #8242

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions deepspeed/runtime/engine.py
100755 → 100644
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1287,9 +1287,6 @@ def zero_model_persistence_threshold(self):
def zero_gather_16bit_weights_on_model_save(self):
return self._config.zero_config.gather_16bit_weights_on_model_save

def zero_grad_hooks(self):
return self._config.zero_config.grad_hooks

def zero_legacy_stage1(self):
return self._config.zero_config.legacy_stage1

Expand Down
6 changes: 0 additions & 6 deletions docs/_pages/config-json.md
100755 → 100644
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,12 +531,6 @@ Enabling and configuring ZeRO memory optimizations
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| Initialize fp32 master weights from fp32 copies in checkpoint (no precision loss) or from model's fp16 copies (with precision loss). This can be used to initialize optimizer state even when checkpoint is missing optimizer state. | `True` |

<i>**grad_hooks**</i>: [boolean]

| Description | Default |
| ----------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| For use with ZeRO stage 1, enable backward hooks to reduce gradients during the backward pass or wait until the end of the backward pass. | `True` |

***round_robin_gradients***: [boolean]

| Description | Default |
Expand Down
Loading