fix: catch FileExistsError when checkpointing to distributed fs

[akx]

What does this PR do ?

Fixes spurious racey FileExistsErrors when creating directories for checkpointing.
This can happen on distributed filesystems (virtiofs in my case).

A multi-hour distributed run was killed at step 297/300, leaving other ranks hung, by

[rank8]:   File "nemo_automodel/components/launcher/interactive.py", line 93, in _run_recipe_in_process
[rank8]:     return recipe.run_train_validation_loop()
[rank8]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank8]:   File "nemo_automodel/recipes/llm/train_ft.py", line 981, in run_train_validation_loop
[rank8]:     self.save_checkpoint(
[rank8]:   File "nemo_automodel/recipes/base_recipe.py", line 310, in save_checkpoint
[rank8]:     self.checkpointer.save_on_global_ranks(getattr(self, key), key, path)
[rank8]:   File "nemo_automodel/components/checkpoint/checkpointing.py", line 1397, in save_on_global_ranks
[rank8]:     _ensure_dirs(state_dir, process_group=self.process_group)
[rank8]:   File "nemo_automodel/components/checkpoint/checkpointing.py", line 2022, in _ensure_dirs
[rank8]:     _create_dirs(*dirs)
[rank8]:   File "nemo_automodel/components/checkpoint/checkpointing.py", line 2011, in _create_dirs
[rank8]:     os.makedirs(directory, exist_ok=True)
[rank8]:   File "<frozen os>", line 225, in makedirs
[rank8]: FileExistsError: [Errno 17] File exists: '/mnt/shared/ckpt/case/epoch_1_step_297/rng'

Changelog

• Catch FileExistsErrors in _create_dirs.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
  • Race condition practically untestable. No tests.
• Did you add or update any necessary documentation?
  • No, just a bugfix.

Additional Information

• Related to perf(checkpoint): reduce distributed save overhead #3369 though the bug existed before it.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[akoumpa]

/ok to test 8fb2c5c