`GRPOTrainer` can crash with `AttributeError` for `Callable` `reward_func.__name__`

[jamesbraza]

Reproduction

GRPOTrainer._prepare_inputs on the non-Module reward_func.__name__: https://github.com/huggingface/trl/blob/v0.15.2/trl/trainer/grpo_trainer.py#L658

The latent crash here is __name__ is not guaranteed to be present on a Callable, for example with unittest.mock.Mock:

>>> from unittest.mock import Mock
>>> Mock().__name__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jamesbraza/.pyenv/versions/3.12.8/lib/python3.12/unittest/mock.py", line 662, in __getattr__
    raise AttributeError(name)
AttributeError: __name__

A more robust implementation is:

reward_func_name = getattr(reward_func, '__name__', str(reward_func))

Happy to make a PR for this.

System Info

n/a

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete

[qgallouedec]

Good point, I'll be happy to receive a PR for this :)

[shirinyamani]

I've commented on your PR!

[jamesbraza]

Hi @shirinyamani thanks for the PR comment, but I think you're misunderstanding here, can you reopen this issue? This issue still stands.

Future-House#9 was about resolving #3018 on a fork, and by happenstance I fixed this issue in that PR too. However, I am not going to open that PR into actual trl as it was too hacky.

[qgallouedec]

Hey @jamesbraza, sorry for the slow follow-up, catching up on the backlog.

The code moved since v0.15.2 (the __name__ lookup now happens in __init__ rather than _prepare_inputs), but the same crash path still exists, see

trl/trl/trainer/grpo_trainer.py

Lines 406 to 409 in 51c6d3c

	if isinstance(reward_funcs[i], nn.Module): # Use Module over PretrainedModel for compat w/ compiled models
	self.reward_func_names.append(get_config_model_id(reward_funcs[i].config).split("/")[-1])
	else:
	self.reward_func_names.append(reward_funcs[i].__name__)

and your getattr(reward_func, '__name__', str(reward_func)) suggestion is the right fix. This issue is open for contributions

[omkar-334]

hey @qgallouedec I'd like to work on this

[qgallouedec]

Hey @jamesbraza, sorry for the very slow loop back on this, catching up on the backlog.

Good news: this is now fixed on main. The reward-name lookup no longer does an unguarded reward_func.__name__; it goes through a new get_callable_name helper that falls back safely for callables without a __name__ (mocks, callable class instances, functools.partial, etc.): see

trl/trl/trainer/utils.py

Lines 224 to 232 in 126ce88

	def get_callable_name(func: Callable) -> str:
	"""
	Return a display name for a callable, supporting the picklable reward forms: module-level functions,
	[`functools.partial`](https://docs.python.org/3/library/functools.html#functools.partial) (unwrapped to the wrapped
	function's name), and callable class instances (which fall back to their class name).
	"""
	while isinstance(func, functools.partial):
	func = func.func
	return getattr(func, "__name__", type(func).__name__)

used at

trl/trl/trainer/grpo_trainer.py

Line 524 in 126ce88

self.reward_func_names.append(get_callable_name(reward_funcs[i]))

It landed in #6313. Verified locally that get_callable_name(Mock()) returns 'Mock' instead of raising. Closing as fixed, thanks for the clear report and the suggested fix, which is essentially what shipped! And thanks @omkar-334 for offering to pick it up; this one got resolved before you started, sorry for the crossed wires.