Skip to content

fix: open job-instance.pickle in binary mode in LSF contrib (#3330) - #3443

Open
1102Aryan wants to merge 3 commits into
spotify:masterfrom
1102Aryan:fix/lsf-runner-pickle-binary-mode
Open

fix: open job-instance.pickle in binary mode in LSF contrib (#3330)#3443
1102Aryan wants to merge 3 commits into
spotify:masterfrom
1102Aryan:fix/lsf-runner-pickle-binary-mode

Conversation

@1102Aryan

@1102Aryan 1102Aryan commented Jul 19, 2026

Copy link
Copy Markdown

Description

lsf_runner.py opened job-instance.pickle in text mode ("r"), which fails on
Python 3 because pickle.load() requires a binary stream. This PR:

  • luigi/contrib/lsf_runner.py: open the pickle file with "rb" instead of "r"
  • luigi/contrib/lsf.py: write the pickle file with "wb" instead of "w" in
    _dump(), and use bytes literals in the __main__-module .replace() call,
    since pickle.dumps() returns bytes on Python 3

Motivation and Context

Fixes #3330. Any LSF task run through lsf_runner.py on Python 3 crashes with
TypeError: a bytes-like object is required, not 'str' (or UnicodeDecodeError,
depending on pickle protocol and platform) before the job's work() method can run.

Have you tested this? If so, how?

I reproduced the reported error on master with a script that pickles a job object
and calls lsf_runner.do_work_on_compute_node() — it fails exactly as described in
the issue. With this fix, the same round-trip succeeds and the job's work() method
runs, including the __module__ == "__main__" dump path. The existing LSF tests
(test/contrib/lsf_test.py) and ruff lint checks also pass.

I have also included a unit test that round trips the pickle through _dump() and lst_runner.do_work_on_compute_node()

@1102Aryan
1102Aryan requested a review from dlstadther as a code owner July 19, 2026 13:51
@1102Aryan
1102Aryan requested a review from a team July 19, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TypeError: a bytes-like object is required, not 'str' when loading pickle in text mode

1 participant