Skip to content

fix: avoid flushing a missing capture stream when tracking URLs - #3453

Open
cpruijsen wants to merge 1 commit into
spotify:masterfrom
cpruijsen:fix/issue-3131
Open

fix: avoid flushing a missing capture stream when tracking URLs#3453
cpruijsen wants to merge 1 commit into
spotify:masterfrom
cpruijsen:fix/issue-3131

Conversation

@cpruijsen

Copy link
Copy Markdown

Description

Stop the external-program tracking subprocess from calling flush() on a missing capture file at EOF. The iterator sentinel is now b"" (binary PIPE EOF) instead of "", and flush runs only if a capture file exists.

ExternalProgramTask tees stdout/stderr through a PIPE so it can scrape a tracking URL. Writes to the capture TemporaryFile were already skipped when capture_output=False. The empty-read branch still called file_to_write.flush(), which raises AttributeError: 'NoneType' object has no attribute 'flush' in the child process. Spark tasks always enable this tracker, so the traceback shows up on yarn/spark-submit runs even though it does not change the task result.

Fixes #3131

Decision: treat EOF as the end of the tracker and skip flush when there is no capture file.
Alternative: only guard the existing flush() and leave the "" sentinel / sleep loop.
Why: the unguarded flush is what the issue reports, and the else branch only ran because "" never matches b"". Can switch to the one-line guard if that is preferred.

Motivation and Context

#3131

PySparkTask on YARN logs file_to_write.flush() on None from _track_url_by_pattern. The reporter already noted it does not change the job result; the child exception is still noise on every tracked run without a capture file.

Have you tested this? If so, how?

I have included unit tests.

test_tracking_process_exits_cleanly_when_capture_output_disabled fails without the source change (AssertionError: 1 != 0, child traceback is the issue's AttributeError) and passes with it (exitcode == 0). Existing tracking tests in test/contrib/external_program_test.py and the Spark client/cluster tracking tests still pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AttributeError: 'NoneType' object has no attribute 'flush'

1 participant