[LIVY-1082] Add config to disable Kubernetes driver pod log polling

[soumyadeeplogin]

What changes were proposed in this pull request?

SparkKubernetesApp currently fetches the driver pod's Kubernetes log (pods/<driver>/log) on every poll cycle to populate live log lines in /sessions/:id/log and /batches/:id/log responses. On large clusters, or when driver logs are already collected externally (e.g. via a centralized log pipeline / Spark History Server), this is wasted request volume against the Kubernetes API server, and can be a source of transient errors during pod lifecycle transitions.

This PR adds a new config, livy.server.kubernetes.driver-log-polling.enabled (default true, preserving existing behavior), that when set to false skips the pods/<driver>/log fetch while Livy continues to poll pod state and diagnostics normally.

Why are the changes needed?

To let operators opt out of the driver log poll when it's unnecessary, reducing Kubernetes API server load and avoiding transient log-fetch errors during pod lifecycle transitions, without losing pod state/diagnostics monitoring.

Does this PR introduce any user-facing change?

Yes. A new optional config livy.server.kubernetes.driver-log-polling.enabled is added (default true). When explicitly set to false, /sessions/:id/log and /batches/:id/log will no longer include live Kubernetes driver log lines; pod state and diagnostics continue to be polled and reported as before.

How was this patch tested?

Added/updated unit tests in SparkKubernetesAppSpec:

• should enable driver log polling by default
• resolveDriverAppLog - should skip fetching the log when driver log polling is disabled
• resolveDriverAppLog - should fetch the log when driver log polling is enabled

Ran mvn -pl server -am test -Dtest=SparkKubernetesAppSpec -DwildcardSuites=org.apache.livy.utils.SparkKubernetesAppSpec; all 14 tests in the suite pass.

Was this patch authored or co-authored using generative AI tooling?

Yes, this patch was co-authored using Claude Code (Anthropic).

[soumyadeeplogin]

Filed LIVY-1082 for this change.

[soumyadeeplogin]

Hi @gyogal / @roczei — could one of you approve the pending CI workflow run for this PR (fork-contributor approval gate)? Happy to address any review feedback as well. Thanks!

[codecov-commenter]

Codecov Report

❌ Patch coverage is 22.22222% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.28%. Comparing base (501e822) to head (fe8e84c).
⚠️ Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
...ala/org/apache/livy/utils/SparkKubernetesApp.scala	0.00%	7 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master     #557       +/-   ##
=============================================
- Coverage     68.68%   53.28%   -15.41%     
+ Complexity     1218      845      -373     
=============================================
  Files           106      106               
  Lines          6815     6816        +1     
  Branches        836      845        +9     
=============================================
- Hits           4681     3632     -1049     
- Misses         1666     2742     +1076     
+ Partials        468      442       -26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gyogal]

Thanks for the update @soumyadeeplogin, the change LGTM now!