Skip to content

Extract the consumer semaphore release sites into overridable methods - #19817

Open
deepthi912 wants to merge 1 commit into
apache:masterfrom
deepthi912:upsert/expose-parallel-consumption-policy
Open

deepthi912 wants to merge 1 commit into
apache:masterfrom
deepthi912:upsert/expose-parallel-consumption-policy

Conversation

@deepthi912

@deepthi912 deepthi912 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

buildSegmentInternal and downloadSegmentAndReplace each release the consumer semaphore when the parallel consumption policy allows the next segment of the partition to run during that phase:

if (_parallelSegmentConsumptionPolicy.isAllowedDuringBuild()) {
  closeStreamConsumerAndReleaseSemaphore();
}
...
if (allowedDuringDownload) {
  closeStreamConsumerAndReleaseSemaphore();
}

This moves each guarded release into a protected method, so a subclass can hold the semaphore for one phase and leave the other alone. Today that is not expressible: closeStreamConsumerAndReleaseSemaphore() is package private and shared by both sites, _parallelSegmentConsumptionPolicy is private, and a subclass overriding the shared method cannot tell which site called it without tracking its own state.

No behaviour change — the conditions and call order are unchanged.

🤖 Generated with Claude Code

buildSegmentInternal and downloadSegmentAndReplace each release the consumer
semaphore when the parallel consumption policy allows the next segment to run
during that phase. Move each guarded release into a protected method so a
subclass can hold the semaphore for one phase without re-deriving the policy
or distinguishing call sites itself.
@deepthi912
deepthi912 force-pushed the upsert/expose-parallel-consumption-policy branch from 2ab7ff4 to 1db8fd9 Compare October 9, 2026 20:10
@deepthi912 deepthi912 changed the title Make _parallelSegmentConsumptionPolicy protected Extract the consumer semaphore release sites into overridable methods Oct 9, 2026
@deepthi912 deepthi912 added real-time Related to realtime table ingestion and serving ingestion Related to data ingestion pipeline labels Oct 9, 2026
@codecov-commenter

codecov-commenter commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.61%. Comparing base (910f9d5) to head (1db8fd9).

Files with missing lines Patch % Lines
...a/manager/realtime/RealtimeSegmentDataManager.java 25.00% 5 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19817      +/-   ##
============================================
- Coverage     68.63%   68.61%   -0.02%     
  Complexity     1486     1486              
============================================
  Files          3526     3526              
  Lines        230143   230147       +4     
  Branches      36562    36561       -1     
============================================
- Hits         157963   157925      -38     
- Misses        59837    59868      +31     
- Partials      12343    12354      +11     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-25 68.61% <25.00%> (-0.02%) ⬇️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 68.61% <25.00%> (-0.02%) ⬇️
unittests 68.61% <25.00%> (-0.02%) ⬇️
unittests1 58.31% <25.00%> (-0.02%) ⬇️
unittests2 40.21% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ingestion Related to data ingestion pipeline real-time Related to realtime table ingestion and serving

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants