fix: increase streaming chunk size for CSV and Excel view downloads

[jacalata]

Summary

Unifies streaming chunk size across all download paths in Endpoint. Previously:

• DownloadableMixin._download_content (workbook / datasource / flow) used a hard-coded 1024-byte chunk since the mixin was introduced.
• Views._get_view_csv / _get_view_excel and CustomViews._get_custom_view_csv also used hard-coded 1024 bytes.
• A TSC_CHUNK_SIZE_MB env var / config.CHUNK_SIZE_MB knob existed for uploads (fileuploads_endpoint) but was never applied to downloads.

This PR:

• Adds DownloadableMixin._stream_content(url, req_options) -> Iterator[bytes] for endpoints whose callers want to lazily iterate the response (Views, CustomViews).
• Routes both _download_content and _stream_content through config.CHUNK_SIZE_MB * BYTES_PER_MB (default 50 MB).
• Migrates Views and CustomViews CSV/Excel to use _stream_content.

Effect: the 1 KB default (which required ~200,000 generator iterations for a 200 MB CSV, causing apparent hangs when callers materialize the full stream via b''.join(view.csv)) is replaced by the tunable TSC_CHUNK_SIZE_MB knob. The default of 50 MB is what upload paths already use; whether that default is right for interactive download streaming is a separate design question — see the Copilot-review thread and issue #1374 comment history.

Closes #1374

Test plan

• Existing test_populate_csv, test_populate_excel, test_filter_excel, test_populate_csv_default_maxage all pass

🤖 Generated with Claude Code

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
tableauserverclient
__init__.py	5	0	100%
config.py	15	0	100%
datetime_helpers.py	25	1	1	96%
exponential_backoff.py	20	0	100%
filesys_helpers.py	31	0	100%
namespace.py	26	3	3	88%
tableauserverclient/bin
__init__.py	2	0	100%
_version.py	358	212	212	41%
tableauserverclient/helpers
__init__.py	1	0	100%
logging.py	2	0	100%
strings.py	31	1	1	97%
tableauserverclient/models
__init__.py	46	0	100%
collection_item.py	41	7	7	83%
column_item.py	55	32	32	42%
connection_credentials.py	35	11	11	69%
connection_item.py	94	14	14	85%
custom_view_item.py	144	21	21	85%
data_acceleration_report_item.py	54	1	1	98%
data_alert_item.py	158	4	4	97%
data_freshness_policy_item.py	155	15	15	90%
database_item.py	207	36	36	83%
datasource_item.py	300	12	12	96%
dqw_item.py	104	5	5	95%
exceptions.py	4	0	100%
extensions_item.py	132	4	4	97%
extract_item.py	44	4	4	91%
favorites_item.py	69	8	8	88%
fileupload_item.py	19	0	100%
flow_item.py	149	10	10	93%
flow_run_item.py	71	0	100%
group_item.py	89	6	6	93%
groupset_item.py	49	7	7	86%
interval_item.py	182	32	32	82%
job_item.py	187	10	10	95%
linked_tasks_item.py	79	1	1	99%
location_item.py	29	2	2	93%
metric_item.py	129	13	13	90%
oidc_item.py	63	3	3	95%
pagination_item.py	34	1	1	97%
permissions_item.py	111	12	12	89%
project_item.py	207	31	31	85%
property_decorators.py	100	18	18	82%
reference_item.py	26	2	2	92%
revision_item.py	59	1	1	98%
schedule_item.py	209	6	6	97%
server_info_item.py	37	7	7	81%
site_item.py	636	13	13	98%
subscription_item.py	101	2	2	98%
table_item.py	119	18	18	85%
tableau_auth.py	61	25	25	59%
tableau_types.py	27	1	1	96%
tag_item.py	15	0	100%
target.py	6	0	100%
task_item.py	56	2	2	96%
user_item.py	310	18	18	94%
view_item.py	220	16	16	93%
virtual_connection_item.py	64	8	8	88%
webhook_item.py	69	1	1	99%
workbook_item.py	362	16	16	96%
tableauserverclient/server
__init__.py	9	0	100%
exceptions.py	4	0	100%
filter.py	29	1	1	97%
pager.py	33	1	1	97%
query.py	143	15	15	90%
request_factory.py	1335	195	195	85%
request_options.py	386	5	5	99%
server.py	188	23	23	88%
sort.py	6	0	100%
tableauserverclient/server/endpoint
__init__.py	35	0	100%
auth_endpoint.py	77	11	11	86%
custom_views_endpoint.py	150	12	12	92%
data_acceleration_report_endpoint.py	21	0	100%
data_alert_endpoint.py	94	23	23	76%
databases_endpoint.py	111	30	30	73%
datasources_endpoint.py	323	33	33	90%
default_permissions_endpoint.py	44	3	3	93%
dqw_endpoint.py	45	16	16	64%
endpoint.py	219	20	20	91%
exceptions.py	77	6	6	92%
extensions_endpoint.py	31	0	100%
favorites_endpoint.py	94	22	22	77%
fileuploads_endpoint.py	51	0	100%
flow_runs_endpoint.py	62	9	9	85%
flow_task_endpoint.py	21	2	2	90%
flows_endpoint.py	198	53	53	73%
groups_endpoint.py	126	9	9	93%
groupsets_endpoint.py	72	7	7	90%
jobs_endpoint.py	67	9	9	87%
linked_tasks_endpoint.py	37	0	100%
metadata_endpoint.py	88	14	14	84%
metrics_endpoint.py	55	6	6	89%
oidc_endpoint.py	42	1	1	98%
permissions_endpoint.py	44	3	3	93%
projects_endpoint.py	178	24	24	87%
resource_tagger.py	127	35	35	72%
schedules_endpoint.py	119	11	11	91%
server_info_endpoint.py	36	10	10	72%
sites_endpoint.py	130	27	27	79%
subscriptions_endpoint.py	56	14	14	75%
tables_endpoint.py	110	36	36	67%
tasks_endpoint.py	63	6	6	90%
users_endpoint.py	183	8	8	96%
views_endpoint.py	147	9	9	94%
virtual_connections_endpoint.py	113	10	10	91%
webhooks_endpoint.py	54	9	9	83%
workbooks_endpoint.py	338	22	22	93%
TOTAL	12004	1423	88%

[jorwoods]

Should this use the config.CHUNK_SIZE_MB as the download size rather than a fixed 64mb?

[bcantoni]

Code Review

Summary: Raises the iter_content chunk size in ViewsEndpoint._get_view_csv / _get_view_excel from a hardcoded 1024 bytes to config.CHUNK_SIZE_MB * BYTES_PER_MB (50 MB default), fixing the effective hang reported in #1374 for large (~200 MB) view downloads.

Strengths

• Correct root-cause fix — 1 KB chunks meant ~200k generator iterations for a 200 MB download; a much larger chunk size removes that overhead.
• Reuses existing config infrastructure (config.CHUNK_SIZE_MB, BYTES_PER_MB) rather than a new magic number, matching the pattern already used in fileuploads_endpoint.py, datasources_endpoint.py, and custom_views_endpoint.py. This also gives users the existing TSC_CHUNK_SIZE_MB env var as a tuning knob.
• Small, surgical change with no other behavior differences — low risk.
• black --check passes on the changed file.

Issues / Gaps

Inconsistent chunk size left in place elsewhere. The same hardcoded 1024-byte pattern still exists, unfixed, in:

• custom_views_endpoint.py:234 (_get_custom_view_csv) — directly parallel to _get_view_csv, the exact method being fixed here, and likely has the same large-download-hang risk.
• flows_endpoint.py:235,242
• datasources_endpoint.py:1069,1076
• workbooks_endpoint.py:1131,1138

Worth confirming whether _get_custom_view_csv is intentionally out of scope for this PR, since it's the closest analog to the code being changed.

No test asserts the new chunk size. test_populate_csv, test_populate_csv_default_maxage, test_populate_excel, and test_filter_excel all mock small fixture files and only assert the reassembled content is correct — they'd pass identically whether the chunk size were 1024 bytes or 50 MB, so they don't protect this change from regressing. test_fileuploads.py:85-90 has a good existing pattern for this (sets TSC_CHUNK_SIZE_MB via env and asserts the actual chunk length) that could be adapted here, or alternatively mock/patch iter_content and assert it's called with the expected size.

The PR description's test plan checkbox is also still unchecked.

Correctness

config.CHUNK_SIZE_MB * BYTES_PER_MB = 50 * 1024 * 1024 ≈ 50 MB by default — consistent with the existing upload chunk-size default. No correctness or security concerns with the change itself.

---

🤖 Generated with Claude Code

[bcantoni]

@jacalata kind of funny how claude code keeps finding issues with a claude code aided PR :) Consider doing those consistency points if they make sense.

[jacalata]

Yea, clearly I need to run a more adversarial set of claudes! Turned out the consistency points were important: since I opened this PR, #1803 landed on development and extracted the streaming-to-file logic used by workbooks/datasources/flows into DownloadableMixin._download_content — but still using hardcoded 1024-byte chunks. Same performance bug applies to those endpoints, and custom_views_endpoint._get_custom_view_csv had the same 1024 too.

Expanded the PR scope:

• DownloadableMixin._download_content — chunk size now config.CHUNK_SIZE_MB * BYTES_PER_MB. Benefits workbooks, datasources, flows.
• New sibling DownloadableMixin._stream_content — for endpoints that yield Iterator[bytes] instead of writing to a file. Same chunk-size config.
• Views and CustomViews migrate to _stream_content, removing their duplicate streaming loops.
• Regression tests added for both mixin methods.

Callers using b"".join(view.csv) or iterating lazily continue to work identically; only the internal chunk size changes.

[Copilot]

Pull request overview

This PR refactors how binary and export content is streamed from the server by routing view/custom-view CSV/Excel downloads through a shared DownloadableMixin._stream_content helper and by changing the iter_content() chunk size used during streaming.

Changes:

• Introduces DownloadableMixin._stream_content() and updates Views/CustomViews CSV/Excel fetchers to use it.
• Updates DownloadableMixin._download_content() to stream using a configurable chunk size instead of the previous 1 KB chunks.
• Adds regression tests asserting that streaming/download paths call requests.Response.iter_content() with the configured chunk size.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/test_workbook.py	Adds a regression test ensuring workbook downloads stream using the configured chunk size.
test/test_view.py	Adds a regression test ensuring view CSV streaming uses the configured chunk size.
tableauserverclient/server/endpoint/views_endpoint.py	Mixes in DownloadableMixin and routes CSV/Excel view streaming through _stream_content().
tableauserverclient/server/endpoint/endpoint.py	Adds _stream_content() and changes streaming chunk size logic in the shared download/stream helpers.
tableauserverclient/server/endpoint/custom_views_endpoint.py	Mixes in DownloadableMixin and routes custom-view CSV streaming through _stream_content().

Suppressed comments (1)

tableauserverclient/server/endpoint/endpoint.py:405

• _stream_content also uses config.CHUNK_SIZE_MB * BYTES_PER_MB as its iter_content() chunk size, which is currently tuned/documented for upload chunking (TSC_CHUNK_SIZE_MB) and defaults to 50 MB. For view/custom-view CSV/Excel streaming this is likely too large for responsive iteration; consider using a smaller, download-specific chunk size (e.g., 64 KiB) and keeping upload chunk sizing separate.

        write it to a file. Complements _download_content, which writes to disk.
        """
        chunk_size = config.CHUNK_SIZE_MB * BYTES_PER_MB
        with closing(
            self.get_request(url, request_object=request_object, parameters={"stream": True})  # type: ignore[attr-defined]
        ) as server_response:
            yield from server_response.iter_content(chunk_size)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jacalata]

Correct that the code uses config.CHUNK_SIZE_MB * BYTES_PER_MB (default 50 MB), not the 65,536 bytes the PR body still claims. Sequence of events: the PR started as a bump from 1024 to 65,536 in the two view helpers, but grew into unifying chunk sizing across all download paths via DownloadableMixin._stream_content + _download_content using the same config.CHUNK_SIZE_MB knob. The PR body is out of date; updating it now.

Two things worth separating:

1. Consistency: this PR makes view/custom-view streaming use the same chunk size as workbook/datasource/flow downloads (pre-existing use of config.CHUNK_SIZE_MB on line 378). That's intentional and correct — one knob for everything, tunable via TSC_CHUNK_SIZE_MB.

2. Whether 50 MB is the right default: legitimate concern for interactive streaming of small CSVs (iter_content(50 MB) on a 500 KB response yields one chunk with the whole body — functionally fine but defeats lazy processing). This affects both the new _stream_content and the pre-existing _download_content. Retuning the default is a separate change, worth doing but not scoped to this PR.