Skip to content

Zipdownload: add ZipStream support for large exports - #10175

Draft
gurnec wants to merge 6 commits into
roundcube:masterfrom
gurnec:zipstream
Draft

Zipdownload: add ZipStream support for large exports#10175
gurnec wants to merge 6 commits into
roundcube:masterfrom
gurnec:zipstream

Conversation

@gurnec

@gurnec gurnec commented May 15, 2026

Copy link
Copy Markdown
Contributor

(The stricken features below have been moved to #10260.)

This includes several improvements to zipdownload. Feel free to cherry-pick what looks useful, or ask me to split this into multiple PRs, or whatever makes most sense.

  • Currently the entire zip is written to the output buffer before sending which can easily cause an OOM. Instead, flush every 512 KiB.
  • Update the .htaccess, see 1972a9e for details.
  • For maildir exports, change the modified-time of each .eml file to the IMAP internal time to allow sorting extracted emails.
  • Change the default charset in the config to UTF-8, see 286c893.
  • The b332416 refactor commit shouldn't change any behavior (if it does it's a bug), it exists solely to make the later commits easier to follow.

The biggest change is adding optional support for the ZipStream library. This allows streaming emails from the storage backend to the web server while zipping them without any temporary files and without much additional RAM overhead.

It's feasible to download entire INBOXes of multiple GB with this update. Without, temporary disk space of around 130% of the total email size is required.

Potential issues so far:

  • No config option to choose which method, ZipArchive or ZipStream. Instead ZipStream is always chosen if available with a fallback to ZipArchive which may be good enough? Resolved as unnecessary.
  • ZipArchive is required to show the menu items even when ZipStream is available. This is done to avoid autoloading ZipStream just for displaying the menu, though it's not ideal.
  • No new unit tests, though E2E does test basic ZipStream downloads which may be sufficient? edit: the improved E2E tests in Zipdownload: fix escaping in Mbox downloads #10151 help to address this.
  • No documentation yet, should probably add something to the zipdownload config or README. And CHANGELOG.md?
  • I added maennchen/zipstream-php as a suggest to the main composer.json, I'm not sure if I should have only added it to the zipdownload composer.json?
  • Is there any interest in adding maennchen/zipstream-php to the Makefile for the complete .tar?

I'm happy to make any requested changes given some feedback. In the mean time, I'll loosely quantify the "without much additional RAM" claim in a follow-up.

@gurnec

gurnec commented May 16, 2026

Copy link
Copy Markdown
Contributor Author

This is the result of entirely unscientific testing on a small VM, running Dovecot/Apache/php-fpm and little else, with one vcpu and about 600MB free RAM. I ran each test a few times in a row, which may have warmed up the disk cache, and took the best results from each.

I recorded three metrics for each test (the column groups): RAM used in MB, time taken in seconds, and zip files size in MB. For time and RAM, I took them from the php-fpm status page.

I ran three test scenarios (the columns):

  1. Attachment download - an email with two attachments, one 24.5 MB and one 0.5 MB (the worst case scenario given a max message size of about 34 MB)
  2. Maildir export, all emails
  3. Mbox export, all emails

I tested four implementations (the rows), each also included PR #10151.

  1. ZipArchive - nothing else
  2. Zipstream w/strings - the "initial ZipStream support" commit ab7d10a
  3. Zipstream w/memory - a commit (not online) which uses php://memory files instead of strings
  4. Zipstream w/fibers - this entire PR

The email corpus (for test scenarios 2 & 3) includes:

  • 10996 emails
  • 830 MB
  • 389 "From " lines which needed escaping
  • 4 emails > 34 MB in size (these are the worst-case emails which push up memory usage for implementations 2 & 3)
  • 67 emails > 512 KB in size

I would expect implementations 2 & 3 to scale upwards in RAM usage in proportion to the size of the largest email (or attachment for that test) included in the download, and for implementations 1 & 4 to remain constant.

Finally I should note that it's "unfair" to compare ZipArchive to ZipStream given that the former uses its default zip compression level (of 9), and the latter specifies a level of 1.

Here are the results:

Attach. RAM Maildir RAM Mbox RAM Attach. time Maildir time Mbox time Attach. zip Maildir zip Mbox zip
ZipArchive 34 10 16 1.5 89.6 84.5 25 331 324
Zipstream w/strings 119 131 n/a 2.2 53.2 n/a 25 354 n/a
Zipstream w/memory 95 97 95 2.6 54.0 56.6 25 354 348
Zipstream w/fibers 26 28 34 1.8 51.2 48.6 25 354 348

@alecpl

alecpl commented Jul 12, 2026

Copy link
Copy Markdown
Member

This includes several improvements to zipdownload. Feel free to cherry-pick what looks useful, or ask me to split this into multiple PRs, or whatever makes most sense.

It would make sense to keep ZipStream part separated.

Potential issues so far:

* [ ]  No config option to choose which method, ZipArchive or ZipStream. Instead ZipStream is always chosen if available with a fallback to ZipArchive which may be good enough?

I'm fine with choosing ZipStream automatically, but I'd add architecture check, so it does not get used on 32bit system.

gurnec added 4 commits July 13, 2026 15:37
Mbox format not yet implemented
Using a custom StreamInterface to transfer emails between $imap->get_*
and ZipStream::addFile* in 512 KiB chunks reduces memory usage when
larger emails or attachments (34MiB) are present to about a fifth
of the version without the custom StreamInterface.

This also makes implementing the Mbox format easy.
@gurnec

gurnec commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@alecpl I've separated out the non-ZipStream changes, and will put them in a new PR. If that PR ends up getting merged, I'll have to update this one to use the set-mtimes-inside-maildir-zip feature; for now the mtimes are just the export-creation time.

I added the 32/64-bit check in 3503723.

@gurnec gurnec changed the title Zipdownload: various improvements to this plugin Zipdownload: add ZipStream support for large exports Jul 13, 2026
@gurnec

gurnec commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Just a quick note: the changes to rcube_message.php‎ will become unnecessary if Roundcube updates to Guzzle 8.x (which updates guzzlehttp/psr7 to 3.x which fixes an issue addressed by that workaround).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants