Zipdownload: add ZipStream support for large exports - #10175
Conversation
|
This is the result of entirely unscientific testing on a small VM, running Dovecot/Apache/php-fpm and little else, with one vcpu and about 600MB free RAM. I ran each test a few times in a row, which may have warmed up the disk cache, and took the best results from each. I recorded three metrics for each test (the column groups): RAM used in MB, time taken in seconds, and zip files size in MB. For time and RAM, I took them from the php-fpm status page. I ran three test scenarios (the columns):
I tested four implementations (the rows), each also included PR #10151.
The email corpus (for test scenarios 2 & 3) includes:
I would expect implementations 2 & 3 to scale upwards in RAM usage in proportion to the size of the largest email (or attachment for that test) included in the download, and for implementations 1 & 4 to remain constant. Finally I should note that it's "unfair" to compare ZipArchive to ZipStream given that the former uses its default zip compression level (of 9), and the latter specifies a level of 1. Here are the results:
|
It would make sense to keep ZipStream part separated.
I'm fine with choosing ZipStream automatically, but I'd add architecture check, so it does not get used on 32bit system. |
Mbox format not yet implemented
Using a custom StreamInterface to transfer emails between $imap->get_* and ZipStream::addFile* in 512 KiB chunks reduces memory usage when larger emails or attachments (34MiB) are present to about a fifth of the version without the custom StreamInterface. This also makes implementing the Mbox format easy.
|
Just a quick note: the changes to rcube_message.php will become unnecessary if Roundcube updates to Guzzle 8.x (which updates guzzlehttp/psr7 to 3.x which fixes an issue addressed by that workaround). |
(The stricken features below have been moved to #10260.)
This includes several improvements to zipdownload. Feel free to cherry-pick what looks useful, or ask me to split this into multiple PRs, or whatever makes most sense.Currently the entire zip is written to the output buffer before sending which can easily cause an OOM. Instead, flush every 512 KiB.Update the.htaccess, see 1972a9e for details.For maildir exports, change the modified-time of each.emlfile to the IMAP internal time to allow sorting extracted emails.Change the default charset in the config to UTF-8, see 286c893.The biggest change is adding optional support for the ZipStream library. This allows streaming emails from the storage backend to the web server while zipping them without any temporary files and without much additional RAM overhead.
It's feasible to download entire INBOXes of multiple GB with this update. Without, temporary disk space of around 130% of the total email size is required.
Potential issues so far:
No config option to choose which method, ZipArchive or ZipStream. Instead ZipStream is always chosen if available with a fallback to ZipArchive which may be good enough?Resolved as unnecessary.README.AndCHANGELOG.md?maennchen/zipstream-phpas asuggestto the maincomposer.json, I'm not sure if I should have only added it to the zipdownloadcomposer.json?maennchen/zipstream-phpto theMakefilefor the complete .tar?I'm happy to make any requested changes given some feedback. In the mean time, I'll loosely quantify the "without much additional RAM" claim in a follow-up.