Add cross-platform sendfile uploads with fallback to normal send#794
Add cross-platform sendfile uploads with fallback to normal send#794j-c-m wants to merge 1 commit into
Conversation
|
I've considered supporting sendfile way back, and dropped it. Checked with AI and it agrees that it's not more efficient than mmap due to pieces being 16kb. Lately I've been considering adding an BEP-10 extension to allow for larger piece messages, as that seems the best way to increase throughput. The new BT spec is too complicated for most clients to bother implementing. |
61cd4d4 to
2e4de39
Compare
|
I agree and don't expect to see much of a network performance/throughput difference. What I am targeting is less memory and possible cpu pressure during the pure uploading phase. Going to continue refining this and will update here. |
d75a942 to
bbe3d9f
Compare
|
Cleaned up, letting it soak test seeding on a large number of torrent files. Looking good so far, moderate effect as it can't be active on encrypted sends. |
bbe3d9f to
56e472d
Compare
|
This has been working OK for me. I think it is a useful option to reduce memory pressure in some scenarios, moving out of draft. |
|
While it reduces memory pressure, does it improve throughput? |
Most of the time, it performs how you would expect. On filesystems where the sendfile() is a true zero-copy it is faster (~2x), less memory, less cpu (~1/2). On filesystems where it's not, it's slightly worse than mmap(piece)->send(16k) for max throughput. When network limited, which seems to be more real-world, the slower max throughput case uses slightly more cpu but still less user process memory. Linux/EXT4 - faster |
|
I've been doing refactoring of various parts of libtorrent since becoming active again. The core of peer connections and mmap handling is very stable and not something I want to touch yet without good reason. I'd prefer to have it be tested irl for a while before merging. |
|
I will continue to let it run for more testing, the changes in peer_connection_base my intent was to keep current send path logic the same unless sendfile is enabled, the diff makes it look worse than it is. |
56e472d to
a1c1d80
Compare
|
update for 09b150f m_fileDesc -> file_descriptor() |
Introduce a unified sendfile_stream() abstraction for zero-copy piece uploads on Linux, FreeBSD, and macOS/Darwin, each with its own syscall signature detected at configure time. When pieces.sendfile is enabled on plaintext peers, uploads resolve file fd and offset via FileList::at_file() without mmap, falling back to chunk mapping only for encryption, padding, or sendfile errors. Expose pieces.sendfile, pieces.sendfile.set, and sendfile stats via ChunkManager for rtorrent configuration.
a1c1d80 to
90f5cf2
Compare
I've just begun testing this, really experimental, opening as a draft if you have any comment.
Introduce a unified sendfile_stream() abstraction for zero-copy piece uploads on Linux, FreeBSD, and macOS/Darwin, each with its own syscall signature detected at configure time.
When pieces.sendfile is enabled on plaintext peers, uploads resolve file fd and offset via FileList::at_file() without mmap, falling back to chunk mapping only for encryption, padding, or sendfile errors.
Expose pieces.sendfile, pieces.sendfile.set, and sendfile stats via ChunkManager for rtorrent configuration.