High-performance UDP peer-to-peer communication application with real-time threading, lock-free data structures, and event-driven architecture.
- Overview
- Application Architecture
- Project Structure
- Module Details
- Build
- Usage
- Coding Conventions
- License
Agent Team Test is a UDP peer-to-peer communication test application designed for low-latency, high-throughput packet exchange between two nodes. Each node transmits periodic packets containing a lifesign counter and monitors the remote peer for communication loss or unstable timing.
Key characteristics:
- Real-time threading with SCHED_FIFO and CPU affinity
- Lock-free SPSC ring buffers for inter-thread communication
- epoll-based event loop for timer-driven operations
- CRC32-protected application packet format
- Graceful shutdown via POSIX signal handling (SIGINT / SIGTERM)
βββββββββββββββββββββββ UDP βββββββββββββββββββββββ
β β ββββββββββββββββββββΊ β β
β Node A β Peer-to-Peer β Node B β
β (agent_team_test) β Communication β (agent_team_test) β
β β β β
βββββββββββββββββββββββ βββββββββββββββββββββββ
src:port A src:port B
dst:port B dst:port A
Each node is an independent instance of the same binary. Node A's destination is Node B's source, and vice versa.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β main (Entry Point) β
β - CLI argument parsing β
β - Object wiring & initialization β
β - Event loop lifecycle β
ββββββββββββββββββ¬βββββββββββββββ¬βββββββββββββββ¬ββββββββββββββββββββββ€
β app β event β socket β thread β
β β β β β
β AppPacket β EventLoop β UdpNode β UdpThreadManager β
β ArgParser β β β LockFreeRingBuffer β
β SignalHandler β β β β
ββββββββββββββββββ΄βββββββββββββββ΄βββββββββββββββ΄ββββββββββββββββββββββ€
β stats β
β LatencyStats TerminalUI β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β timer β
β TimerHandle β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The application runs three execution contexts:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Process β
β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Main Thread β β
β β - Event loop (epoll) β β
β β - TX timer callback βββΊ encodes packet βββΊ TX Ring Buffer β β
β β - Comm monitor timer callback β β
β β - Signal handler check β β
β ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β ββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββββββ β
β β RX Thread β β TX Thread β β
β β CPU Core: 2 β β CPU Core: 3 β β
β β SCHED_FIFO, Prio: 80 β β SCHED_FIFO, Prio: 70 β β
β β SIGINT/SIGTERM blocked β β SIGINT/SIGTERM blocked β β
β β β β β β
β β do { β β do { β β
β β recvfrom(socket) β β pop(TX Ring Buffer) β β
β β push(RX Ring Buffer) β β sendto(socket) β β
β β invoke RX callback β β } while (running) β β
β β } while (running) β β β β
β ββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββββββ β
β β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
| Thread | Role | Scheduling | Priority | CPU Core |
|---|---|---|---|---|
| Main | Event loop, timers | SCHED_OTHER | default | any |
| RX Thread | Socket receive | SCHED_FIFO | 80 | 2 |
| TX Thread | Socket transmit | SCHED_FIFO | 70 | 3 |
Transmit Path
βββββββββββββ
TX Timer βββΊ AppPacket::encode() βββΊ TX Ring Buffer βββΊ TX Thread βββΊ sendto()
(100 ms) (main thread) (lock-free) (SCHED_FIFO) (kernel)
Receive Path
ββββββββββββ
recvfrom() βββΊ RX Ring Buffer βββΊ RX Callback βββΊ AppPacket::decode()
(kernel) (RX Thread) (RX Thread) (RX Thread)
(lock-free) ββ lifesign update
ββ interval measurement
ββ stability check
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
βββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββ€
β unique_id (32-bit) β Header
βββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββ€
β lifesign (16-bit) β data_length (16-bit) β
βββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββ€
β β
β data (0..256 bytes) β Payload
β β
βββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββΌββ€
β crc32 (32-bit) β Footer
βββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ΄ββ
- unique_id: Identifies the sending node
- lifesign: Auto-incremented counter per transmission
- data_length: Payload length in bytes (max 256)
- data: Application payload
- crc32: CRC32 over header + data
agent_team_test/
βββ CMakeLists.txt # Build configuration (C++26, CMake 3.20+)
βββ Makefile # Convenience targets: build, clean, run, test_node_1, test_node_2
βββ LICENSE # MIT License
βββ README.md # This file
β
βββ include/ # Public headers
β βββ app/
β β βββ AppPacket.hpp # Packet encode/decode, comm monitoring
β β βββ ArgParser.hpp # CLI argument parsing
β β βββ SignalHandler.hpp # POSIX signal handler (singleton)
β βββ event/
β β βββ EventLoop.hpp # epoll-based event loop
β βββ socket/
β β βββ UdpNode.hpp # UDP socket wrapper
β βββ thread/
β β βββ LockFreeRingBuffer.hpp # SPSC lock-free ring buffer (template)
β β βββ UdpThreadManager.hpp # RX/TX thread lifecycle management
β βββ timer/
β βββ timer.hpp # timerfd wrapper
β
βββ include/ # Public headers (continued)
β βββ stats/
β βββ LatencyStats.hpp # Percentile statistics (header-only template)
β βββ TerminalUI.hpp # Split-screen ANSI terminal dashboard
β
βββ src/ # Implementation
β βββ app/
β β βββ main.cpp # Entry point, object wiring
β β βββ AppPacket.cpp # Packet codec, CRC32, stability logic
β β βββ ArgParser.cpp # --src / --dst argument parsing
β β βββ SignalHandler.cpp # sigaction setup, callback dispatch
β βββ event/
β β βββ EventLoop.cpp # epoll_wait loop, fd registration
β βββ socket/
β β βββ UdpNode.cpp # socket/bind/connect/send/recv
β βββ thread/
β β βββ UdpThreadManager.cpp # pthread create, affinity, SCHED_FIFO
β βββ timer/
β βββ Timer.cpp # timerfd_create, timerfd_settime
β
βββ config/ # Runtime configuration (reserved)
βββ script/ # Utility scripts (reserved)
| Class / Function | Responsibility |
|---|---|
main() |
Wires all objects, runs event loop until shutdown |
AppPacket |
Encode/decode packets with CRC32 integrity |
| Track lifesign, measure interval, detect comm loss | |
ArgParser |
Parse --src <addr>:<port> --dst <addr>:<port> from CLI |
SignalHandler |
Singleton; installs SIGINT/SIGTERM via sigaction() |
Thread-safe shutdown flag with std::atomic |
|
Registers callback to stop EventLoop on signal |
|
| Supports registering additional shutdown callbacks |
| Class | Responsibility |
|---|---|
EventLoop |
Wraps Linux epoll for multiplexed I/O |
| Registers file descriptors with callbacks | |
| Drives timer expiration and (optionally) socket I/O |
| Class | Responsibility |
|---|---|
UdpNode |
Creates UDP datagram socket (SOCK_DGRAM) |
Sets SO_REUSEADDR for quick restart |
|
| Binds to source address/port | |
Connects to destination (enables send/recv) |
|
| Exposes file descriptor for epoll or thread I/O |
| Class | Responsibility |
|---|---|
TimerHandle |
Wraps Linux timerfd for high-resolution timers |
| Supports one-shot and periodic modes | |
Accepts std::function<void()> callbacks |
|
| Exposes fd for epoll registration |
| Class | Responsibility |
|---|---|
UdpThreadManager |
Creates and manages dedicated RX and TX pthreads |
Blocks SIGINT/SIGTERM in worker threads (pthread_sigmask) |
|
Configures CPU affinity (pthread_setaffinity_np) |
|
| Configures SCHED_FIFO real-time scheduling | |
| Tunes SO_RCVBUF / SO_SNDBUF socket buffer sizes | |
| Sets SO_RCVTIMEO for clean RX thread shutdown | |
| Handles ECONNREFUSED as transient (peer not ready) | |
| Provides packet counters and drop statistics | |
LockFreeRingBuffer |
SPSC ring buffer (template, header-only) |
Cache-line aligned (alignas(64)) to prevent false sharing |
|
| Acquire/release memory ordering for thread safety | |
| Default: 1024 slots x 2048 bytes per slot |
| Class / Template | Responsibility |
|---|---|
LatencyStats<> |
Collects nanosecond latency samples in a lock-free circular buffer |
| Computes p50, p95, p99, p99.9, p99.99 percentiles on demand | |
| O(1) recording, O(N log N) computation via snapshot + sort | |
| Cache-line aligned atomics, header-only template | |
RAII ScopedMeasurement for automatic timing |
|
TerminalUI |
Split-screen ANSI terminal with pinned dashboard (upper 7 lines) |
| Scrolling packet log in lower region | |
| Mutex-protected output for thread safety (RX thread + main thread) | |
| Dashboard shows count, min, p50, p95, p99, p99.9, max per metric |
Three latency metrics are measured in UdpThreadManager:
- TX Send Latency β
sendto()syscall duration - RX Processing Latency β
recvfrom()return through queue push + callback - RX Inter-Packet Interval β time between consecutive received packets (jitter)
See README_LATENCY_BENCHMARKING.md for full architecture, percentile methodology, and performance analysis guide.
- Linux (kernel 2.6.27+ for
timerfd,epoll) - GCC 14+ or Clang 18+ (C++26 support)
- CMake 3.20+
# Build
make build
# Clean
make clean
# Run (requires root for SCHED_FIFO)
make runOr manually:
mkdir -p build && cd build
cmake ..
make# Node A
sudo ./build/agent_team_test --src 127.0.0.1:5000 --dst 127.0.0.1:6000
# Node B (separate terminal)
sudo ./build/agent_team_test --src 127.0.0.1:6000 --dst 127.0.0.1:5000Or use the convenience make targets (can start in either order):
# Terminal 1
make test_node_1
# Terminal 2
make test_node_2sudo is required for real-time scheduling (SCHED_FIFO). Alternative:
sudo setcap cap_sys_nice+ep ./build/agent_team_test=== High-Performance UDP Configuration ===
Source: 0x7F000001:5000
Destination: 0x7F000001:6000
RX Thread: CPU core 2, priority 80 (SCHED_FIFO)
TX Thread: CPU core 3, priority 70 (SCHED_FIFO)
SO_RCVBUF: 2097152 bytes
SO_SNDBUF: 1048576 bytes
==========================================
[TX] Lifesign: 1, Queued: 32 bytes (TX queue: 0)
[RX] UniqueId: 0x12345678, Lifesign: 1, DataLen: 15, Interval: 100023 us
Press Ctrl+C for graceful shutdown.
This project follows safety-critical coding practices:
- MISRA-like boolean checks:
if (condition == true)/if (condition == false) - Always use braces: All
if,else,while,forstatements use{} - Single point return: Each function has exactly one
returnstatement at the end - Prefer
do-while: Loop bodies execute at least once (except event loops) - No
continue: Loop flow is controlled via conditions, notcontinuestatements - Explicit comparisons: Pointers compared with
nullptr, booleans withtrue/false
MIT License - Copyright (c) 2026 Tawan Thintawornkul