Protocol Performance Analysis

In distributed systems, the protocol stack sits on the critical path of every request. The choice between TCP, REST, and gRPC is not merely an API preference — it is a binding contract with the OS scheduler, the kernel network stack, and the memory allocator.

The gap in existing benchmarks: Most protocol comparisons report only end-to-end latency under a single load condition. They do not explain why protocols differ, nor how overhead evolves with message size and concurrency. Without this, protocol selection is intuition-driven rather than evidence-based. This work instruments at the hardware counter, flame graph, and syscall level to expose root causes rather than symptoms.

Inspired by Raghavan et al.'s network stack overhead analysis, which showed that even loopback communication incurs measurable kernel crossing costs. Our contribution extends this with per-protocol hardware counter comparison and controlled anomaly investigation.

All three implement an identical echo service. Client sends a protobuf Payload{id, data[]byte, timestamp}. Server echoes with updated timestamp. Same logical work — overhead differences are purely protocol mechanics.

TCP consistently 2–3.8× faster. At 4KB, REST (55.4µs) exceeds gRPC (50.4µs) — reversing expected ordering. This is the Zone 1→2 crossover in latency, confirmed by dense sweep (see dense latency section). See anomaly section for hardware counter confirmation.

At 64B all three scale approximately linearly. The REST/TCP ratio stays stable at 1.5–1.9× — indicating REST overhead is largely fixed per request at small message sizes. This is average latency; tail behavior tells a completely different story — see next section.

At concurrency=1 with 64B messages, TCP achieves approximately 49,000 req/s, versus 24,000 for REST and 21,000 for gRPC. At higher concurrency, all three protocols converge — the bottleneck shifts from protocol overhead to server CPU capacity.

The "knee" occurs between c=10 and c=50 for TCP and REST. gRPC remains flat throughout.

To understand why REST latency exceeds gRPC at 4KB, hardware counters were collected at 1KB, 4KB, and 16KB with c=1 and 10,000 requests.

Attached perf record to running client processes (120K–137K samples). Both protocols spend ~77–79% blocked waiting on network I/O. Of active CPU samples:

Generated using perf record at 1KHz + call graph capture + FlameGraph tools. Wider = more CPU time. Flame graph file sizes alone reflect complexity: TCP 149KB, REST 241KB, gRPC 332KB.

gRPC wins on context switches at high concurrency, but loses badly on cache misses. At c=500, gRPC generates 27× more cache misses than TCP. HTTP/2 per-stream state (flow control windows, HPACK tables) for 500 concurrent streams far exceeds L1/L2 capacity — constant cache thrashing. This partially offsets gRPC's multiplexing benefits.

TCP's user/sys split is constant at all concurrency levels — always kernel-dominated. REST and gRPC shift toward user space at scale: the bottleneck becomes their own protocol logic and synchronization, not the kernel network stack.

Observed: gRPC page fault count jumped discontinuously at exactly 32KB message size — a step function, not a gradual increase.

Observed: REST vs gRPC page fault ordering is non-monotonic across message sizes — three behaviorally distinct zones.

At 4KB, c=1: REST (55.4µs) > gRPC (50.4µs) — reversing expected ordering. Reproduced across multiple independent runs. Confirmed by hardware counter analysis showing REST consuming 15% more cycles and 18% more cache references at 4KB than gRPC — the inverse of the 1KB relationship.

With maxInflight=50, at most 50 goroutines hold the semaphore at any moment. The remaining 450 block on the Go channel — outside the kernel, in user space — without touching the kernel send buffer. Queue discipline moves from kernel to application layer, making it measurable and controllable.

REST over HTTP/1.1 adds 6 user-space function call layers over raw TCP. This translates to 6× more memory accesses, 3× more context switches, and 36× more mutex lock operations for identical 64B payloads. The overhead is largely additive and fixed per request at small message sizes — REST is essentially a constant 2× overhead tax on TCP.

gRPC introduces greater per-request complexity (2.7× more CPU cycles, 8 user-space layers), but its architectural features prevent catastrophic degradation at scale. The additional overhead is a worthwhile investment when P99 tail latency matters.

• Go runtime inseparability: GC and goroutine scheduler costs cannot be fully isolated from protocol costs without a bare-metal C++ baseline. Partially mitigated by function-level perf attribution per protocol.
• Single-node only: Network latency would dominate µs-level differences in a distributed deployment. Relative ratios, tail latency patterns, and synchronization costs remain applicable.
• Symmetric echo design: Both send-path and receive-path overhead are combined in each measurement. Asymmetric designs (small ACK response, or large server-initiated response) would isolate each direction independently — left as future work.
• Client-side profiling only: All perf stat, flame graph, and syscall measurements were collected on the client process. Server-side profiling would reveal server-side protocol overhead separately — left as future work.
• Go allocator threshold: Definitive confirmation of the 32KB hypothesis requires recompiling Go runtime with modified _MaxSmallSize. Two indirect experiments provide strong but indirect evidence.
• Warmup: One warmup request per client. gRPC's HPACK table initialization and HTTP/2 stream 0 setup may not be fully amortized in a single warmup request.