From Sockets to Packets: Inside the Network Stack

An application writes bytes to a socket, but the network transmits packets. Between those two views, the host kernel maintains connection state, copies or references buffers, divides streams into transport segments, wraps them in network and link headers, queues work for a driver, and coordinates with a network interface card. The receive path reverses that transformation while validating, ordering, and delivering data to the correct socket.

The useful mental model is a chain of bounded ownership boundaries. A successful socket write usually means the kernel accepted bytes, not that a peer received them. A packet accepted by a network interface is not yet an application message. When any stage cannot keep up, pressure must propagate backward or data will queue without bound. Following those boundaries explains partial writes, readable events, TCP flow control, packet drops, and why adding application concurrency can make a saturated host slower.

Treat a Socket as State, Not a Packet Pipe #

A socket is a kernel-managed communication endpoint exposed through a process-local handle. For TCP, the application sees an ordered byte stream. Calls such as write or send append bytes; calls such as read consume bytes. TCP does not preserve the application’s call boundaries. Two writes can be delivered by one read, and one write can require many reads. Applications that need records must define framing with lengths, delimiters, or a higher-level protocol.

The socket object connects several kinds of state:

• Local and remote addresses and ports identify the connection with its network protocol.
• Send and receive buffers hold bytes accepted from one side but not yet consumed by the other.
• Sequence numbers track which stream bytes have been sent, acknowledged, received, and delivered.
• Timers govern retransmission, delayed acknowledgments, keepalive when enabled, and connection teardown.
• Error and readiness state tells the application which operations can make progress.
• Routing, policy, and device references guide packets through this host’s stack.

A file descriptor is only one reference to that object. It can be duplicated, inherited, or shared across threads, so closing one descriptor need not destroy the connection immediately. Conversely, process exit lets the kernel release references and perform protocol teardown according to socket options and pending state.

Blocking mode changes what the calling thread does when a buffer cannot satisfy an operation. A blocking send may sleep until space is available. A nonblocking send returns the number of bytes accepted or indicates that progress would block. The protocol state and bounded buffers are the same; only the application-facing waiting strategy changes.

Readiness is a hint about current state, not a reservation. Between an event notification and the next call, another thread may consume data or an error may occur. Correct event-driven code drains operations until they would block, handles partial progress, and treats hang-up and error events as first-class outcomes.

Maintain TCP’s Connection and Sequence State #

TCP turns unreliable IP datagrams into an ordered, reliable byte stream between two endpoints. Establishment creates shared initial sequence-number state. The active side typically moves through a connect-in-progress state, while a listening host tracks half-open handshakes and creates an established child socket when the exchange completes. The listening socket remains separate and accepts completed connections from a bounded queue.

Once established, every byte occupies a sequence position. The sender tracks bytes written by the application, bytes transmitted, and bytes cumulatively or selectively acknowledged. The receiver tracks the next expected sequence and may retain out-of-order segments until a gap arrives. Duplicate data is recognized by sequence range rather than delivered twice to the application.

Two independent windows limit sending. The receiver advertises how much receive-buffer capacity it can currently accept, providing flow control. The sender also maintains a congestion window based on evidence about the network path. The usable in-flight amount is constrained by both. Receiver capacity protects one host’s socket; congestion control protects the path. Neither is an application-level queue limit.

Acknowledgment confirms transport-level receipt under TCP’s rules. It does not mean the peer application has read, parsed, persisted, or acted on those bytes. An application protocol needs its own acknowledgment when that stronger outcome matters.

Connection shutdown is directional. A graceful close sends an end-of-stream indication after queued data, and the peer can still send in the other direction until it also closes. States during teardown preserve retransmission and protect against delayed segments being mistaken for a later connection. An abortive reset instead reports failure and can discard pending data. Code that treats every end-of-stream as a complete application transaction risks accepting truncated records.

State consumes memory and timer work even when no application thread is running. Large numbers of idle, half-open, or closing connections can exhaust descriptor limits, socket memory, connection tracking, or accept queues. Timeouts and admission controls should reflect protocol and product semantics rather than serving as arbitrary cleanup values.

Trace the Send Path from Bytes to Segments #

Suppose an application calls send with a 100 KiB buffer on an established blocking TCP socket. The language runtime reaches the syscall boundary. The kernel resolves the descriptor, validates the user range and flags, checks socket state, and attempts to place data into the socket’s send memory. A common buffered path copies bytes into kernel-managed packet buffers, although zero-copy and page-referencing facilities can alter ownership and completion rules.

If sufficient send-buffer space exists, the call may accept all 100 KiB and return before transmission. If only 24 KiB fits, a nonblocking call may report 24 KiB; the application must retain and later submit the remaining 76 KiB. A blocking call can wait for acknowledgments and queue drainage to free space, but signals, deadlines, and errors can still produce partial progress. Retrying the original full buffer would duplicate the accepted prefix.

TCP assigns sequence ranges and decides when queued bytes are eligible for transmission. It considers the peer’s receive window, congestion state, pacing, outstanding data, and small-write policies. The transport constructs logical segments no larger than the effective maximum segment size for the route. IP adds source and destination addressing and other network-layer fields. The link layer prepares frames for the selected local interface and next hop.

This description is logical because offloads move work across the kernel/NIC boundary. With segmentation offload, the kernel can hand a large transport payload plus metadata to the NIC, which emits multiple wire-sized frames. Checksum offload similarly lets hardware fill or verify checksums. Packet captures taken above the hardware boundary can therefore display apparently oversized packets or incomplete checksums that never appear that way on the wire.

Routing determines the local egress interface and next hop for a destination. Neighbor discovery resolves the next hop’s link-layer address on applicable networks. Policy, firewall, tunneling, and virtual-interface stages may transform or reject traffic before it reaches the physical driver. This article follows what one host does after a connection endpoint is chosen; choosing among service instances is a separate load-balancing concern.

Hand Work from the Kernel to the NIC #

The network stack eventually places transmit descriptors into a driver-managed ring shared with the NIC. A descriptor points to packet data or a scatter/gather list and includes offload metadata. The driver notifies the device that new work is available. The NIC fetches descriptors and payload through direct memory access, performs requested offloads, and queues frames for physical transmission.

The ring is bounded. If software produces descriptors faster than the device reclaims them, the transmit queue stops accepting work. Queueing disciplines may shape, prioritize, or drop packets before the driver. The socket’s queued data then drains more slowly, eventually reducing available send-buffer space and applying pressure to the application. This is the desired direction of feedback.

Multiqueue NICs expose several transmit and receive rings so work can be distributed across CPUs. Hashing packet headers can keep one flow on a consistent receive queue while spreading different flows. CPU affinity for interrupts and packet processing influences cache locality. Poor placement can make packet data bounce between cores; overly rigid placement can overload one queue while others remain idle.

Transmit completion means the NIC has finished with a descriptor and the kernel can reclaim associated memory. It does not prove that the remote endpoint received the frame. TCP acknowledgment later advances transport state. For zero-copy sends, the completion notification that permits buffer reuse may likewise concern local ownership, not remote application processing.

Offloads trade host CPU work for batching and device complexity. They can substantially reduce per-packet processing, but they affect observability and can interact with tunnels, checksums, or unsupported features. Disabling them can help isolate a diagnostic question but changes the workload and should not become a default remedy without evidence.

Walk the Receive Path Back to a Socket #

On receive, the NIC writes frame data into buffers previously made available by the driver and records completed descriptors in a receive ring. Rather than interrupting the CPU for every packet under load, a common design uses interrupt moderation and polling: an interrupt schedules bounded packet processing, polling drains a batch, and interrupts are re-enabled when the queue becomes quiet. This trades a little latency at low traffic for much lower interrupt overhead at high traffic.

The host validates link, network, and transport headers as the frame moves upward. It checks destination and policy, handles IP-level processing, and identifies the TCP connection by protocol and endpoint tuple. TCP validates checksums as applicable, processes acknowledgments, updates windows and timers, orders payload by sequence number, merges adjacent ranges where possible, and places newly deliverable bytes in the socket receive queue.

Receive offloads can combine adjacent packets into larger in-kernel units before upper-layer processing. As on transmit, a capture point can show an implementation artifact rather than wire framing. Firewalls, virtual switches, containers, tunnels, and host routing add hooks and transformations, each with its own queues and drop counters.

When data reaches an empty socket queue, the kernel marks the socket readable and wakes an eligible waiter or makes an event-poll operation report readiness. The application still has to be scheduled. Wake-to-run delay can dominate even when packet processing is fast. Once running, read copies available stream bytes into application memory on a conventional path and frees receive-buffer capacity.

Freeing capacity allows the receiver to advertise a larger window. That update lets the peer send more, closing the transport backpressure loop. If the application stops reading, the receive queue fills, the advertised window shrinks, and the peer eventually cannot advance despite a healthy physical link. This is flow control working correctly, not packet loss.

Drops can occur before TCP owns the data: the NIC ring may have no available buffers, packet-processing backlog may overflow, policy may reject traffic, or memory allocation may fail. TCP can recover from some loss through retransmission, but recovery adds delay and work. Datagram protocols may expose loss directly because they do not provide stream retransmission.

Work Through a Backpressure Example #

Consider a TCP sender producing fixed 16 KiB chunks. Its application queue is bounded to 256 KiB, and the socket currently has 64 KiB of writable send capacity. The peer stops reading for a while, so acknowledgments may continue for already received data but its advertised receive window eventually approaches zero.

The first four chunks can fill the illustrative 64 KiB of socket capacity. Further nonblocking sends report that they would block. The application may hold at most sixteen additional chunks in its 256 KiB queue. Once both bounds are full, the producer must pause, reject work, spill according to an explicit durable policy, or cancel the connection. Continuing to allocate chunks merely relocates the queue into the heap.

chunk size                  = 16 KiB
currently writable socket  = 64 KiB  -> 4 chunks
bounded application queue  = 256 KiB -> 16 chunks
maximum accepted backlog   = 20 chunks in this snapshot

When the peer resumes reading, it frees receive memory and advertises more window. The sender transmits additional queued bytes; acknowledgments free send memory; writable readiness fires; and the application drains its queue until send would block again. A level-triggered loop should continue while progress is possible. An edge-triggered loop must be especially careful to drain fully or it may wait forever for an edge that already occurred.

The 320 KiB snapshot is not a universal connection bound. Kernel autotuning can change buffer limits, bytes may be in flight or queued in lower layers, framing adds state, and application objects can exceed payload size. The example instead establishes an ownership rule: every queue has a measured limit and a defined full behavior.

There are two common mistakes. Registering for writable events continuously when no data is pending causes a busy loop because sockets are often writable. Buffering an unlimited response per slow client lets a small number of readers consume the server’s memory. Enable write interest only with pending data, use high- and low-water marks, and propagate pause upstream before the hard limit.

Locate Backpressure and Failure Boundaries #

Several independent controls can stop progress, and each has different evidence:

Increasing a socket buffer can absorb bursts, but it also permits more queued bytes per connection and can increase latency before overload becomes visible. Larger rings can reduce drops while building deeper queues. More network threads can help until they compete for CPU, locks, and cache. Tuning should target a measured boundary and preserve an explicit memory budget across the maximum connection count.

TCP head-of-line behavior is inherent to its ordered stream: later bytes cannot be delivered past a missing earlier sequence range. Opening many connections can isolate streams but adds state and may evade intended fairness. Application multiplexing over one TCP stream needs its own policy for large records and slow consumers.

Failure handling must respect ambiguous progress. A connection reset after send accepted bytes does not tell the sender whether the peer application processed them. Reconnecting and replaying can duplicate effects. Consequential protocols need request identifiers, idempotent handling, and application acknowledgments. TCP provides reliable ordered transport for a connection, not exactly-once business execution.

Observe and Verify the Host Stack #

Begin at the application boundary. Record accepted bytes, partial operations, queue depth, pause duration, connection states, timeouts, and errors. At the socket layer, inspect send and receive queue occupancy, retransmission and window-limited indicators where available, accept-queue overflow, and memory pressure. At the interface, collect per-queue packets, bytes, drops, errors, ring exhaustion, and driver resets. CPU usage, soft-interrupt or packet-processing time, scheduling latency, and affinity complete the host picture.

Packet capture answers header, sequence, acknowledgment, and timing questions, but choose the capture point deliberately. Host offloads, namespaces, tunnels, and virtual interfaces can make two points show different packet shapes. Captures can contain secrets and impose overhead; filter narrowly, limit duration, and store them under access controls.

A focused verification plan for the worked example is:

1. Run a sender and a receiver on a disposable test network with explicit application queue limits.
2. Make the receiver pause reads while keeping the connection open.
3. Assert that sender memory remains bounded and the producer reaches its documented full behavior.
4. Observe the receive queue fill, advertised window contract, and sender writable state without assuming exact buffer sizes.
5. Resume reads and verify byte-for-byte framing, ordering, and eventual queue drainage.
6. Repeat with partial sends, resets, half-close, small buffers, and CPU pressure on the receive path.

Use sequence-numbered records and checksums as the correctness oracle. Inject fragmentation of application writes and reads because TCP is a byte stream. A load test should vary connection count and slow-reader proportion while tracking total socket memory, retransmissions, drops, queue latency, and completed application records. Any numerical result belongs to that documented host, kernel, NIC, offload configuration, and workload.

The host network stack is not a transparent tube. It is a set of protocol states and bounded queues that translate application bytes into packets and back. Socket calls transfer ownership at the user/kernel boundary, TCP tracks reliable sequence space and flow, segmentation and offloads shape work for the NIC, receive processing reconstructs deliverable bytes, and backpressure connects a slow reader all the way to the producer. Once those boundaries are visible, network behavior can be diagnosed as a specific queue, state, or handoff rather than an unexplained delay “somewhere on the wire.”