TCP Three-Way Handshake And Connection State

Why This Matters

A TCP connection starts with three segments: SYN, SYN-ACK, ACK. The SYN and FIN flags each consume one sequence number, even when they carry no payload. If the client sends SYN seq=0x9a7c1020, then its first data byte is numbered 0x9a7c1021.

Servers can hold thousands of half-open connections in SYN-RECEIVED. A SYN flood targets that memory. SYN cookies move enough state into the server's sequence number so the server can accept a later ACK without keeping a per-connection record.

Core Definitions

The Three Packets

Use client port 12345 and server port 443. Let the client choose ISN x = 0x9a7c1020, and the server choose ISN y = 0x3b021440.

The SYN flag consumes sequence space. No payload byte uses x; the first client payload byte uses x+1.

A minimal TCP header is 20 bytes. The following header bytes omit IP addresses and use a zero checksum field for readability. Byte 12 has data offset 5, meaning 5 * 4 = 20 header bytes. Byte 13 holds the flags.

Client SYN, source port 12345, dest port 443

30 39 01 bb   9a 7c 10 20   00 00 00 00   50 02 fa f0   00 00 00 00
|src |dst |   sequence      | acknowledgment |do fl|window|checksum|urg|
                              flags 0x02 = SYN

The SYN-ACK changes both sequence and acknowledgment fields.

Server SYN-ACK

01 bb 30 39   3b 02 14 40   9a 7c 10 21   50 12 fa f0   00 00 00 00
                              flags 0x12 = SYN + ACK

The final ACK has no SYN bit and does not consume sequence space.

Client ACK

30 39 01 bb   9a 7c 10 21   3b 02 14 41   50 10 fa f0   00 00 00 00
                              flags 0x10 = ACK

ISN Randomization And Blind Spoofing

Old TCP implementations used predictable ISNs, such as a global counter incremented every few microseconds. A blind attacker who cannot see replies could spoof a source IP, send a SYN, and then guess the server's SYN-ACK sequence number well enough to send the final ACK. If the receive window accepts $w$ sequence numbers, a random 32-bit guess succeeds with probability about $w / 2^{32}$.

For a 64 KiB window, $w = 65536$, so one guess succeeds with probability $2^{-16}$. That is too high if the attacker can send many guesses. RFC 6528-style ISNs use a secret and the four-tuple, with a time-varying component, so different connections do not reveal the next ISN for a spoofed connection.

A common form is:

Here $M(t)$ is a monotonically increasing clock-like value and $F_K$ is a keyed function. The exact function varies by implementation.

The TCP State Machine

TCP is more than just a send and receive loop. Every segment is interpreted through a state.

Passive open:
CLOSED -> LISTEN -> SYN-RECEIVED -> ESTABLISHED

Active open:
CLOSED -> SYN-SENT -> ESTABLISHED

Active close:
ESTABLISHED -> FIN-WAIT-1 -> FIN-WAIT-2 -> TIME-WAIT -> CLOSED

Passive close:
ESTABLISHED -> CLOSE-WAIT -> LAST-ACK -> CLOSED

Simultaneous close:
ESTABLISHED -> FIN-WAIT-1 -> CLOSING -> TIME-WAIT -> CLOSED

A listening server creates a half-open entry after receiving a SYN. That entry stores the client address, ports, ISNs, selected MSS, window scale, timestamps, and retransmission state for the SYN-ACK. If the final ACK arrives, the entry becomes an established connection and accept() can return a socket.

In a normal blocking server, the socket API exposes this split.

int fd = socket(AF_INET, SOCK_STREAM, 0);
bind(fd, (struct sockaddr *)&addr, sizeof(addr));
listen(fd, 128);

for (;;) {
    int cfd = accept(fd, NULL, NULL);   // returns after handshake completes
    // cfd is ESTABLISHED from the application's view
}

The backlog is not one queue in every kernel. Implementations often separate incomplete handshakes from completed sockets waiting for accept(). The specification describes observable TCP behavior, while the kernel picks data structures.

TIME-WAIT And Address Reuse

The endpoint that sends the first FIN usually enters TIME-WAIT after it receives the peer's FIN and sends the final ACK. The wait lasts $2 \cdot \mathrm{MSL}$. With MSL 60 seconds, TIME-WAIT lasts 120 seconds; with MSL 120 seconds, it lasts 240 seconds.

Two jobs justify the wait. First, the final ACK might be lost. If the peer retransmits its FIN, the TIME-WAIT endpoint can retransmit the ACK. Second, delayed packets from the old four-tuple must expire before a new connection with the same four-tuple starts.

The pain is port reuse. Suppose a client has 28,232 ephemeral ports for a fixed remote (IP, port). If it actively closes 500 connections per second and TIME-WAIT lasts 120 seconds, it has about

Connections in TIME-WAIT, more than the port range. Connection attempts to the same server can fail with address-in-use errors until old entries expire. SO_REUSEADDR changes binding rules for local addresses, but it does not make old sequence numbers harmless.

SYN Floods And SYN Cookies

A SYN flood sends many SYNs, often with spoofed source addresses. The server replies with SYN-ACKs and waits in SYN-RECEIVED. Each half-open entry consumes memory and timers. If the incomplete queue fills, real clients may not get service.

SYN cookies encode the minimum server state into the server ISN. A simplified layout looks like this:

32-bit server ISN under SYN cookies

bits 31..27   5-bit time counter
bits 26..24   encoded MSS option
bits 23..0    MAC over client IP, ports, server IP, time, secret

When the final ACK arrives, the server subtracts 1 to recover the cookie. It checks the time and MAC. If they match, it reconstructs a connection without having stored a half-open entry.

Worked example with a toy 8-bit MAC, not a real cookie:

time = 17
mss_index = 2
mac = 0x35a912

cookie = (17 << 27) | (2 << 24) | 0x35a912
       = 0x8a35a912

server sends SYN-ACK seq = 0x8a35a912
client sends final ACK ack = 0x8a35a913
server verifies ack - 1 = 0x8a35a912

The tradeoff is options. If the server stores no half-open state, it can only recover options encoded in the cookie. Some TCP options may be disabled or reduced while cookies are active.

Windows, Congestion Control, And Nagle

TCP uses a sliding receive window and a congestion window. The sender's usable amount is bounded by both:

Let MSS be 1460 bytes, rwnd = 65535, cwnd = 10 * MSS = 14600, and bytes in flight be 8760. Then

The sender can transmit four full-sized segments, because $4 \cdot 1460 = 5840$.

Reno grows cwnd by roughly one MSS per round-trip during congestion avoidance and halves it on loss. CUBIC uses a cubic function of time since the last loss, which is less tied to RTT than Reno. BBR estimates bottleneck bandwidth and minimum RTT, then paces at an estimated bandwidth-delay product. These algorithms change cwnd or pacing; they do not change the three-way handshake.

Nagle's algorithm reduces tiny segments. If an application writes 1 byte at a time and unacknowledged data is already in flight, TCP holds later small writes until an ACK arrives or enough data accumulates for an MSS. Latency-sensitive protocols often disable it.

int one = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one));

TCP_NODELAY does not disable congestion control, retransmission, or the receive window. It only changes coalescing of small writes.

Connection Teardown And Half-Close

FIN consumes one sequence number, just like SYN. If a client has sent bytes through sequence 2000, its FIN uses seq=2001. The server ACKs 2002.

A half-close means one direction is closed while the other direction remains open. In C, shutdown(fd, SHUT_WR) sends FIN but still allows reads.

send(fd, request, request_len, 0);
shutdown(fd, SHUT_WR);          // send FIN, no more writes

while ((n = recv(fd, buf, sizeof(buf), 0)) > 0) {
    fwrite(buf, 1, n, stdout);  // peer can still send data
}
close(fd);

This pattern matters for protocols where end-of-input is signaled by EOF on the byte stream. close(fd) closes both directions from the application view; shutdown(SHUT_WR) preserves the read side.

Key Result

The TCP connection model is a pair of sequenced byte streams controlled by state, windows, and timers.

1. SYN and FIN each consume one sequence number.
2. ACK values name the next expected sequence number.
3. A sender may transmit at most $\min(\mathrm{rwnd}, \mathrm{cwnd}) - \mathrm{flight}$ new bytes.
4. TIME-WAIT lasts $2 \cdot \mathrm{MSL}$ so the final ACK can be retransmitted and old duplicates can expire.
5. The four-tuple cannot be treated as fresh until old packets for that four-tuple are outside the sequence and time windows.

These invariants explain the visible behavior: accept() returns only after the final ACK, FIN-WAIT-2 can persist when the peer never sends FIN, and a just-closed client port can remain unusable for the same server.

Common Confusions

Exercises

References

Canonical:

• W. Richard Stevens and Kevin R. Fall, TCP/IP Illustrated, Volume 1: The Protocols (2nd ed., 2011), ch. 13, ch. 14, ch. 17 - TCP connection establishment, timeout behavior, and options
• Andrew S. Tanenbaum and David J. Wetherall, Computer Networks (5th ed., 2011), §6.5 - TCP service model, state machine, windows, and congestion overview
• W. Richard Stevens, Bill Fenner, and Andrew M. Rudoff, UNIX Network Programming, Volume 1: The Sockets Networking API (3rd ed., 2003), ch. 4, ch. 7 - socket calls, listen, accept, shutdown, and socket options
• Douglas E. Comer, Internetworking with TCP/IP, Volume 1 (6th ed., 2013), ch. 13 - TCP segments, sequence numbers, and connection management
• James F. Kurose and Keith W. Ross, Computer Networking: A Top-Down Approach (8th ed., 2021), §3.5, §3.7 - TCP connection management and congestion control survey

Accessible:

• RFC 9293, Transmission Control Protocol (TCP) - current TCP specification that consolidates earlier TCP RFCs
• RFC 6528, Defending against Sequence Number Attacks - ISN generation for blind-spoofing defense
• Dan J. Bernstein, SYN Cookies - original public description of the SYN cookie mechanism

Next Topics

• /computationpath/sockets-api
• /computationpath/tcp-congestion-control
• /computationpath/epoll-and-event-loops