How to Determine Maximum Segment Size (mss) in Tcp/ip

by

Table of Contents

Understanding Maximum Segment Size (MSS) in TCP/IP Networks

Maximum Segment Size (MSS) is a parameter of the Options field of the TCP header that specifies the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. Understanding and properly configuring MSS is crucial for optimizing network performance, preventing packet fragmentation, and ensuring reliable data transmission across TCP/IP networks. This comprehensive guide explores what MSS is, how it works, and the various methods and tools you can use to determine and optimize MSS values in your network infrastructure.

What Is Maximum Segment Size (MSS)?

MSS (maximum segment size) limits the size of packets, or small chunks of data, that travel across a network, such as the Internet. More specifically, MSS measures the non-header portion of a packet, which is called the payload. This is an important distinction because it does not count the TCP header or the IP header (unlike, for example, the MTU for IP datagrams).

The MSS is agreed on during the TCP handshake: both devices communicate the size of the packets they are able to receive. This negotiation happens at the beginning of every TCP connection and the value cannot be changed after the connection is established. Each endpoint announces its own MSS value independently, and the communication uses the appropriate MSS for each direction of data flow.

MSS vs. MTU: Understanding the Difference

While MSS and Maximum Transmission Unit (MTU) are closely related concepts, they operate at different layers of the network stack. MSS is a layer 4, or transport layer, metric. It is used with TCP, a transport layer protocol. However, it is closely related to MTU, which applies to layer 3 (the network layer).

A critical difference in behavior is that if a packet exceeds a device’s MTU, it is broken up into smaller pieces, or “fragmented.” In contrast, if a packet exceeds the MSS, it is dropped and not delivered. This makes proper MSS configuration essential for avoiding packet loss and ensuring smooth network operations.

How MSS Is Negotiated During TCP Connection Establishment

The MSS negotiation process occurs during the TCP three-way handshake, which establishes a connection between two devices. MSS negotiation occurs during the initial phase of a TCP connection: SYN Packet: The initiating device sends a SYN packet containing its MSS value. SYN-ACK Packet: The receiving device responds with a SYN-ACK packet containing its MSS value.

Where a host wishes to set the maximum segment size to a value other than the default, the maximum segment size is specified as a TCP option, initially in the TCP SYN packet during the TCP handshake. This announcement mechanism allows each endpoint to communicate its receiving capabilities to the other side.

The MSS can be used completely independently in each direction of data flow. The result may be quite different maximum sizes in the two directions. This asymmetric capability allows for optimized performance in scenarios where the two endpoints have different network characteristics or constraints.

Default MSS Values

When no MSS option is explicitly specified during the TCP handshake, default values are used. The default TCP Maximum Segment Size for IPv4 is 536. For IPv6 it is 1220. These conservative default values ensure basic compatibility across diverse network paths, though they are rarely optimal for modern networks.

Calculating MSS from MTU

The MSS value is typically derived from the Maximum Transmission Unit (MTU) of the network interface. Understanding this relationship is essential for proper network configuration and troubleshooting.

The Basic MSS Calculation Formula

The MSS value to send is: MSS = MTU − TCPHdrLen − IPHdrLen This formula accounts for the overhead introduced by both the TCP and IP headers that must be included in every packet.

TCP headers are almost always 20 bytes long. IP headers include information such as the source and destination IP address, and they are also 20 bytes long. Therefore, for standard headers without options, the total overhead is 40 bytes.

Standard Ethernet MSS Calculation

For standard Ethernet networks, a network router has an MTU of 1,500, meaning it only accepts packets up to 1,500 bytes long. The router’s MSS should be set to 1,460 bytes. This calculation follows the formula: 1500 (MTU) – 20 (IP header) – 20 (TCP header) = 1460 bytes MSS.

With an MTU of 1500 bytes and 40 bytes for IP and TCP headers, the typical MSS for Ethernet is 1460 bytes. This is the most common MSS value you’ll encounter in modern TCP/IP networks.

Accounting for TCP Options

While the basic calculation assumes 20-byte headers, real-world scenarios can be more complex. TCP options size (Variable 0–320 bits, in units of 32 bits) must be deducted from MSS size if TCP options are enabled. For example, TCP Time Stamps are enabled by default on Linux platforms. When TCP timestamps are in use, the maximum MSS value should be 1460. However this value may be lowered to 1448 due to TCP Timestamps being used.

Methods to Determine MSS in Your Network

There are several practical methods and tools you can use to determine the MSS values being used in your network. Each method has its own advantages and use cases.

Method 1: Using Wireshark for Packet Analysis

Wireshark is one of the most powerful tools for analyzing TCP connections and determining MSS values. The MSS option is found only in the SYN and SYN/ACK packets of the TCP connection establishment three-way handshake. This makes it straightforward to identify MSS values in captured traffic.

To determine MSS using Wireshark:

  1. Start a packet capture on the network interface you want to analyze
  2. Initiate a TCP connection (such as browsing to a website or starting a file transfer)
  3. Stop the capture and filter for TCP traffic using the filter: tcp.flags.syn == 1
  4. Examine the SYN and SYN-ACK packets in the three-way handshake
  5. Expand the TCP options section in the packet details pane
  6. Look for the “Maximum segment size” option, which will display the MSS value in bytes

MSS stands for Maximum Segment Size which specifies the largest amount of data that a device can receive in a single TCP segment. The TCP MSS does not include the IP header or the TCP header. MSS is negotiated in the three way handshake between the client and the server at the beginning of a TCP connection.

Method 2: Analyzing MSS Values in Network Performance

You can analyze the MSS value for insight into network performance and see how efficiently data is being transferred. When troubleshooting performance issues, anything less than 1,000 bytes is worth investigating. A server responds with a MSS of 512, which is much smaller than the 1,460 than we were expecting. This means that you will have to send three 512-byte segments instead of one 1,460-byte segment.

Small MSS values can significantly impact network efficiency by requiring more packets to transmit the same amount of data, increasing overhead and potentially reducing throughput.

Method 3: Using Ping Tests to Determine Path MTU

While ping doesn’t directly show MSS, it can help you determine the Path MTU, from which you can calculate the appropriate MSS. You can ping the end device with the “don’t fragment” bit set. This will only work if ICMP is supported by all the intermediate devices.

To find the optimum MSS/MTU value of the network, do the ‘ping test’ to find the MTU, and then reduce 40 from that value and set that as tcp-mss in the firewall policy. For a 1500-byte MTU, 1500-28 (20 bytes=IP header and 8 bytes=ICMP header) = 1472.

On Windows, you can use the following command:

ping -f -l 1472 destination_address

On Linux or macOS:

ping -M do -s 1472 destination_address

If the packet needs to be fragmented but the DF bit is set, the MTU is less than 1500 bytes. Suppose it is possible to ping with a size of 1400 but not with a size of 1401, this means that the MTU value is 1400+28=1428 bytes. The solution is either to fix the MTU in the network device or to set the TCP MSS in the firewall policy to 1428-40=1388 bytes.

Method 4: Checking Device and Router Configurations

Many network devices allow you to view and configure MSS settings directly. On Cisco routers, for example, you can view the MTU of an interface using:

show interface [interface-name]

This command displays the current MTU configuration, from which you can calculate the expected MSS value. Different vendors have different commands and configuration options for viewing and adjusting these parameters.

Method 5: Using Network Diagnostic Tools

Several specialized network diagnostic tools can report MSS values and help with Path MTU Discovery. You can check the PMTU and MSS values that each agent is using when connecting to a target. Tools like ThousandEyes, PathPing, and tracepath can provide valuable insights into the MSS and MTU characteristics of network paths.

MSS in Special Network Scenarios

Different network configurations and technologies require special consideration when determining and configuring MSS values.

PPPoE Networks

The PPP over Ethernet (PPPoE) standard supports an MTU of only 1492 bytes. The disparity between the host and PPPoE MTU size can cause the router in between the host and the server to drop 1500-byte packets and terminate TCP sessions over the PPPoE network.

The optimum value for the max-segment-size argument is 1452 bytes. This value plus the 20-byte IP header, the 20-byte TCP header, and the 8-byte PPPoE header add up to a 1500-byte packet that matches the MTU size for the Ethernet link.

VPN and IPsec Tunnels

VPN connections add additional headers that reduce the available space for data. In VPNs, MSS is adjusted to account for additional encapsulation headers, preventing fragmentation. The exact overhead depends on the encryption and authentication algorithms used.

There is an extra IPSec header. It depends on the encryption and authentication algorithm. The AES-256 and SHA1 have a maximum of 73 bytes, and another algorithm might have slightly different bytes. For a 1500-byte MTU, the MSS for a TCP over IPSec tunnel is 1500-20 (IP header) -20 (TCP header)-73 (IPsec header) = 1387.

GRE Tunnels

Generic Routing Encapsulation (GRE) tunnels add their own overhead. In the case of GRE tunneling, a 24-byte header is added to the original packet in order to send it to a new destination. If the original packet was larger than 1,476 bytes, this could make the new packet exceed the typical 1,500-byte MTU.

For GRE tunnels, the recommended MSS is typically 1436 bytes (1500 MTU – 40 bytes TCP/IP headers – 24 bytes GRE header = 1436 bytes).

Jumbo Frames

For Ethernet v2, the MTU is 1500 bytes, but if Jumbo Frames are used the MTU can go up to 9000 bytes. When jumbo frames are enabled across the entire network path, the MSS can be increased proportionally, allowing for more efficient data transfer by reducing the number of packets needed to transmit large amounts of data.

Understanding and Implementing MSS Clamping

MSS clamping is a technique used to prevent fragmentation and packet loss by adjusting the MSS value advertised during TCP connection establishment.

What Is MSS Clamping?

MSS clamping can be achieved by configuring the server to apply an MSS clamp: during the TCP handshake, the server can signal the MSS for packets it is willing to receive, “clamping” the maximum payload size from the other server. This technique is particularly useful when intermediate devices add encapsulation headers that reduce the effective MTU.

TCP MSS clamping enables you to reduce the maximum segment size (MSS) value used by a TCP session during a connection establishment through a VPN tunnel. Network devices such as routers and firewalls can intercept TCP SYN packets and modify the MSS option to a lower value that accommodates the actual path constraints.

Why MSS Clamping Is Necessary

Occasionally, a router along a network path has an MTU value set lower than the typical 1,500 bytes. This can result in packet loss and can be difficult to discover. MSS clamping provides a proactive solution to this problem.

When a TCP traffic goes through any kind of VPN tunnel, additional headers are added to the original packet to keep it secure. For IPSec tunnel mode, additional headers used are IP, ESP, and optionally UDP (if a port translation is present in the network). Because of these additional headers, the size of the encapsulated packet goes beyond the MTU of the VPN interface.

Configuring MSS Clamping on Different Platforms

Cisco IOS Routers:

Use the ip tcp adjust-mss command in interface configuration mode to specify the MSS value on the intermediate router of the SYN packets to avoid truncation. The command is applied in interface configuration mode:

interface GigabitEthernet0/0
ip tcp adjust-mss 1452

The ip tcp adjust-mss command is effective only for TCP connections passing through the router. This means it works bidirectionally on traffic traversing the interface.

Linux with iptables:

To clamp MSS to path MTU (dynamic, recommended), use: iptables -t mangle -A FORWARD -p tcp –tcp-flags SYN,RST SYN -j TCPMSS –clamp-mss-to-pmtu This automatically adjusts the MSS based on the interface MTU.

For a specific MSS value:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380

Juniper Networks:

To diminish the likelihood of fragmentation and to protect against packet loss, you can decrease the TCP MSS size using the tcp-mss statement. Configuration is done at the interface or routing level depending on the platform.

Best Practices for MSS Clamping

  • Apply clamping symmetrically: Configure MSS clamping on both ends of a tunnel or VPN connection to ensure consistent behavior in both directions
  • Use dynamic clamping when possible: Automatic MSS clamping based on interface MTU reduces configuration errors and adapts to changes
  • Account for all encapsulation overhead: Calculate the total overhead from all protocols in the path (VPN, tunneling, VLAN tags, etc.)
  • Test thoroughly: After implementing MSS clamping, verify that connections work properly and that no fragmentation occurs
  • Document your settings: Keep clear records of MSS clamping configurations and the reasons for specific values

Path MTU Discovery (PMTUD) and Its Relationship to MSS

Path MTU Discovery is a mechanism that allows hosts to determine the maximum MTU along a network path, which directly affects the optimal MSS value.

How PMTUD Works

To avoid fragmentation, end-hosts typically run PMTUD for each destination, or are able to process ICMP “Packet too big” messages (default on Linux), and adjust the maximum TCP segment size (MSS) accordingly. The process involves sending packets with the Don’t Fragment (DF) bit set and listening for ICMP error messages that indicate when a packet is too large for a link in the path.

PMTUD Challenges and Limitations

Unfortunately, PMTUD doesn’t always work reliably in real-world networks. Many firewalls and security devices block ICMP messages, including the “Packet Too Big” messages that PMTUD relies on. This can create “black holes” where packets are silently dropped without any notification to the sender.

When PMTUD can’t be performed, you’ll end up with either fragmented packets, dropped packets, or both. Rather than trying to change the MTU on the host itself, which is tedious and error prone, an intermediate device can inspect TCP packets and modify the MSS that is signaled to a value that fits within the MTU of the path. For example, if there is encryption and tunneling, you may lose close to a 100 bytes in some scenarios and many people would configure a MTU of 1400 and MSS of 1360 in such scenarios.

MSS Clamping as a PMTUD Alternative

Another (perhaps more efficient) technique is MSS clamping, where middle boxes actually change the value of MSS in active TCP connections. This approach works even when ICMP is blocked and provides a more reliable solution for preventing fragmentation in complex network environments.

MSS misconfigurations can cause a variety of network problems. Understanding how to identify and resolve these issues is essential for network administrators.

Common MSS Problems

Symptom: Connections establish but data transfer fails or is very slow

This classic symptom often indicates an MSS or MTU mismatch. The TCP handshake completes successfully (using small packets), but when larger data packets are sent, they exceed the path MTU and are dropped. If the DF (do not fragment) bit is set, then the packet can be dropped, which can cause delays or slowness in the network.

Symptom: Some websites work while others don’t

This can occur when some servers send smaller packets that fit within the path MTU while others send larger packets that get dropped. The inconsistency makes this particularly difficult to diagnose without packet analysis tools.

Symptom: VPN connections are unstable or slow

VPN tunnels add overhead that reduces the effective MTU. If MSS is not adjusted to account for this overhead, packets will be fragmented or dropped, causing poor performance or connection failures.

Diagnostic Steps

  1. Capture and analyze TCP handshakes: Use Wireshark to examine the MSS values being negotiated. Look for unusually small MSS values or mismatches between the two directions.
  2. Test with different packet sizes: Use ping with the DF bit set and varying packet sizes to determine the actual path MTU.
  3. Check for ICMP filtering: Verify that ICMP messages, particularly “Packet Too Big” messages, can reach the endpoints. Blocked ICMP can prevent PMTUD from working.
  4. Review intermediate device configurations: Check routers, firewalls, and VPN gateways for MTU and MSS settings. Look for misconfigurations or missing MSS clamping rules.
  5. Test with MSS clamping: Temporarily implement MSS clamping with a conservative value (such as 1360 bytes) to see if it resolves the issue.

Using Wireshark for MSS Troubleshooting

Wireshark provides powerful capabilities for diagnosing MSS-related problems. Key things to look for include:

  • TCP retransmissions: Excessive retransmissions may indicate packets are being dropped due to MTU/MSS issues
  • ICMP “Fragmentation Needed” messages: These indicate that packets are too large for a link in the path
  • Sequence number gaps: Missing sequence numbers suggest packet loss, which could be due to oversized packets
  • MSS values in SYN packets: Compare the advertised MSS values with what you expect based on the network configuration

MSS Optimization Strategies

Optimizing MSS values can significantly improve network performance and reliability. Here are key strategies to consider.

Balancing Efficiency and Compatibility

TCP throughput is primarily affected by the MSS, round-trip time, and average packet loss probability, with the segment size ideally set as large as possible to maximize throughput and minimize fragmentation, provided the network link supports the required MTU.

Larger MSS values improve efficiency by reducing the number of packets needed to transfer data and minimizing per-packet overhead. However, they must be balanced against the constraints of the network path to avoid fragmentation and packet loss.

Environment-Specific Recommendations

Standard Internet connections: Use MSS of 1460 bytes (for 1500-byte MTU)

PPPoE connections: Use MSS of 1452 bytes (for 1492-byte MTU)

IPsec VPNs: Use MSS of 1360-1400 bytes depending on encryption overhead

GRE tunnels: Use MSS of 1436 bytes (for 1476-byte effective MTU)

Multiple encapsulations: Calculate cumulative overhead and subtract from base MTU

Testing and Validation

Testing with Path MTU Discovery (PMTUD): PMTUD helps determine the smallest MTU size along a path, ensuring MSS is aligned with the network’s capabilities. Configuring MSS Clamping: Routers and firewalls can use MSS clamping to adjust MSS values dynamically, accommodating network constraints.

After implementing MSS changes, thoroughly test various scenarios including:

  • Large file transfers
  • Web browsing to various sites
  • Video streaming
  • VoIP or real-time communications
  • Application-specific protocols used in your environment

Advanced MSS Concepts

Asymmetric MSS Values

MSS can have a different value in either direction and that frames exceeding the MTU may cause packets (which encapsulate segments) to be fragmented by the network layer, while segments exceeding the MSS are simply discarded. This asymmetry allows each endpoint to optimize for its own network conditions and capabilities.

In practice, asymmetric MSS values are common when connecting devices with different network characteristics, such as a mobile device on a cellular network communicating with a server on a high-speed data center network.

MSS and Network Performance

The MSS value directly influences data transfer efficiency by limiting the amount of data in each TCP segment, which is particularly important for networks with constraints on packet size. The relationship between MSS, throughput, and network efficiency is complex and depends on multiple factors including round-trip time, packet loss rate, and congestion control algorithms.

Reducing the MSS increases the number of packets and introduces extra TCP headers, which decreases network efficiency by consuming bandwidth on headers that were not originally needed. More packets also mean more processing, which can lower overall data throughput. This is why finding the optimal MSS value for your specific network environment is important.

MSS in Complex Network Topologies

Modern networks often involve multiple layers of encapsulation and tunneling. If two or more overlay technologies run in sequence (e.g., HCX followed by NSX), each adds overhead. Identifying the largest possible encapsulated packet size from all initiators helps determine if the chosen MTU accommodates all overheads. If overhead still causes fragmentation, disabling double encryption (when not required) reduces overhead, minimizing the need to increase MTU further.

In such environments, careful planning and calculation of cumulative overhead is essential. Each layer of encapsulation must be accounted for when determining the appropriate MSS value.

Security Considerations

While MSS is primarily a performance and compatibility parameter, it also has security implications that network administrators should be aware of.

MSS-Based Attacks

Attackers can potentially exploit MSS handling in various ways:

  • MSS manipulation: Malicious actors might advertise extremely small MSS values to degrade performance or cause resource exhaustion
  • Fragmentation attacks: By manipulating MSS and MTU values, attackers might trigger fragmentation that can be used for evasion or denial of service
  • Fingerprinting: MSS values can be used as part of OS fingerprinting techniques to identify systems

Protective Measures

  • Implement minimum MSS values on edge devices to prevent extremely small values
  • Monitor for unusual MSS patterns that might indicate attacks or misconfigurations
  • Use stateful firewalls that can validate MSS values against expected ranges
  • Ensure ICMP filtering doesn’t completely block legitimate PMTUD messages while still protecting against ICMP-based attacks

Future Developments and Considerations

As networks continue to evolve, MSS handling and optimization will remain important. Several trends are shaping the future:

  • IPv6 adoption: IPv6 has different default MSS values and header sizes, requiring updated calculations and configurations
  • Increased encryption: More widespread use of encryption adds overhead that must be accounted for in MSS calculations
  • Software-defined networking: SDN technologies enable more dynamic and automated MSS optimization
  • Cloud and edge computing: Diverse network paths and varying MTU support require flexible MSS handling
  • 5G and mobile networks: Mobile networks often have different MTU characteristics that affect optimal MSS values

Conclusion

Determining and optimizing Maximum Segment Size (MSS) is a critical aspect of TCP/IP network management. Proper MSS configuration prevents fragmentation, reduces packet loss, and ensures efficient data transmission across diverse network environments. By understanding the relationship between MSS and MTU, using appropriate diagnostic tools like Wireshark and ping tests, and implementing MSS clamping where necessary, network administrators can significantly improve network performance and reliability.

Whether you’re managing a simple network or a complex infrastructure with multiple layers of encapsulation, the principles outlined in this guide w