Network connectivity tuning to avoid network read/write failures and increase performance

TCP Offload Engine is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards.  This technology has been known to cause unstable connections.  This results in dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately slow or failed NetBackup backups.

There are two types of TCP offloads - stateful and stateless.

Stateful offload:
TCP Chimney is a stateful offload. TCP Chimney offload enables TCP/IP processing to be offloaded to network adapters that can handle the TCP/IP processing in hardware.

The Windows 2003 TCP Chimney setting and several Network Card (NIC) settings have been known to interfere with successful inter process communication and connectivity over TCP/IP networks.

Stateless offload:
RSS and NetDMA are stateless offloads. Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs.

To read more about this and other issues which are directly related to the introduction of Windows 2003 SP2 and the Scalable Networking Pack, examine these Microsoft Articles.
 

  https://support.microsoft.com/kb/912222 - The Microsoft Windows Server 2003 Scalable Networking Pack release
 

  https://support.microsoft.com/kb/945977 - Some problems occur after installing Windows Server 2003 SP2
 

  https://support.microsoft.com/kb/948496/en-us - An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
 

  https://support.microsoft.com/kb/936594/ - You may experience network-related problems after you install Windows Server 2003 SP2 or the Scalable Networking Pack on a Windows Server 2003-based computer
 

  https://support.microsoft.com/kb/939421 - Best practices and known issues when you install Windows Server 2003 Service Pack 2 on a Windows Small Business Server 2003-based computer
 

 

Note: In September 2008, Microsoft released a Scalable Networking Pack (SNP) for Windows 2003 which addresses several critical problems with NIC's running TCP Offload Engine and TCP Chimney Offload.  It is best to apply this SNP first, and only if problems persist, proceed with disabling the Offload Engines as directed below.
 

  https://support.microsoft.com/kb/950224 - A Scalable Networking Pack (SNP) hotfix rollup package is available for Windows Server 2003
 

 

Important Note For Servers In A Clustered Environment Or That Use Network Interface Card Teaming

It is of vital importance to determine if a server is a member of a cluster BEFORE making any changes to the TCP Offload Engine settings described in this article. Examples include Windows Server Failover Cluster nodes and SQL Always On Availability Groups replicas. Some cluster applications require TCP Offload Engine to be enabled on each cluster node or replica for proper functionality. Disabling any TCP Offload Engine settings on cluster nodes or replicas could adversely affect network performance for cluster-aware applications and/or operating systems. As such, it is recommended not to edit any TCP Offload Engine settings for servers that are nodes or replicas in a clustered environment without first consulting the cluster application documentation. If the cluster documentation clearly confirms TCP Offload Engine settings can be changed without any negative effects, then proceed with the changes after creating a plan to roll back the changes if needed. When in doubt, do NOT make any changes to the TCP Offload Engine settings.

Similar consideration should be given to servers using Network Interface Card (NIC) teaming. Some NIC teaming applications require TCP Offload Engine to be enabled on each NIC for proper functionality. Disabling any TCP Offload Engine settings on teamed NICs could adversely affect network performance for cluster-aware applications and/or operating systems. As such, it is recommended not to edit any TCP Offload Engine settings for servers that use NIC teaming without first consulting the NIC teaming documentation. If the NIC teaming documentation clearly confirms TCP Offload Engine settings can be changed without any negative effects, then proceed with the changes after creating a plan to roll back the changes if needed. When in doubt, do NOT make any changes to the TCP Offload Engine settings.

 

How to enable or disable TCP Offload at the operating system level:
 

The TCP chimney setting can be modified on systems running Windows 2003 SP2 or Windows 2003 SP1 with "The Microsoft Windows Server 2003 Scalable Networking Pack release" installed  To check the current setting, observe this registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

EnableTCPChimney
Type: REG_DWORD
Values: 1 (enabled) 0 (disabled)

This value can be enabled or disabled using these command line commands:
Netsh int ip set chimney ENABLED
Netsh int ip set chimney DISABLED

Veritas Support recommends setting TCP offload to Disabled.

How to enable or disable TCP Offload at the hardware/driver level:
In addition to what can be done through the Windows OS, individual network cards have attributes associated with their drivers which can toggle TCP offload settings from the main CPU to the processor on the NIC.

To list out which TCP offloads are enabled per network card, run this command:
Netsh int ip show offload

Here are examples of settings from 2 different NIC cards:

Example 1 output from Netsh int ip show offload :
Offload Options for interface "Local Area Connection" with index: 10005:
TCP Transmit Checksum
IP Transmit Checksum
TCP Receive Checksum
IP Receive Checksum
TCP Large Send

In order to toggle these settings, go to Windows Device Manager > Network Adapters > (NIC) > Advanced tab

Example 1 Advanced tab settings:
 

Example 1 map :
TCP Transmit Checksum = Offload Transmit TCP Checksum
IP Transmit Checksum     = Offload Transmit IP Checksum
TCP Receive Checksum   = Offload Receive TCP Checksum
IP Receive Checksum      = Offload Receive IP Checksum
TCP Large Send              = Offload TCP Segmentation



Example 2 output from Netsh int ip show offload :
Offload Options for interface "Local Area Connection" with index: 20003:
TCP Transmit Checksum
IP Transmit Checksum
TCP Receive Checksum
IP Receive Checksum
TCP Large Send

Example 2 Advanced tab settings:
 

Example 2 map :
TCP Transmit Checksum = Checksum Offload
IP Transmit Checksum     = Checksum Offload
TCP Receive Checksum   = Checksum Offload
IP Receive Checksum      = Checksum Offload
TCP Large Send              = Large Send Offload


Veritas Support recommends setting all these values to Off/Disabled/None  (or whatever value means "off")

NOTE: As can be seen above, different NIC manufacturers and drivers may name these attributes differently.  Experimentation can be done to identify which settings map to which names by changing the value in the Attribute section of the NIC properties and then rerunning Netsh int ip show offload to see what changed.

Note: Modifying any of these settings will temporarily disconnect the NIC from the network.  Care should be taken when making this adjustment so as not to disrupt any critical activity.  Also, consider directly interfacing with the machine rather then using Remote Desktop Control.

Note: Modifying these settings for most NIC's does not require a reboot to engage the setting.  But if this is a great concern, check with your NIC vendor to see if a reboot is necessary, or experiment on a non-production machine before modifying a production machine.

How to observe the Offload State of a TCP connection:
The command netstat with the switch " -t"  can be used to show the Offload State of a network connection.

Example output from the netstat -ab -t command:
Proto Local Address Foreign Address State      PID Offload State
TCP   server:epmap  server:0        LISTENING  816 InHost

Proto Local Address  Foreign Address  State       Offload State
TCP   Server:netbios name:2151        ESTABLISHED Offloaded

Flow Control Recommendations for NICs:
What is Flow Control?
"Flow Control" is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames. PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting.

It is recommended to set Flow Control on the NIC to "Generate and Respond".  Flow Control can also be modified within the Network Card Properties > Advanced tab.  See Example 3.

Example 3: