LLT supports Jumbo Frames

Problem

LLT supports Jumbo Frames

Solution

LLT supports Jumbo Frames

LLT queries the NICs configured under it for the mtu size LLT should use while transmitting packets over these NICs. If the configured mtu sizes of the NICs configured under LLT are different, LLT uses the lowest mtu among them for transmitting packets on all these NICs. GigE or 10GigE NICs support jumbo mtu sizes of around 9192 bytes. When LLT is configured over such interfaces, LLT can use mtu size of 9192 bytes. Thus, LLT supports jumbo frames.

If the underlying interfaces support jumbo frames, LLT will start making use of them automatically, without the user having to change any settings. If for any reason it is necessary to turn off the use of jumbo frames, please refer to the steps described later in the document.

How to identify whether LLT is using jumbo frames or not?

We can check whether LLT is making use of jumbo frames by checking the value of mtu by querying LLT. The exact commands appear below. A value of 1500 indicates that LLT is currently not using jumbo frames. A value of about 9190 indicates that LLT is using jumbo frames.

Example of LLT not using jumbo frames:

[root@vcslx007 ~]# /opt/VRTSllt/lltshow -l -1 | egrep "valid= 1|mtu=" | grep -v "mtu= 0"
valid= 1  usable= 3  tag= "lan1"
sap= 0xCAFE (51966)  mtu= 1500  addrlen= 6
valid= 1  usable= 3  tag= "lan2"
sap= 0xCAFE (51966)  mtu= 1500  addrlen= 6
valid= 1  usable= 3  tag= "lan3"
sap= 0xCAFE (51966)  mtu= 1500  addrlen= 6

Example of LLT using jumbo frames:

[root@vcslx007 ~]# /opt/VRTSllt/lltshow -l -1 | egrep "valid= 1|mtu=" | grep -v "mtu= 0"
valid= 1  usable= 3  tag= "lan1"
sap= 0xCAFE (51966)  mtu= 9192  addrlen= 6
valid= 1  usable= 3  tag= "lan2"
sap= 0xCAFE (51966)  mtu= 9192  addrlen= 6
valid= 1  usable= 3  tag= "lan3"
sap= 0xCAFE (51966)  mtu= 9192  addrlen= 6

To see what mtu value LLT is providing to its clients, use the following command. This value depends on the minimum mtu among all the links and the space required for LLT's own packet header.

[root@vcslx007 ~]# lltstat -c | grep "overall mtu"
   overall mtu: 1452


Known Symptoms of network mis-configuration while using LLT over Jumbo Frames


Not all switches support jumbo frames, and thus they drop packets which are more than their supported max mtu size. All switches support mtu size of 1500 bytes. Also, not all NICs support jumbo frames, and therefore they also drop packets which are more than their supported max mtu size.

Due to such dropped packets, we can see the following symptoms:

·      The HA Daemon (had) will not properly start on more than one node at a time. The large cluster config message exceeds 1500 bytes, and is dropped by the switch during the membership change protocol.
·      The I/O Fencing (vxfen) driver will not initialize on more than one node at time.

While the above symptoms don't definitely indicate that the underlying network infrastructure does not support jumbo frames, they are good indicators of a network misconfiguration and administrators should try the below steps and check whether the above mentioned symptoms go away.

Steps to resolve such mis-configuration issues

To ensure that using LLT with jumbo frames (i.e. large mtu size of around 9192 bytes) will work across the cluster,

·      It's necessary to ensure that all networking elements (switches, NICs, etc.) between any two network end points support jumbo frames and are configured to accept jumbo frames.

If some of the networking elements (switch, peer NIC, etc.) can't be configured to accept jumbo frames,

·      Configure LLT to use a lower mtu size by specifying it in the /etc/llttab file. E.g.

From:
link nxge0 /dev/nxge:0 - ether - -

To:
link nxge0 /dev/nxge:0 - ether - 1500

This change should be done for all NICs configured under LLT in the /etc/llttab and LLT should be restarted.

OR

·      You may also be able to restrict the layer 2 MTU size in the NIC Driver. The instructions for 'How to enable/disable Jumbo frames' should be obtained from the particular O/S vendor for the machine being configured.