NFS file lock operation hangs when NFS share is mounted using a certain NFS server IP addresses using FileStore
Problem
NFS file lock operation hangs when NFS share is mounted using a certain FileStore NFS server IP addresses.
Error Message
The NFS client application issuing the NFS file lock request hangs.
On a Linux NFS client, an strace of the program issuing the file lock request will show the following:
nfs_client$ strace -fFr /usr/bin/flock fileA -c /bin/true
....
open("fileA", O_RDONLY|O_CREAT|O_NOCTTY, 0666) = 3
flock(3, LOCK_EX <<< the NFS client process hangs here
Cause
In FileStore versions up to 5.7P1, NFS file lock requests are serviced by one dedicated lock serving node in the FileStore cluster. NFS file lock requests received by the other nodes will be forwarded to the dedicated lock serving node. The forwarding is implemented by iptables Network Address Translation (NAT) rules.
For example:
In the following two-node FileStore cluster, the dedicated NFS file lock serving node is sydn8k_02 and the iptables settings will look like the following. NFS file lock requests received by sydn8k_01 will be forwarded to sydn8k_02. Port 4045 is the Network Lock Manager (NLM) port.
sydn8k_01# iptables -L -t nat -nv Chain PREROUTING (policy ACCEPT 44534 packets, 4528K bytes) pkts bytes target prot opt in out source destination 0 0 DNAT udp -- bond+ * 0.0.0.0/0 0.0.0.0/0 udp dpt:4045 to:172.24.134.2:4045 (172.24.134.2 is sydn8k_02) 0 0 DNAT tcp -- bond+ * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4045 to:172.24.134.2:4045 0 0 DNAT udp -- pubeth+ * 0.0.0.0/0 0.0.0.0/0 udp dpt:4045 to:172.24.134.2:4045 0 0 DNAT tcp -- pubeth+ * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4045 to:172.24.134.2:4045 Chain POSTROUTING (policy ACCEPT 322 packets, 21712 bytes) pkts bytes target prot opt in out source destination 0 0 SNAT udp -- * priveth0 0.0.0.0/0 172.24.134.2 udp dpt:4045 to:172.24.134.120 (172.24.134.120 is sydn8k_01) 0 0 SNAT tcp -- * priveth0 0.0.0.0/0 172.24.134.2 tcp dpt:4045 to:172.24.134.120 Chain OUTPUT (policy ACCEPT 322 packets, 21712 bytes) pkts bytes target prot opt in out source destination |
The following NAT rules on sydn8k_02 are not related to NLM. They are for the web servers (Port 80 and 8080).
sydn8k_02# iptables -L -t nat -nv Chain PREROUTING (policy ACCEPT 71209 packets, 7452K bytes) |
rpcinfo shows that Port 4045 is for nlockmgr.
# rpcinfo -p program vers proto port service ... 100021 1 udp 4045 nlockmgr 100021 3 udp 4045 nlockmgr 100021 4 udp 4045 nlockmgr 100021 1 tcp 4045 nlockmgr 100021 3 tcp 4045 nlockmgr 100021 4 tcp 4045 nlockmgr ... |
Due to the enabling of Reverse Path Filtering (rp_filter) some of these lock requests fail to reach the NFS file lock serving node and causes the NFS client application issuing the lock request to hang. The lock forwarding will only work for the first public network interface on the node receiving the lock request. Lock requests received from other interfaces will be blocked by rp_filter. The problem doesn't affect the dedicated lock serving node because forwarding is not required.
For example:
sydn8k_01 has 4 public networking interface. Only those lock requests received through pubeth0 (10.251.63.67) and pubeth0:0 (10.251.63.81) will be forwarded successfully. Other lock requests received through pubeth1 to pubeth3:0 will be blocked and the NFS client application will hang.
sydn8k_01# ifconfig -a | egrep '^[^ ]|inet addr:' .... pubeth0 Link encap:Ethernet HWaddr 84:2B:2B:4C:B9:30 <<< first public interface inet addr:10.251.63.67 Bcast:10.251.63.255 Mask:255.255.248.0 pubeth0:0 Link encap:Ethernet HWaddr 84:2B:2B:4C:B9:30 inet addr:10.251.63.81 Bcast:0.0.0.0 Mask:255.255.248.0 pubeth1 Link encap:Ethernet HWaddr 84:2B:2B:4C:B9:32 |
For example, an NFS client mounting a NFS share using pubeth0 address 10.251.63.67 can lock a file successfully.
10.251.63.67:/vx/fs1 on /sydn8k_01/fs1_pubeth0 type nfs (rw,addr=10.251.63.67) nfs_client$ flock fileA -c /bin/true open("fileA", O_RDONLY|O_CREAT|O_NOCTTY, 0666) = 3 |
Using pubeth0:0 address 10.251.63.81 also worked.
10.251.63.81:/vx/fs1 on /sydn8k_01/fs1_pubeth0_0 type nfs (rw,addr=10.251.63.81) nfs_client$ flock fileA -c /bin/true open("fileA", O_RDONLY|O_CREAT|O_NOCTTY, 0666) = 3 |
But using other public IP addresses on sydn8k_02 will cause flock() to hang.
10.251.63.70:/vx/fs1 on /sydn8k_01/fs1_pubeth3 type nfs (rw,addr=10.251.63.70) <<< pubeth3 on sydn8k_01 nfs_client$ flock fileA -c /bin/true <<< hanged flock(3, LOCK_EX <<< hanged
nfs_client$ flock fileA -c /bin/true <<< hanged flock(3, LOCK_EX <<< hanged |
Solution
1. Disable the Reverse Path Filtering on all the node in the FileStore cluster.
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
2. In order to make the change permanent, update the /etc/sysctl.conf.
# cat /etc/sysctl.conf
...
# disable route verification on all interfaces
net.ipv4.conf.all.rp_filter = 0
....
Note: This issue does not affect FileStore 5.7 MP1 because lock forwarding is not performed anymore.
Applies To
The problem affects Filestore version up to 5.7P1. The problem doesn't affect Filestore 5.7MP1 where lock forwarding is not performed anymore.
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.