Flex worm disk pool is down - 2074

Problem

Flex WORM disk volume is down displaying a 2074 error. 

• All jobs may fail going to the WORM instance, or some jobs may work intermittently. 
• Restarting services allows for volume to come up, but only temporarily as it will go back to a down state eventually. 
• Spad logs may also show a lot of broken pipe messages, with no apparent reason. 

Error Message

These errors may be observed in the spad log:

February 18 17:20:41 TRACE [139864141084416]: CRMapError: called by sendn (../crmsg.cpp:2178), errno = 32
February 18 17:20:41 ERR [139864141084416]: 25099: _crBinaryMessageSend2: Error sending data: broken pipe
February 18 17:20:41 INFO [139864141084416]: SetCAUsageSyncFlag: set cause sync flag
February 18 17:20:41 TRACE [139864141084416]: Sending binary message:  EARLY_ABORT 1 0 0 0 0: 99
February 18 17:20:41 TRACE [139864141084416]: sendn: count 48, enter
February 18 17:20:41 TRACE [139864141084416]: CRMapError: called by sendn (../crmsg.cpp:2178), errno = 32
February 18 17:20:41 ERR [139864141084416]: 25099: Could not send binary message: sent -1 instead of 48 bytes. Cause: broken pipe
February 18 17:20:41 ERR [139864141084416]: 25099: Session start request from nbumedia.local:40514 could not be honored (broken pipe)

Output from a gstack of the spoold process running on the Flex host. Notice the gethostbyaddr function call it is trying to perform. 

root@flexhost01 hostadmin]# gstack 49927
Thread 1 (process 49927):
#0  0x00007fc1371cdddd in poll () from /lib64/libc.so.6
#1  0x00007fc124267f62 in __res_context_send () from /lib64/libresolv.so.2
#2  0x00007fc124265394 in __res_context_query () from /lib64/libresolv.so.2
#3  0x00007fc12447b5df in _nss_dns_gethostbyaddr2_r () from /lib64/libnss_dns.so.2
#4  0x00007fc12447b7d5 in _nss_dns_gethostbyaddr_r () from /lib64/libnss_dns.so.2
#5  0x00007fc1371f3160 in gethostbyaddr_r@@GLIBC_2.2.5 () from /lib64/libc.so.6
#6  0x00007fc1371fca0b in getnameinfo () from /lib64/libc.so.6
#7  0x00007fc139e94a72 in ?? ()
#8  0x0000000000000000 in ?? ()
 

Cause

A DNS server was configured on the Flex host, but the ports to the DNS server were blocked on the network.

Solution

Open the ports to the DNS server.