NetBackup nbpem process shuts down causing numerous status 50's.

Problem

A NetBackup master server nbpem process shuts down triggering status 50's at versions 8.1.2/3.1.2 or 8.2.  The nbpem log shows an orderly shutdown after 10 failures to communicate with the nbproxy email listener.

Error Message

The nbpem log messages seen at DebugLevel=5 when this issue occurs are as follows:
 

nbpem fails to connect to the nbproxy email listener:
 

8/29/2019 01:31:19.764 NBProxy::mailClientStatus CORBA call failed: CORBA exception: TIMEOUT (IDL:omg.org/CORBA/TIMEOUT:1.0) -- system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'
TAO exception, minor code = 3e (timeout during recv; low 7 bits of errno:  62 Timer expired), completed = MAYBE

8/29/2019 01:31:19.764 NBProxy::mailClientStatus call has timed out(PemCORBACall.cpp:138)
8/29/2019 01:31:19.764 nbproxy (/usr/openv/var/nbproxy_pem_email.ior) has failed, will restart
 

The error will be seen another 9 times.

After the 10th occurrence, the nbpem process will shutdown:

8/29/2019 01:31:30.907  NBProxy::mailClientStatus CORBA call failed: CORBA exception: TRANSIENT (IDL:omg.org/CORBA/TRANSIENT:1.0) -- system exception, ID 'IDL:omg.org/CORBA/TRANSIENT:1.0'
OMG minor code (2), described as '*unknown description*', completed = NO

8/29/2019 01:31:30.907   V-116-217 nbproxy ({0}) has exceeded {1} sequential failures
8/29/2019 01:31:30.907  Exit status set to -5(PemService.cpp:433)
8/29/2019 01:31:30.907  =========================================================(OrbService.cpp:739)
8/29/2019 01:31:30.907            Sending Event to Shutdown Service(OrbService.cpp:740)
8/29/2019 01:31:30.907  =========================================================(OrbService.cpp:741)

8/29/2019 01:31:30.908  sending self: 15(AsyncSignalDispatcher.cpp:583)
8/29/2019 01:31:30.908  shutting down the service: block = 0(OrbService.cpp:772)
 

Cause

This is the current design of the nbpem process.  

Solution

A binary to work around the issue is available in the following etracks:

• 8.2:  ET 3983739
• 8.1.2 / 3.1.2: ET 3984097 (Flex Appliance)
• 8.1.2 / 3.1.2 ET 3984324