Duplication and Image Cleanup Failures - error 2060405 Wrong MSDP multi-domain configuration is detected

Problem

Duplications and image cleanup jobs may fail with status 83/1, reporting error 2060405 Wrong MSDP multi-domain configuration is detected and/or Warning More than one master server is configured.

Error Message

Duplication Job shows status 83:

DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=244968) Storage Server Error: (Storage server: PureDisk:app-5250-01) MBGetTime: Failed to complete gettime webservice ((unknown)) V-454-44
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=244968) sts_open_server failed: error 2060405 Wrong MSDP multi-domain configuration is detected. This NBU domain should use a different MSDP user to access the MSDP storage server. See NetBackup Deduplication Guide for details.
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=244968) failure to open sts for storage server app-5250-01: plug-in reports error 2060405 Wrong MSDP multi-domain configuration is detected. This NBU domain should use a different MSDP user to access the MSDP storage server. See NetBackup Deduplication Guide for details.
...
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=245252) Storage Server Error: (Storage server: PureDisk:app-5250-01) MBGetTime: Failed to complete gettime webservice ((unknown)) V-454-44
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=245252) sts_open_server failed: error 2060017 system call failed
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=245252) failure to open sts for storage server app-5250-01: plug-in reports error 2060017 system call failed
...
DD-MMM-YYYY HH:MM:SS - granted resource MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=dp_disk_app-5250-01;Path=PureDiskVolume;StorageServer=app-5250-01;MediaServer=app-5250-02
DD-MMM-YYYY HH:MM:SS - Error bpduplicate (pid=3562907) host app-5250-02 backup id nbclient01_1693361131 optimized duplication failed, media open error (83).
DD-MMM-YYYY HH:MM:SS - Error bpduplicate (pid=3562907) Duplicate of backupid nbclient01_1693361131 failed, media open error (83).
DD-MMM-YYYY HH:MM:SS - Error bpduplicate (pid=3562907) Status = no images were successfully processed.
DD-MMM-YYYY HH:MM:SS - end Duplicate; elapsed time 0:00:06
no images were successfully processed (191)

Image Cleanup Job shows status 83:

DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=161766) Storage Server Error: (Storage server: PureDisk:app-5250-01) MBGetTime: Failed to complete gettime webservice ((unknown)) V-454-44
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=161766) sts_open_server failed: error 2060405 Wrong MSDP multi-domain configuration is detected. This NBU domain should use a different MSDP user to access the MSDP storage server. See NetBackup Deduplication Guide for details.
DD-MMM-YYYY HH:MM:SS - Critical bpdm (pid=161766) failure to open sts for storage server app-5250-01: plug-in reports error 2060405 Wrong MSDP multi-domain configuration is detected. This NBU domain should use a different MSDP user to access the MSDP storage server. See NetBackup Deduplication Guide for details.
...
media open error (83)

Image Cleanup Job shows status 1:

DD-MMM-YYYY HH:MM:SS - Warning bpdbm (pid=3584815) More than one master server is configured: nbprimarycluster01 (1000002), nbprimarynode02 (1000004). Multiple master servers are not supported and may cause issues in future releases.
DD-MMM-YYYY HH:MM:SS - Info bpdbm (pid=3584815) deleted 0 expired records, compressed 0, intelligent catalog archived 0, tir removed 0, deleted 0 expired copies
The requested operation was partially successful. (1)

Cause

1. On standalone primary servers, there are multiple master entries listed in the EMM database; e.g. one valid entry and one or more rogue master entry(s) may exist.
2. On clustered primary servers, the active node may not have host mappings aliases listed for the virtual cluster name. (e.g. shortname & FQDN)
3. In relation to the 2 cause items above, there may also be hung processes, or stale cache entries on the appliance media/storage servers. Hence, fixing those items and clearing the host cache will probably not resolve the issue completely, but may reduce the impact. (e.g. some duplications may be successful and image cleanup jobs may delete some expired records & copies). However, NetBackup and PBX services should be restarted on these hosts.

Solution

1. On standalone primary servers, use the nbemmcmd command to delete the rogue master entry(s) from the EMM database.
   • Example: nbemmcmd -deletehost -machinename roguemaster -machinetype master
   • For any issues removing this host, please see the Related Article (100013542) below
2. On clustered primary servers, add host mappings aliases for the virtual cluster name to the active node; suggest confirming the same exists on the passive node(s).
   • For more details on Adding shared or cluster mappings, please see the NetBackup™ Security and Encryption Guide.
3. After performing either of the solution items above, it is recommended to restart NetBackup and PBX services on the appliance media/storage servers.
   • To perform these steps via the CLISH menu, see the Related Article (100038055) below.