What data to gather if a nbdecommission or nbemmcmd -deletehost fails on a Windows master server.

Problem

What data to gather if a nbdecommission or nemmcmd -deletehost fails on a Windows master server.

Error Message

Snippet from admin logs:

17:08:51.200 [5472.5652] <2> : NBEMMCMD -deletehost -machinename NBmaster -machinetype cluster
17:08:51.215 [5472.5652] <2> emmlib_initialize: (-) Connection attempt #<0>
17:08:51.215 [5472.5652] <4> ValidateConnectionID: (-) Created new Connection ID 0
17:08:51.215 [5472.5652] <2> emmlib_initializeEx: (-) Connecting to the Server <Nbmaster> Port <1556>, App <nbemmcmd> OrbOpt <No extra otions>
17:08:51.231 [5472.5652] <2> Orb::setDebugLevelFromVxul: Orb logging configuration level set to 10(Orb.cpp:2261)
17:08:51.231 [5472.5652] <2> Orb::init: checking if NBAC is enabled(Orb.cpp:508)
17:08:54.164 [5472.5652] <16> emmlib_DeleteHost: (0) DeleteMachine failed, emmError = 2050023, nbError = 0
17:08:54.164 [5472.5652] <16> nbemmcmd: (-) Translating DSM_ERR_NOTARGETMACHINE(2050023) to 196 in the Media context
17:08:54.164 [5472.5652] <16> nbemmcmd: (-) Translating DSM_ERR_NOTARGETMACHINE(2050023) to 196 in the Media context
17:08:54.179 [5472.5652] <16> nbemmcmd: Run time failure: The function returned the following failure status: 
17:08:54.179 [5472.5652] <4> emmlib_uninitialize: (0) Uninitialize connection <0> and shutdown Orb if no other connections exists
17:08:54.179 [5472.5652] <4> DeleteConnectionID: (-) Cleared Connection ID <0> for <>
17:08:54.179 [5472.5652] <4> uninitialize: (0) We still have <0> open connections
17:08:54.179 [5472.5652] <4> shutdown_orb: (-) Destroying the orb

This is a sample error log snip it. This data would need to be collected if any media server deletion is failing.

Cause

Depending on what is causing the command failure a SQL statement may be needed from NetBackup Engineering to resolve the issue.

Solution

This is a list of information that needs to be gathered and sent to the NetBackup Support Engineer working your case:

1. Verify that the following logs are in place, if the directory does not exist create it:
   • install_path\Veritas\NetBackup\logs\admin
   • install_path\Veritas\NetBackup\logs\nbemm
   • install_path\Veritas\NetBackup\logs\bpdbm
2. Ensure that Veritas Unified Logging (VxUL) is turned up on the master server:
   • Open an Administrator: Command Prompt (run as administrator), change directories (cd) to the NetBackup bin directory (install_path\veritas\netbackup\bin), and run these commands:
     • vxlogcfg -a -p 51216 -o 111 -s DebugLevel=1
     • vxlogcfg -a -p 51216 -o 321 -s DebugLevel=6
     • vxlogcfg -a -p 51216 -o 178 -s DebugLevel=5
       • Note: All three OIDs write to file id 111, i.e. 51216-111.
3. Ensure that legacy logging is turned up on master server:
   • Open the NetBackup Administration GUI.
   • Go to the host properties of master\media server.
   • Select the logging tab.
   • Uncheck robust logging and set global logging value to 5.
   • Stop/Start the NetBackup services/daemons.
4. At this point try to delete the media server/application cluster/etc from EMM, or use the nbdecommission command utility to decommission a media server.
   • (Option 1) - Open an Administrator: Command Prompt (run as administrator) on the master server, change directories (cd) to the NetBackup admincmd directory (install_path\veritas\netbackup\bin\admincmd), and run these commands:
     • To try to delete the media server/application cluster/etc from the EMM database:
     • nbemmcmd -listhosts -verbose
       • Note: This output will give you the exact machinename and machinetype.
     • nbemmcmd -deletehost -machinename <servername> -machinetype <machine type>
   • (Option 2) - To try to use nbdecommission to decommission a media server.
     • nbdecommission -oldserver <hostname> -newserver <hostname>
   • If the above commands fail, collect logs per the next step.
5. To collect logs on the master server:
   • Make a directory on the master server on the drive where you have the most space available. The directory must be empty.
   • Open an Administrator: Command Prompt (run as administrator), change directories (cd) to the NetBackup support directory (install_path\veritas\netbackup\bin\support), and run this command:
     • nbcplogs -d 1d -l nbemm,admin,bpdbm <destination_directory>
     • Example: nbcplogs -d 1d -l nbemm,admin,bpdbm L:\Logs
   • This command will copy the nbemm, admin, and bpdbm log from one day, along with the nbsu to the destination directory. It will prompt you to ensure that enough disk space is available in the destination directory.
   • Then, compress/zip the destination directory output, and upload it to the respective case via Veritas Support Portal.
6. Run an online backup of the NetBackup EMM database (NBDB) on the master server.
   • Instructions to run an on-line backup of the NetBackup EMM database:
     1. On the master server create the directory you wish to use to dump the online EMM database, for this example we'll use C:\online.
     2. Open an Administrator: Command Prompt (run as administrator), change directories (cd) to the NetBackup bin directory (install_path\veritas\netbackup\bin), and run this command:
        • nbdb_backup -online C:\online
        • Note: This was the directory created in step one. It must exist before running this command.
   • Once this command is issued an online backup of the EMM database will take place.  Once the command prompt returns the online backup is complete.  It may take a few minutes to run. When it completes add the online backup output to the <destination_directory> selected above, so all files needed can be zipped into one package for uploading.
   • Then, compress/zip the destination directory output (online.zip), and upload it to the respective case via Veritas Support Portal.
7. Run an unload of the NetBackup EMM database (NBDB) on the master server.
   • Open an Administrator: Command Prompt (run as administrator), change directories (cd) to the NetBackup bin directory (install_path\veritas\netbackup\bin), and run this command:
   • Create the destination directory for the database unload, for this example we'll use C:\nbdb_dump.
     • nbdb_unload C:\nbdb_dump
   • Then, compress/zip the destination directory output (nbdb_dump.zip), and upload it to the respective case via Veritas Support Portal.
8. Ensure that Veritas Unified Logging (VxUL) is turned down to default level on the master server:
   • Open an Administrator: Command Prompt (run as administrator), change directories (cd) to the NetBackup bin directory (install_path\veritas\netbackup\bin), and run this command:
     • vxlogcfg -a --prodid 51216 -o all -s DebugLevel=1
       • Note: This will turn down VxUL logging.
9. Ensure that legacy logging is turned down on master server:
   • Open the NetBackup Administration GUI.
   • Go to the host properties of master\media server.
   • Select the logging tab.
   • Uncheck robust logging and set global  logging value to 0.
   • Stop/Start the NetBackup services/daemons.

To recap here is the complete list of NetBackup information that needs to be collected for these issues.

• NetBackup logs: bpdbm, admin, nbemm
• NetBackup Support Utility (NBSU) output
• NetBackup EMM database online backup
• NetBackup EMM database unload

Applies To

 Any supported NetBackup Windows master server environment.