What data to gather if a nbdecommission or nbemmcmd -deletehost fails on a UNIX or Linux Primary Server.

Problem

What data needs to be gathered if a nbdecommission or nbemmcmd -deletehost command fails on a UNIX or Linux Primary Server?

Error Message

Snippet from legacy admin log from the Primary Server:

17:16:18.654 [4334] <2> : NBEMMCMD -deletehost -machinename nbmaster -machinetype master
17:16:18.679 [4334] <2> emmlib_initialize: (-) Connection attempt #<0>
17:16:18.679 [4334] <4> ValidateConnectionID: (-) Created new Connection ID 0
17:16:18.679 [4334] <2> emmlib_initializeEx: (-) Connecting to the Server <nbmaster> Port <1556>, App <nbemmcmd> OrbOpt <No extra otions>
17:16:18.698 [4334] <2> Orb::setDebugLevelFromVxul: Orb logging configuration level set to 10(Orb.cpp:2261)
17:16:18.698 [4334] <2> Orb::init: checking if NBAC is enabled(Orb.cpp:508)
17:16:29.845 [4334] <16> emmlib_DeleteHost: (0) DeleteMachine failed, emmError = 2007072, nbError = 0
17:16:29.846 [4334] <16> nbemmcmd: (-) Translating EMM_ERROR_SQLIntegrityConstraintViolation(2007072) to 193 in the Media context
17:16:29.846 [4334] <16> nbemmcmd: (-) Translating EMM_ERROR_SQLIntegrityConstraintViolation(2007072) to 193 in the Media context

This is just a sample error log snippet. This data will need to be collected if any media server deletion fails.

Snippet from unified nbemm (EMM) log from the Primary Server:

5/14/2014 17:16:29.840 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 6 [DbConnection::MapErrorCode] Err -1, native -198, SqlState 23000(DbConnection.cpp:911)
5/14/2014 17:16:29.840 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 1 [DbConnection::Execute] SQL - retval=DB_ERROR_SQLIntegrityConstraintViolation(2007072) retdal=-1 native=<-198> sqlerror=<[Sybase][ODBC Driver][SQL Anywhere]Primary key for row in table 'EMM_Machine' is referenced by foreign key 'FK_ADTR_AUD_R_MACHINE_EMM_MACH' in table 'ADTR_AuditRecord'> sqlstate=<23000>(DbConnection.cpp:402)
5/14/2014 17:16:29.840 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 1 [DbConnection::Execute] stmt=<DELETE FROM EMM_Machine WHERE MachineKey = 1000329>(DbConnection.cpp:404)
5/14/2014 17:16:29.840 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [SSRDbConnection::CloseOdbcView] Removing OdbcView from a set(DbConnection.cpp:1729)
5/14/2014 17:16:29.840 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnection::CloseOdbcView] Current count = 0(DbConnection.cpp:332)
5/14/2014 17:16:29.840 [Debug] NB 51216 nbemm 111 PID:527 TID:1078012224 File ID:111 [No context] 3 [MachineObject::Delete] retval = < 2007072 >
5/14/2014 17:16:29.840 [Debug] NB 51216 nbemm 111 PID:527 TID:1078012224 File ID:111 [No context] 1 [MachineConfigImpl::~MachineConfigImpl()]  Exiting
5/14/2014 17:16:29.840 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnection::rollbackTransaction] TransLevel 0(DbConnection.cpp:1100)
5/14/2014 17:16:29.843 [Debug] NB 51216 nbemm 111 PID:527 TID:1078012224 File ID:111 [No context] 5 [MachineObject::~MachineObject] 
5/14/2014 17:16:29.844 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [SSRGuard::~SSRGuard] Releasing connection(DbConnectionManager.cpp:1832)
5/14/2014 17:16:29.844 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnectionManager::ReleaseSSRConnection] releasing connection Id=<4>(DbConnectionManager.cpp:1176)
5/14/2014 17:16:29.844 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnectionManager::ReleaseSSRConnection] RqstStmtCount is <0> (DbConnectionManager.cpp:1180)
5/14/2014 17:16:29.844 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnectionManager::ReleaseSSRConnection] Queue size (step 1) is <2>(DbConnectionManager.cpp:1207)
5/14/2014 17:16:29.844 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnectionManager::ReleaseSSRConnection] Queue size (step 2) is <2>(DbConnectionManager.cpp:1230)
5/14/2014 17:16:29.844 [Debug] NB 51216 321 PID:527 TID:1078012224 File ID:111 [No context] 5 [DbConnectionManager::ReleaseSSRConnection] Signaling(conn) thread t@ 1118767424(DbConnectionManager.cpp:1242)
5/14/2014 17:16:29.844 [Diagnostic] NB 51216 nbemm 111 PID:527 TID:1078012224 File ID:111 [No context] 1 V-111-1062 [MachineImpl::DeleteMachine] retval 2007072

Cause

Depending on what is causing the command to fail, a SQL statement may be needed from NetBackup Engineering to resolve the issue.

Solution

This is a list of information that needs to be gathered and sent to the NetBackup support Engineer working on your case.

1)  Verify that the following logs are in place:

/usr/openv/netbackup/logs/admin
/usr/openv/netbackup/logs/bpdbm
/usr/openv/logs/nbemm

If these directories do not exist, they can be created with the following commands:

/usr/openv/netbackup/logs/mklogdir admin,bpdbm
mkdir /usr/openv/logs/nbemm

2)  Run the commands below to change to the NetBackup bin directory and increase debug log levels:

cd /usr/openv/netbackup/bin
./vxlogcfg -a -p 51216 -o 111 -s DebugLevel=1
./vxlogcfg -a -p 51216 -o 321 -s DebugLevel=6
./vxlogcfg -a -p 51216 -o 178 -s DebugLevel=5

All three OIDs write to log file id 111, i.e. 51216-111-...

3)  Ensure that legacy logging is turned up on the Primary Server:

• Open the NetBackup Administration GUI.
• Go to the host properties of Master/Media Server.
• Select the Logging tab.
• Uncheck robust logging and set global logging value to 5.
• Stop/Start the NetBackup daemons/services.

4)   At this point, try to delete the Media Server from EMM, or use the nbdecommission utility to decommission the Media Server.

(Option 1) -  Open a command prompt on the Primary Server, and change directories to the NetBackup admincmd directory

cd /usr/openv/netbackup/bin/admincmd

Run this command and check the output, as it will give you the exact machinename and machinetype of the Media Server to use in the delete host command:

./nbemmcmd -listhosts -verbose

To try to delete the Media Server from the EMM database using this command:

./nbemmcmd -deletehost -machinename <servername> -machinetype <machine type>

(Option 2) To try to use nbdecommission command to decommission a media server.

./nbdecommission -oldserver <hostname> -newserver <hostname>

If the above commands fail, we will need to collect logs and zip and send them to Veritas.

5)  Get a NBDB unload output from the EMM database from the Primary Server.

Open the command prompt on the Media Server and change to the NetBackup DB bin directory.

cd /usr/openv/db/bin

Create the destination directory for the unloaded database output. For example, we'll use the /tmp/nbdb_dump directory.

./nbdb_unload /tmp/nbdb_dump

Gather the entire /tmp/nbdb_dump directory, to include its contents.

6)  To collect logs:

-  Make a directory on the Primary Server on the drive where you have the most space available. The directory must be empty.
-  Open a command prompt on the Primary Server and change directories to the NetBackup support directory:

cd /usr/openv/netbackup/bin/support

Option 1)  ./nbcplogs -d 4h -l nbemm,admin,bpdbm <destination_directory> (example:  /tmp/logs)
 
This command will copy the nbemm, bpdbm and admin log from 4 hours along with the nbsu to the destination directory.  It will prompt you to ensure that enough disk space is available in the destination directory. 

If this command fails, zip the destination directory output and send it to Veritas.
 
Option 2)  ./nbcplogs -l bpdbm,admin,nbemm -d 4h -f <salesforce case number>
Example:  nbcplogs -l bpdbm,admin,nbemm -d 4h -f 05540451

This command will copy the nbemm, bpdbm and admin log from 4 hours along with the nbsu to the Veritas FTP server.  It will prompt you to ensure that enough disk space is available. 

 7)  We will also need an online backup of the EMM database.

cd /usr/openv/db
mkdir nbdb_backup_online
./bin/nbdb_backup -dbn NBDB -online /usr/openv/db/nbdb_backup_online

Once this command is issued, an online backup of the EMM database will take place.  Once the command prompt returns, the online backup is complete.  It may take a few minutes to run. 

Gather this output.

8)  Then open a command prompt and change directories to the NetBackup bin directory.

cd /usr/openv/netbackup/bin

To turn down the VxUL logging using this command:

./vxlogcfg -a --prodid 51216 -o all -s DebugLevel=1

9) Ensure that legacy logging is turned down on Primary Server:

• Open the Netbackup Administration GUI.
• Go to the host properties of Master/Media Server. 
• Select the logging tab.
• Uncheck robust logging and set global  logging value to 0.
• Stop/Start the NetBackup daemons/services.

To recap here is the complete list of NetBackup information that needs to be collected for these issues.

• NetBackup logs:
  • bpdbm
  • admin
  • nbemm
• EMM database unload
• EMM database online backup
• NBSU

Applies To

Any supported NetBackup UNIX or Linux Primary Server environment.