How to use the NBServerMigrator tool

Description

NBServerMigrator (NBSM) tool provides the ability to migrate a NetBackup Primary server from one supported operating system to any another supported operating system.  To see which operating systems are supported on each NetBackup version, please see https://www.veritas.com/support/en_US/article.100040093

Supported migration path examples include but are not limited to:

• NetBackup Appliance primary to Windows/Linux build your own (BYO) Primary, Flex instance Primary.
• Windows primary migration to Linux build your own (BYO) Primary, flex instance primary, NetBackup appliance primary.
• Linux build your own (BYO) Primary migration to flex instance primary, NetBackup appliance primary, windows.
• Solaris primary to Linux build your own (BYO) Primary, flex instance primary, NetBackup appliance primary, windows (BYO).
• The source primary server must be on NetBackup 8.1.2 - 10.4.0.1 to be migrated using this NBServerMigrator 2.5 tool.
• For migrations with NetBackup version 10.2.0.1 and above as the Source primary server, the following OS are not supported:
  - NBA (NetBackup Appliance) as a target primary server.
• The source primary server can be migrated to a target server at a higher version. The NBSM tool will both migrate to the new target server and will also upgrade it to the higher version of NetBackup installed on the target server.
• The target primary server can be on NetBackup 8.1.2 - 10.4.0.1. 
• Locale must be set to English
   

Prerequisites:

1. The Source and Target Primary servers must use the SAME hostname.
   
   For example, if the hostname of the source is the fully qualified domain name (FQDN), then the target must have the same FQDN host name. Execute the following commands and confirm the hostname outputs match.   
   
   Linux: 
/usr/sbin/sysctl kernel.hostname
/usr/bin/hostname

Windows:
C:\Windows\System32\HOSTNAME.EXE  
   
   For further information, see:  https://www.veritas.com/content/support/en_US/article.100060652.html

2. The Source and Target Primary servers must have matched settings for EMM Servername and NetBackup SERVER
   
   Run the following commands and confirm the outputs are identical.
   
   Linux:
/usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts | egrep 'server'
/usr/openv/netbackup/bin/admincmd/bpgetconfig -h SERVER | grep 'SERVER'

Windows: 
<install_path>\Veritas\netbackup\bin\admincmd\nbemmcmd -listhosts
<install_path>\Veritas\netbackup\bin\admincmd\bpgetconfig -h SERVER | findstr "SERVER"

3. The NetBackup version on the Target primary must be at same version (or higher) as Source Primary
   
   The NetBackup version installed on the target must be either the same version as the source, or a higher version than the source. A higher version of NetBackup on the target means that a NetBackup upgrade will be performed during the migration process. Migration to a lower version is unsupported and will fail if a lower version is detected on the target server.

4. The “tar” archiving utility must be installed on both source and target Primary servers.
   
   For Windows 2016 Primary servers and lower, the TAR utility will need to be downloaded to both primary servers.
   
   GNU TAR download link: https://ftp.gnu.org/gnu/tar/tar-1.12.msdos.exe
   
   After downloading TAR, save the file tar-1.12.msdos.exe as ‘tar.exe’ in the C:\WINDOWS directory, or to another directory in the system PATH. If a custom location is used, update the system or user PATH variable to add 'tar.exe' to the command path.
   
   BSD TAR download link: http://libarchive.org/downloads/libarchive-v3.5.2-win64.zip
   
   After downloading BSD TAR, save the file <download_location>\libarchive\bin\bsdtar.exe as ‘C:\Windows\System32\tar.exe’ or to another directory in the system PATH. If a custom location is used, update the system or user PATH variable to add 'tar.exe' to the command path.

5. If Solaris OS is used for either source or target Primary server, the following items must be present: 
   
   Confirm that the the CSWsunmath and CSWcommon packages are installed on the Solaris source Primary server. If the packages are missing, install the Solaris libsunmath.so.1 library from OS media.
   
   Refer to page # 91 of the NetBackup Primary Server Migration Guide, Release 2.5 for more details about the Solaris server requirements.

6. Always run the migration steps using the 'root' or Windows admin-privileged user.
   
   Linux/Unix: Using 'root' user is required
   
   Windows: Use Local or Domain Administrator or a user with Administrator rights. Always use 'Run as Adminsitrator' to start the command prompt that executes the tool. 

7. Role Based Access Control (RBAC) Supported configurations
   
   If RBAC is configured and used on the Source Primary server, take appropriate actions for the migration as determined in the section "About RBAC" on page 53 in the NetBackup Primary Server Migration Guide, Release 2.5.

8. Install EEB 4112070 if the NetBackup version is 10.1.1 (source and/or target).
   
   If the NetBackup version is 10.1.1, install the EEB 4112070 on the server.

9. For migrations with NetBackup version 10.1.1 and below as the source primary server and NetBackup version 10.2.0.1 and above as the target primary server, please do the following:
   a) Run UTF-8 Checker Tool on the source primary server:
   https://www.veritas.com/support/en_US/article.100055151
   
   b) Install Sybase on the Target primary server (present in the <install_path>/nbsm/prerequisites folder).
      i) VRTSsybaseasa.rpm for Linux in the path /usr/openv/asa (by default) (Only applicable to Linux BYO. Not applicable to Flex Appliance/NBFS).
      ii) "Veritas NetBackup Sybase ASA Database Server" MSI for Windows in the path <NetBackup_installation_path>Veritas/NetbackupDB.
   
   Note: For Migrations from NetBackup 10.2.x.x (Source primary server) and above, the UTF-8 check and installing Sybase on the Target primary server is not required.

10. For migrations from (i.e. source primary server) NetBackup version 10.2.x.x and above, install the following Emergency Engineering Binary (EEB) on the target primary server(as per the target primary server's NetBackup version):

• a) EEB 4158556 for 10.2.0.1
• b) EEB 4158555 for 10.3
• c) EEB 4158554 for 10.3.0.1
• d) EEB 4168184 for 10.4
• e) EEB 4168185 for 10.4.0.1

11. If the Target primary server is a Flex Scale, then:
    a)  Run the command on the target primary server /usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts
    b)  Make a list of all the server hostnames mentioned "media" and save in a file (example: media_list.txt) inside the nbsm folder.
    (This will be used in the step 26 of "Post Migration")

Migration Steps in the NetBackup Primary Server Migration Process:

A. Deploy NBSM on source and target primary servers.

B. Generate Pre-migrate report.

C. Initiate server to server data transfer.

D. Migrate Certificates using vxmigrate.

E. Overwrite.

F. Post Overwrite steps.

G. Generate Post-migration report.

A. Deploy NBSM on source and target primary servers.

1. Create a directory :
   /usr/openv/nbsm in Unix/Linux/Appliance
   <install_path>\veritas\nbsm in windows
   /mnt/nbdata/usr/openv/nbsm in Flex Appliance and Flex Scale.

2. The "nbsm" folder must have 755 permission on BYO/NBA/Flex. On Windows, it should be created by a user with "Administrator" privilege.

3. Unzip the NBServerMigrator binaries, and copy the contents of the relevant OS version folder into the nbsm folder created in step 1. For Windows, install the VCRuntime, present inside the “prerequisites” folder if required.

4. If the source primary server is at NetBackup version 10.1.1 or below and the target primary server is at NetBackup 10.2.0.1 or above, then, install the Sybase binaries present in the “prerequisites” folder.

B. Generate Pre-migrate report.

1. Open a root/administrator command prompt, ‘cd’ into the extracted nbsm directory.

2. Execute the Pre-migration report on Target Primary server as below:
   nbsm_report -premigrate -target
   Target will prompt to run a command in source

3. "1.Copy sourcesecret.tar file from target 'nbsm' folder to source 'nbsm' folder"
    "2.Start nbsm_report on source with the following command - 'nbsm_report -source -premigrate  -target_ip <IP> '

4. You will see a sourcesecret.tar file which would get generated on the target inside "nbsm" folder.

5. Copy the sourcesecret.tar file from Target to Source in "nbsm" folder. 

6. Run the command, on the Source Primary server

7. Example: 'nbsm_report -source -premigrate -target_ip xx.xx.xx.xx'
   This will generate the premigration report on that target server in html format, inside the nbsm folder.

8. Please open the report in the browser and resolve any issues that exist (if any).

9. After resolving the issues (if any), please remove the reportTemp folder on both systems, and remove the report.html file on the target primary.  IF more than one week has passed since running the last report, then also remove the sourcesecret.tar from both source and target, and the sourcessl folder from the Source primary. After cleanup, run the report again.

C. Initiate server to server data transfer.

10.  Open a root/administrator command prompt, ‘cd’ into the extracted nbsm directory.

11.  On the Target primary server, initiate the data transfer:

nbsm_target -data_transfer
This will prompt the following to be run on the Source primary server:
[INFO] Please run below commands on source:
[INFO] 1. "nbsm_source -target_ip 192.168.2.48 -target_name nbmaster2.nbulab -data_transfer"

12.  Run the command on the source primary server. Once data transfer is complete, proceed with the next steps.

Note: This step creates a folder named "temp" in the nbsm directory on source and target primary servers. It contains configuration files of NBSM. Do not delete this folder until it is explicitly mentioned to do so.

D. Vxmigrate - Use Vxmigrate when NBCA or NBCA+ECA are configured. The Vxmigrate migrates NBCA certificates only.

13.  Stop the NetBackup services on Source and Target Primary servers.
       Note: For Flex scale (NBFS), disable the health check before stopping the NetBackup services:
                 /opt/veritas/vxapp-manage/nbu-health disable

14.  Vxmigrate binaries are available in the compressed NBServerMigrator binaries.
       For Windows, install the Microsoft VC Runtime shipped with NBServerMigrator binaries in “\vxmigrate\prerequisites” folder.
       Note: This VC Runtime is specific to Vxmigrate and is needed specifically for vxmigrate, even with nbsm_X.

15.  Run the command on Source and Target Primary servers:

vxmigrate -o export -c /usr/openv/var/global/vxss/eab/data/root/.VRTSat/profile/VRTSatlocal.conf -f /usr/openv/nbsm/vxmigrate/export_source.out -p P@ssw0rd -d -l /usr/openv/nbsm/vxmigrate/vxmigratelog.log

 Where on UNIX/Linux::
 /usr/openv/nbsm/vxmigrate/export_source.out is the file name in which the output file is created. Any name can be given. Give different names on source and Target Primary servers.

Where on Flex Appliance:

/mnt/nbdata/usr/openv/nbsm/vxmigrate/export_source.out is the file name in which the output file is created. Any name can be given. Give different names on source and Target Primary servers.

and

/mnt/nbdata/usr/openv/nbsm/vxmigrate/vxmigratelog.log is the file name in which the operation is logged

Where on Windows:

C:\VERITAS\nbsm\vxmigrate\export_source.out

Note: Do not run vxmigrate with Power Shell prompt. Executing the vxmigrate import command with power shell may not produce the required output file. Use the Windows Command Prompt that is started with “Run as Administrator” instead.

For Windows path of VRTSatlocal.conf is:<install_Path>\NetBackup\var\global\vxss\eab\data\systemprofile\VRTSatlocal.conf

Note: For NBFS/Flex, please provide a persistent path for the file in -f switch.

16.  Copy the export_source.txt file from the source to the target.  When copying from Windows to Linux, a binary tranfer mode is required.

17.  Using the export_source.txt file just copied to the target, run the import command on the Target Primary server:

vxmigrate -o import -c /usr/openv/var/global/vxss/eab/data/root/.VRTSat/profile/VRTSatlocal.conf -f /usr/openv/nbsm/vxmigrate/export_source.out -p P@ssw0rd -d -l /usr/openv/nbsm/vxmigrate/vxmigratelog.log

18.  Start NetBackup Services.
       Note:  For Flex Scale (NBFS), enable the health check before starting the NetBackup services:
                   /opt/veritas/vxapp-manage/nbu-health enable

        Note: For Target server on NetBackup version 9.1 and higher with SERVICE_USER configured, if, after step 18, NBWMC does not start, then please run the following:
                  chown -R <SERVICE_USER> /usr/openv/var/global/vxss/eab/data
                  After this, please start the NetBackup Services again.

E. Overwrite

Note: During the migration process, a NOexpire touch file is created on the target server to ensure no data cleanup will take place until the migration is confirmed to be complete.  The cleanup routine at the end of the migration process will remove this.  For information on NOexpire, see:  https://www.veritas.com/support/en_US/article.100002418

19.  To initiate migration, run the following on the target primary server

nbsm_target -overwrite -keep_src_cert
Note: If we are not migrating NBCA or ECA, then please run nbsm_target -overwrite

20.  It will initiate a series of Modules:

a) get_env_info
b) Run_precheck
c) Connect_server: at this point, it will display:
    Please run below command on source:
   "nbsm_source -target_ip xx.xx.xx.xx -target_name xxxxxx -migrate -ignore_disk"

21.  Please run the provided command on the Source Primary server.

22.  Once the tool is completed successfully, review the logs for error or warning messages to determine if action is required or if the message can be ignored. For log location, see the Troubleshooting section below, letter B.
Note: There are a few warning messages which are common to all testing and can be ignored.
 

F. Post Overwrite

23. Regenerate the certificates on the Target Primary server.

UNIX/Linux:

 1) /usr/openv/netbackup/bin/nbwmc -terminate

 2) /usr/openv/netbackup/bin/admincmd/nbcertconfig -u -i

 3) /usr/openv/netbackup/bin/admincmd/nbcertconfig -m

 4) /usr/openv/netbackup/bin/admincmd/nbcertconfig -t -f

 5) On 8.3 and above: /usr/openv/netbackup/bin/admincmd/nbcertconfig -s -f

 6) /usr/openv/wmc/bin/install/configureWmc

 7) /usr/openv/wmc/bin/install/configureCerts

 8) /usr/openv/wmc/bin/install/setupWmc

 9) /usr/openv/netbackup/bin/nbwmc -start

  Windows:

 0) set WEBSVC_PASSWORD=<nbwebsvc password>

 1) C:\Windows\System32\sc.exe stop "NetBackup Web Management Console"

 2) <Install_Path>\NetBackup\bin\admincmd\nbcertconfig -u -i

 3) <Install_Path>\NetBackup\bin\admincmd\nbcertconfig -m

 4) <Install_Path>\NetBackup\bin\admincmd\nbcertconfig -t -f

 5) On 8.3 and above: <Install_Path>\NetBackup\bin\admincmd\nbcertconfig -s -f

 6) <Install_Path>\NetBackup\wmc\bin\install\configureWmc

 7) <Install_Path>\NetBackup\wmc\bin\install\configureCerts

 8) <Install_Path>\NetBackup\wmc\bin\install\setupWmc

 9) C:\Windows\System32\sc.exe start "NetBackup Web Management Console"

24.  Using a re-issue token, regenerate the CA and Certificate on the Target Primary server.

Note: If the target server is Windows or RHEL (with root user) then skip to sub-step (a). Refer NBServerMigrator’s admin guide for more details.

If nbsm is to be executed as a ‘non-root’ user on NBA, Flex, NBFS, or BYO Linux, the user must be an RBAC principal with NetBackup Administrator privileges.

To add the non-root user as an RBAC principal, run the following:
Linux:
bpnbaz -AddRBACPrincipal -User unixpwd:PrimaryName:USERNAME
Flex Appliance(path needs to be included):
/usr/openv/netbackup/bin/admincmd/bpnbaz -AddRBACPrincipal -User unixpwd:PrimaryName:USERNAME
Windows Local account:
bpnbaz -AddRBACPrincipal -User nt:PrimaryName:USERNAME
Windows Domain account:
bpnbaz -AddRBACPrincipal -User nt:DOMAIN:USERNAME

If the target server is Windows or RHEL (with root user) then skip to the next step. Refer NBServerMigrator’s admin guide for more details.

a) bpnbat -login -loginType WEB
Note: If the bpnbat command fails, alternatively, login to the WEBUI and generate a reissue token:  security > Certificates > select the primary server's hostname > Generate Reissue Token
           After this, proceed with step c.
b) nbcertcmd -createToken -name token_name -reissue -host <target_Primary_server_name>
c) nbcertcmd -getCACertificate
d) nbcertcmd -getCertificate -token <reissue_token> -force

25.  Check if the communication with the Client/Media servers is happening successfully.

a) If the Target primary server and the Source primary server have the same IP address, then skip to point "b".
     If Target and source primary server have different IP addresses, then edit the IP address in the Hosts File on the client/Media server ( redirecting it to the Target Primary server).
b) Add the IP/Hostname of the Client/Media server in the hosts file of the Target Primary server.
c) bpclntcmd -clear_host_cache (on the client/Media  server)
d) bpclntcmd -pn (on the client/Media server)
e) bptestbpcd -client <client_name> -verbose (on the Target Primary server).

26.  If the target is NBFS(If no, then move to step 27), then we need to redeploy the certificates on the Media servers of the NBFS. Please do the following:

a) Update the bp.conf of the target primary server with the entries of the Media servers (which were present on the flex scale before starting the migration).
Note: The hostnames of media servers can be obtained from point 11 (in Prerequisites).

b) Reconfigure the certificate on the management gateway of the flex scale.

c) run the following commands on the Media server:
  /usr/openv/netbackup/bin/nbcertcmd -getCACertificate
  /usr/openv/netbackup/bin/nbcertcmd -getCertificate -force

d) if it asks for a token, please generate one using :
    https://www.veritas.com/support/en_US/doc/21733320-127424841-0/v120724194-127424841
    and then run /usr/openv/netbackup/bin/nbcertcmd -getCertificate -token <> -force

G. Post Migration Report and Cleanup

27.  Run the Post Migration report on the Target Primary server:

nbsm_report -post_manipulation
It will generate the report in HTML format in the nbsm folder. Analyze it to compare the difference between the Source primary server (pre-migration) and the Target Primary server (post-migration).

28.  Activate the Backup Policies and SLPs.

nbsm_target -activate_policies
nbsm_target -activate_slps

29.  After ensuring all operations have been completed successfully and the system is operating as expected, run cleanup:

NOTE:  This operation will remove all temporary files from the migration process, and also removes the NOexpire touchfile.

nbsm_target -clean_up

Troubleshooting:

a) Logs related to report:

For NBFS: /mnt/nbdata/usr/openv/nbsm/ReportTmp

For Unix: /usr/openv/nbsm/ReportTmp
For Windows: <install_path>\veritas\nbsm\ReportTmp

b) Logs related to migration:

For Unix: /usr/openv/nbsm/logs
For Windows: <install_path>\veritas\nbsm\logs
For Flex and NBFS: /mnt/nbdata/usr/openv/nbsm/logs

c) NetBackup database-related logs are logged in

  i) If the NetBackup version on the target server is 10.1.1 or lower:
   server.log
   Windows: <install_path>\Veritas\NetBackupDB\log
   Unix: /usr/openv/netbackup/db/conf/log

  ii) If the NetBackup version on the target server is 10.2.0.1 and above:
   postgresql-yyyy-mm-dd.csv
   Windows: <install_path>\Veritas\NetBackupDB\data\logs
   Unix: /usr/openv/db/data/log/

For any NetBackup-related error, check the Veritas NetBackup Logging Reference Guide.

Additional troubleshooting articles:

NbServerMigrator: Existing Technotes (veritas.com)