NetBackup Primary Server upgrade planning and tasks

Description

• This article is intended to assist in planning a NetBackup Primary Server upgrade. 
• The steps are documented to help ensure that the NetBackup Primary Server upgrade is successful. However, even after implementing them, there are some cases where upgrades may fail. 

Points to consider:

1. Have the compatibility lists, hardware requirements, and upgrade guides been reviewed?

• Review the NetBackup upgrade and installation quick reference (with links to resources)

2. How long is the downtime for the upgrade process at the Primary Server and is any rollback plan (in case of failure) included in downtime?

• It is recommended to begin early in the day and early in the week to avoid after-hours and weekend complications when possible.

3. Is there any ongoing issue in this environment that Veritas support should be aware of? 

• Collect any causes and resolutions that were carried out for specific issues, along with any recent changes.

4. Have any upgrade failures been experienced recently?

• Collect any reference case numbers and/or workarounds if applicable.

5. How long are NBU services up and running?

• Document any recent outages and their causes. For longer up times, cycle services before beginning to make sure they restart without error.

Suggested proactive measures:

• Catalog Backup - Take a full catalog backup. Once completed, ensure that DR file and DR package are available. Please verify if the DR package password is available and make sure it is valid. Admins can validate the passphrase with the -testpassphrase switch with the nbhostidentity command.

• Job Database Size - Reduce the job database size before upgrading. This action minimizes the amount of processing required to perform the association and minimizes the effect on web services performance. Reduce the job database size before upgrade 

• NBDB Rebuild - Perform NBDB database rebuild before upgrade as it will reduce the size of the database and the upgrade will be completed in less time. How to rebuild the NetBackup Database (NBDB) 

• NBDB Validate - Prior to upgrading, we recommend running “nbdb_admin -validate -full” to validate the Sybase SQL Anywhere schema. If a database is not valid, any number of issues can occur. In one example, corruption might cause rows at a table that violate foreign key constraints. S All corruption issues should be fixed before migration. How to perform a consistency check of the NetBackup EMM and BMR databases using the command line or NetBackup NBDbAdmin(Windows) or dbadm(UNIX/Linux) utilities

• nbcheck - NetBackup uses a preinstallation program that does a check at the beginning of installation or upgrades. The check looks for any known problems that you can eliminate so the operation can succeed. The checks that are performed are developed from customer input on the previous problems that were encountered during installations and upgrades. Ensure that nbcheck output is without any error. NetBackup installation nbcheck: not ok missing_binaries_noncritical / not ok missing_binaries_critical

• Upgrade to NBU 10.1.1 (and higher) – We recommend upgrading to NetBackup 10.1.1(and higher point releases such as 10.4.0.1) as the nbcheck utility from this version includes an extensive list of additional checks which has been added as per feedback from previous upgrade failure experiences. 

• NetBackup Services are down – Ensure that all NetBackup services are down (except pbx) before attempting to upgrade a primary server. In case of a cluster, please ensure that the NBU service group is frozen first and then only stop the NBU services on the active node of the cluster using NBU commands.

• Antivirus - Disable antivirus software. If the antivirus software cannot be disabled, we should exclude NetBackup files and folders from virus scan events. More information about excluding NetBackup files and folders is available. General recommendations for virus scanner exclusions working with NetBackup

• Infoscale Compatibility in Case of Cluster -Make sure the target NBU version is compatible with Infoscale software.  Veritas InfoScale 7.4.2 Software Compatibility List - Windows 

• Cluster Compatibility - Confirm cluster type VCS HA or GCO (applicable for Infoscale + NBU environment) - 
   
  If it is a GCO cluster, then make sure you and your customers are aware of the GCO upgrade procedure. Upgrading a globally clustered (GCO) NetBackup master server to NetBackup 8.1 and later

• Cluster Environment - For cluster environment only, freeze the NetBackup group and stop NetBackup services with NBU commands only on both nodes (except pbx).

• NetBackup Web Services user account - It is recommended to save the details of the user account(username and password) that you use for NetBackup Web Services. A primary server recovery requires the same NetBackup Web Services user account and credentials that were used when the NetBackup catalog was backed up. 

• 3rd Party Application - Ensure that any 3rd party applications that usually interact with NetBackup is not sending are requests to the primary server. For example, any user-directed backup. Failure to stop these applications may result in unexpected behavior. Observed behavior includes aborted upgrades and application failures.

Additionally, it is possible switch the SERVICE_USER from non-root user to root and again switch it from root to non-root user before the upgrade (this will require downtime ).This activity will ensure that user ownership and group ownership of all NBU files are correct before the upgrade and prevent issues such as:  

• Upgrade from 10.0.01 to 10.1 failed at 99 percent
• After upgrade to NetBackup 10.1.1, the Primary server's nbatd service won't start / won't stay running 

Note:- Be aware that the nbserviceusercmd command is not supported on Flex Appliance, NetBackup Flex Scale, and NetBackup on any Kubernetes Service platform. The Kubernetes Service platforms include Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (EKS), etc.

• User access to registry – At Windows Primary, the user using which customer is logging at the primary server and triggering the upgrade should have read/write access at the registry.

• Mini reset steps before upgrade (Proactive) – This is a proactive step and not required in every case. However, this will ensure that we are refreshing the certificates before attempting to upgrade. In an ideal scenario, nbcheck will identify any problem with certificates and we may need to run it anyhow in that case. The article below provides steps to perform mini reset steps automatically. How to perform automatic mini reset of NetBackup certificates using configureCerts script

• Restart NetBackup services before or reboot Primary Server -  This allows us to identify and fix any lingering existing issue that is present  in the environment beforehand which may cause an upgrade to fail. For example, if nbwmc is not able to start after restart, we can take corrective action beforehand.

• nbhealthchecker – Fix any issue identified by the nbhealthchecker beforehand for the primary server. This utility runs every night by bpcd, and the anomalies detected by this tool are reported as alerts/notifications in NetBackup WebUI.

• NetBackup Consistency Check(NBCC) - Only if the NetBackup Primary is using tape media as storage, then it is advisable to run NBCC utility prior to the upgrade. 

• JAVA_PATH at Windows Primary Server – Verify that JAVA_PATH environmental variable is set to the correct installation path. NetBackup web management console service is not starting after the NetBackup master server upgrade from 8.2 to 9.1.0.1

• NetBackup UTF-8 Checker Tool – From NetBackup 10.2 onwards, there is a requirement to run the UTF-8 Checker Tool which checks any invalid UTF-8 characters in the NetBackup database before migration to the new database in NetBackup 10.2  NetBackup UTF-8 Checker Tool 

• For the best performance of the database precheck tool, install the EEB Hotfix for the nbdb_unload utility. See the following article:  HOTFIX - NetBackup 8.1.2 - 10.1.1 - nbdb_unload performance impact when upgrading to NetBackup 10.2 
• Install/upgrade pre-check for NetBackup Scale-out Database Services - Review below article if planning to upgrade to NetBackup 10.2 or higher  Install/upgrade pre-check for NetBackup Scale-out Database Services

• Sybase to PostgreSQL upgrade at Cluster - It is recommended that you search for the pgsql_utf8_error.txt file after the successful upgrade of the active node of a clustered primary server. Notification of invalid UTF-8 characters is not displayed on clustered servers. Details on the location of this file are found in the Solution section of this technical article.  Invalid UTF-8 Character Conversion 

• Non-root database user for NBU 10.2+ -While planning to upgrade to NBU 10.2+, for Linux environments, the new NetBackup database must have a non-root user for operations. You are prompted by the non-root user if the root user starts the NetBackup demon. The username must meet the criteria shown. 

1. Root accounts are not allowed. 

2. Accounts with access to the sudo utility are not allowed. 

3. The username must be 1-31 characters. 

4. The username must contain only English characters. 

5. Do not use the nbwebsvc user as the scale-out database user 

6. If the NetBackup daemons already start with a non-root user this same account is used to run the NetBackup database. Reference Article Requirements and need for database user with release 10.2 (or higher) 

• On Windows, the LOCAL SERVICE user is the default user for the “NetBackup Scale-Out Relational Database Manager” and “NetBackup Scale-Out Relational Database Connection Pool Service" services. 

• Port 13787 – At NetBackup 10.2+, The new database has a connection pooler service that, by default, uses port 13787. Confirm that this port is available for database operation before upgrading. Failure to provide valid port numbers renders NetBackup unusable until the port issues are resolved. 

• Snapshot of Primary Server – If the Primary Server is a Virtual machine, please take a VM-level snapshot before attempting the NetBackup upgrade.

• OpsCenter to NetBackup IT Analytics Transition - NetBackup 10.0 is the last NetBackup version to include NetBackup OpsCenter and OpsCenter Analytics.From NetBackup 10.1 onwards , NetBackup IT Analytics is the supported reporting tool.More information is available at FAQ document NetBackup OpsCenter last release and transition to NetBackup IT Analytics FAQ

   Note: Provisions have been made to support OpsCenter 10.0.0.1 with NetBackup 10.1 and higher, but it requires a support exception.

• Known Issue when upgrading NetBackup Primary Server configured with NetBackup Snapshot Manager - Mandatory actions required as explained in the article below while upgrading NetBackup Primary Server to version 10.3 with Snapshot Manager configured Mandatory actions required while upgrading NetBackup to version 10.3 with Snapshot Manager configured

If NetBackup Snapshot Manager exists in an environment, then NetBackup Snapshot Manager should be upgraded first. Specifically for 10.3:

• An EEB 4142147 is available that Veritas recommends to apply on the Primary server, Media server, Backup host server post upgrade to 10.3
• An EEB 4142186 is available that Veritas recommends to apply on the NetBackup Snapshot Manager Server.

Environment specific issues:

• Ensure that you have free disk space on the NetBackup Installation path that is twice the size of the NetBackup Relational database. 
• Large number of files at NetBackup Installation directory. The NetBackup primary server upgrade on Windows appears to hang at the step: Granting full control to NetBackup services over NetBackup files if there are large number of files in the NetBackup installation directory other than <NetBackup_installation_directory>\NetBackup\logs and <NetBackup_installation_directory>\NetBackup\db\images directories. The process of granting permissions to the NetBackup services for the NetBackup installation directory may take longer than expected. During primary server upgrade for NetBackup 9.1 or later on Windows, the upgrade appears to hang
• Rogue files such as 'C:\Program' may conflict with NetBackup - As of version 10.5.1, the NetBackup preinstall checker reports the existence of rogue files if found at installation and upgrade time. Ensure that no such rogue file exist at Primary Server.  Rogue files such as 'C:\Program' may conflict with NetBackup
• /tmp should not be mounted with noexec on the Primary Server. This can be verified with /etc/fstab

# cat /etc/fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel_redhat71-root                              /                         xfs       defaults                 0 0
UUID=cf1d6f54-ae59-499b-ba2c-8054dc897f49       /boot                   xfs       defaults                  0 0
/dev/mapper/rhel_redhat71-tmp                             /tmp                    xfs       defaults, noexec        0 0
/dev/mapper/rhel_redhat71-swap                           swap                   swap    defaults                   0 0

To fix it, either remove noexec from /tmp line at /etc/fstab or use the command below to remount /tmp with the exec flag.

mount -o remount,exec /tmp

 Please always engage the OS administrator to perform such tasks to ensure there is no breach of company policy or security measures.