NetBackup™ Snapshot Manager Install and Upgrade Guide

NetBackup Snapshot Manager agent fails to connect to the NetBackup Snapshot Manager server if the agent host is restarted abruptly.

This issue may occur if the host where the NetBackup Snapshot Manager agent is installed is shut down abruptly. Even after the host restarts successfully, the agent fails to establish a connection with the NetBackup Snapshot Manager server and goes into an offline state.

The agent log file contains the following error:

Flexsnap-agent-onhost[4972] mainthread flexsnap.connectors.rabbitmq: error - channel 1 closed unexpectedly: (405) resource_locked - cannot obtain exclusive access to locked queue ' flexsnap-agent.a1f2ac945cd844e393c9876f347bd817' in vhost '/'

This issue occurs because the RabbitMQ connection between the agent and the NetBackup Snapshot Manager server does not close even in case of an abrupt shutdown of the agent host. The NetBackup Snapshot Manager server cannot detect the unavailability of the agent until the agent host misses the heartbeat poll. The RabbitMQ connection remains open until the next heartbeat cycle. If the agent host reboots before the next heartbeat poll is triggered, the agent tries to establish a new connection with the NetBackup Snapshot Manager server. However, as the earlier RabbitMQ connection already exists, the new connection attempt fails with a resource locked error.

As a result of this connection failure, the agent goes offline and leads to a failure of all snapshot and restore operations performed on the host.

Workaround:

Restart the Veritas NetBackup Snapshot Manager Agent service on the agent host.

NetBackup Snapshot Manager agent registration on Windows hosts may time out or fail.

For protecting applications on Windows, you need to install and then register the NetBackup Snapshot Manager agent on the Windows host. The agent registration may sometimes take longer than usual and may either time out or fail.

Workaround:

To resolve this issue, try the following steps:

Refer to the following for more information:

See Registering the Windows-based agent.

See Restarting NetBackup Snapshot Manager.

Disaster recovery when DR package is lost or passphrase is lost.

This issue may occur if the DR package is lost or the passphrase is lost.

In case of Catalog backup, 2 backup packages are created:

The DR package contains the NetBackup UUID certs and Catalog DB also has the UUID. When you perform disaster recovery using the DR package followed by catalog recovery, both the UUID cert and the UUID are restored. This allows NetBackup to communicate with NetBackup Snapshot Manager since the UUID is not changed.

However if the DR package is lost or the Passphrase is lost the DR operation cannot be completed. You can only recover the catalog without DR package after you reinstall NetBackup. In this case, a new UUID is created for NetBackup which is not recognised by NetBackup Snapshot Manager. The one-to-one mapping of NetBackup and NetBackup Snapshot Manager is lost.

Workaround:

To resolve this issue, you must update the new NBU UUID and Version Number after NetBackup primary is created.

After you get the NBU UUID and Version number, execute the following command on the NetBackup Snapshot Manager host to update the mapping:

/cloudpoint/scripts/cp_update_nbuuid.sh -i <NBU UUID> -v <Version Number>

The snapshot job is successful but backup job fails with error "The NetBackup Snapshot Managers certificate is not valid or doesn't exist.(9866)" when ECA_CRL_CHECK disabled on master server.

If ECA_CRL_CHECK is configured on master server and is disabled then it must be configured in bp.conf on NetBackup Snapshot Manager setup with same value.

For example, considering a scenario of backup from snapshot where NetBackup is configured with external certificate and certificate is revoked. In this case, if ECA_CRL_CHECK is set as DISABLE on master then set the same value in bp.conf of NetBackup Snapshot Manager setup, otherwise snapshot operation will be successful and backup operation will fail with the certificate error.

See Configuring security for Azure Stack .

NetBackup Snapshot Manager fails to establish connection using agentless to the Windows cloud instance

Error 1: <Instance_name>: network connection timed out.

Case 1: NetBackup Snapshot Manager server log message:

WARNING - Cannot connect to the remote host. SMB Connection timeout
 <IP address> <user>

…

flexsnap.OperationFailed: Could not connect to the remote server 
<IP address>

Workaround:

To resolve this issue, try the following steps:

Case 2: NetBackup Snapshot Manager log message:

WARNING - Cannot connect to the remote host. WMI Connection 
timeout <IP address> <user>

…

flexsnap.OperationFailed: Could not connect to the remote 
server <IP address>

Workaround:

To resolve this issue, try the following steps:

Case 3: NetBackup Snapshot Manager log message:

Exception while opening SMB connection, [Errno Connection error 
(<IP address>:445)] [Errno 113] No route to host.

Workaround:: Verify if the cloud instance is up and running or not in inconsistent state.

Case 4: NetBackup Snapshot Manager log message:

Error when closing dcom connection: 'Thread-xxxx'"

Where, xxxx is the thread number.

Workaround::

To resolve this issue, try the following steps:

Error 2: <Instance_name>: Could not connect to the virtual machine.

NetBackup Snapshot Manager log message:

Error: Cannot connect to the remote host. <IP address> Access denied. 

Workaround::

To resolve this issue, try the following steps:

NetBackup Snapshot Manager cloud operations fail on a RHEL system if a firewall is disabled

The NetBackup Snapshot Manager operations fail for all the supported cloud plugins on a RHEL system, if a firewall is disabled on that system when the NetBackup Snapshot Manager services are running. This is a network configuration issue that prevents the NetBackup Snapshot Manager from accessing the cloud provider REST API endpoints.

Workaround:

Backup from Snapshot job and Indexing job fails with the errors

Jun 10, 2021 2:17:48 PM - Error mqclient (pid=1054) SSL
Connection failed with string, broker:<hostname>
Jun 10, 2021 2:17:48 PM - Error mqclient (pid=1054) Failed SSL
handshake, broker:<hostname>
Jun 10, 2021 2:19:16 PM - Error nbcs (pid=29079) Invalid
operation for asset: <asset_id>
Jun 10, 2021 2:19:16 PM - Error nbcs (pid=29079) Acknowledgement
not received for datamover <datamover_id>

and/or

Jun 10, 2021 3:06:13 PM - Critical bpbrm (pid=32373) from client
<asset_id>: FTL - Cannot retrieve the exported snapshot details
for the disk with UUID:<disk_asset_id>
Jun 10, 2021 3:06:13 PM - Info bptm (pid=32582) waited for full
buffer 1 times, delayed 220 times
Jun 10, 2021 3:06:13 PM - Critical bpbrm (pid=32373) from client
<asset_id>: FTL - cleanup() failed, status 6

This can happen when the inbound access to NetBackup Snapshot Manager on port 5671 and 443 port gets blocked at the OS firewall level (firewalld). Hence, from the datamover container (used for the Backup from Snapshot and Indexing jobs), communication to NetBackup Snapshot Manager gets blocked. This results in the datamover container not being able to start the backup or indexing.

Workaround:

Modify the rules in OS firewall to allow the inbound connection from 5671 and 443 port.

Agentless connection fails for a VM with an error message.

Agentless connection fails for a VM with the following error message when user changes the authentication type from SSH Key based to password based for a VM through the portal:

User does not have the required privileges to establish an 
agentless connection

This issue occurs when the permissions are not defined correctly for the user in the sudoers file as mentioned in the above error message.

Workaround:

Resolve the sudoers file issue for the user by providing the required permissions to perform the passwordless sudo operations.

When NetBackup Snapshot Manager is deployed in private subnet (without internet) NetBackup Snapshot Manager function fails

This issue occurs when NetBackup Snapshot Manager is deployed in private network where firewall is enabled or public IP which is disabled. The customer's information security team would not allow full internet access to the virtual machine's.

Workaround:

Enable the ports from the firewall command line using the following commands:

firewall-cmd --add-port=22/tcp

firewall-cmd --add-port=5671/tcp

firewall-cmd --add-port=443/tcp

Restoring asset from backup copy fails

In some of the scenarios it is observed that the connection resets intermittently in Docker container. Due to this the server sends more tcp payload than the advertised client window. Sometimes Docker container drops SYN+ACK packet from new TCP connection handshake. To allow these packets, use the nf_conntrack_tcp_be_liberal option.

If nf_conntrack_tcp_be_liberal = 1 then the following packets are allowed:

If nf_conntrack_tcp_be_liberal = 0 then those are also rejected as invalid.

Workaround:

To resolve the issue of restore from backup copy, use the nf_conntrack_tcp_be_liberal = 1 option and set this value on node where datamover container is running.

Use the following command for setting the value of nf_conntrack_tcp_be_liberal:

sysctl -w net.netfilter.nf_conntrack_tcp_be_liberal=1

Some pods on Kubernetes extension progressed to completed state

Workaround:

Disable Kubernetes extension.

Delete listener pod using the following command:

#kubectl delete pod flexnsap-listener-xxxxx -n <namespace>

Enable Kubernetes extension.

User is not able to customize a cloud protection plan

Workaround:

Create a new protection plan with the desired configuration and assign it to the asset.

Podman container not starting or containers are not up after reboot

On RHEL 8.x platform, restarting container or machine reboot, the container displays the following error message:

# podman restart flexsnap-coordinator 47ca97002e53de808cb8d0526ae033d4b317d5386ce085a8bce4cd434264afdf:
 "2022-02-05T04:53:42.265084989+00:00 Feb 05 04:53:42 flexsnap-coordinator flexsnap-coordinator[7] 
agent_container_health_check flexsnap.container_manager: INFO - Response: b'{""cause"":""that name is already in use"",""message"":
""error creating container storage: the container name \\""flexsnap-agent.15bd0aea11164f7ba29e944115001d69\\"" is already in use by 
\\""30f031d586b1ab524511601aad521014380752fb127a9440de86a81b327b6777\\"". You have to remove that container to be able to reuse that
 name.: that name is already in use"",""response"":500}\n'"

Workaround:

Check if there is a file with IP address entry mapping to the container that could not be started at/var/lib/cni/networks/flexsnap-network/ file system location.

[ec2-user@ip-172-31-44-163 ~]$ ls -latr /var/lib/cni/networks/flexsnap-network/ total 16 -rwxr-x---. 1 root root 0 Jan 22 12:30 lock drwxr-xr-x. 4 root root 44 Jan 22 12:30 .. -rw-r--r--. 1 root root 70 Feb 4 14:47 10.89.0.150 -rw-r--r--. 1 root root 70 Feb 4 14:47 10.89.0.151 -rw-r--r--. 1 root root 70 Feb 4 14:47 10.89.0.152 -rw-r--r--. 1 root root 11 Feb 7 11:09 last_reserved_ip.0 drwxr-xr-x. 2 root root 101 Feb 7 11:13 . [ec2-user@ip-172-31-44-163 ~]$

From the above directory , delete the duplicate IP address file and perform the stop and start operation as follows:

Stop the container: #podman stop <container_name>

Start the container:#podman start <container_name>

After starting the start/stop services, NetBackup Snapshot Manager, RabbitMQ and MongoDB containers are still in the starting state

It was observed that flexsnap-mongodb and flexsnap-rabbitmq containers did not go into healthy state. Following is the state of flexsnap-mongodb container:

[ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='{{json .Config.Healthcheck}}'
flexsnap-mongodb {"Test":["CMD-SHELL","echo 'db.runCommand({ping: 1}).ok' 
| mongo --ssl --sslCAFile /cloudpoint/keys/cacert.pem 
--sslPEMKeyFile /cloudpoint/keys/mongodb.pem flexsnap-mongodb:27017/zenbrain --quiet"],
"Interval":60,"Timeout":30000000000,"Retries":3} [ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='
{{json .State.Healthcheck}}' flexsnap-mongodb {"Status":"starting","FailingStreak":0,"Log":null} [ec2-user@ip-172-31-23-60 log]$

Workaround:

Run the following #podman CLI(s) command:

[ec2-user@ip-172-31-23-60 log]$ sudo podman healthcheck run flexsnap-mongodb

[ec2-user@ip-172-31-23-60 log]$ sudo podman ps -a

CONTAINER ID   IMAGE                        COMMAND                         CREATED      STATUS                  PORTS                           NAMES

fe8cf001032b  localhost/veritas/   flexsnap-fluentd:10.0.0.0.9817         2 days ago    Up 45 hours ago      0.0.0.0:24224->24224/tcp        flexsnap-fluentd

2c00500c1ac6  localhost/veritas/   flexsnap-mongodb:10.0.0.0.9817         2 days ago    Up 45 hours ago (healthy)                            flexsnap-mongodb

7ab3e248024a  localhost/veritas/   flexsnap-rabbitmq:10.0.0.0.9817        2 days ago    Up 45 hours ago (starting)                           flexsnap-rabbitmq

[ec2-user@ip-172-31-23-60 log]$ sudo podman healthcheck run flexsnap-rabbitmq

[ec2-user@ip-172-31-23-60 log]$ sudo podman ps -a

CONTAINER ID   IMAGE                        COMMAND                        CREATED       STATUS                      PORTS                         NAMES

fe8cf001032b  localhost/veritas/    flexsnap-fluentd:10.0.0.0.9817        2 days ago    Up 45 hours ago          0.0.0.0:24224->24224/tcp     flexsnap-fluentd

2c00500c1ac6  localhost/veritas/    flexsnap-mongodb:10.0.0.0.9817        2 days ago    Up 45 hours ago (healthy)                             flexsnap-mongodb

7ab3e248024a  localhost/veritas/    flexsnap-rabbitmq:10.0.0.0.9817       2 days ago    Up 45 hours ago (healthy)                             flexsnap-rabbitmq

 [ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='{{json .State.Healthcheck}}' flexsnap-mongodb

{"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2022-02-14T07:32:13.051150432Z","End":"2022-02-14T07:32:13.444636429Z","ExitCode":0,"Output":""}]}

[ec2-user@ip-172-31-23-60 log]$ sudo podman container inspect --format='{{json .State.Healthcheck}}' flexsnap-rabbitmq

{"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2022-02-14T07:32:46.537804403Z","End":"2022-02-14T07:32:47.293695744Z","ExitCode":0,"Output":""}]}

[ec2-user@ip-172-31-23-60 log]$

Certificate generation would fail while registering NetBackup Snapshot Manager with NetBackup

Starting NetBackup Snapshot Manager release 9.1.2, NetBackup certificate generation will happen synchronously with registration in register API of NetBackup Snapshot Manager. Hence, any failure in certificate generation will cause failure while registering NetBackup Snapshot Manager with NetBackup, that is ading or editing the NetBackup Snapshot Manager entry from Web UI. These certificates are used for datamover which is launched for operations like backup from snapshot, restore from backup, indexing (VxMS based), and so on. Hence, if certificate generation fails, these jobs cannot be performed. Hence NetBackup Snapshot Manager on cloud VMs cannot connect to NetBackup on lab VMs, hence the registration will fail, and hence NetBackup Snapshot Manager cannot be added to NetBackup.

Workaround:

To add NetBackup Snapshot Manager in such scenario requires to skip certificate generation on NetBackup Snapshot Manager by adding the following entry in /cloudpoint/flexsnap.conf file:

[client_registration] skip_certificate_generation = yes

Default timeout of 6 hours is not allowing restore of larger database (size more than 300 GB)

Workaround:

Configurable timeout parameter value can be set to restore larger database. The timeout value can be specified in /etc/flexsnap.conf file of flexsnap-coordinator container. It does not require restart of the coordinator container. Timeout value would be picked up in next database restore job.

User must specify the timeout value in seconds as follows:

docker exec -it flexsnap-coordinator bash root@flexsnap-coordinator:/# cat /etc/flexsnap.conf [global] target = flexsnap-rabbitmq grt_timeout = 39600

Agentless connection and granular restore to restored host fails when the VM restored from backup has 50 tags attached to it

Workaround:

(For AWS) If a Windows VM restored from backup has 50 tags and platform tag does not exists, user can remove any tag that is not required and add the Platform: windows tag.

For few GKE versions, failed pod issues are observed in namespace

Following few failed pods in namespace is observed with failure status as NodeAffinity:

$ kubectl get pods -n <cp_extension_namespace>

NAME                                        READY     STATUS       RESTARTS     AGE
flexsnap-datamover-
2fc2967943ba4ded8ef653318107f49c-664tm        0/1     Terminating    0          4d14h
flexsnap-fluentd-collector-c88f8449c-5jkqh    0/1     NodeAffinity   0          3d15h
flexsnap-fluentd-collector-c88f8449c-ph8mx    0/1     NodeAffinity   0          39h
flexsnap-fluentd-collector-c88f8449c-rqw7w    1/1     Running        0          10h
flexsnap-fluentd-collector-c88f8449c-sswzr    0/1     NodeAffinity   0          5d18h
flexsnap-fluentd-ftlnv                        1/1     Running        3 (10h ago)10h
flexsnap-listener-84c66dd4b8-6l4zj            1/1     Running        0          10h
flexsnap-listener-84c66dd4b8-ls4nb            0/1     NodeAffinity   0          17h
flexsnap-listener-84c66dd4b8-x84q8            0/1     NodeAffinity   0          3d15h
flexsnap-listener-84c66dd4b8-z7d5m            0/1     NodeAffinity   0          5d18h
flexsnap-operator-6b7dd6c56c-cf4pc            1/1     Running        0          10h
flexsnap-operator-6b7dd6c56c-qjsbs            0/1     NodeAffinity   0          5d18h
flexsnap-operator-6b7dd6c56c-xcsgj            0/1     NodeAffinity   0          3d15h
flexsnap-operator-6b7dd6c56c-z86tc            0/1     NodeAffinity   0          39h

However, these failures do not affect the functionality of NetBackup Snapshot Manager Kubernetes extension.

Workaround:

Manually clean-up the failed pods using the following command:

kubectl get pods -n <cp_extension_namespace> | grep NodeAffinity | awk '{print $1}' | xargs kubectl delete pod -n <cp_extension_namespace>

Plugin information is duplicated, if NetBackup Snapshot Manager registration has failed in previous attempts

This occurs only when NetBackup Snapshot Manager has been deployed using the MarketPlace Deployment Mechanism. This issue is observed when the plugin information is added before the registration. This issue creates duplicate plugin information in the CloudPoint_plugin.conf file.

Workaround:

Manually delete the duplicated plugin information from the CloudPoint_plugin.conf file.

For example, consider the following example where the duplicate entry for GCP plugin config is visible (in bold) in CloudPoint_plugin.conf file:

{
    "CPServer1": [
      {
        "Plugin_ID": "test",
        "Plugin_Type": "aws",
        "Config_ID": "aws.8dda1bf5-5ead-4d05-912a-71bdc13f55c4",
        "Plugin_Category": "Cloud",
        "Disabled": false
      }
    ]
  },
  {
    "CPServer2": [
      {
        "Plugin_ID": "gcp.2080179d-c149-498a-bf1f-4c9d9a76d4dd",
        "Plugin_Type": "gcp",
        "Config_ID": "gcp.2080179d-c149-498a-bf1f-4c9d9a76d4dd",
        "Plugin_Category": "Cloud",
        "Disabled": false
      },
      {
        "Plugin_ID": "gcp.2080179d-c149-498a-bf1f-4c9d9a76d4dd",
        "Plugin_Type": "gcp",
        "Config_ID": "gcp.2080179d-c149-498a-bf1f-4c9d9a76d4dd",
        "Plugin_Category": "Cloud",
        "Disabled": false
      }
    ]
  }

Plugin information is duplicated, if cloned NetBackup Snapshot Manager is added into NetBackup

This occurs only when cloned NetBackup Snapshot Manager is added into NetBackup during migration of NetBackup Snapshot Manager to RHEL 8.6 VM. Cloning of NetBackup Snapshot Manager uses existing NetBackup Snapshot Manager volume to create new NetBackup Snapshot Manager. This creates duplicate entry into CloudPoint_plugin.conf file.

Workaround:

Manually edit and delete the duplicated plugin information from the CloudPoint_plugin.conf file.

For example, consider the following example where the duplicate entry for Azure plugin config is visible (in bold) in CloudPoint_plugin.conf file:

{
    "CPServer1": [
      {
        "Plugin_ID": "config10",
        "Plugin_Type": "azure",
        "Config_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521",
        "Plugin_Category": "Cloud",
        "Disabled": false
      }
    ]
  },
  {
    "CPServer2": [
      {
        "Plugin_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521",
        "Plugin_Type": "azure",
        "Config_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521",
        "Plugin_Category": "Cloud",
        "Disabled": false
      },
       {
    "cpserver101.yogesh.joshi2-dns-zone": [
      {
        "Plugin_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521",
        "Plugin_Type": "azure",
        "Config_ID": "azure.327ec7fc-7a2d-4e94-90a4-02769a2ba521",
        "Plugin_Category": "Cloud",
        "Disabled": false
      },
      {
        "Plugin_ID": "AZURE_PLUGIN",
        "Plugin_Type": "azure",
        "Config_ID": "azure.4400a00a-8d2b-4985-854a-74f48cd4567e",
        "Plugin_Category": "Cloud",
        "Disabled": false
      }
    ]
  }
 ]
}

Backup from Snapshot operation using Snapshot Manager version 10.0 deployed in Azure fails due to SSL cert error

Backup from Snapshot operation using Snapshot Manager version 10.2 deployed in Azure fails due to SSL cert error related to CRL (curl).

Workaround:

Add ECA_CRL_CHECK = 0 in Snapshot Manager bp.conf file and ensure that Azure endpoints are accessible from media server.