Troubleshoot PaaS workload protection and recovery issues | Troubleshooting protection and recovery of cloud assets | NetBackup™ Web UI Cloud Administrator's Guide | Veritas™

NetBackup™ Web UI Cloud Administrator's Guide

Last Published:
Product(s): NetBackup (10.2)
  1. Managing and protecting cloud assets
    1.  
      About protecting cloud assets
    2.  
      Limitations and considerations
    3. Configure Snapshot Manager's in NetBackup
      1.  
        Configure a third-party CA certificate
      2.  
        Add a Snapshot Manager
      3. Add a cloud provider for a Snapshot Manager
        1.  
          IAM Role for AWS Configuration
      4.  
        Associate media servers with a Snapshot Manager
      5.  
        Discover assets on Snapshot Manager
      6.  
        Edit a Snapshot Manager
      7.  
        Enable or disable a Snapshot Manager
      8.  
        (Optional) Add the Snapshot Manager extension
    4. Managing intelligent cloud groups
      1.  
        Create an intelligent cloud group
      2.  
        Delete an intelligent cloud group
    5. Protecting cloud assets or intelligent cloud groups
      1.  
        Customize or edit protection for cloud assets or intelligent groups
      2.  
        Remove protection from cloud assets or intelligent groups
    6.  
      Cloud asset cleanup
    7.  
      Cloud asset filtering
    8.  
      AWS and Azure government cloud support
    9. About protecting Microsoft Azure resources using resource groups
      1.  
        Before you begin
      2.  
        Limitations and considerations
      3. About resource group configurations and outcome
        1.  
          Examples of resource group configurations
      4.  
        Troubleshoot resource group permissions
    10. About the NetBackup Accelerator for cloud workloads
      1.  
        How the NetBackup Accelerator works with virtual machines
      2.  
        Accelerator forced rescan for virtual machines (schedule attribute)
      3.  
        Accelerator backups and the NetBackup catalog
      4.  
        Accelerator messages in the backup job details log
    11.  
      Configuring backup schedule for cloud workloads
    12.  
      Backup options for cloud workloads
    13.  
      Snapshot replication
    14.  
      Configure AWS snapshot replication
    15.  
      Using AWS snapshot replication
    16.  
      Support matrix for account replication
    17.  
      Protect applications in-cloud with application consistent snapshots
    18. Protecting PaaS assets
      1.  
        Prerequisites for protecting PaaS assets
      2. Installing the native client utilities
        1.  
          Installing the MySQL client utility
        2.  
          Installing sqlpackage client utility
        3.  
          Installing Postgres client utility
      3.  
        Configuring the storage server for instant access
      4.  
        About incremental backup for PaaS workloads
      5.  
        Limitations and considerations
      6.  
        Discovering PaaS assets
      7.  
        Viewing PaaS assets
      8.  
        Managing PaaS credentials
      9.  
        View the credential name that is applied to a database
      10. Add credentials to a database
        1.  
          Creating an IAM database username
        2.  
          Creating a system or user managed identity username
      11.  
        Add protection to PaaS assets
      12.  
        Perform backup now
  2. Recovering cloud assets
    1.  
      Recovering cloud assets
    2.  
      Perform rollback recovery of cloud assets
    3. Recovering PaaS assets
      1.  
        Recovering non-RDS PaaS assets
      2.  
        Recovering RDS-based PaaS asset
      3.  
        Recovering Azure protected assets
      4.  
        Recovering duplicate images from AdvancedDisk
  3. Performing granular restore
    1.  
      About granular restore
    2.  
      Supported environment list
    3.  
      List of supported file systems
    4.  
      Before you begin
    5.  
      Limitations and considerations
    6.  
      Restoring files and folders from cloud virtual machines
    7.  
      Restoring volumes on cloud virtual machines
    8.  
      Troubleshooting
  4. Troubleshooting protection and recovery of cloud assets
    1.  
      Troubleshoot cloud workload protection issues
    2.  
      Troubleshoot PaaS workload protection and recovery issues

Troubleshoot PaaS workload protection and recovery issues

Backup fails with error: 3808 Cannot check if the database exists.

You can see the following message in the Activity Monitor:

AuthorizationFailed -Message: The client '<clientId> '<objetId>' does not have authorization to perform action 'Microsoft.Sql/servers/databases/read' over scope '<resoourceId>' or the scope is invalid. If access was recently granted, please refresh your credentials.

Explanation: This error occurs, when the Snapshot Manager and NetBackup are deployed in AKS, and:

  • The media server pod node pool is a different node pool from the Snapshot Manager node pool

  • Managed Identity is enabled in the Snapshot Manager Virtual Machine Scale set

Workaround: Do any of the following:

  • In the media server that you are using for backup and restore, enable Managed Identity on the Scale set. Also, assign required permission in the role attached to this managed identity.

  • Create a storage unit on the MSDP server, and use only those media servers that has Managed Identity enabled on Scale configuration.

Backup fails when the database or the resource group has Read-only lock applied, and partially successful when Delete lock is applied.

Explanation: This issue occurs if the Read-only lock or Delete lock attribute is applied to the database or the resource group.

Workaround: Before performing any backup or restore, remove any existing Read-only lock and Delete lock attributes from the database or the resource group.

Status Code 150: Termination requested by administrator

Explanation: This appears when you manually cancel a backup or a restore job from the activity monitor and a database is created on the portal during the partial restore operation.

Workaround:Manually clean up the database on the provider portal, and temporary staging location at the universal share mount location under specific directory created by database name.

Stale status messages in Activity monitor

Explanation: If the Snapshot Manager container service restarts abruptly; the provider protected restore jobs may remain in the active state and you may not see the updated status on the activity monitor details page.

Workaround: Restart the workflow containers using the following command in the Snapshot Manager:

docker restart flexsnap-workflow-system-0-min flexsnap-workflow-general-0-min

After restarting the containers, the restore jobs are updated with the latest status in the activity monitor.

Status Code 233: Premature eof encountered

Explanation: Appears if the client name used for backup exceeds the length of 255 characters.

The bpdbm logs confirms the same by displaying the following error message:

db_error_add_to_file: Length of client is too long. Got 278, but limit is 255. read_next_image: db_IMAGEreceive() failed: text exceeded allowed length (225)

Note:

This is observed when the primary server is RHEL.

Workaround:Rename the database such that the client name fits within the length of 255 characters.

Error: Broken pipe (32), premature end of file encountered EXITING with status 42, network read failed

Explanation:Occurs during backup if the client name used is big in length, due to this the file path length of catalog image exceeds 256 chars and fails with the above error message in activity monitor.

The bpdbm logs confirms the same by displaying the following error message:

<16> db_error_add_to_file: cannot stat(\\?\C:\Program Files\Veritas \NetBackup\db\images \azure-midb-1afb87487dc04ddc8fafe453dccb7ca3+ nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+ testdb_bidinet02\1656000000\tmp\catstore\ BACKUPNOW+141a73e7-cdc4-4371-823a-f170447dba2d_ 1656349831_FULL.f_imgUserGroupNames0): No such file or directory (2) <16> ImageReadFilesFile::get_file_size: cannot stat(\\?\C:\Program Files\Veritas\NetBackup\db \images\azure-midb-1afb87487dc04ddc8fafe453d ccb7ca3+nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+testdb_ bidinet02\1656000000\tmp\catstore\BACKUPNOW+141a73e7-cdc4-4371 -823a-f170447dba2d_1656349831_FULL.f_imgUserGroupNames0): No such file or directory (2) <16> ImageReadFilesFile::executeQuery: Cannot copy \\?\C:\Program Files\Veritas\NetBackup\db\images\azure-midb-1afb87487dc04ddc8fafe453dccb7 ca3+nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+testdb_bidinet02\1 656000000\tmp\catstore\BACKUPNOW+141a73e7-cdc4-4371-823a-f170447d ba2d_1656349831_FULL.f_imgUserGroupNames0

Note:

This is observed when the primary server is Windows.

Workaround: Rename the database such that the file path length fits within the length of 256 characters.

Status Code 3801: Cannot complete the requested operation.

Explanation:NetBackup is not able to successfully carry out the requested operation.

Recommended action: Please refer to the activity monitor details for possible reason of failure.

Status Code 3817: Cannot complete the pre-backup operation

Explanation: The error message seen in dbagentsutil logs as,pg_dump: error: query failed: ERROR: permission denied for table test;pg_dump: error: query was: LOCK TABLE public.test IN ACCESS SHARE MODE;Invoked operation: PRE_BACKUP failed

Occurs when you try to backup a database which has multiple tables with different roles. If tables have at least one different owner, other than the database owner, and it is not a member the database owner role, then the backup may fail.

Recommended action: You must have a role that has access to all tables inside the database which you want to backup or restore.

For example, say that we wanted to backup the School database which has two tables.

  • student, owner is postgres

  • teacher, owner is schooladmin

Create a new role. Say, NBUbackupadmin

Run the following command to create the role:

postgres=> CREATE USER NBUbackupadmin WITH PASSWORD '***********';

CREATE ROLE

To make this new role a member of postgres and schooladmin role, run:

postgres=> GRANT postgres TO NBUbackupadmin;

GRANT ROLE

postgres=> GRANT schooladmin TO NBUbackupadmin;

GRANT ROLE

Note:

You must have a role who is either owner or member of the owner of the table, for all tables inside database.

Backup fails with Status 40 (Network connection broken)

Explanation: Backups fail due to loss of connectivity to the media server.

Recommended action: You can restart the backup job if the policy has checkpoints enabled. Once the network issue is resolved, select the incomplete backup job in web UI and click Resume. The job resumes from the point it was stopped. If checkpoint is not enabled in the policy, the job shows up as a failed job in the web UI.

Backup job fails with the error: "Failed to backup database"

Explanation: The Job details contain additional details: ManagedIdentityCredential authentication unavailable. The requested identity is not assigned to this resource. The allocated media server doesn't have any Managed Identity attached to it.

Recommended action: If you use system or user managed identity for the PaaS Azure SQL and Managed Instances, apply the same set of permissions/rules to the media server(s) and the snapshot manager. If you use user managed identity, attach the same user managed identity to the media server(s) and the snapshot manager.

Error code 3842 - The requested backup type for the corresponding PaaS asset is unsupported.

Differential incremental backup is supported for only for Azure SQL server. When you select an unsupported backup type, this error appears.

Error code 3843 or 3844 - Failed to disable CDC.

Appears when you do not have permissions to enable or disable CDC.

Explanation: Give NetBackup the necessary permissions to enable or disable CDC in your Azure environment.

Note:

Do not enable CDC manually. Provide the permissions to NetBackup for enable or disable CDC.

Error: Client restore EXIT STATUS 5: the restore failed to recover the requested files Cloud policy restore error (2824)
Error: ERR - Failed to restore database [<db_name>] with name [<db_name>]. ERR - Failed to open file ''. Errno = 12: client restore EXIT STATUS 5: the restore failed to recover the requested files

Explanation: Occurs during restore if the backup image is generated on 10.2 media and restore goes to an older (< 10.2) media server.

Workaround: Change the restore media to 10.2 and remove the older media from storage.