NetBackup™ for Cloud Object Store Administrator's Guide
- Introduction
- Managing Cloud object store assets
- Planning NetBackup protection for Cloud object store assets
- Prerequisites for adding Cloud object store accounts
- Configuring buffer size for backups
- Permissions required for Amazon S3 cloud provider user
- Permissions required for Azure blob storage
- Permissions required for GCP
- Limitations and considerations
- Adding Cloud object store accounts
- Manage Cloud object store accounts
- Scan for malware
- Protecting Cloud object store assets
- About accelerator support
- About incremental backup
- About dynamic multi-streaming
- About policies for Cloud object store assets
- Planning for policies
- Prerequisites for Cloud object store policies
- Creating a backup policy
- Policy attributes
- Creating schedule attributes for policies
- Configuring the Start window
- Configuring the exclude dates
- Configuring the include dates
- Configuring the Cloud objects tab
- Adding conditions
- Adding tag conditions
- Examples of conditions and tag conditions
- Managing Cloud object store policies
- Recovering Cloud object store assets
- Troubleshooting
- Reduced acceleration during the first full backup, after upgrade to version 10.5
- After backup, some files in the shm folder and shared memory are not cleaned up.
- After an upgrade to NetBackup version 10.5, copying, activating, and deactivating policies may fail for older policies
- Backup fails with default number of streams with the error: Failed to start NetBackup COSP process.
- Backup fails or becomes partially successful on GCP storage for objects with content encoding as GZIP.
- Recovery for the original bucket recovery option starts, but the job fails with error 3601
- Recovery Job does not start
- Restore fails: "Error bpbrm (PID=3899) client restore EXIT STATUS 40: network connection broken"
- Access tier property not restored after overwriting the existing object in the original location
- Reduced accelerator optimization in Azure for OR query with multiple tags
- Backup failed and shows a certificate error with Amazon S3 bucket names containing dots (.)
- Azure backup jobs fail when space is provided in a tag query for either tag key name or value.
- The Cloud object store account has encountered an error
- The bucket is list empty during policy selection
- Creating a second account on Cloudian fails by selecting an existing region
- Restore failed with 2825 incomplete restore operation
- Bucket listing of a cloud provider fails when adding a bucket in the Cloud objects tab
- AIR import image restore fails on the target domain if the Cloud store account is not added to the target domain
- Backup for Azure Data Lake fails when a back-level media server is used with backup host or storage server version 10.3
- Backup fails partially in Azure Data Lake: "Error nbpem (pid=16018) backup of client
- Recovery for Azure Data Lake fails: "This operation is not permitted as the path is too deep"
- Empty directories are not backed up in Azure Data Lake
- Recovery error: "Invalid alternate directory location. You must specify a string with length less than 1025 valid characters"
- Recovery error: "Invalid parameter specified"
- Restore fails: "Cannot perform the COSP operation, skipping the object: [/testdata/FxtZMidEdTK]"
- Cloud store account creation fails with incorrect credentials
- Discovery failures due to improper permissions
- Restore failures due to object lock
Restore failed with 2825 incomplete restore operation
Not all objects are restored from the backup image. Restore failed with 2825 incomplete restore operation.
Explanation:
This error can occur for multiple reasons. The most likely scenario for this error is when a cloud API initiated by NetBackup during a restore returns an error like an HTTP 400 status code (Bad Request). The reasons can vary with each cloud vendor. For example, GCP supports different Content-Language metadata as compared to AWS. In some cases, the error can also occur depending on the features enabled or disabled on a specific cloud account or bucket.
nbcosp log shows the following message:
{"level":"warn","error":"InvalidArgument: Invalid argument.\n\tstatus code: 400, request id: , host id: ","object key":"meta-user-defined/t2.rtf","time"…}
nbtar log shows the following message:
15:56:15.739 [22496.22496] <16> operation_to_cloud_by_type: ocsd reply with error, error_code: 400 15:56:15.739 [22496.22496] <16> CloudObjectStore::InitMultiPartUpload: operation_to_cloud_by_type() failed, status=3600 15:56:15.739 [22496.22496] <16> CloudObjectStore::ObjectOpen: InitMultiPart Upload call failed with status = 3600 15:56:15.739 [22496.22496] <16> cCloudApiRestoreHandler::writeOpen: ERR - ObjectOpen failed with error code [3600]
Workaround:
When the error is not fatal, the restore job is a partial success. Check the Activity Monitor to see the list of objects that cannot be restored. Try restoring to a different location (bucket/container or different account) to check if the problem is with the destination cloud account or bucket settings.
When the error is fatal, the restore job fails. Check the nbcosp logs to determine the object for which the restore has failed. Use granular object selection for the next restore, and skip the earlier failed object while selecting the objects.
Refer to your cloud provider documentation to check if you use a feature or any metadata that the cloud vendor does not support completely, or if it needs any more configuration. Fix the object with the right attributes in the Cloud object store and start a new backup job. Once this backup completes, the objects can be restored without this workaround.