Image Cleanup job finishes with status 1 and 2060069 because the WORM expiration date is reflecting the completion time of fragments for the copy.

Problem

Image Cleanup job finishes with status 1 and 2060069 because the WORM expiration is set reflecting the completion time of fragments for the copy. The job details show the error: Failed to complete deleteimage webservice ((unknown)) V-454-47.

Error Message

Job Details

Dec 17, 2021 12:00:20 PM - Info bpdm (pid=154836) Failed to delete WORM locked image client1_1638486030_C1_IM: error 2060069.
Dec 17, 2021 12:00:20 PM - Critical bpdm (pid=154836) Storage Server Error: (Storage server: PureDisk:server.local) MBDeleteImage: Failed to complete deleteimage webservice ((unknown)) V-454-47

The spad session log for bpdm:

bpdm/spad/020222.log:20:02:47 ERR [372116] [140611802953472]: 25128: deleteImage: WORM image(/mnt/msdpcat/databases/catalog/2/client1//policy1/client1_1638486030_C1_F1.info) is NOT expired, time remaining in seconds: 1643753033, current time:1643753018

Note: This log is located in <storage path>/log/spad/<media server host name>/bpdm/spad/<date>.log

Cause

The WORM lock time is longer than the expiration time. This calculation is the difference between the copy1 backup and the most recent copy creation time.

• If the SLP experiences backlog, then the image locks will be "Copy creation time + retention time".
• For backups (not SLP operations) the delay time is the duration of the backup i.e. The difference from the backup start time in the backupid to the last fragment of image is completed on storage. "Backup ctime + delay time = Worm lock retention".

Fundamentally, WORM locks will be longer than the expiration time by the difference from the first copy's backup time and this copy's creation time, meaning that if a SLP operation is delayed by SLP backlog, the image locks will be "Copy creation time + retention”. For backups, the delay time is the real time to create the image (which could be days in some cases).

From NetBackup version 10.0 onwards this will still be the case, but NetBackup will respond to the offset by adjusting the expiration time in the catalog to reflect the actual worm lock duration. Then cleanup will only try to expire the image once the WORM lock is released and the files are truly removable.

Note: The system time on the storage server (MSDP WORM container) is not relevant since there must be protection against an attack on the time server. The WORM lock duration does not depend on the date/time of the system. This is the reason for the calculations above and there are no adjustments to the WORM lock duration for the delay time, because this cannot be assumed to be accurate.

Solution

As described above, NetBackup is working as designed and the images will be deleted by image cleanup after the delays are accounted for.