Veritas Bug ID: ET 4078688

Problem Description: NetBackup 10.0.0.1 MSDP Preferred EEB Bundle

Version:  NetBackup 10.0.0.1

Installation Location:  Media Server, Client

Note:  If client side deduplication is used, this EEB should be installed on the client and MSDP server. All servers that are MSDP storage pools or can write to a MSDP storage pool should also have this EEB installed.

This Hotfix contains multiple Emergency Engineering Binaries which fix problems with MSDP Servers, NetBackup Appliances, and Flex Appliances.

Fixes Included in earlier versions of this NetBackup 10.0.0.1 MSDP EEB Bundle:

ET 4042622: Spoold crash while idle within an hour after being started
ET 4044194: Msdpcheck will not progress due to errors in CRQP relating to 'DecryptSoKey'
ET 4045989: Enhance crstats type of history to support cloud LSU
ET 4046228: Ushare backup failed while dumping data to Ushare, and there are lots of ERRORs in vpfsd log
ET 4048644: Enable compaction check dispite refdb state
ET 4048714: Run 'dcscan -a' hang
ET 4049897: Ushare Backup Error 40
ET 4050140: Vpfsd hang in user space after several rounds of dump/backup test
ET 4050311: Cachedb integrity check return error in case the db is busy
ET 4050705: Error in mounting few vpfs shares using loop devices
ET 4050954: Ushare crash after upgrade to 8.3.0.2
ET 4051431: Fix ushare backup issues
ET 4051604: Avoid calling chown on data filesystems whenever the MSDP WORM container is launched
ET 4051681: Remove the command option 'keydictutil --refresh'
ET 4051777: Files on snapshot corrupt with incorrect lmdb size
ET 4052988: SQL VLD cannot find boundary issue
ET 4053266: Soft link returned error
ET 4054341: Disable VLD if Oracle stream handler is enabled
ET 4055303: During dump data, turn on and turn off ingest mode some times, find some files with size 0 in UShare, but their original size are not 0
ET 4055933: Unable to do universal share sweep
ET 4056373: Fix a potential data loss issue for NBFS, aka SpoolerProtectAllDCs does not always protect all DCs as expected on multi-node setup
ET 4056402: When multiple instances, delete Ushare IA via Restful API, delete succeed, but vpfsd log has many related ERR message
ET 4056494: Tlog processing hangs after every 2 days on newly deployed 5250 Appliance running on 4.1 release
ET 4056713: Unable to configure universal share on a NetBackup instance
ET 4057843: Not a valid utf8 char error for the delete reference
ET 4058092: Handle create file group exception
ET 4058267: Catalog upload improvement
ET 4059625: Critical bpdm Storage Server Error: async_get_job_status: Replication started but failed to complete successfully: __process_refop_batch: Could not get reference operation status for SO : invalid argument
ET 4060495: Multiple versions of podb.journal and uldb.journal files were uploaded after running worm backups on Azure and HCP cloud
ET 4060776: Fix empty image name in job stats
ET 4061671: MSDP plugin does not enforce WORM lock immediately per fragment
ET 4062585: ERR 'CRTaskRefOpProcess: failed to protect DO xxx for dsid 3'
ET 4062823: Can't find SO X of DO Y
ET 4063757: MSDP Cloud - Seedutil - Poor performance, low dedupe rate for new clients
ET 4064005: BetterDR failed with po.db and podb.journal mark deleted and reported 'decrypt_podb_file():490, file error'
ET 4064340: NFS Time out errors for ushare on the client
ET 4064614: VM become unavailable intermittently after instant access restore successfully
ET 4065291: Backup fail when client side dedupe is used on a MSDP-C storage unit
ET 4065924: High number of pending EVENTS and replication is not working for over month
ET 4066300: Remove validate NFS client
ET 4066564: The progress info of replication job is wrong
ET 4066657: Image Cleanup status 1
ET 4067149: Msdpcheck can't identify corrupt images even though SO/DO is corrupt
ET 4067336: Need reload data container if can't find a SO/DO, because spoold may add more SO/DO after vpfsd open data container files
ET 4067425: Disable dcoffset for non-generic mode
ET 4067678: Add image deletion logging from SPAD side
ET 4067818: Spad memory usage exceeds 100 GB
ET 4068537: High memory utilization more than 90% after upgrade to 9.1 and performance degraded
ET 4071213: Unable to collect MSDP-Config from WORM container v16.0
ET 4073781: Large delays seen in duplication jobs to msdpc worm
ET 4075400: Replication error
ET 4076511: Need to trigger ushare checkpoint before backup to successfully backup all the files within ushare
ET 4076512: Replication failing due to worm lock error, which actually seems to be a access denied error
ET 4077867: Oracle DB backups (txndb,mddb,mgtdb,stdb) using NBU 10 Universal shares failed
ET 4078436: Spad threads waiting in Flex because of slow check in IsLocalBrowseEnable()
ET 4078880: Failed to apply WORM lock for backup/duplicate images
ET 4079044: Change to config2 run backup, lots of ERR in log 'Could not handle reference operation type 13'
ET 4079058: SPWS does not check vSphere certificate
ET 4080107: Instant Access for a VM with more than 15 drives on a PVSCSI controller fails
ET 4080388: Handle xfs dirty log and enhancement for executing command error message
ET 4080396: Expect get_image_worm_prop to fail with STS_ENOENT when PO was deleted
ET 4081598: Vpfsd report a variety of ERROR message
ET 4082935: 5.0 vpfsd core dumping
ET 4083403: Checkpoint exception handling for vpfsd
ET 4083906: Don't continue to do backfill if the data container is missing
ET 4085570: HSTS Missing from HTTPS Server
ET 4085977: Unable to duplicate some images back to MSDP from DD
ET 4085989: 'catdbutil --worm disable' not work
ET 4086305: Vmware IA rollback and create instance access vm failed after change IP from vlan324 dual stack to another vlan 316 Ipv6 only
ET 4086817: Potential file corruption when enlarge a small file to a big file in universal share
ET 4087672: Container mount namespace or SELinux policy issue on RHEL 8
ET 4088425: Use different default work thread number for normal mode and recovery mode
ET 4089240: Compaction check hang in case of data error
ET 4089931: VMware Linux clients fail incremental backups with STATUS 84 to Dedupe storage
ET 4091148: The command 'dedupe compaction start' report 'CRControlDSCompactStart failed: no route found'
ET 4091847: Support a special AIR case of multi-domain
ET 4092028: Universal Share mounts are not responding and production is down
ET 4093633: NFS world-readable files for ushare in Flex
ET 4095843: a2cCacheBuild takes too much time
ET 4096253: Ushare created before apply EEB backup failed at second time retry meet error network connection broken
ET 4097535: Lock contention between the Backfill thread and the fuse_do_work thread
ET 4097542: Instant access VM can't power on after worm container relocate to another node
ET 4098505: Dir track list is too big caused the performance issue and backup failure
ET 4099214: MSDP WORM images not identifiable as IMMUTEABLE after close, prior to setting INDELIBLE
ET 4099245: Check spad/spoold running permission in WORM environment
ET 4099530: NB_CC_AMZCrypt_812_Mig_SRV fail
ET 4100110: Ushare is slow
ET 4100217: After upgrading 3.0.0.1 to 3.1, rshell cmd 'support MSDP-log collect' reported /usr/bin/find: /msdp/data/dp1/pdvol/log/ocsd_storage: Permission denied
ET 4103255: For multiple vpfs instance, oracle copilot snapshot fail (156 Snapshot error encountered) if ushare is NOT on default vpfs instance /mnt/vpfs
ET 4103846: AWS WORM - many more verify operations than uploads, ocsd high CPU use and slow upload performance causing on prem storage to fill
ET 4104171: Some restores fail with 'bad file descriptor' status 83 after CC to MSDPC migration
ET 4106431: Orphan image cleanup
ET 4106538: Empty POs were not deleted for vpfsd and it caused file leak
ET 4107007: Ushare Protection Point Backups Fail with status 87
ET 4107282: NBDB crash intermittently on Flex
ET 4107472: The scan for Windows client backup folder with encryption data fails 'Failed to create the instant access mount'
ET 4107492: If VLD is enabled for ushare, sqlpackage extract action will fail if output file is saved on ushare
ET 4108120: Write error is not correctly handled for the bloom filter
ET 4108272: Orphan image cleanup
ET 4109502: File rename operation is not handled correctly for rsync
ET 4110248: Use big compaction check batch size to improve performance
ET 4110363: Orphan image could not be cleaned up with bpstsinfo cmd
ET 4111702: Scan 50 millions files hang due to cata2map hang with race condition of background space monitor thread
ET 4111955: Forced compaction doesn't work if there are a lot of data in MSDP
ET 4112037: AzureSQL backups hung and ushare is not accessible
ET 4112641: Vpfs write report IO error
ET 4112745: Alta Production: Backup from Snapshot job fails consistently with disk down error
ET 4112849: Vpfsd report EEROR message 'Failed to open env for db'
ET 4113395: Intermittent Oracle RMAN backup failed to MSDP with error status 84 and 39;error 2050022 software error 39;
ET 4113961: Slow backup performance on MSDP after upgrading to netbackup v10.0.0.1 from 9.1.0.1
ET 4114201: BackupIoRequest::ReportIoError: write failure on backup device, Operating system error 59(An unexpected network error occurred.)
ET 4114274: Cannot detect data container and lock file format is incorrect without special operations
ET 4114934: Enable direct IO with bdev for container platforms
ET 4115681: Copilot umount snapshot may cause primary share into bad status
ET 4116165: Dedupe shell ActiveDirectory configure will reset the smb.conf
ET 4116266: Skip file group deletion while ushare backup snapshot is being used to prevent potential data loss
ET 4116660: Backups of SQL07 Universal share are corrupt
ET 4117163: Don't get extent map by loading lmdb file
ET 4118121: Vpfsd core dump when NB_Decommission_MediaServer
ET 4118234: Do imagecheck for 4 GB image on cloud lsu, the tool stuck for a long time
ET 4118349: Backup and background task deadlock
ET 4118521: Stop backfill while taking snapshot
ET 4120242: Cleanup ushare backup snapshot copies may cause memory leak
ET 4120871: VM IA error 4004, /usr/openv/pdde/pdopensource/lib/libz.so.1:no version information available
ET 4123575: MSDP failing to clean-up log files
ET 4124651: Utility to recover (DR) from Azure soft deletion in case NBU/MSDP catalogs are gone by mistake
ET 4127761: Vpfsd crash on the instance netmediaprd01
ET 4128845: Spad hang due to lock leak
ET 4128953: All jobs hang on Win 2k19 high stream IO
ET 4130738: File path may be missing while rename a file in ushare
ET 4130860: Fix oracle copilot with ushare backup from snapshot fail with status 40/6 issue, only close file once when DumpBinary
ET 4130976: The command 'dcscan -a' hits 'too many open files' error
ET 4132111: Too many logs in online check log files
ET 4133919: Msdpcldutil failed to deal with the DELETE items in podb journal file, which leads to failure of image importing
ET 4136539: Avoid using garbage entries in extent map to generating references
ET 4139144: Ushare backup reports no such file or directory during ushare dump and backup concurrently
ET 4139748: Ushare backups always fail on 40 error, pdplugin log ERR: 'PDSTS: pd_test_for_open_server: ip-10-168-49-25.ec2.internal isn't embedded, and get_agent_cfg_file_path_for_mount failed:'
ET 4142962: Vpfsd hang during concurrent dumping data to ushare from 2 clients, the following ushare backup also hang
ET 4143329: Adjust the online check batch size
ET 4147534: Move extent failure due to not enough space
ET 4153909: Restore of Postgresl failed with error 2824 Failed to get export path and vpfs server, Instant Access API failed with error code: 2894
ET 4159647: Flush race condition observed while dumping data to UShare accelerator for RDS PGSQL Database

ET 4159725: After upgrade to Access 8.2, jobs start but do not progress
ET 4153102: Fix the boost_interprocess issue
ET 4160369: Universal share backup failing with status 40, vpfs logs show 'size should not be 0, but its size is 0'

Version 20 Adds:

ET 4161572: MSDP - UShares - SQL - mount slow
ET 4164426: After parallel duplication to MSPD-C WORM only first copy created gets as WORM locked