Veritas Bug ID: ET 4102406

Problem Description: NetBackup 10.1.1 MSDP Preferred EEB Bundle

Version:  NetBackup 10.1.1

Installation Location:  Media Server, Client

If client side deduplication is used, this EEB should be installed on the client and MSDP server. All servers that are MSDP storage pools or can write to a MSDP storage pool should also have this EEB installed.

This Hotfix contains multiple Emergency Engineering Binaries which fix problems with MSDP Servers, NetBackup Appliances, and Flex Appliances.

Fixes Included in this NetBackup 10.1.1 MSDP EEB Bundle:

In previous versions:

ET 4098505: dir track list is too big caused the performance issue and backup failure
ET 4100110: enhancement for mssql vld segmentation
ET 4097542: Instant access VM cannot power on after worm container relocate to another node
ET 4097535: See lock contention between the Backfill thread and the fuse_do_work thread
ET 4103523: fix vm browse issue, cata2map signal handling issue and assignment issue
ET 4103650: Port the fix for DTE mem leak to DD 17.1
ET 4103255: FLEX/NBA/BYO oracle copilot ushare support multiple vpfs instance
ET 4092554: Cluster resiliency for recover mode
ET 4097301: ignore status when getting recover mode nodes.
ET 4097573: get engine name instead of node host when getting recover mode nodes.
ET 4097838: Fix error message in non-cluster environment.
ET 4099020: add trace for cluster resiliency.
ET 4099129: filter SOFPs when it is in a recover or shutdown mode node.
ET 4099866: check if it is in recover mode before forward DC protection.
ET 4100941: fix volume donw error when getting KMS property.Incident
ET 4101970: support resiliency of recover mode in cluster.
ET 4100334: when create an image, the engine should be selected again for the old engine might be in recover mode.
ET 4106038: Properly handle .map files after CloudCatalyst to MSDP cloud migration
ET 4106579: return KMS error if there is a KMS issue instead of no space
ET 4106393: Should delete local container files after doing cloud compaction recovery.
ET 4109857: UShare dump hang and got ERRORs in vpfsd logs, seems that there is deadlock issue
ET 4109883: Fix circular wait locks issue
ET 4108993: Refresh nodes' mode in the phase of protecting DO before job finished
ET 4109502: file rename operation is not handled correctly for rsync
ET 4103846: Add the config to enable/disable cloud download cache file verfiy. By default, disable it
ET 4099112: Fix cata2map uid/gid overflow issue
ET 4109506: back port ushare backup fix to 10.1.1 EEB bundle
ET 4108752: UShare hang during dumping data
ET Porting fix of ET4107549, make startpoint always be first connection of 'current/new' set of connections
ET Porting fix of ET4106732, in vpfsd some place doesn't flush bin file after writing SO
ET 4104583 4100088: Back port data integrity enhancement for vpfsd running on cloud LSU1
ET 4108120: write error is not correctly handled for the bloom filter
ET 4106538: empty POs were not deleted for vpfsd and it caused file leak
ET 4107492: if vld is enabled for ushare, sqlpackage extract action will fail if output file is saved on ushare
ET 4107456: add refresh context map before checking remote data container protection

ET 4099245: Add runtime permission check and refuse to work in case of incorrect permissions
ET 4110248: Use big compaction check batch size to improve performance

ET 4109784: Fix cata2map coredump issue which is due to big SO split and read issue, fix some cata2map logging issue
ET 4113932: Removed early return that did not return spws CRQ back to the pool
ET 4107282: Upload the podb.journal instead of po.db when set image retention.
ET 4113858: file name is too long , change it to a short name
ET 4100193: do not close the image immediately incase it used by other thread
ET 4112773: Add retries to VxFS VX_SETEXT ioctl on failure if errno is EINTREINTR indicates that the system call was interrupted by a signal
ET 4104646: MSDP - WORM - Orphan Image Cleanup - Compliance Mode
ET 4112938: Fix incorrect permissions for history directorymsdpsvc:users -> msdpsvc:pduser
ET 4110390: MSDP API supports to add, delete and list samba user in AD mode
ET 4112849: vpfsd reported EEROR message: Failed to open env for db
ET 4112800: back port fix: Fix loop device conflict issue, fix unable to delete livemount issue
ET 4112721: Create aws or minio cloud failed in Disk volume with the specified name already exists. But disk volume name not used
ET 4112641: vpfs writes file reported IO error
ET 4111955: Forced compaction doesn't work if there are a lot of data in MSDP
ET 4111708: malware scan 50 millions files hang due to cata2map hang with race condition of background space monitor thread
ET 4111420: Fix a regression issue in cata2map when analyzing windows stream data
ET 4112037: AzureSQL backups hung. Ushare is not accessible
ET 4111794: master spad query local IRE queue.
ET 4108993: Add protection for data container in recover mode node.
ET 4113055: Fix issue getLivemount share status inactive because of race condition
ET 4114274: fail to mount vpfs may reset the file change list
ET 4114208: Retry EINTR failure from fstatfs for VxFS
ET 4111537: fix IA failure whenever the file's size is bigger than 50G
ET 4114638: if file name or directory name is too long ,change it to short name
ET 4112377: support to stop backfill with signal 
ET 4094472: reduce the not necessary protect share to spoold when destruct a file 
ET 4112641: vpfs writes file reported IO error
ET 4112849: vpfsd reported EEROR message: Failed to open env for db

ET 4112377: support to stop backfill with signal
ET 4112641: vpfs writes file reported IO error
ET 4094472: reduce the not necessary protect share to spoold when destruct a file
ET 4114201: BackupIoRequest::ReportIoError: write failure on backup device, Operating system error 59(An unexpected network error occurred.)
ET 4112849: vpfsd reported EEROR message: Failed to open env for db
ET 4115681: copilot umount snapshot may cause primary share into bad status
ET 4114934: enable direct IO with bdev for rhel8+ and all container based platforms
ET 4109784: fix IA issue when enabled TIR+ and diff/cum schedule type
ET 4115411: PDSTS: impl_write_image: max image size exceeded (51200 MB)
ET 4111420: add rx for dir during IA
ET 4116266: skip file group deletion while ushare backup snapshot is being used to prevent potential data corruption
ET 4112745: Load the config for cloud WORM storage before check the status.
ET 4116660 Fixes
ET 4116165: MSDP-7089 dedupe shell ActiveDirectory configuration fixes
ET 4118349: backup and background task deadlock
ET 4111420: add more logs on cata2map to logging extent locate failure
ET 4117747: Some image cleanup jobs exit with status 88
ET 4118480: usharecli read all OST image from C1F1, C1F2, C1F3 ... C1FX, not just read thefirst C1F1
ET 4118521: stop backfill while taking snapshot
ET 4118819: fix incorrect extent size for ADS stream.
ET 4119969: Ready pem from older path for NBU upgraded from older versions.
ET 4118676: fix a few issues on cata2map and lmdb
ET 4120242: cleanup ushare backup snapshot copies may cause memory leak
ET 4118121: add try catch for file system release
ET 4119452: After ushare backup, loop device might be hold by mtstrmd and other processes and could not release

ET 4096276: Fix Acc backup fail in upgrade case.
ET 4085562: Fix memory leaks  
ET 4124651: Utility to recover (DR) from Azure soft deletion in case NBU/MSDP catalogs are gone by mistake
ET 4132111: Too many logs in online check log files    
ET 4128845: Spad hang due to lock leak    
ET 4118234: Close dc data file to clean up the open files and free memory.
ET 4131754: If users touch compaction.all in mount point and trigger force full compaction, add a log message to indicate the full compaction has been started.
ET 4130976: Close the DC iterator when all entries are accessed
ET 4130860: Fix oracle copilot with ushare backup from snapshot fail with status 40/6 issue (errno 9 when dump file's SO list)
ET 4113961: Add option in pd.conf to allow MSSQL backup using multi-stream
ET 4130518: set vpfs dsid encryption when dsid 0 is set during initialization.
ET 4130738: file path may be missing while rename a file in UShare
ET 4127761: snapshot may mount incorrect head volume
ET 4125603: Add engine down check to improve scache hookup performance
ET 4124834: Backport kek_tag_reporting and crcontrol createnewkek option, includes incidents 4118545, 4116618, 4113795, and 4115764.
ET 4128953: spoold's loopback noblocking socket stop work in windows 2019
ET 4133919: Verify the podb.journal in msdpcldutil.
ET 4126667: Initial config of MSDP fails on build 119 of DD 17.1 Flex container deployed on Access
ET 4123495: High memory RSS usage of vpfsd process after running Ushare dump and backups for several hours, cause pod/spoold restart

ET 4104218: Media Write Error(84)
ET 4138610: Media Server instance off-line due to /mnt/nbdata run out of inodes
ET 4126907: compaction needn't call loadIndexFromContainer() if s-p cache is enabled

ET 4139144: Ushare backup reports no such file or directory during ushare dump and backup concurrently.
ET 4133654: 1. fix malware scan high memory issue 2. fix malware scan get extent issue
ET 4141039: NetBackup Flex WORM Universal Share: incomplete MSDP cleanup or poor deduplication - PO size incorrect
ET 4133206: Ushare accelerator crashes while doing multiple types of operations against it

ET 4148056: Replication from MSDP-C pool are failing due to missing IM image component for all the SLPs.
ET 4146413: update im if tir or data vector is empty
ET 4110247: Validate host only if secure comms is enabled
ET 4147534: move extent failure due to not enough space
ET 4143329: Adjust the online check batch size
ET 4124639: During scalability case running, found ushare access hang issue
ET 4139748: chdir() should not be called from DeleteTree() especially in a multi-threaded process, otherwise CWD of the process would be wrong forever.
ET 4142962: vpfsd hang during concurrent dumping data to ushare from 2 clients, the following ushare backup also hang
ET 4145149: Improve SQL backup performance in NBFS
ET 4130378: create journal directory may fail
ET 4148056: Replication from MSDP-C pool are failing due to missing IM image component for all the SLPs.
ET 4136423: Check the storage worm mode when delete PO.
ET 4135096: WORM flag set to enable for backup images on non-worm MSDP pool on Access Appliance

ET 4153909: bad recovering file causes vpfsd cannot be repaired autmaticly 
ET 4146413: always update IM 
ET 4153152: ignore filename having surrogates
ET 4132244: Fix delete share may fail if some shares are initializing
ET 4153102: fix boost interprocess security issue. use programdata folder
ET 4127387: UShare dump failed and VPFS reported lots of stack trace ERRR, after rename failure
ET 4156775: increase inode space usage from 25% (default) to 80%
ET 4145688: limit malware scan retention to 3 days

ET 4155592: Fix vpfs_mounts hang when /etc/resolv.conf is empty.
ET 4158690: cata2map log improvement
ET 4113395: Ignore the verification on ctx->buf. Continue the backup.

ET 4158690: cata2map log improvementcata2map log save under storage path <storage>/log/vpfs/<proc_name>/<proc_name>.log
ET 4159164: missing nginx log directory
ET 4159647: Data corruption observed while dumping data to ushare accelerator for RDS PGSQL Database
ET 4156433: Mlock cache_root to forbid it being swap out
ET 4159426: MSDP-C disk pool volume is down after upgrade.
ET 4160369: Universal share backup failing with status 40, vpfs logs show 'size should not be 0, but its size is 0'
ET 4161572: MSDP - UShares - SQL - mount slow
ET 4159340: Split the init into two parts, create threads only if all other components have been initialized
ET 4163635: Set default WORM min = 1 day, max = 1 year in spadb
ET 4158447: BOCHK prod site Many Malware scan job Hangs, potential memory leak issue has in vpfsd
ET 4162340: Notify memory allocator to release memory if it reserve too much memory
ET 4158136: Log more details for I/O error

ET 4164426: After parallel duplication to MSPD-C WORM only first copy created gets as WORM locked
ET 4163270: BOCHK prod site, some of BackupIDs had 'Failed to create the instance access mount (4001).' error for Malware Scan
ET 4098027: change log level to lower level
ET 4176868: Spad crash due to incorrect lock
 

Version 26 Adds:

ET 4181003: Fix plain password in pdregister
ET 4181061: Fix 'crcontrol --memallocator release' hung
ET 4177104: The Instant Access of Malware Scan was deleted during scanning
ET 4175374: Fix chown for folder: if user or group is not provided, just keep original value
ET 4173259: Let merge continue if it has been canceled too many times
ET 4175807: Add try except when try to uuid of the device with blkid
ET 4171758: Check the uuid of device instead of the device name