Description
This article will provide an overview of the important components of Malware scan functionality in NetBackup.
- NetBackup 10 introduces malware scanning of images for Windows and Standard policies, that are backed up to an IA enabled MDSP pool.
- Scanning images is a manual action, unless triggered by anomaly detection (this option requires configuration and is a separate feature).
- The feature has many dependencies, and is an orchestration of several NetBackup features including Instant Access/MSDP/API/AV engine.
- If NetBackup finds malware in a supported backup image, it can find the last known good image that is malware free.
- This feature is supported for all of the workloads listed below as of NetBackup 10.5.
- Unstructured Data workloads
- Standard
- MS-Windows
- NAS-Data-Protection
- Universal shares
- Kubernetes
- Structured Data workloads
- VMware
- Cloud
- Cloud-Object-Store
For best results, the MSDP EEB Bundle for the respective version of NetBackup should be installed on all of the associated NetBackup servers:
- 10.1 https://www.veritas.com/content/support/en_US/downloads/update.UPD446179
- 10.1.1 https://www.veritas.com/content/support/en_US/downloads/update.UPD405441
- 10.2 https://www.veritas.com/content/support/en_US/downloads/update.UPD907533
- 10.2.0.1 https://www.veritas.com/content/support/en_US/downloads/update.UPD631427
- 10.3 https://www.veritas.com/content/support/en_US/downloads/update.UPD815396
- 10.3.0.1 https://www.veritas.com/content/support/en_US/downloads/update.UPD126292
- 10.4 https://www.veritas.com/content/support/en_US/downloads/update.UPD107290
- 10.4.0.1 https://www.veritas.com/content/support/en_US/downloads/update.UPD448583
- 10.5 https://www.veritas.com/support/en_US/downloads/update.UPD455501
libguestfs
- Libguestfs is a set of tools and libraries for accessing and modifying virtual machine (VM) disk images.
- It was part of the VRTSpddei.rpm package in the Media Server software installation and required for malware scan of VMware backup images.
- From NBU 10.4 onward, VRTSpddei.rpm is no longer installed as part of a fresh installation on a BYO (Bring Your Own) Server Environment
- It is required to install libguestfs binaries manually as per the article: VRTSpddei rpm and libguestfs in NetBackup BYO
Scan Host Server:
- The Scan host Server does not need to have any NetBackup software installed on it.
- Malware Scanner Software needs to be installed on Scan Host Server.
- An RSA key will be needed to add the scan host to the credentials section in the NetBackup Web UI.
- To retrieve this, run the following command from the media server to the scan host:
ssh-keyscan <<hostname>> 2>/dev/null | grep ssh-rsa | awk '{print $3}' | base64 -d | sha256sum
- Pre-Requisite for Linux Scan Host - Prerequisites for Linux scan host
- Pre-Requisite for Windows Linux Host - Prerequisites for Windows scan host
- Scan host can be configured automatically using one of the following options:
- PowerShell: Scan host configuration using PowerShell
- Shell: Scan host configuration using Shell
- Ansible: Scan host configuration using Ansible
- The Scan Host Server can be configured with a non-root or non-administrator user.
- By default, three parallel scans are supported per scan host and this limit is configurable as per Configure resource limits for malware detection
- A Scan host pool is a group of scan hosts, and configurations must be performed from the NetBackup Web UI before the scan host configuration is complete.
- During Malware Scanner software installation on the Scan Host Server, it will ask to provide the path of Malware Scanner installation.
- Ensure the scan host engine path is defined in the system path:
Windows example :
Linux example:
Malware Scanner Software:
- Malware Scanner SDK binaries need to be installed on the Scan Host Server.
- The only supported malware scanners are:
- Symantec Protection Engine
- Microsoft Defender Antivirus
- NetBackup Malware Scanner (Avira)
- Scanner Configuration steps are available in the following articles:
- How to configure NetBackup Malware Scanner at Linux Server
- How to configure a NetBackup Malware scan-host on Linux using share type NFS and Avira malware scanner
- How to configure NetBackup Malware Scanner at Windows Server
- How to configure Symantec Protection Engine at Windows and Linux Server
- How to configure Microsoft Defender Antivirus at Windows Server
Instant Access:
- During Malware Scan, two instant access jobs start. The first job is for mounting, and the second job is for unmounting.
- Instant Access creates a VPFS (Veritas provisioning file system) image of the backup image present at MSDP disk.
- The VPFS image is created by leveraging the de-duplicated data blocks stored in the MSDP. This involves mapping the logical structure of the backup image to the physical storage blocks in the MSDP.
- The VPFS image is then mounted on the scan host server. The mounting process involves creating a virtual disk that represents the VPFS image. This virtual disk is then attached to the server, making the file system accessible.
- Once the VPFS image is mounted, it is exposed as a read-write file system. This allows the malware scanner to access the data almost instantaneously.
- Pre-requisite - MSDP build-your-own (BYO) server prerequisites and hardware requirements to configure Instant Access
Dynamic Scan:
- NetBackup 10.3 or later provides support for the dynamic scan feature.
- Dynamic Scan is supported on Standard, MS-Windows, Cloud-Object-Store and NAS-Data-Protection workloads.
- Dynamic scan optimizes instant access and scan performance for MSDP (for large number of files in the backup), and improves the instant access time as well as the scan performance.
- Dynamic scan is enabled by default.
- Dynamic scan can be turned off manually by adding the parameter below in the spws configuration file (<storage path>/spws/etc/spws.cfg) on the MSDP server:
[spws]
EnableDynamicScan=False
Differences between Traditional malware scans and Dynamic scans:
Key scanning procedure | Traditional malware scan using Instant Access mount points | Dynamic scan |
---|---|---|
Instant access stage | Analyzes the tar stream, then builds each file's header and extent map file (LMDB database). This is time consuming when there is a large number of files in the backup. |
Restores TIR (catalog database) and IM (image metadata) information from data fragments. |
Instant access share (NFS/SMB) is mounted and user tries to list or access the file. | Accesses the file header and reads the attributes from it. | Queries the directory from the catalog database to get all of the files and directories which are under this directory. It can also query each file and directory attribute to the output. |
Scan host opens a file | Opens and loads the LMDB database. | Builds the index in memory and reads directly from the data container:
|
Scan host reads a file | Searches from LMDB database and reads from data container. | If the storage server is from a 3rd party storage vendor, it reads the data through the OST interface directly. If the storage server is PureDisk, it searches from the mapping table and reads data from the data container. |
Mirror Servers:
- NetBackup Malware Scanner (Avira) mirror server must be configured for signature updates only when the scan host server does not have internet access.
- For NetBackup Malware Scanner, updates happen over HTTPS (https://oem.avira-update.com/update). Please provide access to this URL from the Mirror Server.
- The Mirror Server requires web server software (nginx, Apache etc.) installed and configured on it.
Example steps using nginx :-
a. Install nginx: yum install nginx
b. Edit /etc/nginx/nginx.conf
c. Add the following to the server block
location /home/update-path {
autoindex on;
root /;
}
Note: Here /home/update-path is being used as an example path to store Avira VDFs.
- This is the same path as the installation directory.
- This path can be modified as per requirement (/home/update-path is only an example)
d. Reload nginx
nginx -s reload
systemctl status nginx
- Enter the following in the browser to test the URL
http://{{MIRROR_SERVER_HOST}}/home/update-path
- Verify whether the path and the present files are accessed via the URL.
- If yes, execute the steps from: https://www.veritas.com/content/support/en_US/doc/21733320-165970098-0/v160413170-165970098
File Hashes:
- This feature is available from NetBackup 10.5 and higher versions only.
- Only Standard, MS-Windows and DNAS policies are supported to generate file hashes.
- The Accelerator option must be enabled in order to generate file hashes.
- This option allows users to search files by a file’s hash.
- The file’s hash is calculated based on the file’s content using the SHA256 algorithm.
- The file’s hash is calculated at backup stage.
- When backup job is finished, the file’s hash is stored in NetBackup catalog on the Primary Server then moved to search server(File hash server).
- One use case is to find malware files in the backup images.
- There are 3rd party authorities that periodically publish file hashes of known malware files.
- Enable File Hash functionality by selecting the calculate file hash check box in the policy attributes under the Use Accelerator option:
- This feature may affect the backup performance depending on the client configuration such as CPU and memory.
- If the CPU has SHA extensions, the hash calculation is faster than the CPU without SHA extensions.
- The NetBackup catalog is expected to increase by 20% or more.
- You can configure to delete the file hash information automatically from the NetBackup catalog after it is copied to the file hash server by adding the line AUTO_CLEAN_FILE_HASH_FROM_CATALOG = 1 in the bp.conf file on the primary server.
- To list all the backups that have file hash enabled, run the following command
Windows:<install_path>\veritas\netbackup\bin\admincmd\bpcatlist -file-hash-present
Linux: /usr/openv/netbackup/bin/admincmd/bpcatlist -file-hash-present
Process Flow of how File Hash works:
File Hash Server (also known as Search Server):
- The File Hash Server is designed to receive file hash information from a Primary Server on a periodic basis.
- The file hash information of backup images gets transferred from the Primary Server to the File Hash Server
- The File Hash server must be on a Red Hat Enterprise Linux media server, but it cannot be on a MSDP storage server.
- This means that the search server and MSDP storage server cannot be on the same media server.
- The search server also cannot be configured on a NetBackup Primary Server.
- A NetBackup domain can only have one search server.
- On the search servers, nginx must be installed and configured before installing the NetBackup media server software.
- Use the fhdb_config.sh script to configure a file hash search server. It takes one parameter which is the storage location (/search in this example) for file hash search files. The command below needs to be run on the File Hash Server:
#/usr/openv/pdde/pdcr/bin/fhdb_config.sh --hash-storage-path=/search
After the file hash search server is configured, run the command shown below on the Primary Server so that the Primary Server knows which media server is acting as the File Hash server:
#/usr/openv/netbackup/bin/goodies/nbfhsmgr -config <File_Hash_Server>
- On the Primary Server the /usr/openv/netbackup/bp.conf will display a new entry for the FILE_HASH_SERVER after the above configuration is complete.
- The primary server will now periodically send the file hash information to the search server.
- This time interval can be configured in /usr/openv/netbackup/bp.conf file on the primary server.
- The value of this parameter represents the amount of time in seconds for the interval that a primary server sends the file hashes to the search server.
- This configuration parameter is:
FILE_HASH_BID_BATCH_INTERVAL = 86400
- File Hash Server Configuration is explained here: Configuring the file hash server
Searching for files using the file hash:
- On the Search tab of the Catalog window, click the Action drop-down list and select “File hash search” from the list:
- In the File hash search window, input the list of file hashes into the search box at the bottom of the screen.
- One hash occupies one line.
- Each hash must be 64 characters.
Validating the scan host pool configuration
- On the left, click Detection and reporting > Malware detection.
- On the Malware detection page, click Malware detection settings on the top-right corner.
- After adding a new scan host or an existing scan host, on the Manage malware scanner hosts page, select the desired scan host and click Validate configuration from the action menu..
- On the Validate configuration page, enter the details to search and select an image to validate the configuration.
Note: Validating the configuration is only supported for Standard policy backup images.
- Select the backups to scan and click on Validate configuration.
- It is recommended to use backup images with a smaller number of files.
- For larger backups, IA creation may be delayed and the test scan might fail.
- After successful validation, click Finish.
- The Malware scanner host pools page will be displayed with the list of added scanner hosts.
Malware scanning for OST and AdvancedDisk
Support is available for malware scan for unstructured data only:
- Standard
- Windows
- DNAS backup policies with OST (DataDomain)
- Advanced Disk storage
Other workloads e.g. VMware etc are not supported.
Important Tuning and Configuration Parameters
More information about the following commands can be found in the Administrator's Guide or Security and Encryption Guide for the installed version of NetBackup:
MALWARE_SCAN_OPERATION_TIMEOUT
- This parameter is used to configure the duration of the scan operation that is allowed to run before a timeout occurs.
- By default, the scan operation timeout value is 2880 minutes (2 days) and it is configurable:
- The minimum supported value is 60 minutes (1 hour) and
- The maximum supported value is 43200 minutes (30 days).
- This parameter is only applicable on the media server.
- Example - MALWARE_SCAN_OPERATION_TIMEOUT = 2880
- Use nbgetconfig or nbsetconfig commands to view, add, or change the value of the timeout.
MALWARE_DETECTION_TIMEOUT_PERIOD
- This parameter is used to configure the duration of the scan operation that is allowed to run before timeout happens.
- By default scan operation timeout value is 48 hours (2 days).
- This parameter is only applicable on the Primary Server
- Example - MALWARE_DETECTION_TIMEOUT_PERIOD = 48
- Use nbgetconfig or nbsetconfig commands to view, add, or change the value of the timeout.
MALWARE_DETECTION_CLEANUP_PERIOD
- Malware detection performs automated cleanup of scan jobs which are older than 30 days in batches
- Cleanup runs every 24 hours after NetBackup has started.
- The value for this setting is in the number of days.
- Although any integer value greater then 0 is accepted, it is recommended not to set a value of more than 6 months (180 days).
- If an invalid value is configured, then the default value of 30 days is used.
- To switch off the cleanup period altogether use a value of 0.
- Example - MALWARE_DETECTION_CLEANUP_PERIOD = 30
- Modify the configuration parameters in the bp.conf file on the primary server.
MALWARE_DETECTION_CLEANUP_BATCH_SIZE
- It controls the automatic cleanup batch job size.
- The default value is 500.
- You can set any value between 1 to 5000 for the batch size.
- Example - MALWARE_DETECTION_CLEANUP_BATCH_SIZE = 600
- Set the parameter in the bp.conf file.
FAIL_SAFE_SCAN_RETRY_COUNT
- This parameter is used by the primary server to attempt the scan job on a different scan host from the same scan host pool.
- The following are the permissible values for the FAIL_SAFE_SCAN_RETRY_COUNT parameter:
Value Description
0 Disabled
1 Default retry count (By default, the fail safe scan would be triggered once and the maximum configurable value in bp.conf is 3)
Instant Access tuning parameters for malware scanning
- Perform the following to update the Linux kernel on the Storage Server.
- Add the following parameter with the value to the /etc/sysctl.conf file:
vm.max_map_count=262144
- After adding or changing the value, reload the configuration file using the command:
#sysctl -p
Resource limit for Malware Detection
- This parameter defines how many parallel malware scans per scan host can run simultaneously.
- It can be configured from the Web UI only (see the Web UI Administrator's Guide for more information).
- Configuration instructions see: Configure resource limits for malware detection
Viewing scan result failures:
- When viewing a scan result failure on the Web UI, hover over the failed entry to reveal more details of the error:
Useful commands:
1. Once NetBackup has scanned an image and identified and recorded malware in that image, it is possible to restore only the clean files and skip the infected files using the bpcleanrestore command. The bpcleanrestore command identifies the Malware infected backup images in a given date range, skips the infected files and restores the clean data.
This command is especially important during recovery from a malware attack, since the Java and/or WebUI UI may not yet be functional.
Examples:
Windows: <install_path>\veritas\netbackup\bin\goodies\bpcleanrestore -restorecleandata -C myclient -S mymaster -t 0 -p myclientpolicy -st 01/01/2024 -et 01/10/2024
Linux: /usr/openv/netbackup/bin/goodies/bpcleanrestore -restorecleandata -C myclient -S mymaster -t 0 -p myclientpolicy -st 01/01/2024 -et 01/10/2024
2. Once the vpfs_share image is created on the storage server, you can use the mount command with verbose settings to troubleshoot the mount issues.
You can identify the share name from the virt-access.log and run the mount command on the Scan Host Server.
Example:
# sudo mount -vvv -t nfs -o ro,soft,timeo=600 <storage_server>:/mnt/vpfs_shares/36_1/36_1673545045 /tmp/malware-testmount.nfs: timeout set for Fri Jan 13 10:28:23 2023
mount.nfs: trying text-based options 'soft,timeo=600,vers=4.2,addr=2600:::::13,clientaddr=2600:::::::b95f'
mount.nfs: mount(2): No such file or directory
mount.nfs: trying text-based options 'soft,timeo=600,addr=2600:::::13'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 2600:80b:210:112::13 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 2600:80b:210:112::13 prog 100005 vers 3 prot UDP port 20048
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting <storage_server>:/mnt/vpfs_shares/36_1/36_1673545045
3. Display the NFS server mount by running the showmount command on the Scan Host Server:
#showmount -e <storage_server>
4. Verify connectivity on the ports below from the Scan Host Server to the Storage Server using curl commands:
#curl -vvv telnet://<MSDP_Storage_Server>:2049
#curl -vvv telnet://<MSDP_Storage_Server>:20048
#curl -vvv telnet://<MSDP_Storage_Server>:111
#curl -vvv telnet://<MSDP_Storage_Server>:867
For additional information see: The Smart Use of Malware Scanning in NetBackup
Also review the Related articles section below for a list of known issues with resolutions.