Important configuration, components and features of NetBackup Anomaly Detection

Description

This article will describe important configuration, components and features of Anomaly Detection:

A. Anomaly Detection Services at NetBackup

There are 3 processes that run under the Anomaly Detection module. 

• nbanomalymgmt
• nbanomalydetect
• nbanomalyalert

Once the upgrade or fresh installation is completed on Primary Server, Anomaly management service (nbanomalymgmt) starts automatically.

Two remaining services (nbanomalydetect and nbanomalyalert) will not start automatically.

B. How to enable/disable Anomaly Detection?

In the NetBackup Web UI, go to:

Detection and reporting > Anomaly detection -> Anomaly detection settings > Backup anomaly detection settings > Click Edit > Enable anomaly detection activities settings > Select

• Enable only for unstructured data - Enables anomaly detection for the following policy types: Standard, MS-Windows, NAS-Data-Protection, and Universal share.

 Note:-This is the default configuration for the fresh NetBackup 10.4 installation

•  Enable - Enables anomaly detection for all policy types except for the ones that are excluded in the Advanced settings > Disable policy type or specific features for machine learning.
•  Disable - Disables anomaly detection in NetBackup for all workload types.

C. Anomaly configuration to enable automatic malware scanning

1. Anomaly detection flow can trigger malware scan for those anomalies that have high severity
2. To enable automated scan for images on which the anomaly was detected, create the following configuration file at Primary Server

• Windows:<install_path>\veritas\var\global\anomaly_detection/anomaly_config.conf
• Linux:/usr/openv/var/global/anomaly_detection/anomaly_config.conf

3. Add the following contents in the anomaly_config.conf configuration file:

[AUTOMATED_MALWARE_SCAN_SETTINGS]
ENABLE_AUTOMATED_SCAN=1
ENABLE_ALL_CLIENTS=1
SCAN_HOST_POOL_NAME=<scan_host_pool_name>   # Default pool name

#Use specific pool for mentioned clients
NUM_CLIENTS_BATCH_SPECIFIED=2

4. All settings should be under [AUTOMATED_MALWARE_SCAN_SETTINGS]

• ENABLE_AUTOMATED_SCAN= 1 
  • Setting the value to 1 will start the Malware scan on the Anomalies generated with a high score.
• ENABLE_ALL_CLIENTS=1
  • This setting will enable all clients for scans.
  • If this value is 0, then scanning will happen only on the clients mentioned under ENABLE_SCAN_ON_SPECIFIC_CLIENT_<Batch_Number>
• NUM_CLIENTS_BATCH_SPECIFIED=<batches>
  • Defines the number of batches for different host scanning pools. 

D. Automated malware scan of anomaly tagged images in their DR or IRE environment (Secondary domain)

1.  In the secondary domain NBU Web UI, go to:

Detection and Reporting -> Anomaly detection -> Anomaly detection settings -> Backup anomaly detection settings -> Enable automatic scan for imported copies.

2. Follow the same steps listed in the section above to enable automatic malware scanning in the secondary domain.
3. All the other configurations are common for imported images and self-images of a domain, i.e. severity, pool score, ransomware ext.

E. Entropy and File Attributes

1. Entropy is the measure of randomness in a file. This is the method for measuring the uncertainty in a series of numbers of bytes.
2. Normal data will have less randomness compared to encrypted data. This randomness measure will help to find encrypted data and improve the quality of detected anomalies.
3. Ransomware attacks typically encrypt the data on the system. Threat vectors that encrypt files tend to abruptly change the entropy. Calculating the entropy of files helps to determine if the entropy has increased or not at the time of backup, so that a ransomware attack can be reported immediately.

4. NetBackup runs a real-time, in-line ransomware detection process (nbinlinerwdetect) that analyzes the entropy and the following file attributes, and detects anomalies based on:

• File size
• File extension
• File permission
• File access time
• File modified time
• File change time

5. To use the computation of entropy and file attributes feature, the NetBackup Primary Server and media server must be 10.4 or later, and NetBackup clients must be 10.3 or later.

6. Job types that the computation of entropy and file attributes feature supports:

• Backup
• Backup from Snapshot

7. Policy types that the computation of entropy and file attributes feature supports:

• Cloud-Object-Store
• FlashBackup
• FlashBackup-Windows
• MS-Windows
• Standard
• VMware (excluding Continuous Data Protection plan)

8. Protection plans that the computation of entropy and file attributes feature supports:

• VMware (excluding Continuous Data Protection)

9. COMPUTE_IMAGE_ENTROPY enables computation of entropy and file attributes in NetBackup that enhances cyber resiliency. This parameter needs to be enabled at the Primary Server with the following 3 options as values:

• ALWAYS
  • Computation of entropy and file attributes is always enabled.This is the default value.
• NEVER
  • Computation of entropy and file attributes is always disabled.
• IF_MANAGED_BY_ALTA
  • Computation of entropy and file attributes is enabled if Veritas Alta View manages the associated NetBackup primary server.
  • If Veritas Alta View does not manage the primary server, computation is disabled.

F. Client Offline Anomaly Detection Extension

1. This extension adds the capability to detect the failed backup of any malware attacked (attacks that uses encryption technology) clients.
2. When a backup of a malware attacked client occurs, then the backup fails with a specific status code 7647.
3. This extension checks for any backup failed with status code 7647 and reports it as an anomaly with high severity on the system anomaly tab and also raises global notification.
4. It is enabled from the Web UI:

Detection and Reporting > Anomaly Detection > Anomaly Detection Settings > System anomaly detection configuration > System Anomaly Detection > Enable Anomaly detection of offline clients

5. Once the anomaly is generated, this extension will raise a notification indicating the abnormal backup failure with the client details.
6. For example, if a client backup fails abruptly with error code 7647 due to a corrupted certificate, an anomaly gets generated.

7. Additionally, this extension will apply legal hold flag on all backed up images of that particular client, so any expiration attempt will be paused.
8. The process holds the images with fixed hold name as NBANEXCLIENTOFF_HOLD. 
9. After images are put on hold, users cannot expire those images unless the hold is removed manually.
10. Information about such images will be available in the log folder below on the Primary Server:

• Windows: \<install_path>\Veritas\logs\nbanexclientoff\
• Linux: /usr/openv/netbackup/logs/nbanexclientoff/

G. Image Expiry Anomaly Detection Extension

1. Image expiry anomaly detection extension adds the capability to detect any unusual manual image expiration, or any unusual image expiry date modification (causing early expiration). This capability will capture any destructive operations like image expiration.
2. It uses a machine-learning based model to form a normal trend of users who are expiring images manually or changing the expiry date. It will use the last 3 months of image expiry data from audit records. Only operations which are done manually will be used.
3. It accumulates the number of expirations/modifications done in a given time period window and forms a normal behavior done per user. And for a new time window, if there are image expiration /modifications done which are beyond the normal / usual pattern, this extension will raise the anomaly.
4. It forms a window of 5 mins per user to see how many images that user expired or modified. If it determines there are any expirations done by any user which are beyond the threshold, it will throw an anomaly warning.
5. Also, if any new user starts performing image expirations which the ML model has not seen in the past, it will generate anomalies for this activity as well.
6. This functionality needs to be enabled from the Web UI:

Detection and Reporting > Anomaly Detection > Anomaly Detection Settings > System anomaly detection configuration > System Anomaly Detection > Detect anomalies for image expiration operation

7. Multi Person authentication for image expiration can be enabled optionally, with a default retention of 52 weeks for such flagged images. When it is enabled, a ticket will be generated for each expiration, and it will need to be approved:

8. Below is an example of an anomaly notification when images were expired manually:

H. Minimum observation requirement for detection to start

1. Before NetBackup 10.4, this feature needs a minimum of 30 backups as a baseline for starting the detection.

• Even if the 30 backups are done in a single day, detection will start.

2. From NetBackup 10.4 and later, there is one more requirement to have training data for a minimum of one month.

• If 30 jobs are available for training, but those jobs all fall in the same month, detection will not start. 
• Once the training jobs run beyond a single month, then detection will start.

I. Rules-based Anomaly Detection

1. Rules engine-based anomaly detection allows you to define certain rules. If the threshold values that are defined in the rule are exceeded, anomalies are generated. For example, an anomaly is generated if a certain number of failed login attempts occur in a specified time period.
2. For each rule, you can configure the following parameters: execution frequency, query period, and threshold.
3. For the latest rules file, go to the Veritas Download Center and download the rules file (.zip) for which you want to generate anomalies.
4. After extracting the zip file, you will get the rules in json format.
5. To upload the rules from the NBU Web UI, go to:

Detection and reporting > Anomaly detection > Anomaly detection settings > System anomaly detection configuration > Rules-based anomaly detection

6. Click on Upload rules to upload the latest rule .json file downloaded in point 3:

7. Select the checkbox "Detect anomalies using NetBackup anomaly detection rules" and it will display all the available rules.

8. Select the rules that you want to enable and for which you want to generate anomalies.
9. Click Enable. NetBackup will generate anomalies that meet the rule criteria.

J. Risk engine-based anomaly detection

1. In a zero trust architecture, risk needs to be continually assessed and security defenses need to be introduced dynamically when the types of actions being invoked introduce risk to either the data protection platform or to the protected data.
2. NBU 10.3 introduced a rules engine that alerted users to risk as actions were performed in the data protection platform.
3. It introduces a risk engine that intercepts data destruction/security setting modification actions dynamically and introduces MFA or MPA security controls based on the risk assessed from multiple criteria. This allows NetBackup to be proactive in identifying risky operations and approve/reject them automatically.
4. The configuration is available in the NetBackup Web UI:

      Detection and reporting  > Anomaly detection > anomaly detection setting > system anomaly detection configuration > Risk engine-based anomaly detection

1. Detect suspicious image expiration - Use this option to detect when images are expired in an unusual or a suspicious manner. By default, a system anomaly is generated when the risk engine detects an unusual or a suspicious image expiration attempt and allows the operation to proceed. However, for additional security, you can configure multi-person authorization for such image expiration attempts, where an MPA approver needs to approve the operation.
2. Secure Critical Operations - Use this option to protect critical operations such as modifying global security settings and creating API key. When you select this option, you are required to re-authenticate yourself by entering the one-time password that you see in the authenticator application on your smart device before you perform the given critical operations. Ensure that you have configured multifactor authentication for your user account. If multifactor authentication is not configured, you are not prompted to re-authenticate.
3. Detect possible session hijack - Use this option to detect if there is a possible user session hijack by a malicious source.The risk engine detects if the same user session token is used by another IP address, and sends a maximum of 10 alerts per day.Click Edit and select the check box to terminate the user session when the risk engine detects that there is a possible session hijack.

K. Important Tuning and Configuration Parameters

Please note that these parameters need to be set in the following configuration file in Primary Server:

• Windows: <install_path>\veritas\var\global\anomaly_detection/anomaly_config.conf
• Linux: /usr/openv/var/global/anomaly_detection/anomaly_config.conf

TRIGGER_SCAN_FOR_LOW_SEVERITY=1 

• Setting value of 1 will trigger malware scan for low severity anomaly
• Setting value of 0 will not scan will not be triggered for low severity anomaly, this is also the default value.

TRIGGER_SCAN_FOR_MEDIUM_SEVERITY=1

• Setting value of 1 will trigger malware scan for medium severity anomaly
• Setting value of 0 will not scan will not be triggered for medium severity anomaly.This is also the default value.

TRIGGER_SCAN_FOR_SCORE_GREATER_THAN=2.5

• Setting the value eg. 2.5 will trigger malware scan for a score greater than equal to the given anomaly score. This is also the default value.
• The anomaly score should be a positive value and greater than -1.

TRIGGER_SCAN_FOR_RANSOMWARE_EXT_IMAGES=1

• When this is enabled with value 1,we will get the list of backup files from bpdbm
• For each backup file, backup file name would be parsed to get the extension and then it would get compared against the known ransomware extension
• Anomaly will be generated once the ransomware extension is detected for a particular backup id. 

DISABLE_DYNAMIC_PLUGIN_LOADING=0 

• This will allow you to start/stop the extensions dynamically.
• The value of 1 is to stop the loaded extensions and 0 is to allow the extensions to load dynamically.

ANOMALY_PLUGIN_PATH_ADDITIONAL=<path> 

• This setting will allow the ability to start/stop the extensions dynamically from a different location.
• The default location is netbackup/bin

LIVE_DATA_GATHER_FREQUENCY_IN_MIN=15 (default)

• Use this option to specify the frequency with which the data should be gathered for anomaly detection.

PURGE_DATA_INTERVAL_IN_MONTHS=12 (default)

• Use this option to clean up the database after the specified number of months.
• The maximum number of months that you can set is 12.