Enterprise Vault™ Classification using the Veritas Information Classifier
- About this guide
- Preparing Enterprise Vault for classification
- Setting up Veritas Information Classifier policies
- Defining and applying Enterprise Vault classification policies
- Defining classification policies
- Running classification in test mode
- Using classification with smart partitions
- Appendix A. Enterprise Vault properties for use in custom field searches
- Appendix B. PowerShell cmdlets for use with classification
- Appendix C. Classification cache folder
- Appendix D. Migrating from FCI classification to the Veritas Information Classifier
- Appendix E. Monitoring and troubleshooting
About policy conditions
A condition specifies the criteria that an item must meet for the Veritas to consider it a match. Your policies can contain any number of conditions.
This topic provides information on the following:
All conditions have this basic form:
property operator value
For example, in the following condition, "Content" is the property, "contains text" is the operator, and "Stocks" is the value:
The property specifies the part or characteristic of an item that you want to evaluate: its content, title, modified date, file size, and so on. When you choose a property from the list, the options in the two other fields change to suit it. For example, if you choose the "Modified date" property, the other fields provide options with which you can set one or more dates. For properties such as "Content" the available operators are as follows:
contains text
matches regex
matches pattern
is similar to
contains exact data match in
language is
contains entity
sentiment score
At the right of each condition, you can specify the minimum number of times that an item must meet the criteria for the Veritas to consider it a match.
Various applications that you use in your organization may add custom property information to the items that you want to classify. For example, when Enterprise Vault processes an item, it populates a number of the item's metadata properties with information and stores this information with the archived item: the date on which Enterprise Vault archived the item, the number of attachments that it has, and so on.
If you know the name of a property that particularly interests you, you can enter it as a custom field in your policy conditions.
While creating a policy if a required property is not available in the property list, you can create a new property by using custom property fields.
To create a new property, use custom property fields while creating or editing a policy as follows:
- Set the other fields as per steps given in the topic Creating policies.
- Under Conditions section, from the Property drop-down list, select a required custom property field: Custom date field, Custom number field, or Custom string field.
- Specify the name for the new custom property.
Note:
Custom property name must be same as the metadata property name as identified by text extraction engine, for example Apache TIKA. In case of Veritas Enterprise Vault, custom property name must match with one of the indexing properties.
- Complete the rest of the steps to create a policy.
The new policy is created with a new custom property.
Use the Veritas 's YAML file to add a custom property under the property list on the UI.
The metadataDefinitions section of YAML file lists all the existing properties in the property list as follows:
The following table shows the data structure for an existing property:
Property Item | Description |
---|---|
name | Specifies the metadata property recognized by the text extractor engine like Apache TIKA. In case of Veritas Enterprise Vault, specify the indexing properties captured. |
displayName | Name of the property as displayed in the property list on the UI, for example "Title". |
type | Associated property type, for example String, Datetime, or Number. |
aliases | Specifies the additional metadata properties to be mapped to displayName. |
To make this property available in UI under policy condition page
- Add the new property details as shown in the previous table to the metadataDefinitions section in YAML.
- Restart the Veritas service of respective application.
Observe the following guidelines when you set up a condition to look for specific words or phrases in the items that you submit for classification:
The condition can look for multiple words or phrases, if you place each one on a line of its own. An item needs to contain just one word or phrase in the list to meet the condition.
Select
to find only exact matches for the uppercase and lowercase characters in the specified words or phrases.Select
to find instances where the specified words or phrases are contained within other ones. For example, if you select this option, the word enter matches enters, entertainment and carpenter. If you clear the option, enter matches only enter.Similarly, if you select
, the phrase call me matches call media and recall meeting, but not surgically mend.You can place the proximity operators NEAR and BEFORE between two words in the same line. For example, tax NEAR/10 reform matches instances where there are no more than ten words between tax and reform. sales BEFORE/5 report matches instances where sales precedes report and there are no more than five words between them. The number is mandatory in both cases.
Note:
These proximity operators may not work as expected when evaluating formatted data, such as tables and spreadsheets. The conversion process that this data undergoes before it is classified can swap the order of the table cells. For example, suppose that a spreadsheet contains the word sales in one cell and report in the cell immediately to the right. This should match the operator sales BEFORE/5 report but may not do so after the spreadsheet has been converted, because the conversion process has transposed the two words.
Word and phrases can include the asterisk (*) and question mark (?) wildcard characters. As part of a word, an asterisk matches zero or more characters. On its own, the asterisk matches exactly one word. A question mark matches exactly one character. For example:
stock* matches stock, stocks, and stockings.
*ock matches stock and clock.
*ock* matches stock and clocks.
??ock matches stock and clock, but not dock.
sell * stock matches sell the stock and sell some stock, but not sell stock.
You can use wildcards in combination with the NEAR and BEFORE operators. For example:
s?l? BEFORE/1 stock* matches sold the stock, sell stocks, and sale of stockings.
Select
if you want to exclude specific words or phrases while evaluating the policy condition criteria.When you select this option, along with the inclusion terms, you can also define the terms that you want to excluded from the matching criteria.
For example, assume that a document contains a sample text "
". You want to hit on the terms "spoof, spoofed, spoofing" only, and want to avoid or exclude the terms spoofed email account, an email spoof. In such a scenario, you can provide the keywords "spoof, spoofed, spoofing" in the inclusion terms field, and provide the terms "spoofed email account, an email spoof" in the exclusion terms field as shown in the sample image below.This field allows only keywords based exclusion. It means the input field for exclusion terms will only accept keywords, and not regular expression, pattern, and so on.
Keyword based exclusion works only for the scenario where every inclusion term is completely contained within an exclusion term.
The group-level condition proximity option will not be available for a group if any of the underlying conditions has exclusions.
To use the exclusion policy condition,
A regular expression, or regex for short, is a pattern of text that consists of ordinary characters (for example, letters a through z) and special characters, called metacharacters. The pattern describes one or more strings to match when searching text. For example, the following regular expression matches the sequence of digits in all Visa card numbers:
\b4[0-9]{12}(?:[0-9]{3})?\b
Your regular expressions must conform to the Perl regular expression syntax.
See the online Help for the Veritas for extensive information on this syntax.
You may find it helpful to build and test your regular expressions using the free online tool at https://regex101.com. This tool displays an explanation of your regular expression as you type it, and also lists all matches between the regular expression and a test string of your choice. The default regular expression flavor, pcre (php), is compatible with the Veritas .
Note:
Looking for regular expression matches is considerably slower than looking for matches for specific words or phrases. You can greatly improve performance and accuracy by looking for instances where both types of matches occur in proximity to each other. To do this, set up an All of condition group that contains both a regular expression condition and a condition for finding specific words and phrases, and specify the required distance within which matches must occur. The Veritas first evaluates the condition and only then looks for a regular expression match.
A pattern match evaluates the selected item property against an existing Veritas pattern. Depending on the selected pattern, you may be able to set the confidence levels that you are willing to accept. A high confidence level is likely to produce fewer but more relevant matches.
Note the following if you do not get the expected results when you test a policy that makes use of a built-in pattern:
It is important to check that your test item meets the pattern confidence levels. For example, by default, the Credit Card Policy looks for content that matches the pattern "Credit/Debit Card Number" with medium to very high confidence. To meet the requirements of the medium confidence level, an item must contain either of the following:
A delimited credit card number (one that contains spaces or dashes between the numbers).
Both a non-delimited credit card number and one or more credit card keywords, such as "AMEX" or "Visa".
So, an item does not meet these requirements if it contains a non-delimited credit card number but it does not also contain credit card keywords.
After you click Test classification results window may fail to highlight some or all of the matches. This is a known issue with certain patterns only. A future version of the Veritas will correct the issue.
to view the results of a test, the
Unlike most classification techniques that rely on pattern matching to identify sensitive data, Exact Data Match (EDM) triggers a classification response when the actual data that needs to be protected is detected. By matching on the exact data, this reduces the rate of false positives and allows for much higher levels of accuracy in automatic classification. EDM uses a fingerprint method whereby an extract of a database or table is provided as source file in either CSV or TXT format. The table is ingested, and rules are created that indicate a match when one or more columns of a single row are detected in proximity. EDM is ideal when the identification of discrete customer data, employee data, and any other sensitive data repository maintained within a table is required.
To classify information using Exact Data Match
Create an EDM pattern by setting the configuration options and providing the source document (typically containing the desired fields exported from a data store, such as a database). See “To create an Exact Data Match based pattern”.
Use the resulting EDM pattern in any policy to be used for EDM based classification.
Exact Data Match can be enabled or disabled using YAML.
The Exact Data Match feature allows you to detect the specific data sets from a database. For example, employee records. You can match one or more fields and optional fields as per the configured proximity value. It supports large data sets (like database records) and text in all languages and provides data protection by hashing the stored data. The main benefit of using Exact Data Match is to reduce false positives by matching data exactly (unlike pattern-based matching).
For example, if you have the following content in the document to classify:
Name: Teresa M. Brown
Employee ID: 624828
and you are trying to match against the following EDM source document,
Then this will trigger a match.
Exact Data Match provides following benefits:
Provides the ability to detect specific data sets from a database. For example, employee records.
Supports matching of combinations of data. For example, matching one or more fields and optional fields as per configured proximity value.
Supports large data sets like database records.
Provides data protection by hashing of stored data.
Automatically synchronizes the exact data match rule pack (which is required for classification) on the remote classification servers. Manual intervention is not required.
Supports file encryption for the exact data match rule pack files with the mechanism similar to tenant-specific patterns, policies, and tags.
Supports the Min/Max disposition for exact data match type policy conditions while configuring policies.
Supports text in all languages.
To create a policy using an Exact Data Match pattern
- Follow the initial steps for creating or editing a policy as described earlier.
- In the operator list box, select contains exact data match in and then select the required exact data match pattern from the value list box next to it.
Note:
Under Conditions, the Min/Max disposition support is added for exact data match type policy conditions while configuring policies. You can specify the exact or more counts for keywords match. When you select the value more than 1, the Exclude repeats check box appears. If you select this check box, matches that are different from each other.
For example, a credit card condition with a minimum count of two requires two different credit cards in a single document.
- Click Save.
When you test a document against a EDM based policy, Veritas shows the result. Also, the first column of the matching row is highlighted.
Example 1:
If source document content is as follows,
with Exact Data Matching Options as follows:
Name | Value |
---|---|
First row contains column headers | Yes |
Column delimiter | , |
Perform hashing to secure data fields | No |
Use case-sensitive matching | No |
Proximity for matches | 200 |
Minimum columns to match | 2 |
All columns | Not selected |
And if test document content is as follows:
The classification result will show a match for two records Stuart, and James.
Example 2:
For same source document and test document as stated in earlier example, if Minimum Columns value is set to 3 as follows:
Name | Value |
---|---|
First row contains column headers | Yes |
Column delimiter | , |
Perform hashing to secure data fields | No |
Use case-sensitive matching | No |
Proximity for matches | 200 |
Minimum columns to match | 3 |
All columns | Not selected |
The classification result will show a match for single record, that is Stuart. Because all 3 fields from first record is present in test document.
Example 3:
For same source document and test document as stated in first example, if proximity value is set to 50 as follows:
Name | Value |
---|---|
First row contains column headers | Yes |
Column delimiter | , |
Perform hashing to secure data fields | No |
Use case-sensitive matching | No |
Proximity for matches | 50 |
Minimum columns to match | 3 |
All columns | Not selected |
In this case, required words are not within proximity of 50 characters. Therefore the result will show no match.
Classification performance for Exact Data Match based policy depends on following factors.
Number of records to be matched
Number of fields and field size
Data being classified
Number of matches
Proximity and column matches found
Compute hardware and available resources
You can set up a condition to restrict policy matching to items in a particular language. For example, set the condition like the one below to find items whose content is primarily in French:
One of the options in the language list is
. This option matches items that contain at least two languages.To safeguard against the Veritas ignoring items because it cannot determine their primary language, select
. The most common reason why the Veritas may be unable to determine an item's primary language is that the item has a very small amount of content.You can set up a condition to restrict policy matching to content that includes a person name or location.
Note:
The "contains entity" condition will only be available if nlp-service-0.1.6.jar
is used while running the Veritas application. Also, Named Entity Recognition (NER) is available only for English.
For example, set the condition like the one below to find content including the person name.
Note:
Named Entity Recognition (NER) consumes more time and resources compared to normal classification. NER is not suitable for large documents, especially documents bigger than 10 MB.
Risk score and risk level for each classified item is sent to the consuming applications. Consuming applications can analyze this information and support features such as sort, filter, search, and report on items by risk score and/or risk level. By understanding the level of risk, you can optimize efforts on data management, review, and control. You can prioritize activities and resources on items of highest risk.
The risk score and risk level are based on the number of pattern or policy condition hits. Items with more hits are categorized as high risk. Items with fewer hits are categorized as low risk.
In the YAML file, the previously used lowerRiskRuleNameParts parameter is deprecated, and the three new parameters - lowRiskUpperLimit, mediumRiskUpperLimit, and highRiskUpperLimit - are added. These parameters provide control over different risk level definitions based on the risk score value. This configuration defines the upper limit of the risk score range for the low, medium, and high-risk levels.
- It can be zero or greater than zero. By default, it is set to 2.
-It can be any non-zero positive integer. But must be greater than lowRiskUpperLimit value. By default, it is set to 5.
-It can be any non-zero positive integer. But must be greater than mediumRiskUpperLimit value. By default, it is set to 10.
This setting defines the upper limit of risk score range for low, medium, and high risk levels.
Condition | Risk Level |
---|---|
Risk score > highRiskUpperLimit | Very high |
highRiskUpperLimit >= Risk score > mediumRiskUpperLimit | High |
mediumRiskUpperLimit >= Risk score > lowRiskUpperLimit | Medium |
lowRiskUpperLimit >= Risk score >= 1 | Low |
Risk score = 0 | No risk |
The risk information is sent as part of classify response only if following conditions are met:
The matchDetailLevel is configured in classify request as either LOW/MEDIUM/HIGH
The item must have some risk based on the risk score and the risk level limits settings in the YAML file.
The risk computation of potentially sensitive content is based on the degree of hits against patterns or policy conditions and policy risk weight.
Note:
By default, the Risk weight value of all the custom policies and most of the built-in policies is configured as 1. For Subscription policy and all the Language detection policies risk weight is set to 0 by default.
Consider the following example. A document has the following classification result.
Policy name | Pattern name (match count) | Risk weight |
---|---|---|
Policy-1 | Pattern-A (2), Pattern-B (3) | 2 |
Policy-2 | Pattern-B (2), Pattern-C (1) | 0 |
Policy-3 | Pattern-C (3), Pattern D (5) | 1 |
Policy-4 | Pattern-C (1), Pattern E (1) | 5 |
The following table describes a sample scenario for risk score calculation.
Pattern name | Match count | From policy | Risk weight |
---|---|---|---|
Pattern-A | 2 | Policy-1 | 2 |
Pattern-B | 3 | Policy-1 | 2 |
Pattern-C | 3 | Policy-3 | 1 |
Pattern-D | 5 | Policy-3 | 1 |
Pattern-E | 1 | Policy-4 | 5 |
To calculate risk score, the policy with the highest risk weight is considered in case the pattern hitting on item is present in multiple policies as shown in the unique policy match table.
The risk score is a sum of the products of match count and the policy risk weight. The following steps explain the step-level actions for risk score calculation.
: Multiply match count with risk level.
: Repeat step1 for all the rows in the unique rule match table.
: Add the results of step2.
Risk Score = 2*2 + 3*2 + 3*1 + 5*1 + 1*5
= 4 + 6 + 3 + 5 + 5
= 23
Risk is categorized in different risk levels as per the risk score and is described in the About policy conditions section.
In the above example, the risk score is 21. Therefore, the Risk is categorized as:
Sentiment score/Named Entity based policy condition hits does not contribute towards risk score.
Contribution to the total item risk score will be zero due to Subscription policy and any language detection policy as all these policies have risk weight zero by default.
Due to language detection policy hits, some discrepancies may be observed in
results from analyzer overview page and analyzer PDF report. The result of the analyzer overview page is accurate.Following policy conditions contribute to risk score.
Content
Title
Author
Content Type
Recipient
Modified Date
Creation Date
Sensitivity
Category
Size (Bytes)
Custom date field
Custom number field
Custom string field
You can group a set of conditions and nest grouped conditions within other grouped conditions. The group operator that you choose determines whether an item must meet all, some, or none of the conditions in the group to be considered a match. The following group operators are available:
All of. An item must meet all the specified conditions.
Any of. An item must meet at least one of the specified conditions.
None of. An item must not meet any of the specified conditions.
Note:
You can nest a None of group within an All of group to look for certain condition matches while also excluding others. For example, to achieve the effect of "(condition X AND condition Y) BUT NOT condition Z", you would include the X and Y conditions in an All of group and the Z condition in a nested None of group.
n or more of. An item must meet the specified number of conditions.
For an All of group only, you can choose to look for instances where the conditions occur within a specified number of characters of each other. For example, the following condition group looks for instances where the word Goodbye appears within 20 characters of the word Hello:
The text string "You say Goodbye and I say Hello" matches these conditions because there are fewer than 20 characters between the first character of Hello and the first character of Goodbye. Similarly, the string "You say Hello and I say Goodbye" also matches because there are fewer than 20 characters between the ends of the two words. In each case, the spaces count as characters.
Note:
When you conduct within nn characters proximity searches, take care not to duplicate the same search terms across multiple conditions. For example, suppose that you define one condition to look for the names Fred, Sue, and Bob, and a second to look for Joe, Bob, and Sarah. An item that contains a single instance of Bob would match these conditions.
Rather than choose the
option, you can choose . This option looks for instances where the conditions occur within any sequence of characters of the specified number. For example, a condition group that looks for instances where the word Goodbye appears within a 20-character sliding window of the word Hello does not match "You say Goodbye and I say Hello". There are 23 characters between the start of the word Goodbye and the end of the word Hello.More Information