Elasticsearch Snapshots Retention Management in Enterprise Vault 14.4 and later

Description

In Enterprise Vault 14.2, the Elasticsearch indexing engine was introduced, which changed the traditional way of file system backup of indexes to Elasticsearch snapshot. In addition, PowerShell commands were introduced in 14.2 to take snapshots and delete snapshots that are no longer required. Before Enterprise Vault 14.4, the Enterprise Vault administrator had to do the expiry of the index snapshot manually. Enterprise Vault 14.4 or later has introduced the Index Snapshot Expiry feature, a way to schedule and run the expiry of index snapshots based on the age of index snapshots. The age is calculated based on the creation date of index snapshots.

The index snapshot expiry feature is disabled by default, and the Enterprise Vault administrator must configure index snapshot expiry settings to expire index snapshots automatically. The Enterprise Vault administrator must configure the days after index snapshots expire and schedule when index snapshots should expire. 

Index snapshot expiry can be run immediately or scheduled for selected times. Furthermore, it can be run in normal or report mode. Unlike normal mode, in the case of report mode, index snapshots eligible for expiry are only reported and not actually deleted from the Enterprise Vault database and Elasticsearch indexing engine.

The Enterprise Vault indexing service running on the index server is responsible for deleting eligible index snapshots of that index server. Whenever index snapshots expiry runs, index snapshots from the Enterprise Vault database are first deleted and then deleted from the Elasticsearch indexing engine. In case of failure to delete from Elasticsearch indexing engine, the eventual consistency is done by indexing service by retrying 3 times (one retry a day at 12 AM). After 3 times, it will leave an orphan snapshot in the Elasticsearch indexing engine; but the index snapshot has been deleted in the Enterprise Vault database and will not be listed for the restore command.

Configure Index Snapshot Expiry

Schedule index snapshot expiry

Index snapshot expiry can be scheduled for selected times to delete expired index snapshots. Once the schedule hits, expired index snapshots will be deleted, and the index snapshot expiry report will be generated. However, if the Run in report mode option is selected, the expired snapshots will not be deleted, and only the details of index snapshots eligible for expiry will be gathered in the report. 
By default, index snapshot expiry is not scheduled. To schedule index snapshot expiry, follow the below steps.

1. Open Vault Administration Console (VAC).
2. Right-click on Site and select Properties.
3. Go to the Index Snapshot Expiry tab.
4. Enter Expire index snapshot after setting a value between 1 to 365.
   This value will be used to decide whether the index snapshot is eligible for expiry or not. By default, 30 days is set, meaning the index snapshot will be retained only for 30 days after its creation.
   
5. Select the Selected times schedule option.
6. Select the Interval setting value from 1 hour or 15 minutes. As per the interval value, below grid of scheduled time changes. As per the selected interval in the scheduled time grid, index snapshot expiry will run. Index snapshot expiry runs only once in the given interval.
   
   
   
7. Check the Run in report mode checkbox if you want to run the index snapshot expiry in the report mode otherwise, the index snapshot expiry will run in normal mode and delete eligible index snapshots from the Enterprise Vault database and Elasticsearch indexing engine.
   
8. Click Apply to save the settings in the database. 
9. Restart the Enterprise Vault Indexing service on each Enterprise Vault index server in that Enterprise Vault site.

Index snapshot expiry will run at selected times on each Enterprise Vault index server, and an index snapshot expiry report will be generated on each Enterprise Vault index server in that Enterprise Vault site.
Note: Select the Never option from the schedule grid to disable index snapshot expiry.

Immediate index snapshot expiry

Index snapshot expiry can be run immediately in the normal mode or the report mode by using the Run Now option. In normal mode, eligible index snapshots will be deleted from the Enterprise Vault database, and the Elasticsearch indexing engine and expiry report will be generated. In report mode, eligible index snapshots will not be deleted from the Enterprise Vault database and Elasticsearch indexing engine, and only an expiry report will be generated.

To run immediate index snapshot expiry, follow the below steps:

1. Open VAC.
2. Right-click on Site and select Properties.
3. Go to the Index Snapshot Expiry tab.
4. Enter a value between 1 to 365 for the Expire index snapshot after setting.
   This value will be used to decide whether the index snapshot is eligible for expiry or not. By default, 30 days is set, meaning the index snapshot will be retained only for 30 days after its creation.
   
5. Select Normal mode or Report Mode and click the Run Now button.
   This will immediately run the index snapshot expiry on each Enterprise Vault index server under the Enterprise Vault site.
   
   
   
   Index snapshot expiry will run immediately, and an index snapshot expiry report will be generated on each Enterprise Vault index server.
   In schedule or immediate index snapshot expiry, the reports are generated under <EV_INSTALL_DIR>\Reports\IndexSnapshotExpiry\<EV Server Alias>\. The alias of the Enterprise Vault index server is added for the Building Blocks deployment of Enterprise Vault. Inside the above folder based on normal/report mode and schedule/run now mode, another folder will be created, and the report will be generated inside that. For example, the RunNowNormalMode folder is created for the index snapshot expiry run immediately and in the normal mode.

Advanced Index Snapshot Expiry Settings

There are some Index snapshot expiry advanced settings that the Enterprise Vault administrator might have to configure if required. Following are the index snapshot expiry advanced settings:

Delete the latest index snapshot

By default, the latest index snapshot is not deleted in the index snapshot expiry run. To delete the latest index snapshot in the index snapshot expiry run, the above setting must be set to Yes.
To enable the deletion of the latest index snapshot in the index snapshot expiry run, follow the below steps.

1. Open the Enterprise Vault Administration Console.
2. Right-click on Site and select Properties.
3. Go to the Advanced tab and select the Index Snapshot Expiry settings section.
4. Double-click on the Delete latest index snapshot setting.
   
5. Select Yes to enable deletion of the latest index snapshot or select No to disable deletion of the latest index snapshot, then click on OK.
   
6. Apply the settings and restart the Enterprise Vault Indexing service.

Number of index snapshot expiry reports to keep

An index snapshot expiry report is generated on each Enterprise Vault index server after the index snapshot expiry run is completed. Enterprise Vault administrator can configure the number of index snapshot expiry report to be kept, after which the older reports will be purged. The value can be between 1 to 30, and that many numbers of index snapshot expiry reports will be kept in each index snapshot expiry mode. By default, 5 index snapshot reports are kept for each index snapshot expiry mode.

To set the number of index snapshot expiry reports to keep, follow the below steps:

1. Open VAC.
2. Right-click on Site and select Properties.
3. Go to the Advanced tab and select the Index Snapshot Expiry settings section.
4. Double-click on the Number of index snapshot expiry reports to keep setting.
   
5. Enter a number between 1 to 30 to set as the index snapshot expiry report count to keep, then click on OK.
   
6. Apply settings and restart the Enterprise Vault Indexing service.

Troubleshooting

Enterprise Vault administrator can check:

• Events in Event Viewer
• Index snapshot expiry reports are generated in the reports folder
• DTRACE of EVIndexAdminService for more details