How to backup indexes in Enterprise Vault 14.2 and later

Description

Enterprise Vault 14.2 has introduced Elasticsearch as its new indexing engine. All three types of indexes: Alta-Vista (32-bit), Velocity (64-bit), and Elasticsearch (64-bit) can co-exist in the EV environment. The data of all the three types of indexes are stored on the filesystem, within index locations configured for each respective indexing engine. Compared to the earlier indexing engines, Elasticsearch supports the configuration of a single index location. The Elasticsearch index data is stored on the filesystem in a single location configured on the Enterprise Vault index server.

Backup strategy

The backup strategy of the Elasticsearch indexes stored in the Elasticsearch index location is different as compared to the backup strategy of other Non-Elasticsearch indexes. The Elasticsearch engine does not support backing up index data through a filesystem backup.  Instead, it recommends taking snapshots of index data stored in the Elasticsearch index location. Enterprise Vault provides a set of Powershell commands to enable the snapshots of Elasticsearch index data.

Note: The backup strategy of Non-Elasticsearch indexes is unchanged and continues to support the backing up of index data through a filesystem backup.

Snapshot of Elasticsearch indexes

Enterprise Vault supports snapshot of Elasticsearch indexes. To store the snapshots, the Enterprise Vault administrator must configure the index snapshot location on the Enterprise Vault index servers.

The index snapshot location can be configured at each Enterprise Vault index server or each Enterprise Vault site. If you configure the index snapshot location at the Enterprise Vault site level, all the snapshots are stored at a single location. Enterprise Vault recommends that you store all your snapshots at a single location.

Note: Snapshots are incremental when stored in the same index snapshot repository and at the same index snapshot location. If a new index snapshot location is configured, a full snapshot is triggered.

Step 1: Plan storage for the index snapshot location

By default, Elasticsearch compresses snapshot files when stored on the filesystem. Compression is applied only to the metadata files (index mapping and settings) and not to the data files which are already compressed.

The Enterprise Vault Administrator must plan appropriate storage space for the index snapshot location by considering the (a) existing index size and (b) the future index size. Elasticsearch index data size is 8% of the total archived data. Ensure that the disk size of the Elasticsearch index snapshot location is 1.5 times the disk size of the indexed data. For example, if you have a planned disk size of 1 TB for your indexed data, the disk size of the Elasticsearch index snapshot location must be at least 1.5 TB.

Step 2: Configure the index snapshot location

The Enterprise Vault Administrator must configure the index snapshot location on each Enterprise Vault index server in the Enterprise Vault site. Enterprise Vault recommends having a single file share location to store the snapshots.

Use the Set-EVIndexSnapshotLocation PowerShell command to configure the index snapshot location at the Enterprise Vault site, as shown in the example below. Refer to the Enterprise Vault PowerShell Cmdlets guide for more details.

Example:

Set-EVIndexSnapshotLocation -SiteId “18F00EB8DCAE07D49B567698975CE32E91d10000evserver” -Path \\EVStorage\IndexSnapshotLocation

Where,

• SiteId: Entry ID of the Enterprise Vault site
• Path: Path to filesystem where snapshots will be stored

Step 3: Restart the Enterprise Vault Indexing Service on all the Enterprise Vault index servers

After the index snapshot location is configured at the Enterprise Vault site, the Enterprise Vault Administrator must restart the Enterprise Vault Indexing Service on all Enterprise Vault index servers. The restart creates the index snapshot repository in Elasticsearch running on each Enterprise Vault index server. Snapshots are stored in the index snapshot repository.

Step 4: Taking a snapshot of Elasticsearch indexes stored in Elasticsearch index location

Based on the organizational backup schedule, the Enterprise Vault Administrator must take a snapshot of the indexed data. 
Use the New-EVIndexSnapshot PowerShell command to create a new index snapshot at the Enterprise Vault site, as shown in the example below. Refer to the Enterprise Vault PowerShell Cmdlets guide for more details.

Example:
New-EVIndexSnapshot -SiteId “18F00EB8DCAE07D49B567698975CE32E91d10000evserver”

Where SiteId: Entry ID of the Enterprise Vault site.

Note: Snapshots are incremental when stored in the same index snapshot repository and at the same index snapshot location. If a new index snapshot location is configured, a full snapshot is triggered.

Step 5: Backing up the index snapshot location

The Enterprise Vault Administrator may wish to make an independent backup of the index snapshot repository or index snapshot location so that a copy of its contents can be used to recreate the index repository to a point-in-time state later. The contents of the index snapshot repository will be used to restore snapshots of index data for Elasticsearch indexes.

Enterprise Vault recommends regular backing up of the index snapshot repository or index snapshot location. 
The Enterprise Vault Administrator must ensure that the schedules of index snapshot and backup of index snapshot location are not conflicting. The New-EVIndexSnapshot Powershell command does not write to the index snapshot repository or the index snapshot location while their content is being backed up.

To avoid such conflicts, Enterprise Vault recommends that you set the index snapshot location in the backup mode before taking a backup of the index snapshot location. Also, after the backup of the index snapshot location is complete, clear the index snapshot location backup mode. Refer to the Generating PowerShell backup commands for your environment section in the Enterprise Vault Backup and Recovery guide for more details and the associated KB article.

From Enterprise Vault 14.5 onwards, Elasticsearch index snapshots save the State.dat index metadata to the snapshot location. Upon restoring the snapshot, the index metadata in the State.dat files is also restored. This enables the indexing engine to synchronize the index data automatically.

Note:
The snapshot and restore operation is limited only to the Elasticsearch index. If you want the Elasticsearch index and the archived items in storage to be backed up and restored at the same point in time, you must take a snapshot of the entire Enterprise Vault machine and its related machines (such as SQL server, domain controller (DC), storage server, and so on).