In rare circumstances, multiple elasticsearch indexes are created before the actual limit for existing elasticsearch index is reached.

Problem

Enterprise Vault (EV) has a predefined limit for the Elasticsearch index (in terms of number of archives and maximum shard size) which, once reached, triggers the creation of a new Elasticsearch index. In rare circumstances, multiple Elasticsearch indexes are created before the actual limit for the existing Elasticsearch index is reached. Please refer Elasticsearch index to Veritas Enterprise Vault index volume mapping configuration parameters for more information on how Elasticsearch index to EV Index mapping is done.

Error Message

In some cases, the below events can also be seen in EV Event logs:

Log Name:        Enterprise Vault 
Source:          Veritas Enterprise Vault
Event ID:        41352
Task Category:   Index Volumes Processor
Level:            Error
Description:
The processing of the Index Volume has stopped following errors.

Archive name: SMTPJournal01
Task: <none>
Index Volume ID: 1FAE53D6C8AAA8E48A8812B20F6307F4F1110000evsite_44
Reason: <none>
Error Type: NonCritical

Description: Failed to create a new index evserver1_smtp_60. Check if elasticsearch is running and the total active shards count has not exceeded the maximum limit. 

Log Name:        Enterprise Vault 
Event ID:        41309
Task Category:    Index Admin Service
Level:            Error

Description:
The following error message occurred in the core indexing engine: Index creation has reached its maximum capacity. To occupy more indices, please provision a new Enterprise Vault indexing server to resume the indexing operations. 
 

Cause

An index is treated suitable for indexing if it is open and healthy, and the index size is below the threshold limit. In the existing implementation, checks are in place to validate if the index is open, healthy and its size is below the threshold limit. If any of these conditions are not met by the existing indexes, then a new index is created.

The possibility that an index is not in a healthy state could be a temporary issue which may occur due to environmental conditions such as a network glitch or other problems, and cause the new indexes to be created.

Solution

The fix for this issue has been implemented and will be released as a part of EV 14.5.2.

The below configurable settings (EVIndexVolumesProcessor.exe.config in the app settings section) have been introduced to decide when the new Elasticsearch index can be created:

CheckAllIndexesHealthBeforeCreatingNewIndex - By default, this is set to true and will cause an Elasticsearch indexes health check before creating a new Elasticsearch index.
NumberofTimesToCheckIndexHealth - By default, this is set to 3 and will cause a repetitive check for Elasticsearch index health before creating the new index.
WaitInSecsBeforeCheckingIndexHealth - By default, this is set to 300 (Seconds) and will be used as an interval to wait before rechecking the Elasticsearch index health.