'The (synchronisation/rebuild/upgrade) paused as it was overloaded' messages occurring repeatedly

Problem

Synchronisation/Rebuild/Upgrade tasks appear to be stuck and not completing.

The subtask report file shows repeating entries like:

31/05/2011 16:12:57 Adding items to the new index volume.

31/05/2011 16:17:50 The rebuild paused as it was overloaded.
31/05/2011 16:18:50 The rebuild has resumed its processing.

31/05/2011 16:21:01 The rebuild paused as it was overloaded.
31/05/2011 16:22:06 The rebuild has resumed its processing.
 

Cause

Dtrace (of EVIndexVolumesProcessor) is required to determine why the index volume is being reported as overloaded as detailed below:

1) 'Throttling'

{EVSUBTASK} 03/06/2011 08:06:22 The upgrade paused as it was overloaded.

{VELOCITYPROXY} Enumerating Log Entries for 1AB4CF6711106A84C96F9384B9F9E2991_12355 - Symantec.EnterpriseVault.Indexing.IndexingEngineService.VelocitySrvRef.SearchCollectionAuditLogRetrieve

{IndexingWCFProxy`1} Calling SearchCollectionAuditLogRetrieve

{IndexingWCFProxy`1} [97ms] Called SearchCollectionAuditLogRetrieve

{VELOCITYINDEX} [98ms] DoEnumerateAcknowledgements  - IndexVolumeID=1AB4CF6711106A84C96F9384B9F9E29911110000ev_12355

{VELOCITYINDEX}  Did not retrieve any Acknowledgements

{VELOCITYINDEX} No acknowledgements from velocity

{STATUSCHECKER} Did not receive any Additions acknowledgments

{IndexingWCFProxy`1} Calling CollectionBrokerCrawlerOfflineStatus

{IndexingWCFProxy`1} [66ms] Called CollectionBrokerCrawlerOfflineStatus
{STATUSCHECKER} No Additions acknowledgements received for a while, waiting for 43 pending items. Number of unfruitful attempts 10. Indexing engine is probably too busy. Will STOP for now.
{STATUSCHECKER}  worker thread is stopping...

{STATUSCHECKER} Worker Thread has STOPPED

This indicates that items have been sent to the indexing engine but EV has not received confirmation that the items were successfully indexed (or rejected).
This is usually because the indexing engine is very busy.

2) 'CollectionBrokerCrawlerOfflineStatus'

<Agent List thread:14644>    EV-L    {IndexingWCFProxy`1} Calling SearchCollectionReadOnly

<Agent List thread:14644>    EV-L    {IndexingWCFProxy`1} [51ms] Called SearchCollectionReadOnly

<Agent List thread:14644>    EV-L    {IndexingWCFProxy`1} Calling CollectionBrokerCrawlerOfflineStatus

<Agent List thread:14644>    EV-L    {IndexingWCFProxy`1} [74ms] Called CollectionBrokerCrawlerOfflineStatus

<Agent List thread:14644>    EV-M    {SYNCHRONISATIONWORKITEM} The work is item overloaded [11099363C190CB84399E49E4B6E3140A31110000evshadow_14038]

This indicates that the call 'CollectionBrokerCrawlerOfflineStatus' has confirmed that there are items for this index volume that have been sent to the indexing engine that are still waiting to be processed. EV does not send more items until these items have been processed (either successfully indexed or rejected).

The most likely reason for items being stuck in this queue is that the index volume has been unloaded (by the indexing engine) to process other index volumes. When the index volume is loaded by the indexing engine these items are processed.

Solution

1) 'Throttling'

Enterprise Vault will keep checking for confirmation that the sent items have been indexed (or rejected).

If no confirmation is received then Enterprise Vault will resend the items after a period of time (at least an hour).

Excessive amount of 'throttling' may indicate that the indexing engine is being restricted by the available CPU or RAM on the indexing server.

2) 'CollectionBrokerCrawlerOfflineStatus'

Enterprise Vault will keep checking this queue and will continue sending more items when the queue is empty automatically.

In order for this queue to be emptied the indexing engine must load the index volume. This can only happen when there are sufficient resources available (RAM) which may require other loaded index volumes to be unloaded first.

You can force the indexing engine to load the index volume in this state by searching the index volume however this may result in other index volumes that are indexing being unloaded (and therefore will stop those index volumes being updated temporarily).

Applies To

Windows 2008 R2 onwards

Enterprise Vault 10 onwards