Collection-Service.exe will crash periodically generating an Event ID: 1000 in the application log.

Problem

On a daily basis the collection-service.exe will crash generating an event ID 1000 in the application logs. There are no other evident effects from the crash as the collection-service.exe will restart and continue processing.

Error Message

The application log will show the following error:

Type : Error
Date : 12/5/2013
Time : 4:53:25 AM
Event : 1000
Source : Application Error
Category : Application Crashing Events
User : N/A
Computer : EVSERVER.EV.COM
Description:
Faulting application name: collection-service.exe, version: 8.1.1006.1002, time stamp: 0x4fec87c7
Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e
Exception code: 0xc0000005
Fault offset: 0x00000000000245d3
Faulting process id: 0x4140
Faulting application start time: 0x01cef19fa01996d1
Faulting application path: D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\collection-service.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 15313bfa-5d93-11e3-a62d-90b11c27db7c

Analysis of the dump file created will show the following:

FAULTING_IP:
ntdll!RtlReportCriticalFailure+62
00000000`77c540f2 eb00            jmp     ntdll!RtlReportCriticalFailure+0x64 (00000000`77c540f4)

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr ffffffffffffffff)
ExceptionAddress: 0000000077c540f2 (ntdll!RtlReportCriticalFailure+0x0000000000000062)
   ExceptionCode: c0000374
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 0000000077ccb450

DEFAULT_BUCKET_ID:  APPLICATION_FAULT

PROCESS_NAME:  collection-service.exe

ERROR_CODE: (NTSTATUS) 0xc0000374 - A heap has been corrupted.

BUGCHECK_STR:  STATUS_HEAP_CORRUPTION

Review of the system-reporting logs will show the following for one or more index volumes:

Started a [resume] crawl of [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757] in [live].

Could not start the [live] [indexer] ([D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live] for collection [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757]: <log ><log ><error time="1670" date="1389026026" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.
</string>. </error></log></log>

Could not start the [live] [indexer] ([D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live] for collection [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757]: <log ><log ><error time="3432" date="1389026028" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.
</string>. </error></log></log>

Could not start the [live] [indexer] ([D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live] for collection [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757]: <log ><log ><error time="4244" date="1389026029" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.
</string>. </error></log></log>

Could not start the [live] [indexer] ([D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live] for collection [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757]: <log ><log ><error time="5008" date="1389026029" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.
</string>. </error></log></log>

Could not start the [live] [indexer] ([D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service]) in [U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live] for collection [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757]: <log ><log ><error time="5944" date="1389026030" id="SERVICE_STARTUP_RECEIVE" >Failed to receive startup confirmation from the service <string name="command" >D:\Program Files (x86)\Enterprise Vault\EVIndexing\bin\indexer-service</string> in directory <string name="path" >U:\Index\index4\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live</string>: <string name="error" >An existing connection was forcibly closed by the remote host.
</string>. </error></log></log>

The [live] [crawler] for collection [1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757] terminated unexpectedly: [Attempting to start the indexer service failed more than 5 times. [Mon Jan 06 11:33:50 2014]].

Note: The sequence of events in the system-reporting logs will repeat consistently.

Note: Review of task manager for the effected collection-service process will show memory usage increasing then releasing with the overall trend being that memory usage gradually increases over a long period  of time.

Cause

An index volume(s) prevented the indexer from starting. It would continually attempt to start and consume memory until heap corruption was caused.
Attempts to Synchronize or rebuild the problematic index volume result in no progress being shown when monitoring the job and it never goes active. 

U:\Index\index1\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live Joe User

Solution

Resolved this issue by recreating the repository.xml as per https://www.veritas.com/docs/HOWTO56259 and renaming the problematic index volume(s) from

U:\Index\index1\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757\live Joe User  to U:\Index\index1\1ABFEE1D80968E34EAE1F3F2EC6265E0C_29757_OLD\live Joe User

After restarting the indexing service a rebuild of the index volume will complete successfully.

Applies To

Enterprise Vault 10 SP4 with Cumulative Hotfix 1.