Event 41293 Indexing Service start up error: 403 Forbidden

Problem

When attempting to start the Enterprise Vault (EV) Indexing Service the service stops and Event ID 41923 is logged stating HTTP request was forbidden.

Error Message

Source: Enterprise Vault  
Event ID: 41293
Description: An unexpected error has occurred.
Error Summary: Indexing Service start up error  
Error Details: The HTTP request was forbidden with client authentication scheme 'Ntlm'.
The remote server returned an error: (403) Forbidden.
V-437-41293
 

Dtrace Information:
To validate if this is the issue a Dtrace is needed, please perform the following steps on the EV server that has generated Event ID 41293 while re-creating the issue.

1. For more information about Enterprise Vault’s Dtrace Utility please see 100038975
2. Dtrace the process EVIndexAdminService in verbose mode.
3. With the Enterprise Vault Admin Service and Enterprise Vault Directory Service services started, attempt to start the Enterprise Vault Indexing Service through the windows Services.msc console.
4. Allow the service to fail startup.
5. Stop the Dtrace.
6. Open the log file created and search for either of the following the strings:

(EVIndexAdminService) <11020> EV-H {IndexingWCFProxy`1} Exception: The HTTP request was forbidden with client authentication scheme 'Ntlm'. Info: Diag: Type:System.ServiceModel.Security.MessageSecurityException ST:|Server stack trace: |   at System.ServiceModel.Channels.HttpChannelUtilities.ValidateAuthentication(HttpWebRequest request, HttpWebResponse response, WebException responseException, HttpChannelFactory factory)|   at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory factory, WebException responseException, ChannelBinding channelBinding)|   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)|   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)|   at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)|   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)|   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)|   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)|Exception rethrown at [0]: |   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)|   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)|   at Symantec.EnterpriseVault.Indexing.IndexingEngineService.VelocitySrvRef.VelocityPort.RepositoryListXml(RepositoryListXmlRequest request)|   at lambda_method(ExecutionScope , VelocityPort )|   at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method)|   at Symantec.EnterpriseVault.Indexing.Common.IndexingWCFProxy`1.Invoke[TResult](Expression`1 method) Inner:System.Net.WebException: The remote server returned an error: (403) Forbidden.|   at System.Net.HttpWebRequest.GetResponse()|   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

7. Open the most recent IIS logs and examine the logs requests going to the /evindexing/velocity.aspx virtual directory.

POST /evindexing/velocity.aspx v.app=api-soap& 80 - EXTERNAL_SERVER_IP - 403 6 5 0

If the IIS logs show the 403.6 response code then the issue may be present in the environment.

Cause

This issue can occur when localhost resolves to an IP Address other than 127.0.0.1

During startup and regular indexing operations the EV services communicate to the EV indexing engine via the IIS EVindexing virtual directory. By default the IIS EVindexing virtual directory is set to only accept http(s) set to the localhost ip address (127.0.0.1). When http(s) requests are sent to any IP address other than the localhost address the request is rejected with a 403.6 error code. When the request is rejected the Enterprise Vault Indexing Service fails to start.

Solution

1. Open the Windows Hosts file and look for an Alias pointing to the external EV server IP address

Locating the Windows server Hosts file:
https://technet.microsoft.com/en-us/library/ff807405(v=ws.10).aspx

2. The issue occurs when the Hosts file contains alias(s) for the EV server pointing to the external IP address of the EV server. To resolve the issue one of the following modifications may be necessary.

Please note: Option 1 is the recommended solution as it complies with Microsoft recommended host file settings, Option 2 and 3 should only be used in cases where Option 1 is not feasible.

Option 1: (Recommended Solution)

If the network DNS configuration is correct for the EV aliases, then the entries in the host file can be removed.

Option 2:

If the network DNS configuration is known to have issues and the entries are there as a valid work around, the entries can be modified to the local ip address of (127.0.0.1).

Option 3:

If the host file cannot be modified for valid network or troubleshooting reasons, then the IIS IP address restrictions can be modified to include the network IP address of the server.
 

Note: According to the Windows server documentation this file should be empty as it bypasses the network's DNS entries.  If there are entries in the host file it is important to find out why the Aliases are present.  If the entries are present to bypass certain network configurations, contact the network administrator to see if corrections to the network's DNS configuration are necessary.