How to turn on Transport Layer Security (TLS) for SQL server connections from an Enterprise Vault server

Description

In some environments, it may be necessary to require SQL server connections originating from an Enterprise Vault (EV) server to be made over TLS -- providing encryption in transit.

EV does not have any encryption-related settings that control the behavior, but SQL Server and/or SQL client components can be used to turn on encryption in transit.

One of two methods can be used.  Method 1 can be done at the SQL server level. Method 2 can be done per EV server.

Method 1: Force All Connections to the SQL Instance to Use TLS

This method enforces TLS encryption at the SQL Server level, affecting all client connections to the SQL instance.

Steps:

1. Open SQL Server Configuration Manager on the SQL Server or remotely via an MMC snap-in.
2. Navigate to:
   • SQL Server Network Configuration -> Protocols for MSSQLSERVER (or the specific instance name if not the default).
3. Right-click on Protocols for MSSQLSERVER and select Properties.
4. Change the Force Encryption setting to Yes. (Fig 1) 
5. Restart the SQL Server service for the changes to take effect.

Fig 1

Method 2: Configure SQL Components on EV Servers to Force TLS

This method enforces TLS encryption specifically for SQL connections originating from EV servers, without affecting other clients.

Step 1: OLEDB Settings

1. Open the Windows Registry Editor (regedit.exe) on the EV server.

2. For example, on a system with OLEDB version 18,  Navigate to:

   • HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\MSSQLServer\Client\SNI18.0\GeneralFlags\Flag1

3. Change the Value of Flag1 to 1 (Mandatory encryption). (Fig 2)

   • Flag1 = 1: Forces encryption for OLEDB connections.

   • Flag2: Optionally configure this key to control whether the server certificate is trusted (e.g., set to 1 to trust the server certificate).

Fig 2

Step 2: .NET Provider Settings

1. Run the SQL Server Client Network Utility (cliconfg.exe) on the EV server. Both settings may be required.  

   • For 64-bit systems: %systemroot%\system32\cliconfg.exe

   • For 32-bit systems: %systemroot%\syswow64\cliconfg.exe

2. On the General tab:
   • Ensure TCP/IP is listed under Enabled protocols by order.
   • Check the box for Force protocol encryption. (Fig 3) 
3. Click OK to save the changes.

Fig 3