DFS 9.0.0 - The underlying connection was closed: An unexpected error occurred on a send.

On a few client installations we have seen the following error on web service requests

{
	"statusCode": 0,
	"content": "",
	"headers": [],
	"responseUri": null,
	"errorMessage": "The underlying connection was closed: An unexpected error occurred on a send."
}

The symptoms was that everything would work just fine and then suddenly everything would stop working and we would see that every web service call would have that error message.

Resolution

According to Sitecore, this is not an unknown bug. See this article it is related to which security protocols the server accepts and basically it can be resolved by adding an initialization pipeline with the following code:

public void Process(PipelineArgs args)
{
  System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12 | System.Net.SecurityProtocolType.Tls11 | System.Net.SecurityProtocolType.Tls | System.Net.SecurityProtocolType.Ssl3;
}

We still saw some issues after we added this, so the next step was to investigate any custom ccode that was in the Sitecore installation. It turned out that some custom code (written by the Sitecore developers) was actually setting this ServicePointManager.SecurityProtocol to something else which was not supported by the server. Seeing as this is a static variable, then changing it will overwrite anything that has been set on initialization and hence we could enter this state again. We expanded the custom client code to support more protocols and the problem was resolved.