10 % of message don't reach the endpoints in BizTalk

See more:

biztalk

server

Microsoft

I am working on a problem plaguing an already established client system.
So don't ask why this is set up this way :)
The client doesn't want any major changes to the system as this is working on production.

The system is set up this way:

1. BizTalk receives a message (let us call this msg) (~1 MB in size) from a Web Service (ws1).
2. An orchestration (orch1) processes it (assigns some variables) and publishes it to MessageBox.
3. Another orchestration (orch2) picks it up and sends a response message (an acknowledgement or let us say a dummy) to ws1 asynchronously using the sync - async approach (correlation EPMRR token).
4. The dummy from biztalk is picked up by ws1 and forwarded to another ws2 which enters this into a DB.
5. After the dummy is sent from orch2(only sent from the logical send port... no way of verifying whether this actually reached or not)
6. The orch2 sends the msg to 4 endpoints in parallel. (say ws3, ws4, ws5, ws6).
7. After orch2 receives a response from ws3, ws4, ws5, ws6 these are again sent to ws2 to enter in the DB.

I hope I made it clear enough.

Then:
1. orch2 sends the dummy to ws1 which in turn sends it to ws2. (This is always successful).

2. Sometimes orch2 sends msg to ws3, ws4, ws5, ws6 but doesn't get a response. So it sends the error to ws2 to enter in the DB. (No retries).

And this is the problem:
10% of the messages are either not making it to ws3, ws4, ws5, ws6 or BizTalk received the responses but they are not making it to ws2.

So we are only able to see the dummy but not the response for msg.

The error is:
"The underlying connection was closed: An unexpected error occurred on a receive."

I have tried the following:
1. Fiddled with keep - alive.
2. Increased the timeouts upto 20 mins during testing.
3. Checked processor utilization, memory utilization etc.. to see if there is a bottleneck somewhere. Nothing.

No difference.
Also there are no failures on DEV. The issue is on QA, UAT, PROD.

Also the number of messages being sent to BizTalk is 80 messages every 3 minutes. This is fixed.

Need suggestions.

Thanks for your help in advance.