BizTalk WCF-Custom Adapter (SQLBinding) silently consuming messages
This isn’t going to be a brilliantly helpful post, as I’m not 100% sure of the cause of this, but hopefully it should serve to remind me next time around.
We have several Receive Locations set up using the WCF-Custom adapters. These use the sqlBinding binding type, using Typed Polling.
Our polling statement is a stored procedure call where we pick some data to be returned to the adapter, then mark that data as having been sent.
The issue we were having (which only got spotted after going live) was that sometimes some data would go missing. It was being marked as having been sent, but then would show up in the table where it should have. Sort of coinciding with this were timeout messages showing up in the Event Viewer. (I particularly like how the events don’t tell you which receive location just timed out)
Initially, we assumed that the problem was with the stored procedure, and argued over over line, every join, every possible database lock, and every scenario of data corruption. Eventually though, by writing out some debug tables, we were confident enough that the data was being returned by the stored procedure, it just went missing.
After that, we used the Group Hub to look for the messages which we thought we should be receiving. We were polling every 2 minutes, and saw that sometimes there would be a gap in the list of messages. These gaps corresponded to the timestamp we gave the missing records.
This was a really useful breakthrough, as it allowed us to be fairly confident that it was the custom adapter which was swallowing the messages: as we saw them go in, and verified that they didn’t come out.
Eventually, we found some useful information to back this up. (Amazing how knowing roughly what to look for yields such useful information…once you know what to look for)
We came across 2 possible solutions to this. We tried the easiest one first, and it’s worked for us, so I don’t know how effective the second one is.
Solution 1
This sounds madness, but bear with me. In the sqlBinding settings of the adapter that’s causing you issues, set the Receive Timeout to: 24.20:31:23.6470000. I don’t really know why this works, except that it seems to.
Solution 2
Untried, but apparently the issue has been fixed in a Cumulative Update Package for BizTalk Adapter Pack 2010.
Written by Tom Morgan
Tom is a Microsoft Teams Platform developer and Microsoft MVP who has been blogging for over a decade. Find out more.
Hi, how did you come across the timeout value you state in Solution 1 please?
Regards,
Phil
Hi Phil,
It was referenced in a forum post here: http://social.msdn.microsoft.com/Forums/en-US/83e76be7-cea5-4164-89be-8726cd4c58c6/biztalk-2009-sql-wcf-adapter-losing-messages-aka-typedpolling-not-working?forum=biztalkgeneral
Specifically: “[T]hanks everyone for your input on this. None of the suggestions actually seemed to solve our specific issue, and we ended up raising a support call with Microsoft. They got us to use some tracing tools that are not publically available and it turns out that there was a problem with the underlying WCF commnications channel, which was not visible via the usual windows event logs. They suggested a solution that, although very counter-intuitive, seems to have resolved our problem; set the ReceiveTimeout property of the receive adapter to 24.20:31:23.6470000 and restart the service host.”
I’m afraid that’s all the information I have.
-tom