Let me start this post out with a lit­tle story. I am nor­mally a hard­core vir­tu­al­iza­tion and stor­age guy. Some­times my career in this sec­tor brings me into work­ing with stuff I haven’t worked with before because vir­tu­al­iza­tion encom­passes so much. As I con­tinue to work with other teams I learn more and more about what they do every­day. I usu­ally find myself involved in every per­for­mance trou­bleshoot­ing ses­sion and every new project these days. My per­sonal phi­los­o­phy is the IT guy of the future will be truly con­verged just as all the tech­nolo­gies are con­verg­ing into 1 box or “stack”. Spe­cial­ties in smaller sub­sets will fall away and a more spe­cial­iza­tion in every­thing Dat­a­cen­ter may become the norm.

Early Mon­day morn­ing I over­heard a con­ver­sa­tion about con­nec­tion issues with our new Exchange 2010 envi­ron­ment while drink­ing some cof­fee and review­ing my brand new vSphere design. I didn’t think about it very much until my boss came to my desk and asked me to have a look at the prob­lem. Our mes­sag­ing guy was on vaca­tion and I was the only other per­son on staff who had some mes­sag­ing expe­ri­ence. It seems that all of our global and even local offices were com­plain­ing about ran­dom exchange dis­con­nec­tions and also includ­ing email deliv­ery delays from 30 min­utes to 4 hours! It seems Activesync devices and OWA users were not affected by these delays at all. Being always up for learn­ing new stuff I took the challenge.

First let’s start with the quick facts I could put together. We had users in every coun­try we have offices com­plain­ing about the ran­dom dis­con­nec­tions and delays. I had one actu­ally con­firmed in China but had some slight trou­ble get­ting exact user names from the local IT per­son. Also we had con­nec­tions ran­domly dis­con­nect­ing and show­ing dis­con­nected in the lower right hand cor­ner of the out­look client. I did not have any con­fir­ma­tion of who exactly was hav­ing these prob­lems. To start I dug through the event logs on all the servers in the Exchange 2010 envi­ron­ment and the amount of errors I found was over­whelm­ing. To shorten this up a bit and not write a novel most all of those I inves­ti­gated were directly related to run­ning Exchange 2010 SP1 with­out any update rollups in place. There were cor­re­spond­ing KB arti­cles from Microsoft con­firm­ing these fixes in var­i­ous update rollups.

I noticed an Event ID 2915 on our CAS servers that stuck out. I noticed sev­eral EWS and RPC con­nec­tions report­ing “Ses­sion Limit Over Bud­get”. I cor­re­lated this with the Default Throt­tling Pol­icy Exchange 2010 uses. It seems that the more mail­boxes a user opens the more con­nec­tions Exchange cre­ates. It doesn’t some­how trun­cate these con­nec­tions. To under­stand more about the Default Throt­tling Pol­icy see Under­stand­ing Client Throt­tling Poli­cies. So I quickly whipped up a pow­er­shell script that set the Throt­tling Pol­icy defaults to Null so there was no restric­tions (funny Microsoft states as a workaround just to do this if you encounter an issue).
If you are inter­ested in see­ing this script or want me to go deeper about Throt­tling Poli­cies con­tact me, but this arti­cle isn’t quite about this so I will move quickly on.

After the Throt­tling Pol­icy was changed the reported dis­con­nec­tions stopped but the deliv­ery delays con­tin­ued as men­tioned all around the globe. With the other prob­lem out of the way I began to real­ize that the prob­lem seemed very ran­dom. Some users expe­ri­enced it, some not, some couldn’t tell me whether they expe­ri­enced it or not. This is when the hours of fruit­lessly dig­ging through con­fig­u­ra­tions to learn them and read­ing about Exchange 2010 on Google began. I noticed our mail­box servers were set up in an active — active con­fig­u­ra­tion with bidi­rec­tional repli­ca­tion using DAGs. This is when I decided to go back to basics of trou­bleshoot­ing. I went over to my col­league sit­ting next to me and sent var­i­ous test mes­sages to him. All of them were promptly deliv­ered with­out any prob­lems. I noted down what server his mail­box was run­ning on and moved on. Then I walked around the IT depart­ment until I was able to find a col­league that con­firmed they had the deliv­ery delays up to 4 hours. Just for kicks I turned off their cache mode on the Out­look client and their prob­lem mag­i­cally van­ished. Then I turned cached mode back on and left it bro­ken since I was deter­mined to fix it on the server side and not just band aid the prob­lem. When I went back to my desk and noticed what mail­box server the col­league with the delays was expe­ri­enc­ing a light bulb went off and every­thing seemed to be com­ing together. Now all I had to do was note the dif­fer­ences between the 2 servers.

First of all to stop the global issue from occur­ring while I could resolve the prob­lem I failed all the DAG vol­umes over to the 1 server that did not seem to be hav­ing the prob­lem. Reports quickly came in that the prob­lem was resolved. Then I quickly moved on to exam­in­ing dif­fer­ences between the 2 servers. After com­par­ing win­dows updates between boxes I noticed that some updates from Feb­ru­ary were recently applied to both servers, how­ever, there was 1 dif­fer­ence. It seems Microsoft KB2393802 was applied to one server but not the other. I googled regard­ing this but only found one vague thing about delays in Exchange 2010 mail deliv­ery men­tioned in the mid­dle of a tech­net arti­cle relat­ing to this patch, but noth­ing offi­cial at all from Microsoft. I removed the patch, rebooted, and tested with a test mail­box data­base run­ning on the server I had cre­ated for this pur­pose. The prob­lem was fixed as I thought.

I tried to research on what about this patch could be caus­ing this prob­lem but came up with noth­ing. If any of you read­ers have an idea please com­ment and let me know your thoughts! I have attempted to con­tact Microsoft regard­ing this issue so they could pos­si­bly append to the KB arti­cle but they cur­rently have not replied.

Ed Czer­win