Saturday, September 8, 2007

Exchange 2007 Took Out 1700 Mailboxes For 25 Minutes

We have a 4+1 cluster and about 28K maiboxes. One of the storage groups started throwing 1022 logon errors and no one could access their mailbox. I started getting hammered with calls and walk-ups. All the services were still running. I had to take the server offline and back up to get it cleared.

The 1022's followed numerous 8230's and then one 623 just before the logon errors. The 8230's are a known issue with recurring calendar items that have been expanded by OWA. Don't know much more than that except that it has caused us to go down a couple of times previously due to rampant transaction log create and disk space issue. This time the store went down first.

I'm using an event log monitoring app to e-mail me on 8230's and then I'm using PowerPoof to clear the corrupt calendar item out. I hope to have an automated solution up soon but I'm having difficulty since event log monitoring likes to happen locally to the servers and PowerPoof uses Outlook which can't (shouldn't) be installed on servers. More later...

2 comments:

sgriffin said...

Powerpoof only needs MAPI - it doesn't have to be Outlook's MAPI - Exchange's MAPI would work.

skyh00k said...

So you can use profiles without installing Outlook?