Exchange: Something Went Wrong

Fixing Exchange

Now, I’ve taken a couple Exchange courses. They cover all the bases… expect when things go wrong. That’s why it’s nice to have labs… today in my Lab I discovered I was unable to get email from my exchange server, neither from activeSync nor Outlook Web App (OWA).

Something went wrong alright… first thing I noticed was my disk had run out of space… whoops. Hahaha. Expand the drive, reboot and… Something Went Wrong…

Sigh…. alright event viewer… what ya got for me…

Unable to mount…. I guess it didn’t like what happened to the DB after the disk ran out of space… some quick googling (1 and 2 copy cats… and can’t even tell you the DB file locations…) and one more, more personal blog post.

Exchange Default DB File Locations

If you are using Exchange Server 2000 & 2003, you can locate your EDB files at:

C:\Program Files\Exchsrvr\MDBDATA\Priv1.edb
C:\Program Files\Exchsrvr\MDBDATA\Pub1.edb

If you are using Exchange Server 2007, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Mailbox Database.edb
C:\Program\Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Public Folder Database.edb

If you are using Exchange Server 2010, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V14\Mailbox Database\Mailbox Database.edb
C:\Program Files\Microsoft\Exchange Server\V14\Public Folder Database\Public Folder Database.edb

If you are using Exchange Server 2013, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox database Name\Mailbox database Name.edb

If you are using Exchange Server 2016, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database Name.edb

Repairing the Exchange Mailbox Database

Which they all tell you to use a cool old “tool” eseutil. which seemed straight forward, ensure you run an elevated cmd or you won’t have access to the directory path of the exchange DB file. In my case I used the Exchange 2013 path which was the version used in my lab.

I also moved the log files:

move *.log c:\temp

Yeah… that took a lil while.

Mounting the Exchange Mailbox Database

Once it’s repaired used Exchange Mgmt Shell to mount it:

Whoops, silly me, since everyone said to stop the information store service I did… so after starting the service, and rerunning the command it succeeded.

Which resulted in:

Much better!

Summary

  1. Check the source of the Database Corruption. (Mine was Disk Space)
  2. Stop the Information Store Service
  3. Check the Validity of the Mailbox Database (eseutil /mh)
  4. Repair if required (eseutil /p)
  5. Restart the Exchange Information Store Service
  6. Mount the Mailbox Database

Hope this helps someone.

Testing Active Sync

I’ll keep this post short.. I swear this time haha

The Story

I recently blogged about using OPNsense as a reverse proxy for Exchange. This was really nice, and I was able to access Outlook Anywhere, literally from anywhere with owa. However…

The Problem

For some reason I could not connect with Active Sync… Even when I had my phone in the same network pretty much. I thought it might had to do with the self signed certificate (I should have known it was not when selecting “Accept all certificates” under the connections options still didn’t work), so I wastefully exported the certificate and key from my OPNsense server and imported it into the Exchange ECP, and assigned it to the IIS and SMTP services. (I’m probably going to change these back to the self signed as I don’t really have intentions on completing these steps every 60 days.

Since I didn’t want to use my account to test on Microsoft test site (this is a life saver), I used a account and email I was setting up for my colleague… and it passed… I was shocked. (At first I thought it was cause of the certificate changes… I soon found out.. it was not).

So I tested the same connection settings on my phone and it worked!

Woo Hoo!

The Real Problem and Solution

My happiness was sort lived once I attempted to add my account…. which still failed…. what the heck? I had all the settings the same exc…ep…t… my account…. wait a minute…. Google… WTF! Admins can’t use active Sync?!?!? Why isn’t that more specified in the documentation! I was aggravated about this for days… cause of something that was sooo simple!

I’ll cover exporting and importing certificates for other uses in a nother blog post. I just wanted to get this one out cause even though I had configured everything in my previous blog post about using OPNsense as a reverse proxy correctly, I wanted to follow up on why Active Sync was working for me, cause everything else on online guides made it sound so simple, and it rather is… until you find out a little secret GEM MS didn’t tell you about…

Configuring an Anonymous Receive Connector on Exchange 2016

The Story

Well in my previous post I discussed the issue I faced resolving an email problem with one of our development applications in which it was unable to send emails after a recent Exchange upgrade/migration. So initially we were going to simply rebuild our own workflow in-house using ASP .NET Core. Until we noticed that even our own workflows were failing… in this case the answer from the old post which was super vague “reconfigure the receive connector”. Then I somehow stumbled upon my answer through one of my hundreds of google searches… I founds this gem!

OK before I link the gem which will be the source to my answer. I also wanted to point something out real quick here in hopes maybe someone can comment below the answer to this one:

When using Exchange 2016 as an email SMTP relay, and you use a no-reply from address, with an external email address for the destination, how do you query to find out if its gone through, or stuck in que? All I could see in the ECP it always required me to select a mailbox… there’s no mailbox associated with these relayed email messages, so how does one check this?

OK, now for the gem. This guy “Paul Cunningham” He’s… uhhhhhh… He’s uhhhhh… he’s uhhhh a good guy. So I always knew you could use telnet to check certain ports and services… but this was so concise… it nailed the problem…

From my K2 Server or my in house workflow server:

1) Ensure Telnet Client feature is enabled

2) Open cmd prompt or PowerShell:

telnet exchangeServer 25
helo
mail from: user@corp.ca
rcpt to: ExternalUser@gmail.ca

220 EXSERVER.exchange2016demo.com Microsoft ESMTP MAIL Service ready at Thu, 22
Jun 2018 12:04:45 +1000
helo
250 EXSERVER.exchange2016demo.com Hello [192.168.0.30]
mail from: adam.wally@exchange2016demo.com
250 2.1.0 Sender OK
rcpt to: exchangeserverpro@gmail.com
550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

huh, just like the source blog, now why would I be getting that error… I allowed Anonymous users via the check box under the receive connectors security tab… yet Paul does a lil extra step that doesn’t seem to be mentioned elsewhere, and that check box I mentioned is his first line, but then look at the interesting second line….

[PS] C:\>Set-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" -PermissionGroups AnonymousUsers
[PS] C:\>Get-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

Since I was using anonymous settings on the Application server side (K2 in this case) I gave the second PowerShell cmdlet a run from my new exchange server.

Amazingly enough just like the source blog after running the second line (edited to fit my environment obviously) then the rcpt to succeeded!

20 EXSERVER.exchange2016demo.com Microsoft ESMTP MAIL Service ready at Thu, 22
Jun 2018 12:59:39 +1000
helo
250 EXSERVER.exchange2016demo.com Hello [192.168.0.30]
mail from: test@test.com
250 2.1.0 Sender OK
rcpt to: exchangeserverpro@gmail.com
250 2.1.5 Recipient OK

Part 2 – The Solution

If K2 is configured to use EWS, check that stuff out elsewhere, if you landed here from my previous post looking for the answer to the “There is no connection string for the destination email address ‘Email Address'” and wanted to know how that person altered his receive connector:

[PS] C:\>Get-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

Setting Mailbox Subitem Permissions

I had to do this for Resource Items, to allow all staff to view the Calendar of the resource so they could schedule items accordingly. As one of the comments mentions from this Spiceworks post

Set-MailboxFolderPermission "roomname:\Calendar" -User Default -AccessRights LimitedDetails

This can however be expanded on knowing powershell being object oriented, roomname is actually any mailbox, :\ is the delimiter, Calendar is the SubFolder. -User $Group or $User (Groups are usually best practice per IDGLA) and finally -AccessRights, you can specify any of the access rights you see in the pull down option of Outlook.

That’s it. that’s all there is to mangling Mailbox sub-item permissions.

LegacyExchange Annoyance
(Exchange Cross Forest Migration Woes)

The Begining

I’m my quest to completely rebuild my company domain from the ground up (new Domain Controllers (Server 2016 Core), SQL 2016, SharePoint 2016, Exchange 2016….You get the idea) I’ve had to face many interesting challenges (not all blogged about just yet, but if you follow my TechNet posts you’ll see I have plenty of content to write about moving forward if I don’t come across interesting new things, at this rate that seems unlikely… anyway) This time it was another interesting one.

The Weekend

After spending the entire weekend combing over every little detail of the migration (Mail relays, systems that use email, how they send email, the receive connectors they would need (Auth types, TLS sec, etc) I figured I had all my bases covered, and made the switch (all changes including expanding existing server configs to allow mail flow, not hinder any) so the last part of my switch was ensuring most servers/services were using a Fully Qualified Domain Name (FQDN) in their settings/configs for SMTP. So my cut over in this case was very simple change the A host record, and clear all systems DNS cache. To my amazement everything was still working (even ActiveSync cut over without a hiccup)…. Until… the next day…

The Next Day

Out of all things I didn’t anticipate internal email flow to break… I mean… there’s nothing different between Joe.blow@corp.ca and Joe.blow@corp.ca right??!?! Wrong! lol with Microsoft Exchange you are totally, and utterly wrong! First… Read this and read this to understand exactly what I mean. In short internal email likes to use it’s own special address… (give a dirty look)… called X.500 addresses (AKA The LegacyExchangeDN) bunch of garbage, muff cabbage BS…. So instead of everything resolving normally cause all new linked mailboxes had the proper SMTP address (So all other outbound/inbound flowed without issue), user wanting to reply to old emails, or creating new ones and having the TO field be auto populated, they would get a stupid NDR (AKA a bounce back telling them the email can’t be sent to the recipient) cause FFS it can’t just use the SMTP address NOOOOOOOO it uses the stupid Legacy X.500 Address… Gosh Darn *Mumbles* Exchange… in case you can’t tell I dispise email with a passion.

The Search

Anyway, I looked up what the possible solutions were, I wasn’t too happy. For now I was telling people to simply remove the old Auto Populate cache Outlook was using. As for existing Calendar events, turns our all resources fell under the exact same annoying problem, even though I created them all with the same alias’s it wasn’t good enough Exchange was seeing them via their old X.500 addresses (Since all old Calendar Items were imported from Backup), they had new X.500 addresses in the new Exchange Server. So I would have to tell people to remove those resources, and simply re-add them.

The Problem

There is however a problem with this, and that is if someone edits an existing Calendar event and changes the time, the room may already have been booked (the new room) so when the user editing the old re-occurring event goes to re-add the room, it would complain about a conflict. Someone has already booked the “new” room even though it should have been held by the initial booking. Alright so how does one re-map this… well took me a while digging through google like this guy, or this guy (seems everyones a blogger these days), but I found an excellent resource blog that covers the problem, and the solution pretty clearly

The Answer

To keep it short for everyone, and as usual to Paraphrase the solution, which so far is not even working for me :@, even after waiting 18 hours. *UPDATE!* Don’t put in X.500 like the stupid UI tells you to… just put in X500 without the ****ing dot… See my Technet post for details!

1)  Open User and Computers (From linked Domain/ Old Domain)
2) Find any User/Resource/Equipment object that were migrated
3) Right Click and select attribute editor tab (requires advanced view)
4) press "L" and lookup LegacyExchangeDN, double click and copy
5) Open Exchange ECP (New Server)
6) Under Recipients double click migrated mailbox, click email address
7) Add new email, Type X500, paste the address you copied in step 4
8) Wait for the OAB to synchronize across the farm and clients

New-MailboxImportRequest Failed

This is going to be another short post.

Working on an Exchange migration this weekend, I was using our backup software to simply export users mailbox’s from the most recent backup of your old Exchange server, then importing them into the new Exchange server for each mailbox after creation.

I would have loved to have simply selected each user as a whole and import those pst files. However from testing showed it simply created a sub item with the users name and all their folders, instead of properly placing them under the primary parent hierarchy. So I was forced to export Each item individually (Inbox, Sent Items,Drafts, Etc) and Import them. I initially didn’t script this as there were only about 30-40 users I had to migrate, i figured it was easier to just go through the wizards… until I discovered some users created folders outside of their Inbox! Ohhh boy…. Anyway, turns out if you exceed 9 imports for a single mailbox without specifying a special name for it (even after they succeed) you will get en error as follows:

“The name must be unique per mailbox. There isn’t a default name available for a new request owned by mailbox xyz”

The solution was easy enough to find a good band-aid indeed.

Get-MailboxImportRequest -status completed | Remove-MailboxImportRequest

However sometimes in my case I found I was still getting there error even though I cleared all completed import requests (with default names obviously). I found out I was having a weird bug happen to be where imports where showing as Queued, yet if I piped them into Get-MailboxImportRequestStatstics | Select Status, they reported a status of Completed…(If you want all the details, pipe into Format-List, instead of Select)

Get-MailboxImportRequest -status Queued | Get-MailboxImportRequestStatstics | Select Status

lol I wasn’t sure what to make of this but there was 2 solutions.

  1. Clear the “Queued” imports that are really Completed.
  2. Give your new import a unique name using the -name parameter

I’ll admit though Exchange 2016 is more intuitive to manage then old Exchange 2010.

Trend IMSVA LDAP Settings Account Permissions

If you’re reading this chances are you have read this or this (If you like Wizards) … and this only to ask your self… “What permissions does this “admin” account need?” I mean… it says specify an admin account right?!

“Under LDAP admin, specify the administrator account, the corresponding password and the base distinguished name”

After quick testing, turns out no, any regular domain user account worked here.