Microsoft Exchange Vulns and Buggy Updates

I’ll keep this post short. If you are unaware, there’s been a big hack on exchange servers.

Microsoft Exchange hack, explained (cnbc.com)

I ran the IOC scripts from MS, was I affected, it appears I may have.

Initiated my own lab DRP/BCP. Informed myself that services would be down, and restored AD and Exchange from backups before the logged incidents. Took the OWA Rev proxy rule  down till the servers could be fully patched.

Booted restored VMs, patched, hopefully good to go.

Then doing patch Tuesday updates users laptops start failing to boot after installing KB5000802. All I could find was news of prints causing BSODs classic.. BSODs! In my case it was causing boot crashing, I did my usual trick, but I got a different error, then ran the Windows Start up repair process, which amazingly got it to boot but said it reverted an updated (the one above). i attempted a install again, but same problem. I didn’t want to re-image as it was an VIPs machine, and time was of the essence. I took a whim, and decided to install all the latest drivers from the laptop OEM vendor (In case some was using MS drivers instead), after that tried the update again, and got a successful install.  Phewwww!

Even More PowerShell Fun

The Story

It’s another day, and we all know what that means… yes, another blog post, and even more PowerShell! Can you feel all the power!?!?!

This time it came down to the storage size of my Exchange servers C:\ which turns out to be due to Logs. Logs are great, and best practice is to only clear them if you have a backup copy. Often is the case that logs can be truncated after a backup via VSS by many backup solutions however in my case I could and probably should get that validated with Veeam (as I can’t seem to get that working ‘out of the box’) at the moment. So instead I wanted to know what was “usually” done server side even if someone was not implementing a backup solution.

Source: https://social.technet.microsoft.com/wiki/contents/articles/31117.exchange-201320162019-logging-clear-out-the-log-files.aspx

Neat, but the script is just alright, good for them doing what they want and that’s running it as a scheduled task. Not my goal, but a great source and starting point… let’s have some fun and give this script some roids, much like my last one… I’ll give this a home on GitHub.

Things learned…

  1. Working with the Registry
  2. Determining if Elevated (This is great and I may have a solution to the conundrum in my previous PowerShell post)
  3. Getting a Number, and validating it
  4. Validating Objects by Type
  5. Getting Folder Sizes

Check out my script for all the fun coding bits. I’m a bit tired now as it’s getting late so not much blogging, all more coding. 🙂

ErrorAction Stop Not Stopping Script

Quick Educational note (Source)

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{Write-Host "No Key"}
Write-Host "This should not hit"

Produces:

Well poop… The catch block was triggered but the script did not stop…

Oddly, changing to Throw, which is ugly does make the script stop…

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{throw "No Key"}
Write-Host "This should not hit"

Nice it worked this time, but it’s ugly…

Write-Error is just as ugly, but doesn’t stop the script?

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{Write-Error "No Key"}
Write-Host "This should not hit"

Produces:

Yet if I follow Sages answer in the source, and do a script variable for the stop action it then works???!?!

$ErrorActionPreference = [System.Management.Automation.ActionPreference]::Stop
$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2}
catch{Write-Error "No Key"}
Write-Host "This should not hit"

Those are really weird results, but all still ugly… so it seems even though -ErrorAction Stop causes a non-terminating error to be treated as a terminating error, but depending on what you do in the catch block determines if there’s a break/exit event being done. In my case to have things look nice and actually stop the script I have to do the follow.

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{Write-host "No Key";break}
Write-Host "This should not hit"

Which finally produced the output I wanted. (I could have also used exit in place of break)

Finally!

Validating URLs:

Source:

Good but broken due to the match on the scheme extension:

function isURI($address) {
	($address -as [System.URI]).AbsoluteURI -ne $null
}

function isURIWeb($address) {
	$uri = $address -as [System.URI]
	$uri.AbsoluteURI -ne $null -and $uri.Scheme -match '[http|https]'
}


isURI('http://www.powershell.com')
isURI('test')
isURI($null)
isURI('zzz://zumsel.zum')

"-" * 50

isURIWeb('http://www.powershell.com')
isURIWeb('test')
isURIWeb($null)
isURIWeb('zzz://zumsel.zum')
isURIWeb('hp:') #Return True

 

Better results with:

function isURI($address) {
($address -as [System.URI]).AbsoluteURI -ne $null
}

function isURIWeb($address) {
$uri = $address -as [System.URI]
$uri.AbsoluteURI -ne $null -and $uri.Scheme -match "http|https"
}


isURI('http://www.powershell.com')
isURI('test')
isURI($null)
isURI('zzz://zumsel.zum')

"-" * 50

isURIWeb('http://www.powershell.com')
isURIWeb('test')
isURIWeb($null)
isURIWeb('zzz://zumsel.zum')
isURIWeb('hp:') #Return True

Outlook and the Cache Mode

The Story

Wouldn’t be another post without another day of annoyances… yup, just another day. So recently it’s been reported that when users are working remotely their outlook decides not to open….

Now since they are remote, I have noticed this only happens if the Outlook client is configured in “Online mode”, instead of “Exchange Cached Mode”… see this MS Docs for more information on the different types and when to utilize each mode.

Originally I configured Online Mode, as most users are locally available in the work network and this is not a major problem. Also when using “Cache mode” not all emails show up in Outlook right away, specially if using folders, there generally shows a link “Click here to view more on Microsoft Exchange” this means items that were not cached (depending on the cache time slider this may vary, I choose 12 months), view this support if you can’t see the link in cache mode enabled.

I used to have an issue with this setting, user reported items wouldn’t load even after clicking the link, however I haven’t seen this to be an issue anymore, so I recommend to enable Outlook Cache mode unless you fall under the other points in MS’s Doc linked above.

Finding Users That Are Not Using Cache Mode

So I was now on a new mission… “How do I know who’s not using Cache mode on outlook?” And this is were the rabbit hole began….

First result, same question

Craig Harts reply…

his or response seem like it’s rather easy until…

wtf is this…? Depreciated?!?! C’mon, but there’s an alternative now that’s better right? No…. Thanks MS… seriously… thanks…

So this older post goes over alternatives, not because they didn’t have access to the cmdlet above, but simply cause they didn’t trust the results.. huh…

Problem is this seem to be reg keys used by older outlook, and new outlook seems to use alternative keys… This sure is fun! However, thanks to others that blog and code in their spare time as well, in this case thanks to Jose Espitia much like in my last blog post this is a great start, but usual me, I don’t like expecting anyone to change the source code. In this case you have to provide a file with a computer list, output path, but it’s hard coded…. OK you know what that means! Yeah I usually would create a new GitHub Repo but first…

Much like the older post mentioned they choose to target end user directly to get the most “accurate data”. This however assumes four things:

  1. That Exchange admins are workstation admins, and have elevated rights on all machines.
  2. That all firewalls are configured and permissions to allow remote querying.
  3. That the user’s are in fact online at the time the query is made.
  4. That RemoteRegistry Service is started and running on end machines.

In my Case it was not and disabled on all machines, I didn’t feel it was beneficial enough to introduce a risk for the simple sake of determining a users connection mode on Outlook.

So I decided to go back to Option A from the old original post, parsing all the RPC Access logs… Using this an alternative reference.

Well I gave it a try and looking through all th elogs there was no reference to “Classic” so I guess that’s no longer valid option either.

Looks like I’m stuck on this one, I can’t seem to find a valid way to find this information out server side with MS removal of the Get-LoggingStatistics cmdlet. And attempting to query all ends users has to many restrictions/hoops that I do not wish to implement. In this case I have to simply go around to all users machines, or wait for the to complain when they work remotely.

Thanks Microsoft I really love what your doing for SysAdmins. Taking away mark and putting him on as COO of Azure so all our beautiful tools from SysInternals are now just the way they are, and new tools, don’t need em right, just buy your cloud subscriptions and who needs SysAdmins… 😛

Anyway… that’s it for today. Sorry no advanced scripts form this post, just use Jose’s script if you wish to query end users machines. Just ensure you have RemoteRegistry service running on all end users machines, and not blocking it in the firewall as well. Hoops I have no interest in jumping through.

Cheers!

Exchange: Something Went Wrong

Fixing Exchange

Now, I’ve taken a couple Exchange courses. They cover all the bases… expect when things go wrong. That’s why it’s nice to have labs… today in my Lab I discovered I was unable to get email from my exchange server, neither from activeSync nor Outlook Web App (OWA).

Something went wrong alright… first thing I noticed was my disk had run out of space… whoops. Hahaha. Expand the drive, reboot and… Something Went Wrong…

Sigh…. alright event viewer… what ya got for me…

Unable to mount…. I guess it didn’t like what happened to the DB after the disk ran out of space… some quick googling (1 and 2 copy cats… and can’t even tell you the DB file locations…) and one more, more personal blog post.

Exchange Default DB File Locations

If you are using Exchange Server 2000 & 2003, you can locate your EDB files at:

C:\Program Files\Exchsrvr\MDBDATA\Priv1.edb
C:\Program Files\Exchsrvr\MDBDATA\Pub1.edb

If you are using Exchange Server 2007, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Mailbox Database.edb
C:\Program\Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Public Folder Database.edb

If you are using Exchange Server 2010, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V14\Mailbox Database\Mailbox Database.edb
C:\Program Files\Microsoft\Exchange Server\V14\Public Folder Database\Public Folder Database.edb

If you are using Exchange Server 2013, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox database Name\Mailbox database Name.edb

If you are using Exchange Server 2016, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database Name.edb

Repairing the Exchange Mailbox Database

Which they all tell you to use a cool old “tool” eseutil. which seemed straight forward, ensure you run an elevated cmd or you won’t have access to the directory path of the exchange DB file. In my case I used the Exchange 2013 path which was the version used in my lab.

I also moved the log files:

move *.log c:\temp

Yeah… that took a lil while.

Mounting the Exchange Mailbox Database

Once it’s repaired used Exchange Mgmt Shell to mount it:

Whoops, silly me, since everyone said to stop the information store service I did… so after starting the service, and rerunning the command it succeeded.

Which resulted in:

Much better!

Summary

  1. Check the source of the Database Corruption. (Mine was Disk Space)
  2. Stop the Information Store Service
  3. Check the Validity of the Mailbox Database (eseutil /mh)
  4. Repair if required (eseutil /p)
  5. Restart the Exchange Information Store Service
  6. Mount the Mailbox Database

Hope this helps someone.

Testing Active Sync

I’ll keep this post short.. I swear this time haha

The Story

I recently blogged about using OPNsense as a reverse proxy for Exchange. This was really nice, and I was able to access Outlook Anywhere, literally from anywhere with owa. However…

The Problem

For some reason I could not connect with Active Sync… Even when I had my phone in the same network pretty much. I thought it might had to do with the self signed certificate (I should have known it was not when selecting “Accept all certificates” under the connections options still didn’t work), so I wastefully exported the certificate and key from my OPNsense server and imported it into the Exchange ECP, and assigned it to the IIS and SMTP services. (I’m probably going to change these back to the self signed as I don’t really have intentions on completing these steps every 60 days.

Since I didn’t want to use my account to test on Microsoft test site (this is a life saver), I used a account and email I was setting up for my colleague… and it passed… I was shocked. (At first I thought it was cause of the certificate changes… I soon found out.. it was not).

So I tested the same connection settings on my phone and it worked!

Woo Hoo!

The Real Problem and Solution

My happiness was sort lived once I attempted to add my account…. which still failed…. what the heck? I had all the settings the same exc…ep…t… my account…. wait a minute…. Google… WTF! Admins can’t use active Sync?!?!? Why isn’t that more specified in the documentation! I was aggravated about this for days… cause of something that was sooo simple!

I’ll cover exporting and importing certificates for other uses in a nother blog post. I just wanted to get this one out cause even though I had configured everything in my previous blog post about using OPNsense as a reverse proxy correctly, I wanted to follow up on why Active Sync was working for me, cause everything else on online guides made it sound so simple, and it rather is… until you find out a little secret GEM MS didn’t tell you about…

Configuring an Anonymous Receive Connector on Exchange 2016

The Story

Well in my previous post I discussed the issue I faced resolving an email problem with one of our development applications in which it was unable to send emails after a recent Exchange upgrade/migration. So initially we were going to simply rebuild our own workflow in-house using ASP .NET Core. Until we noticed that even our own workflows were failing… in this case the answer from the old post which was super vague “reconfigure the receive connector”. Then I somehow stumbled upon my answer through one of my hundreds of google searches… I founds this gem!

OK before I link the gem which will be the source to my answer. I also wanted to point something out real quick here in hopes maybe someone can comment below the answer to this one:

When using Exchange 2016 as an email SMTP relay, and you use a no-reply from address, with an external email address for the destination, how do you query to find out if its gone through, or stuck in que? All I could see in the ECP it always required me to select a mailbox… there’s no mailbox associated with these relayed email messages, so how does one check this?

OK, now for the gem. This guy “Paul Cunningham” He’s… uhhhhhh… He’s uhhhhh… he’s uhhhh a good guy. So I always knew you could use telnet to check certain ports and services… but this was so concise… it nailed the problem…

From my K2 Server or my in house workflow server:

1) Ensure Telnet Client feature is enabled

2) Open cmd prompt or PowerShell:

telnet exchangeServer 25
helo
mail from: user@corp.ca
rcpt to: ExternalUser@gmail.ca

220 EXSERVER.exchange2016demo.com Microsoft ESMTP MAIL Service ready at Thu, 22
Jun 2018 12:04:45 +1000
helo
250 EXSERVER.exchange2016demo.com Hello [192.168.0.30]
mail from: adam.wally@exchange2016demo.com
250 2.1.0 Sender OK
rcpt to: exchangeserverpro@gmail.com
550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

huh, just like the source blog, now why would I be getting that error… I allowed Anonymous users via the check box under the receive connectors security tab… yet Paul does a lil extra step that doesn’t seem to be mentioned elsewhere, and that check box I mentioned is his first line, but then look at the interesting second line….

[PS] C:\>Set-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" -PermissionGroups AnonymousUsers
[PS] C:\>Get-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

Since I was using anonymous settings on the Application server side (K2 in this case) I gave the second PowerShell cmdlet a run from my new exchange server.

Amazingly enough just like the source blog after running the second line (edited to fit my environment obviously) then the rcpt to succeeded!

20 EXSERVER.exchange2016demo.com Microsoft ESMTP MAIL Service ready at Thu, 22
Jun 2018 12:59:39 +1000
helo
250 EXSERVER.exchange2016demo.com Hello [192.168.0.30]
mail from: test@test.com
250 2.1.0 Sender OK
rcpt to: exchangeserverpro@gmail.com
250 2.1.5 Recipient OK

Part 2 – The Solution

If K2 is configured to use EWS, check that stuff out elsewhere, if you landed here from my previous post looking for the answer to the “There is no connection string for the destination email address ‘Email Address'” and wanted to know how that person altered his receive connector:

[PS] C:\>Get-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

New-MailboxImportRequest Failed

This is going to be another short post.

Working on an Exchange migration this weekend, I was using our backup software to simply export users mailbox’s from the most recent backup of your old Exchange server, then importing them into the new Exchange server for each mailbox after creation.

I would have loved to have simply selected each user as a whole and import those pst files. However from testing showed it simply created a sub item with the users name and all their folders, instead of properly placing them under the primary parent hierarchy. So I was forced to export Each item individually (Inbox, Sent Items,Drafts, Etc) and Import them. I initially didn’t script this as there were only about 30-40 users I had to migrate, i figured it was easier to just go through the wizards… until I discovered some users created folders outside of their Inbox! Ohhh boy…. Anyway, turns out if you exceed 9 imports for a single mailbox without specifying a special name for it (even after they succeed) you will get en error as follows:

“The name must be unique per mailbox. There isn’t a default name available for a new request owned by mailbox xyz”

The solution was easy enough to find a good band-aid indeed.

Get-MailboxImportRequest -status completed | Remove-MailboxImportRequest

However sometimes in my case I found I was still getting there error even though I cleared all completed import requests (with default names obviously). I found out I was having a weird bug happen to be where imports where showing as Queued, yet if I piped them into Get-MailboxImportRequestStatstics | Select Status, they reported a status of Completed…(If you want all the details, pipe into Format-List, instead of Select)

Get-MailboxImportRequest -status Queued | Get-MailboxImportRequestStatstics | Select Status

lol I wasn’t sure what to make of this but there was 2 solutions.

  1. Clear the “Queued” imports that are really Completed.
  2. Give your new import a unique name using the -name parameter

I’ll admit though Exchange 2016 is more intuitive to manage then old Exchange 2010.