Clearing up Space on an Exchange Server

I wanted to migrate an old exchange server, but the size was way more then I ever expected… so I dusted out my old script…

GitHub – Zewwy/Manage-ExchangeLogs: Script to help ease in managing Exchange logs

inetpub logs ended up being over 50 gigs. etl nearly 5 Gigs, another 5 On the next. all together cut out over 60 Gigs of data space. But lots remained.

I then found a OWA prem folder with lots of data and found a usual blogger covering it’s data removal here: Remove old Exchange OWA files to free up disk space – ALI TAJRAN

“To know which Exchange build number folders we can remove, we have to find which Exchange versions are running in the organization.

$ExchangeServers = Get-ExchangeServer | Sort-Object Name
ForEach ($Server in $ExchangeServers) {
Invoke-Command -ComputerName $Server.Name -ScriptBlock { Get-Command Exsetup.exe | ForEach-Object { $_.FileversionInfo } }
}

After running the above script in Exchange Management Shell, look at the Exchange build number.”

Pretty much run that command and delete all the folders expect the one the system is currently on.

Just like this blogger saved another 10 Gigs off that folder alone, but still heafty, I checked the Exchange DB folder path and found enless log files (guess I have to extend my script. Ended up writing a one liner to grab all the log files and clear then, this was almost another 50 gigs in space saved. We are well over 100 Gigs in space saved/deleted. but there’s still some heft, checking the Exchange client mailbox DB is only 6 gigs (and I”ll see if I can save space there but over all it is peanuts to all the space being used.

Next I found WinSXS folder taking up space. I followed the steps from this blog: What to Do If the Windows Folder Is Too Big on Windows 10? (diskpart.com)

I had already ran diskclean including system files. I ran the DISM commands specified as well but that only brough the 15 gigs of space it was using to roughly 7 Gigs, half is not bad but I was hoping it could do better.

Man wonder if I can just delete it? No! Says Microsoft:

“Don’t delete the WinSxS folder, you can instead reduce the size of the WinSxS folder using tools built into Windows. For more information about the WinSxS folder, see Manage the Component Store.

We following that MS KB pretty runs the same commands as the blog.

I’ll live with that for now, I logged into the mailbox and deleted everything I could, I should have no other mailboxes… wonder what’s got the Exchange DB up to 6 gigs of used space?

I followed another one of Ali’s Blogs: Get mailbox database size and white space – ALI TAJRAN

But that just told me what I had already found out looking at the base file system. He links to another one of this post to use a a script he had written. I checked the script… Clean use of case statement… Well done bud 🙂

Anyway I pipe get mailbox selection into a stats comdlet and then Format the whole list:

(Get-mailbox -resultsize unlimited) | Get-MailboxStatstics | Select *Name,*Count,*Size | FL

This shows All mail items size, and all message and attachment sizes. For me all mailboxes were tiny, and I can get rid of several of them as well. but I don’t think that will change the size on disk…

Got rid of everything down to 2 mailboxes and with 2 gig quotes each no way it could be over 4 gigs, yet it’s still the 6-7 gigs noted eariler.. mhmmm…

So in his blog he basically create a new mailbox DB via the EAC (or Powershell) moves the mailboxes and deletes the old DB. OK I can do that…

So old DB as seen here:

Create new folder path for new DB file (or use a new disk whatever ya want):


Create new DB:

New-MailboxDatabase -Name "DB02" -Server "EX01" -EdbFilePath "C:\DBPath\DB02.edb" -LogFolderPath "C:\LogPath\DB02""

Restart the services.. wait is my new DB and files? Oh I forgot to mount it:

Alright… time to move the mailboxes…

Huh, I was hoping for a process display but I guess it makes sense to throw the job in the background to not be interrupted by signing out or closing the console. Checking resource monitor… Starts chanting… Oh ya Go Beanus, Go Beanus!!

Looks like I/O settled… and…

Get-MoveRequest

Completed nice… size? less then 300 Megs baby.

K, just need to delete the unmount and delete the old DB. What a dink he knew it would fail but I followed his other blog post here:

Cannot delete mailbox database in Exchange Server – ALI TAJRAN

and after running the remove-mailboxdatabase cmdlet it still told me I had to manually delete the file, so I did… I finally got the server to roughly 30 gigs… not bad but I really don’t like that 7 gig SXS folder…

I even cleaned of the SoftwareDistrobution (Windows update Cache folder)

Hope this helps someone, time to hole punch this vmdk and migrate this server.

*Update* The hole punch didn’t work, Why? cause I forgot to run sdelete.
*WARNING* I tried to run sdelete on the VM while it was thin provisioned on a datastore that didn’t have enough storage to will the whole drive, as such the VM errored out with there is no more free space on disk.

It’s like the old adamant goes, things gotta get dirty before they get clean. In this case the drive has to be completed filled (with zeros) before it can be hole punched. Make sure the VM resides on a Datastore with enough actual space if the drive were to be completely filled.

*Update #2* Seems I went down a further rabbit hole then I wanted this to go, unlike my post about hole punching a Linux VM, which was pretty easy and straight forward. This one had a couple extra problems up it’s sleeve.

Problem #1 – Clearing Up Storage Space When Extending Is Not Possible.

This is literally what this whole blog post was about, read it and clear whatever space you can. If you have a physical Exchange server you’re probably done, and all your backups would probably be agent based at this point.

However if you’re virtualized (specifically with VMware/ESXi), then you have some more steps to do, the “hole punching”

Problem #2 – Hole Punched Size Doesn’t Match OS Disk Usage

This is were I want to extend on this blog post cause while the solution seems simple and straight forward. Each step has it’s own caveats and issue to overcome. Let’s discuss these now…

  1. You have to “zero the empty space” before VMware’s tools and properly complete the hole punch. This is only an issue if you happen to be over provisioning the datastore. If so:
  2.  At this point its assumed you cleared as much space as possible, and 2 you have defragged the HDD using the Windows defrag tool, and you have the VM overprovisioned. Simply shrink the partition down to a size that IS available on the datastore, or migrate to a datastore with enough storage. In my case I opted for the first choice to shrink the partition when I hit YET ANOTHER PROBLEM:
  3.  Even though I knew I had cleared the used space down to roughly 30 GBs of space, running the shrink wizard in diskmgmt tool stated it could only shrink the disk to 200GB since “There was a system file preventing further shrinkage”. WTF man, we ran disk cleanup, we cleared SXS folder, we cleared old logs, lots, we cleared the actual Exchange Database files, we disabled, and shrunk the pagefile then re-enabled… What is possibly preventing the shrinkage?

I found this post: windows 7 – Can’t shrink partition due to mystery file – Super User after I looked in the event log for event 259 showing the file in question preventing the shrinkage is “$LogFile::$DATA”… Da Fuck does that mean…

In short.. It’s an NTFS journaling file using “Alternate Data Streams“, or as quoted by Andrew Lambert “The $LogFile file is a part of the NTFS filesystem metadata, it’s used as part of journaling. The ::$DATA part of the name indicates the default $DATA stream of the file. I don’t know why it’s causing a problem, though.”

Bunch of comments about System Restore points, but I checked and there were none. Many other comments mentioning the use of 3rd party tools (no thanks). I can’t seem to locate it but I pretty sure I remember reading a comment somewhere that other NTFS aware applications have the ability to move and correct such things. So here’s my plan of action:

  1. Create a snapshot so I don’t have to recover the whole VM is something goes wrong. (On a slower I/O datastore but one with enough space for whole disk just to be safe).
  2. Boot The Exchange VM but with GParted Live disc connected.
  3. Use GParted to shrink the partition.
  4. Clone the VM. (This is what I don’t get the cloned VM still shows a disk size usage of 70 GBs…. AHhhhhhhhh!!)

Here’s another interesting note, as I stated in point 1 I had this VM on a Datastore based on spindle disk, shown on the ESXi host as “Non-SSD”, and cloned it to an SSD Datastore where it now states to use 70 Gig when the OS boots only having a partitioned disk of 46 Gigs, with 12 Gigs free. Opening the defrag application states defrag not possible cause it’s an SSD, guess let’s run sdelete see what happens?!

sdelete -z c:
sdelete -c c:

The backend VMDK grew from 70 Gigs to 80 Gigs… man wtf is going on… Hole punch it:

vmkfstools -K /vmfs/volumes/SSD/VM/VM.vmdk

You’re tellin me.. the SSD can handle ripping the drive at over 250 MB/s but holepunching causes I/O errors?

Good ol technology never ceasing to piss me off… fine I’ll destroy this VM, move the main spindle drive into a new ESXi host. Which will have an SSD Datastore with more storage and hopefully not on the way out (if it actually is an I/O error, storage/drive failure on the SSD). One sec…

So yeah… even with a larger SSD the copy worked, the hole punch “succeeded” but the drive was still 80 gigs. I made a clone and the vmdk came down to 60 gigs, I still can’t make sense of the roughly 30 gigs of discrepancy. Since the whole idea is to move this to my wireless ESXi host, I’ll see what exporting it as it is now results in the final OVA file size and then update this blog post.

 

Spammed via BCC

Well, whenever I’d check my local email, I noticed a large amount of spam and junk getting sent to my mailbox. The problem was the spammers were utilizing a trick of using BCC, aka Blind Carbon Copy. This means that the actual users it was all sent to (in a bulk massive send, no less) were all hidden from all people that received the email.

Normally people only have one address associated with their mailbox, and thus it would be obvious which address it was sent to, and getting these to stop outside of other technical security measures can be very difficult. It’s very similar to a real-life person who knows where you live and is harassing you, secretly at night by constantly egging your house. You can’t ask them to stop since you don’t know who they are, can’t really use legal tactics because you don’t know who they are. Sop you have to rely on other means, first identification if the person is wished to be identified, or simply move. Both are tough.

In my case I use multiple email addresses when signing up for stuff so if one of those service providers get hacked or compromised, I usually can simply remove the leaked address from my list of email addresses.

However because the spammer was using BCC, the actual to address was changed to a random address.

Take a look at this example, as you can see, I got the email, but it was addressed to jeff.work@yorktech.ca. I do not own this domain so to me it was clearly forged. However, that doesn’t help me in determining which of the multiple email addresses had been compromised.

I figured I’d simple use EAC and check the mail flow section, but for some reason it would always return nothing (broken)?

Sigh, lucky for me there’s the internet, and a site called practical365 with an amazing exchange admin who writes amazing posts who goes by the name Paul Cummingham. This was the post to help me out: Searching Message Tracking Logs by Sender or Recipient Email Address (practical365.com)

In the first image you can see the sender address, using this as a source I provided the following PowerShell command in the exchange PowerShell window:

Get-MessageTrackingLog -Sender uklaqfb@avasters.nov.su

Oh, there we go, the email address I created for providing a donation to heart n stroke foundation. So, I guess at some point the Heart n Stroke foundation had a security breach. Doing a quick Google search, wow, huh sure enough, it happene 3 years ago….

Be wary of suspicious messages, Heart and Stroke Foundation warns following data breach | CTV News

Data security incident and impact on Heart and Stroke constituents | Heart and Stroke Foundation

This is what I get for being a nice guy. Lucky for me I created this email alias, so for me it’s as simple as deleting it from my account. since I do not care for any emails from them at this point, fuck em! can’t even keep our data safe, the last donation they get from me.

Sadly, I know many people can’t do this same technique to help keep their data safe. I wish it was a feature available with other email providers, but I can understand why they don’t allow this as well as email sprawl would be near unmanageable for a service provider.

Hope this post helps someone in the same boat.

Mailbox Offline Exception

Since I need some email from an address I use, I figured I’d have some fun and spin up the ol’ Exchange server.

To my surprise when I attempted to login to OWA (since the front ends were loading just fine) after authentication I would be greeted with “Microsoft.Exchange.Data.Storage.MailboxOfflineException”.

My initial googlings didn’t provide much of good results.

I went to the server and did the usual check services and such, and noticed the root cause. Low Disk Space. I figured extending the logical volume and a reboot would suffice… nope. Problem persisted.

I decided to run the MS Exchange health checker: https://aka.ms/ExchangeHealthChecker

even after getting everything green in the health checker, the problem persisted.

A bit more google fooing and I was able to track down someone with a similar problem on TechNet with some useful guidance to use eseutil.exe to check the database.

The database indeed return “Dirty Shutdown”

ran the repair commands. *Note* you should try to use /r before using /p if it works you don’t need to use /p as it’s a hard recovery and data loss could ensure from it. I didn’t care as it’s use lab data.

K checking again it return “Clean Shutdown” everything I’ve read says it should be able to be mounted now. Failed to Mount….

As a last ditch effort, I try to Google some more if I missed something else. I found this nice post by Eric Simson

Step 1: Backup the Database (my case don’t care)
Step 2: Check Storage
(Was the cause, extended volume to 190GB used out of 250GB)
Step 3: Restart Exchange Services (Yeap, ran health checker)
Step 4: Check Database State (Yeap fixed it)
Step 5: Repair Exchange Database (Yeap fixed it)

Yet even after reboot and using PowerShell AND using accept data loss…

I was about to give up when I had one final idea, I realized that since /p does a hard recovery of the DB even if the log files are lost, and the log files take up a lot of space…

At this point I had well over 50% free space on the server. I ran the repair DB command again just to be safe.

wait.. what .. no error…. guess I was only at 24% free space, and that wouldn’t cut it, I don’t get why considering the -AcceptDataloss was defined.

Go to log in to OWA…. Ehhhh!!! There’s my emails!

Hope this helps someone.

Send an Email using Powershell

Source: Send-MailMessage (Microsoft.PowerShell.Utility) – PowerShell | Microsoft Docs

Build your object….

$mailParams = @{
SmtpServer = 'heimdall.dgcm.ca'
Port = 25
UseSSL = $false
From = 'notifications@dgcm.ca'
To = 'nos_rulz@msn.com'
Subject = ('ON-PREM SMTP Relay - ' + (Get-Date -Format g))
Body = 'This is a test email using ON-PREM SMTP Relay'
DeliveryNotificationOption = 'OnFailure', 'OnSuccess'
}

And then send it….

Send-MailMessage $mailParams

if there’s any pre-reqs required I’ll update this blog post. That should be it though. Easy Peasy Lemon Squeezy.

Azure AD and the ADConnect

*Note this is not supported. Installing Azure AD Sync on a Core server but it appears it does work.

Here’s what I did, I found this MS doc for reference:

  1. I followed this to guide me to make the “primary” tenant.
    no, I did not check either checkbox, **** em!
  2. I read this content to understand the tenant hierarchy.
  3. I added a custom domain (zewwy.ca), it said, sure no problem no federation issues, just verify. (Create a TXT record on the registrar to verify you own domain.)
    *refresh the page and the status will update accordingly.
  4.  I proceeded to download the Azure AD Connect msi file via the provided link after adding the custom domain.
  5. Install: (This was on Server 2016 Core)

2015.. interesting…

Click Accept Next.

Enter the Credentials from Step 1 (or enter the credentials provided by your MSP/CSP/VAR.

Enter the credentials of the local domain, enterprise admin account.

If you wish to do a hybrid Exchange setup check the second checkbox, Not sure how to configure this later but I’m sure there is a way. At this time that was not part of this post’s goals.

There was one snippet I missed, it appears to install a SQL express on the DC.

Then it appears to install a dedicated service.

This is Ground Control to Major Tom…

This is Major Tom to Ground Control… You’ve really made the grade!

They got all my passwords!

wait … it worked…. like what? No Errors?… No Service account creations? It actually just worked?…

Goto azure portal login, use my on prem credentials… and it logged me in….

I’m kind of mind blown right now. Well Guess on the next post can cover possibly playing with M365 services. Stay tuned. 😀

Email Stuck in Exchange Transport in 2022

Happy New Year!

If you are an exchange admin you may want to check out the notice from Microsoft. But you probably already have considering it started in the beginning of the new year: Email Stuck in Exchange On-premises Transport Queues – Microsoft Tech Community

So you probably already implemented this fix.

We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

Using the Automated Solution

  • Download the script here: https://aka.ms/ResetScanEngineVersion
  • Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
  • Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell).

Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked           : 01/01/2022 08:58:22 PM -08:00
LastUpdated           : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001 (note: higher version number starting with 211233... is also OK)
UpdateStatus          : UpdateAttemptSuccessful

Using the Manual Solution

In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization that downloads antimalware updates. Edge Transport servers are unaffected by this issue.

Verify the impacted version is installed
Run Get-EngineUpdateInformation and check the UpdateVersion information. If it starts with “22…” then proceed. If the installed version starts with “21…” you do not need to take action.

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 (or higher)

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

If you want to know why this happened here’s a answer from the comments:

John_C_Kirk – “This wasn’t due to a change on 31st Dec. The problem is caused by an integer overflow error: the anti-malware component is converting the date/time into “YYMMDDHHMM” format and storing it as a signed 32-bit number (max value 2147483648). So, in Dec 2021, the number would start with “2112…” (below the threshold). In Jan 2022, the number would start with “2201…” (above the threshold).”

Two Thumbs up on implementation.

Exchange Certificates and SMTP

Exchange and the Certificates

Quick Post here… If you need to change Certificates on a SMTP receiver using TLS.. how do you do it?

You might be inclined to search and find this MS Doc source: Assign certificates to Exchange Server services | Microsoft Docs

What you might notice is how strange the UI is designed, you simple find the certificate, and in it’s settings check off to use SMTP.

Then in the connectors options, you simply check off TLS.

Any sensible person, might soon wonder… if you have multiple certificates, and they can all enable the check box for SMTP, and you can have multiple connectors with the checkbox enabled for TLS…. then… which cert is being used?

If you have any familiarity with IIS you know that you have multiple sites, then you go enable HTTPS per site, you define which cert to use (usually implying the use of SNI).

When I googled this I found someone who was having a similar question when they were receiving a unexpected cert when testing their SMTP connections.

I was also curious how you even check those, and couldn’t find anything native to Windows, just either python, or openSSL binaries required.

Anyway, from the first post seems my question was answered, in short “Magic”…

“The Exchange transport will pick the certificate that “fits” the best, based on the if its a third party certificate, the expiration date and if a subject name on the certificate matches what is set for the FQDN on the connector used.” -AndyDavid

Well that’s nice…. and a bit further down the thread someone mentions you can do it manually, when they source non other than the Exchange Guru himself; Paul Cunninham.

So that’s nice to know.

The Default Self Signed Certificate

You may have noticed a fair amount of chatter in that first thread about the default certificate. You may have even noticed some stern warnings:

“You can’t unless you remove the cert. Do not remove the built-in cert however. ” “Yikes. Ok, as I mentioned, do not delete that certificate.”-AndyDavid

Well the self signed cert looks like is due to expire soon, and I was kind of curious, how do you create a new self signed certificate?

So I followed along, and annoyingly you need an SMB shared path accessible to the Exchange server to accomplish this task. (I get it; for clustered deployments)

Anyway doing this and using the UI to assign the certificates to all the required services. Deleted the old Self Signed Cert, wait a bit, close the ECP, reopen it and….

I managed to find this ms thread with the same issue.

The first main answer was to “wait n hour or more”, yeah I don’t think that’s going to fix it…

KarlIT700 – ”

Our cert is an externally signed cert that is due to expire next year so we wanted to keep using it and not have to generate a new self sign one.

We worked around this by just running the three PS commands below in Exchange PS

Set-AuthConfig -NewCertificateThumbprint <WE JUST USED OUR CURRENT CERT THUMPRINT HERE> -NewCertificateEffectiveDate (Get-Date)
Set-AuthConfig -PublishCertificate
Set-AuthConfig -ClearPreviousCertificate

 

Note: that we did have issues running the first command because our cert had been installed NOT allowing the export of the cert key. once we reinstalled the same cert back into the (local Computer) personal cert store but this time using the option to allow export of the cert key, the commands above worked fine.

We then just needed to restart ISS and everything was golden. :D”

Huh, sure enough this MS KB on the same issue..

The odd part is running the validation cmdlet:

(Get-AuthConfig).CurrentCertificateThumbprint | Get-ExchangeCertificate | Format-List

Did return the certificate I renewed UI the ECP webUI… even then I decided to follow the rest of the steps, just as Karl has mentioned using the thumbprint from the only self signed cert that was there.

Which sure enough worked and everything was working again with the new self signed cert.

Anyway, figured maybe this post might help someone.

How a Small Mistake Became a Big Problem

Background

This story is about implantation of compliance requirements, and the technical changes made that caused some heartburn. In particular Exchange server and retention policies.

Very simple; Compliance, and regulatory practices.

Retention Policy was becoming enforced. As such on Exchange no less, see here for more information on how to configure retention policies on Exchange.

You might notice you have to create tags of time frames. In this case there wasn’t one already pre-populated with my needs. So you have to create those. You may have also noticed that the only name time frame is all defined with number of days.

Human Error

So long story short, I  wrongly defined the number of days for the length of period I wanted to defined. Simply due to bad arithmetic, I swear I was an ace at math in school. Anyway, after this small mistake on the tag definition, and it was deployed to all Mailboxes. (Yeap there was steps of approval, and wasn’t caught even during pilot users).

Once it was discovered, there were 2 options. 1) Wait till specific people notice and  recover as required, or  2) do it all in one swoop.

Recover deleted messages in a user’s mailbox in Exchange Server | Microsoft Docs

After following this, it was determined that we couldn’t find just the emails from the time frame we needed to restore. This turns out cause all the emails “whenChanged” timestamp all became the same time the retention policy came into effect. So filtering by Date was completely useless.

Digging a Hole

At this point we figured we’d just restore all email, and let the retention policy rerun with the proper time frame tag applied. While this did work, there was a technique or property that was recently added that would have restored the emails into the sub folders in which they were removed from. Instead, all the emails were placed back into users Inbox.

This was a rough burn.  Overall it did work, it just wasn’t very clean and there was some fallout from the whole ordeal.

Hope this story helps someone prevent the same mistakes.

Audit Client Side Outlook Archive Settings

Why…

What You Need to Know About Managing PST Files (ironmountain.com)

How?

[SOLVED] Powershell Script find pst files on network – Spiceworks

From this guys script I wrote a simple script of my own.

As noted in the current issue that it only works running under the users current context. Though I know the results from testing, I can’t source any material on the CIM_DataFile class as to denote how or why this is the case.

For the time being this post will remain short as it’s a work in progress that I haven’t been able to resolve the interactive part of the script, I’m not happy with a requirement of a open share, and a logon script. As the script in its current state isn’t even written to support that design.

Will come back to this when time permits.

Microsoft Exchange Vulns and Buggy Updates

I’ll keep this post short. If you are unaware, there’s been a big hack on exchange servers.

Microsoft Exchange hack, explained (cnbc.com)

I ran the IOC scripts from MS, was I affected, it appears I may have.

Initiated my own lab DRP/BCP. Informed myself that services would be down, and restored AD and Exchange from backups before the logged incidents. Took the OWA Rev proxy rule  down till the servers could be fully patched.

Booted restored VMs, patched, hopefully good to go.

Then doing patch Tuesday updates users laptops start failing to boot after installing KB5000802. All I could find was news of prints causing BSODs classic.. BSODs! In my case it was causing boot crashing, I did my usual trick, but I got a different error, then ran the Windows Start up repair process, which amazingly got it to boot but said it reverted an updated (the one above). i attempted a install again, but same problem. I didn’t want to re-image as it was an VIPs machine, and time was of the essence. I took a whim, and decided to install all the latest drivers from the laptop OEM vendor (In case some was using MS drivers instead), after that tried the update again, and got a successful install.  Phewwww!