Windows MCS and MPIO

I was configuring some iSCSI disk on a Windows server and noticed there were two different options available that seem to provide similar functionality and I had to know… What’s the difference?

Source

THE DIFFERENCE BETWEEN MCS AND MPIO (IN A NUTSHELL):

First we agree upon the common features within both technologies: both serve a multipathing for (iSCSI) I/O-operations utilizing multiple hardware (or OSI Level 1) components, such as Ethernet NICs or iSCSI HBAs. The purpose of multipathing is redundancy and aggregation – how this is implemented depends on the above depicted figures, i.e. through the decision which paths are active and which are passive (or standby, using Microsoft parlance). For the exact definition of the policies, such as round robin, weighted path, fail over only, etc. please refer to “MS iSCSI UG”, p. 41.

Finally here come the condensed definitions for both technologies:

MCS allows the initiator to establish multiple TCP/IP connections to the same target within the same iSCSI session.

MPIO in contrast allows the initiator to establish multiple iSCSI sessions (each having single TCP/IP connection) to the same target, effectively aggregating the duplicate devices into a single device.

If you are not familiar with the terminology (initiator, target, session, connection, initiator port and network portal) please refer to “Multivendor Post” which provides very informative sketches to the iSCSI network architecture.

Now that we know that MCS means effectively several connections within a session and MPIO means multiple sessions the question is when to use what. Mainly you will have to concentrate on two perspectives – vendor support and load balance policy inheritance. The question – or rather schools of thought – about the speed and performance differences are factored out here, because in the author’s opinion these are almost equal and you will probably never get to the point of fully utilizing them. With this said consider the following simple rule of thumb: you can use MCS only when it is supported from the vendor’s SAN and you are not using hardware iSCSI HBAs. In any other case use MPIO. The second thought is – if considering the above conditions you are able to use MCS, but want to apply different load balancing policies to different targets (and effectively LUNs or groups of LUNs) you will still be better off using MPIO. This is because load balancing policies are session adherent. In other words when you are applying policy to MCS it is for the whole session, no matter how many connections are aggregated “beneath” it. On the other side when using MPIO you can set different policies for different LUNs, because the multipathing is using different iSCSI sessions.

 

I’m still trying to wrap my hear around exactly what the source is getting at. But will update the blog when I do some more testing.

CU update not Showing in WSUS

The Story

Today was a bit annoying…

I did my usual updates sync, and approve required updates, and in the past this has included CU’s without much fuss. However today I did my usual and ran check for updates on a member machine, returned clean (which i was expecting as per the results on WSUS console) and then I did the followup “Check for updates from Microsoft Updates” to my dismay the server stated an update was available a CU (KB4516061)… ughhh OK…

Checking WSUS

Decided to double check WSUS, to my dismay, re-syncing, and checking unapproved updates yielded no new updates. But I know there’s a new CU/// what gives?

Doing some reseach I find this is nothing really new and has been a problem for a while due to what could be multiple problems, including apparently packaging certain updates into other updates… how lovely.

As the main answer from that one says you can Import them… Ughhh fine…

*Expectation* Download MSU, click Import update, update gets imported to WSUS, and approve.

*Reality* Well reality is generally always worse then the expectations…

Importing Update into WSUS

So I downloaded the 1.5 Gig KB from MS Catalog and on my MMC snap-in click Import update… What do I get, a Windows Explorer popup asking me where the msu file I want to import is… NO… a link to the MS Catalog website…

Ughh… I already downloaded it what gives…

after a bit more research (honestly software should not be this non-intuitive, but that’s how old software was… non-intuitive…) turns out this “Import Updates” is not even designed for remote use (uhhh isn’t that the whole point of MMC Snap-ins?!?!?) Anyway, OK so people state you have to use it directly on the WSUS server….

FINE,  Log directly into the server and open the WSUS console, click “Import Updates” IE opens and page can’t be loaded. Strange checking the IE security settings the site attempting to be navigated should be trusted.

Even grabbing the direct catalog link and pasting it in this IE window only gave me the option to Download, not “add” and then “view basket”. It turns out the option to add only becomes available after an ActiveX install for something.

Originally I was not getting this, it wasn’t until I read this , I found a golden egg on Technet, and very carefully read the answer:

MS WSUS Product Team:

“Just to let you know, a statement from the WSUS Product Team has been published: WSUS Catalog import failures

“We are aware of the issue and presently working on a fix. In the meantime, the following workarounds can be used to unblock your deployment:

After clicking on the “Import Updates…” option in the WSUS console, an Internet Explorer window will open on the following URL: http://catalog.update.microsoft.com/… &Protocol=1.20
Before proceeding with importing the updates, change the “1.20” protocol value in the URL to the previous protocol value “1.8”. The URL should look like this when you’re done: http://catalog.update.microsoft.com/… &Protocol=1.8″

Uhhh ok… so it turns out on the initial pop-up where you get the Windows can’t display this page:

Change end number to 1.8

Yes, and Yes

Once this page loads, you can add the Active X control at the bottom:

Now you get the add, and view basket, and finally get the import option:

Well that was an annoying morning…

Update Computer Group Membership without Reboot

Source

Purge the computer account kerberos tickets

klist -lh 0 -li 0x3e7 purge

Force the gpo re-evaluation

gpupdate /force

Any previous attempt for access via newly added group membership should work; such as in this example I created a new Group, added this computer object into it, created a gMSA granting the group permission to use it, however the computer was not rebooted since added it into the group which was allowed access to install the gMSA.

PS C:\Windows\system32> New-ADGroup -Name "gMSANewGroup" -SamAccountName gMSANewGroup -GroupCategory Security -GroupScope Domain -DisplayName "gMSANewGroup" -Path "CN=Managed Service Accounts,DC=zewwy,DC=ca" -Description "Members of this group get Access to gMSATest2"
PS C:\Windows\system32> Add-ADGroupMember "gMSANewGroup" -Members "THISCOMP$"
PS C:\Windows\system32> New-ADServiceAccount -name gMSATest2 -DNSHostName gMSATest2.zewwy.ca -PrincipalsAllowedToRetrieveManagedPassword "gMSANewGroup"

Then Attempting to install the gMSA fails as the computer object hasn’t updated its group memberships locally, even though it has replicated throughout the domain, but following the command above to purge the computers tickets worked:

Hope this helps someone who needs to do granular group control but also don’t have the ability to reboot the host machine for service disruptions. 🙂

Quick Managed Service Account Audit

First get the list of gMSAs from AD:

$gMSAlist = Get-ADServiceAccount -filter {samAccountName -like "*"}

Second Determine the systems allowed to use them:

ForEach ($gMSA in $gMSAlist) {(Get-ADServiceAccount $gMSA -properties *).PrincipalsAllowedToRetrieveManagedPassword}

Yay, we know who can use these accounts… but ARE they currently using it. If this returns a Group, look to see the systems in this group, else just access the system in question.

Third, verify the account is in use by listing all the services on the system and the accounts used to run them:

Get-Service | Select -ExpandProperty Name | ForEach{(Get-WmiObject Win32_Service -Filter "Name='$_'") | Select Name, StartName}

The above command simply lists out all the services and the account they run under, it’s not optimal as it is slow, but it gets it all, and if you need a more readable version pipe it into Output-CSV, or apply a more granular filter on the result for the gMSAs in question.

That’s about it, if you don’t see the gMSA listed on any service on the target machine, it’s rather safe to assume that the gMSA is not in use and can be safely removed from AD.

Remove-ADServiceAccount gMSAToBeRemoved

Systems not showing up in WSUS console

When a system doesn’t show up in WSUS, do these steps on the system not showing up:

1) Verify this registry setting (usually set via a GPO):

reg query HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate

2) Verify the System can resolve the hostname of the WSUS server, if specified. If IP, move on.

3) Use telnet against the specified port (if different from 8530), this verifies layer 4 and that not firewall ports are in the way.

4) Ensure the Windows Service is actually running via services.msc

5) All else fails try this:

net stop wuauserv
reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientId /f
reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientIdValidation /f
net start wuauserv
wuauclt /resetauthorization /detectnow
wuauclt /reportnow

Wait a couple mins and hard refresh the WSUS MMC Snap-in. I noticed this trick also works for systems that are in WSUS but won’t report an install percentage of 100%.

I noticed one system was not showing up with 100% install rate, and a yellow icon indicating needed updates still required, however checking for updates on the system kept reporting all updates, even after step 5 a couple times.

So… to get updates WSUS doesn’t have on Desktop based version of Windows there’s usually a nice link that states “Check for updates from Windows Update Online”, but this was a core hyper-v server, and no GUI, so…

*Note ByPassing WSUS on Core, without GPO changes.

reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU /v UseWUServer /t REG_DWORD /d 0
net stop wuauserv
net start wuauserv
sconfig

run option 6 and check for all updates (this assumes the server/system has access to internet servers).

Don’t forget to set the setting off again:

reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU /v UseWUServer /t REG_DWORD /d 1

Then run Step 5 again.

Setting Network Profile from Public to Domain on Core

Get Network interfaces profile

Get-NetConnectionProfile

sometimes when you recover a VM from backup it likes to change to Public even though all the other network settings were recovered successfully.

NLA Reset via Disable/Enable NIC

first thing to try is simply disable and re-enable the NIC once connection to a DC is verified.

Get-NetAdapter | Disable-NetAdapter
Get-NetAdapter | Enable-NetAdapter

All else fails, Set the DNS Suffix

If the NLA still refuses to show Domain status, try setting the DNS suffix…

Get-NetAdapter Ethernet | Set-DNSClient -ConnectionSpecificSuffix "Zewwy.ca" -UseSuffixWhenRegistering $true

Then, again disable, enable the interface.

It should go back to DomainAuthenticated. If not verify the computer is still actually authenticated with a DC with nltest.

Hope this helps someone.

Migrating Users and Passwords
Same NetBIOS Name

Story

*NOTE* All information here is provided as is for educational purposes only, whatever you do in your own environment is on you.

I want to give a bit of a background here, the goal was to migrate users to a new child domain, that was previously a 2 way forest wide trust. In this case scenario, there was no need for a 2 way trust as both domains were owned and operated by the same corporation. A simplified AD structure was conceived, new workflow servers, new clean permissions throughout.

Everything was coming along swimming until, after some extensive research, how to migrate users and their passwords without service interruptions…

With Windows Authentication in the back end there was only one choice, ADMT v3.2 and it’s associated documentation … yeah that’s not a content based website, it’s a download to a doc, that was last updated 2014 with support with Windows Server 2012 R2.

It states the following:

Any DB works (I used SQL 2016 Express)
Other things, read it if you wish to be over/under whelmed
Needs a Trust

Simple Overview

Here’s a simple of an idea of how the Forests are laid out, and you can see where the users are planned to be migrated… just one major problem… You can’t create a trust between forests where the NetBIOS match. And Yes, that is a thread from 2011 unanswered, which I will answer for you tonight. Which you can see from my design, is exactly the problem I was facing. I was initially hoping this could be done without a trust which turned out was the answer which lead me to the answer..

You can do this via trust but not how you might normally think about it.

Build out a new dc in your source domain and allow it to replicate properly, be sure that it is a dc/gc and dns server. Disconnect this dc from the current domain and expect to NEVER connect to this domain again.

Do a metadata cleanup of this dc….

Along with this, How to rename the NetBIOS name, now mix everything into one huge sch-melting pot and what do we get…. this blog post.

Setup

ADMT

*Note don’t bother installing SQL + ADMT until the member server is domain joined to the target domain. In this example Windows Server 2016 is used to host ADMT services. So this server is domain joined to Special.NewDomain.com

First Download ADMT from here

Based on the size of the installer I’m assuming this is an online installer and instead of dicking around trying to find an offline version.. simply connect this server to the internet. However before we begin ADMT requires an SQL instance to utilize, to keep life easy we will install SQL Express on this server and run it locally for the migration.

Installing SQL Express 2016

Now looking at where to find specific version to download and use… I wasn’t sure which version was best
I decided to start with Express Core …
Next, next, next, Mixed Auth (just in case) sa password.

Installing ADMT

Now with SQL Express on the target domain joined server, double click the ADMT v3.2 installer exe.

Accept the EULA. Accept/Decline the CEIP

Use local SQL and….

DB Import… NO, next

Now we should be good to run ADMT (as a Domain admin on Special.NewDomain.com), IF you installed SQL + ADMT before joining to the target domain and did not choose to use mixed auth and have no sa account, follow my previous blog post to recover access to the SQL Express instance, granting your domain admin account access.

Preparing the Source Domain

Now how you choose to accomplish this is entirely up to you. If you wish to go the route suggested by the TechNet Post to create a new DC thats a GC and rip it out of the Forest/Domain via a MetaData Cleanup… be my guest but that’s a lot of work.

Instead I choose to simply create a secondary version of the Special.local DC via a backup, but I could have easily made a clone since it’s all virtualized. So for me it started out like this…

Clearly at this point a trust still can’t be established as NetBIOS names are still the same, however now we have no fear of mucking up the source domain as it’s simply a clone will all users and their passwords still encrypted within AD. So this migration will require 2 things:

1 – A domain rename

2- Password Export Server (Covered later in this post)

Renaming the Source Domain

Changing the IP Address

Since this is a clone, and I was not interested in alternative firewalls outside the windows firewalls I connected the source domain to the same subnet as the target domain to ease life. This requires the IP address to change.

So open Network and Sharing Center and edit the adapter settings accordingly, this will however break the DC’s DNS service. So lets fix that.

Deleting the DC A host Record

On the DC open DNS snap-in, and navigate to the top where the SOA and NS records are, and below that search for the A host record for the DC itself, and delete that record (remember this should be a clone or copy of a DC from the original source DC so no risk should be had here).

Reset DNS settings

ipconfig /flushdns
net stop dns
net stop netlogon
net start dns
net start netlogon

Create a new DNS zone

Open DNS snap-in again on cloned source DC, and create a new DNS Zone for the new domain name.

Complete the wizard.

Configure Domain to Accept new DNS Suffix

– Open ADSI Edit
– Right Click ADSI Edit -> Connect to…
– Leave defaults -> ok
– Expand “Default Naming Context”
– Right Click Domain Parent Object -> Properties

– Enter the new domain name into the msDS-AllowedDNSSuffixes

Enable update DNS Suffix option

Server will reboot after this step, again since it is a clone and not actually hosting AD services for any production need this is no problem, right?

Get the required XML to edit:

Everything I was reading online stated that you need to do all this from a member server, and you need to copy rendom and another application from the System32 from a DC, etc, etc, all a bunch of rubbish… everytime I attempted to follow such guides the rendom command would spit out some lines that seems was supposed to be parsed by something else to provide useful return output. People stated running from System32 directly fixed that issue for them, but not for me. Instead I decided to run all the commands directly from the DC since it was the lowest risk for me. as stated serveral times why above. Sooo…..

– Open CMD as admin on DC
– Run “rendom /list”

Edit the XML file (open via elevated cmd prompt)

save and check by running “rendom /showforest”

It should report the changes you made to the XML file.

Upload the XML to the DC

Prepare DC

Execute Domain Rename

Let DC reboot and then complete the rename.

End the Domain Rename process

So now the setup should look like this…

Now as you can see we no longer have the same NetBIOS name and thus we can create a trust here to migrate users using ADMT and PES yay!

The Trust

Conditional Forwarders

For the trust to work each domain must be reachable by the other domains DC via FQDN. This obviously requires conditional forwarders to be configured for each DC accordingly.

So opening a MMC.exe application, from a member system with RSAT installed, or directly on Special.NewDomain.com if it has the desktop experience. Then Add the DNS snap-in. Add a conditional forwarder, in my case I added NotSpecial.com pointing to the IP address of the cloned and renamed DC.

Then doing the same thing on NotSpecial.com DC, opening the DNS application (or remotely with RSAT), and creating a conditional forwarder that says special.newdomain.com pointed to the IP address of the actual child DC, in the same subnet as in the diagram.

At this point ensure that Target DC (Special.NewDoamin.com) can ping NotSpecial.com, and that Source DC (Special.com) can ping Special.NewDomain.com. If yes, we can now go ahead and build the trust.

Building the Trust

Open domain and trusts. Right click domain and properties:

Click the Trusts Tab, New Trust:

Complete the wizard for both sides of the trust. I had a domain admin account in each source and target domain.

With admin account on each domain and already logged in as domain admin on the NotSpecial domain, wizard completes successfully:

Now with a trust in place, we could start just migrating users, but we need those passwords migrated as well, else we will have a bunch of angry users.

ADMT & PES

Permissions

Nest Special\DomainAdmin into NotSpecial\BuiltIn\Administrators group, as well as into Special\BuiltIn\Administrators group. You might be wondering why? Well I hit this error when attempting to migrate users passwords:

After reading this, I made the changes above and it finally got past this error when attempting to migrate users passwords.

Logged on to ADMT as Special\DomainAdmin

Password Export Server Setup

Step 1) Create the encryption key for the migration:

admt key /option:create /sourcedomain:notspecial.com /keyfile:"C:\path\to\file.pes" /keypassword:*

Step 2) Copy the Key to the NotSpecial.com DC (I used RDP)

Step 3) Grab PES installer from here, and get it on NotSpecial.com DC

You should now have this on NotSpecial.com DC:

Step 4) Install PES running the MSI from an elevated cmd prompt:
If you’re wondering why, I was about to smash a monitor when the installer kept telling me the password was wrong for the encryption file, when I knew for certain I wasn’t putting it in wrong, and someone else blogged about it.

I used a installed using local system account cause again this DC will be shutdown after the migration.

Step 5) Complete the Install and after reboot start the service

ADMT and Migrating Users

At this point we should be officially ready to migrate users, on ADMT open ADMT:

Right click the folder and select the user migration wizard
Populate the domain names and tree source domain controllers should pick up automatically.

Select your users, Pick a target OU, then select to migrate password:

Given you followed the permissions section, this should work:

Keep target state same as source and don’t copy SID as we have no intention of using SID filtering.

These settings worked great for me, change based on your needs.

again these settings worked for me.

After the process…

It worked! However i was amazed even in my first test run, there was one noticeable message in the log:

Rename UPN name user@NewDomain.com to user@NewDomain.com. Cannot create accounts with the same UPN name as another UPN in the enterprise.

Well cause there already exists a user with that UPN at the parent, but why is it picking the parent for setting the UPN? Who knows… but much like that reference you can bulk select users in ADUAC Snap-in, and select the child domain from the drop down text-box.

Summary

  1. Create a Copy of the Source Domain Controller
  2. Rename its Domain
  3. Connect to target domain subnet
  4. create conditional forwarders
  5. create two way trust
  6. Setup ADMT
  7. Setup PES
  8. Migrate Users
  9. Remove Trust, and Shutdown NotSpecial.com DC
  10. Happy Dance

Hope everyone enjoyed this post, and hopefully someone finds it useful.

*Update* Here’s a really good blog on SID FIltering between forest trusts!

Resetting Access to SQLEXPRESS 2016

So today I wanted to rerun a task of using ADMT on a server I configured, I was now connecting to this server via a different domain account then when I had first configured the server in my first test run. Now for the purpose of this particular DB and server purpose I could easily rebuild… but what if… you’re in a situation in which the data and the access to it is much more important.

Also… I was lazy… so I researched.. after a little while (my dev got involved too) it was simply a mission… a purpose a GOAL to figure it out… in the end it was really easy it was just SYNTAX, oh gawd the important of syntax.

Now the SQL team at Microsoft does a lot of wonky things and doesn’t follow standards that most other divisions follow, so it hats off to the walls, if the SQL guys are on mushrooms today better expect some funky changes without notice or documentation. Wait… what… anyway…

My research began.. . stackoverflow.… to… a random blog, archived on the wayback machine ahhh I feel that’d what will become of my blog….

Let me paraphrase everything:

  1. If you are a local admin you can fix this
  2. SYNTAX

My main issue was even though I kept starting the service with the sqlservr -m as the blog posted this resulted in an error, it turns out you have to specify the instance name… so

  1. change DIR to “C:\Program Files\Microsoft SQL Server\MSSQL13.SQLEXPRESS\MSSQL\Binn”
  2. run “sqlservr -s SQLEXPRESS -m”
    *Note* If you are on said mushroom you can skip the space and run “sqlservr -sSQLEXPRESS -m” and it will work too
    At this point there are two lines to watch out for:
    A) SQL service is running in single user mode (-m)
    B) SQL service is running under admin account that is logged in
  3. open another cmd prompt as admin and run “sqlcmd -S COMPUTERNAME\INSTANCENAME -E”
    E.G. sqlcmd -S SQLSERVER\SQLEXPRESS -E
    *NOTE* the computername is super important else you will get a logon failure at the other console screen showing the compueraccount instead of the user account for some reason.

Because of this error I kept trying other things, my dev tried a couple things and thought one of them was enabling TCP/IP stack, but I said this was all local commands and connections with sqlcmd since this SQLEXPRESS doesn’t come bundled with SSMS and there was no member server or client system with SSMS available to connect or use, as to the point of resetting access locally.

once you get the 1> prompt you can just follow the other guides which is:

1> CREATE LOGIN [DOMAIN\USER] FROM WINDOWS
2>go
1>exec sp_addsrvrolemember @loginame=’DOMAIN\USER’,@rolename=’sysadmin’
2>go
1>exit

Yes that is loginame and not loginname, cause mushrooms.

A certificate chain could not be built to a trusted root authority

Today I tried installing .NET 4.6.2 onto an offline WIndows 7 to setup playnite.

Every since I built my PiCade I’ve been huge into loaders, almost all of them seem to rely on RetroArch for an old game emulation which is fine by me. Check out those sites for their respective offerings. My PiCade uses Lakka which is just a different loader that I find is rather well compiled for ARM based devices such as the Pi. They do offer a x86-64 build however I found that since it’s a linux derivative you have to be pretty Linux swanky to do good with it, by that I mean supporting hardware can be a bit more difficult as you have to be able to get the drivers for certain hardware yourself, in my case using an old HP laptop with an ATI card inside it was CPU rendering everything so N64 and Dreamcase would CHUGGGG, even though the system is way more than capable.

Anyway… Playnite only has one dependency for Windows, thats .NET 4.6.2, so when I downloaded the offline installer I didn’t expect issues, until…

Error! A certificate chain could not be built to a trusted root authority.

Ughhh ok… Google… wtf does that mean? MS says

Grab their certificate

install using elevated cmd with…

certutil -addstore root X:\Where\you\saved\the\Cert.crt

Sure enough after this I was successfully able to install .NET 4.6.2