VMware HA down after 6.5 patch

The Story

So the other day I tested the latest VMware patch that was released as blogged about here.

Then I ran the patch on a clients setup which was on 6.5 instead of 6.7. Didn’t think would be much different and in terms of steps to follow it wasn’t.

First thing to note though is validating the vCenter root password to ensure it isn’t expired. (On 6.7u1 a newer)Else the updater will tell you the upgrade can’t continue.

Logged into vCenter (SSH/Console) once in the shell:

passwd

To see the status of the account.

chage -l root

To set the root password to never expire (do so at your own risk, or if allowed by policies)

chage -I -1 -m 0 -M 99999 -E -1 root

Install patch update, and reboot vCenter.

All is good until…

ERROR: HA Down

So after I logged into the vCenter server, an older cluster was fine, but a newer cluster with newer hosts showed a couple errors.

For the cluster itself:

“cannot find vSphere HA master”

For the ESXi hosts

“Cannot install the vCenter Server agent service”

So off to the internet I go! I also ask people on  IRC if they have come across this, and crickets. I found this blog post, and all the troubleshooting steps lead to no real solution unfortunately. It was a bit annoying that “it could be due to many reason such as…” and list them off with vCenter update being one of them, but then goes throw common standard troubleshooting steps. Which is nice, but non of them are analytical to determine which of the root causes caused it, as to actual resolve it instead of “throwing darts at a dart board”.

Anyway I decided to create an SR with VMware, and uploaded the logs. While I kept looking for an answer, and found this VMware KB.

Which funny the resolution states… “This issue is resolved in vCenter Server 6.5.x, available at VMware Downloads.”

That’s ironic, I Just updated to cause this problem, hahaha.

Anyway, my Colleague notices the “work around”…

“To work around this issue in earlier versions, place the affected host(s) in maintenance mode and reboot them to clear the reboot request.”

I didn’t exactly check the logs and wasn’t sure if there actually was a pending reboot, but figured it was worth a shot.

The Reboot

So, vMotion all VMs off the host, no problem, put into maintenance mode, no problem, send host for reboot….

Watching screen, still at ESXi console login…. monitoring sensors indicate host is inaccessible, pings are still up and the Embedded Host Controller (EHC) is unresponsive…. ugghhhh ok…..

Press F2/F12 at console “direct management as been disabled” like uhhh ok…

I found this, a command to hard reboot, but I can’t SSH in, and I can’t access the Embedded Host Controller… so no way to enter it…

reboot -n -f

Then found this with the same problem… the solution… like computer in a stuck state, hard shutdown. So pressed the power button for 10-20 seconds, till the server was fully off. Then powered it back on.

The Unexpected

At this point I was figuring the usual, it comes back up, and shows up in vCenter. Nope, instead the server showed disconnected in vcenter, downed state. I managed to log into the Embedded Host Controller, but found the VMs I had vMotion still on it in a ghosted state. I figured this wouldn’t be a problem after reconnecting to vCenter it should pick up on the clean state of those VM’s being on the other hosts.

Click reconnect host…

Error: failed to login with the vim admin password

Not gonna lie, at  this point I got pretty upset. You know, HULK SMASH! Type deal. However instead of smashing my monitors, which wouldn’t have been helpful, I went back to Google.

I found this VMware KB, along with this thread post and pieced together a resolution from both. The main thing was the KB wanted to reinstall the agents, the thread post seemed most people just need the services restarted.

So I removed the host from vCenter (Remove from inventory), also removed the ghosted VM’s via the EHC, enabled SSH, restarted the VPXA and HOSTD services.

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

Then re-added the host to vCenter and to the cluster, and it worked just fine.

The Next Server

Alright now so now vMotion all the VMs to this now rebooted host. So we can do the same thing on the alternative ESXi host to make sure they are all good.

Go to set the host into maintenance mode, and reboot, this server sure enough hangs at the reboot just like the other host. I figured the process was going to be the same here, however the results actually were not.

This time the host actually did reconnect to vCenter after the reboot but it was not in Maintenance mode…. wait what?

I figured that was weird and would give it another reboot, when I went to put it into Maintenance Mode, it got stuck at 2%… I was like ughhhh wat? weird part was they even stated orphaned ghosted VM’s so I thought maybe it had them at this point.

Googling this, I didn’t find of an answer, and just when I was about to hard reboot the host again (after 20 minutes) it succeeded. I was like wat?

Then sent a reboot which I think took like 5 minutes to apply, all kinds of weird were happening. While it was rebooting I disconnected the host from vCenter (not removed), and waited for the reboot, then accessed this hosts EHC.

It was at this point I got a bit curious about how you determine if a host needs a reboot, since the vCenter didn’t tell, and the EHC didn’t tell… How was I suppose to know considering I didn’t install any additional VIBs after deployment… I found this reddit post with the same question.

Some weird answers the best being:

vim-cmd hostsvc/hostsummary|grep -i reboot

The real thing that made me raise my brow was this convo bit:

Like Wat?!?!?! hahaha Anyway, by this time I got an answer from VMware support, and they simply asked when the error happened, and if I had a snippet of the error, and if I rebooted the vCenter server….

Like really…. ok don’t look at the logs I provided. So ignoring the email for now to actually fix the problem. At this point I looked at the logs my self for the host I was currently working on and noticed one entry which should be shown at the summary page of the host.

“Scratch location not set”… well poop… you can see this KB so after correcting that, and rebooting the server again, it seemed to be working perfectly fine.

So removed from the inventory, ensured no VPXuser existed on the host, restarted the services, and re-added the host.

Moment of Truth

So after ALL that! I got down on my knees, I put my head down on my chair, I locked my hands together, and I prayed to some higher power to let this work.

I proceeded to enable HA on the cluster. The process of configuring HA on both host lingered @ 8% for a while. I took a short walk, in preparation for the failure, to my amazement it worked!

WOOOOOOOOO!!!

Summary

After this I’d almost recommend validating rebooting hosts before doing a vCenter update, but that’s also a bit excessive. So maybe at least try the commands on ESXi servers to ensure there’s no pending reboot on ESXi hosts before initiating a vCenter update.

I hope this blog posts helps anyone experiencing the same type of issue.

 

Creating Custom ESXi Image

Follow these steps

  1. Download Offline Bundle of ESXi Image
  2. Download Drivers E.G The Native ESXi USB NIC drivers
  3. Install PowerCLI (Set-ExecutionPolicy Remotesigned; Import-Module PowershellGet; Install-Module -Name VMware.PowerCLI)
  4. In PowerCLI connect the standard SoftwareDepot by typing:

    Add-EsxSoftwareDepot -DepotUrl <Path to zip>

  5. Get the ImageProfile list:

    Get-EsxImageProfile

  6. Clone standard ImageProfile:

    New-EsxImageProfile -CloneProfile ESXi-6.7.0-8169922-standard -Name MyProfile -Vendor <vendor>

  7.  [Only If Required] If your vib file has Acceptance Level – CommunitySupported, we need to set this Acceptance Level for our ImageProfile:

    Set-EsxImageProfile -ImageProfile MyProfile -AcceptanceLevel CommunitySupported

  8. Add our vib to SoftwareDepot:

    Get-EsxSoftwarePackage -PackageUrl <path to vib>

  9. Add our vib to ImageProfile:

    Add-EsxSoftwarePackage -PackageUrl

Error:

Search result.

Answer driver for specfic version (7.1, need 6.5)

So I downloaded the proper driver but I couldn’t figure out how to pick the right software package since the “get” command was actually already loaded the other driver, so it kept trying to add the 7.1 driver. Only thing I could think of was to close the powershell windows and start fresh…

10. Export ImageProfile to ISO image:

Export-EsxImageProfile -ImageProfile MyProfile -ExportToIso -FilePath

That was it! Sadly the laptop I wanted to use this on was still boot looping, and sadly the USB NIC “Insagnia” didn’t seem to work and was getting NFS4 client failed to load, and not network adapters found on the machine. But was worth a shot.

Using Flash in 2021

The Story

No one should have to use flash…. however, there have been some amazing things that were done with the framework at the height of its time.

Now my issue was more around the fact that VMware, as I use VMware a lot. Happened to choose this framework for their Management Web Interface with 6.0~6.5, with only having fully depreciated it in 6.7. Why did they do this? Cause they didn’t want to rely on a Windows based framework anyway, AKA .NET. Say what you will about Microsoft and Windows, but when you look at the two frame works, it’s pretty clear which is still supported and which is clearly not…

Anyway I digress, if you attempt to use the flash based interface, if you happen to still be on 6.0~6.5 and need access to the flash based interface. Well let’s see…

Wow, that was unreal useless…

Strange same results in new Edge Chromium (well ok that’s not to odd considering they are based on the same engine), I remember just seeing the weird new Flash logo with an information logo.

I thought this might be due cause I never configured the old GPO’s which were used to define allowed sites for flash.

I thought I remember setting it for another thing that used flash and if I navigate to that page…

Clicking Get Flash leads to the same online EOL page… alright. Not exactly the results I was expecting. I swear I remember it popping up that flash logo.. let me try one other machine to validate… Ok all browsers same results, IE simply doesn’t even load the page. I don’t use Firefox. Either way, one of two things happen you get the above snippets, or you get this Flash logo:

Hunt for an Answer

My first google search brought me to a ghacks blog post suggesting to use a web app called ruffle… no thank you.

However lucky for me reading the comments another guy with a way nicer site (no dang ads cluttered everywhere), the guys name is Charles Wilkinson. The rest of this blog post I’ll follow along with Charles’s to see how it plays out.

The Fix

He’s done such a good job with the basic detail I kind don’t even want to paraphrase it, so here’s a direct copy n paste of the How to Fix it from Charles’s blog:

“Reading the Flash Player Administrator’s Guide, in a section called: Administration > Enterprise Enablement we find the official solution.

On any device that we want to enable our legacy app on, we need to edit the mms.cfg file that holds the configuration for Flash Player.

This file can be found under:

  • /Library/Application Support/Macromedia/mms.cfg on OSX
  • C:\Windows\System32\Macromed\Flash\mms.cfg on 32bit Windows OS
  • C:\Windows\SysWOW64\Macromed\Flash\mms.cfg on 64bit Windows OS

This file needs to be replaced with the following content:

# Disable Automatic Updates
AutoUpdateDisable=1
SilentAutoUpdateEnable=0

# Disable prompts to uninstall Flash Player
EOLUninstallDisable = 1

# duplicate actionscript console output
# in browser's console for javascript
TraceOutputEcho=1

# Enable the AllowList feature
EnableAllowList=1

# Normally, the allow list blocks URL requests
# unless the url matches a pattern in the list.
# In preview mode, all requests go unblocked,
# but console output is written for each request
# indicating which pattern it matched or that
# no match was found.
AllowListPreview=0

# Pattern to enable Your Legacy Flash Web App:
AllowListUrlPattern=http://legacy.app.domain.name:8001/

Obviously, you need to replace http://legacy.app.domain.name:8001/ with the URL of your legacy app.

Once this file is saved, hit refresh in your browser and your legacy web app should load. You do not need to restart the browser (at least not when I tested this on OSX with Firefox) – Flash seems to pick these settings up next time you refresh the page.”

Doing the Needful

OK Let’s try it out, my machine is Windows 10 x64, lets navigate to the path mentioned.

I dunno about you, but, I don’t see no mms.cfg

OK, I can’t see much else as to if you need to create this file yourself, or what…

Wait a second….. double reading the Limitations section from Charles post….

Limitations

This fix allows Flash to continue to run, disables the prompts to uninstall and disables automatic updates, however, it does not prevent newer browser versions from removing Flash Support. Users who need to access your legacy app will need to use an older version of Chrome or Firefox with automatic updates disabled. The last versions of browsers supporting Flash are:

Firefox version 84
Microsoft Edge version 87
Chrome version 87

It also seems that Microsoft have released a Windows update that will uninstall Flash: Adobe Flash Removal Update for Windows 10 – KB4577586. Sysadmins will probably want to prevent this update from being installed.

Putting the Pieces Together

Oh.. I’m starting to think the reason I don’t see the flash logo above when I did before is cause I believe the update to remove flash was pushed onto this machine, also the browsers got updated, now on 88.0.705.74

So I literally have to have a system that doesn’t install one particular windows update (if I want to keep it “online”), or use an older machine that is fully offline to get any of these updates, either it be the OS KB updated mentioned or the browser itself updating. Both these requirements are pretty bad.

I should have suspected this, but it sort of slipped my mind, till right now. OK so what are my options…

Option 1

Old copy of a machine, and prevent it from reaching the internet, only access to the devices or URL’s it needs to manage/access. OK so I managed to find a backup/copy/vm of a system that has an older copy of Chrome (version 80) that’s well below the 87… OK but how do I stop it from updating if it does manage to connect to the internet… really just rename the folder update, neat. In my case when I went to go rename told me the file was locked by system… which leaves me to believe there’s a service.. and sure enough there were two, let me just disable these services and then rename the folder…

Weird, even after stopping those services it still won’t let me rename the folder saying its locked by the system…

So after creating a clone of the VM, and disabled the browser updates, and disable windows updates, I navigated to the page, got the “run one time” and it finally tried to load, and I finally got the logo as mentioned on Charles’s blog, that means it’s finally time to try the “hack”.

Open CMD as an admin, and create the file in question:

and fill the table.

 

Not sure if a reboot is required or what lets do one to be safe…

SOB… Chrome updated…. let me try that again…

Well even with Chrome 78, and enabled Flash in settings, and clicked allow on pop-up and I get Download Failed. Sigh… so I grabbed the PPAPI flash installer from the web archive linked in the comments of Charles blog. Installed it and sure enough again, got the logo I posted above, this time a file already existed in the c:\windows\sysWOW64\Macromed\Flash and I edited with the same options mentioned above again…

Same flash logo not sure if I need to reboot to apply or try like the other comments in that blog post stated, and put it in a special appdata location… I’ll try that first and then reboot as a last attempt.

Yes! The Flash based web interface finally loaded!

I have no idea what Option 2 even is at this point…

Soo Summary..

Summary

  1. You need to ensure a Chrome/Chromium based browser Pre-87
  2. If you have MS KB4577586, you need to install the PPAPI flash manually.
  3. Enable Flash within the Browser Settings
  4. Manually edit/create mms.cfg as shown above, and have in both C:\Windows\SysWOW64\Macromed/Flash as well as C:\Users\%Username%\AppData\Local\Google\Chrome\User Data\Default\Pepper Data\Shockwave Flash\System\

I hope someone finds this guide useful… cause I sure found this process painful. 😀

 

VMware vCenter Updates using VAMI

This is a quick post on the latest security release notification from VMware.

VMSA-2021-0002 (vmware.com)

If for whatever reason an update is not possible you can follow these workarounds.

While you can use VUM to distribute updates and patches to ESXi hosts.

You’ll have to use VAMI for updating vCenter.

You can download the latest patches here (vmware account required).

I did this on my lab vCenter,  took a lil while but not bad.

  1. Made a backup of the VCSA using Veeam
  2. Shutdown Veeam or any other backup solution that might use vCenter
  3. Notified anyone that might use vCenter that it would be inaccessible during update
  4. Attached ISO to VCSA VM (You can do as 4sysops did and upload to a datastore, or you can simply open the VCSA console via VMRC, and attach the ISO from your Downloads folder)
  5. Log into VAMI (https://vcsa:5480)
  6. Click Update on left nav, then Update -> Check CD-ROM
  7. The update should be available as the option, then click Stage and Install
  8. Accept the EULA, use/don’t use CEIP, Check I have a backup, Click Install.

It could take an hour or so, then everything is back to running state, here’s the summary page after completion:

You can read the alternative methods such as using CLI, or how to handle a vCenter HA cluster upgrade using the link above to 4sysops guide on upgrading vCenter.

Sorry this post is not as extensive as usual, just a heads up about the latest VMware patches. Stay Safe out there.

 

Get Windows Server out of Stuck Update State

I probably should be a bit more clear, this post will cover how I managed to get a Windows Server 2016 to “check for updates” when it had gone wrong and was stuck looping (checking) and failing where it replaces the “check for updates” button with nothing other than “retry”.

This happened after clicking “Search Microsoft Online for Updates” in which case it found a couple that were not approved by WSUS or not selected as category’s that WSUS actually downloads.

Funny in this case after I did what will be mentioned below, clicking retry did just start checking again, and then stated “Your device is up to date”.

So ok it worked that time, but what I discovered at the time, was that there’s a new command to use on the backend (command line) to do the needful when the UI doesn’t have the appropriate button available. Like usual Microsoft fashion, notifying stakeholders was poor, and so was an documentation.

Now this isn’t the first time I discussed issues around Windows update, in particular around the tool MS has given Syadmins to do the needful; WSUS. Such as this time, when clients are not showing up within WSUS after clearly showing they had applied the GPOs (registries) required and no network issues between them, or this time CU updates weren’t being downloaded by WSUS although clearly the types and categories were fully correct.

In this case however instead the issue was simply what commands to use, as stated within the original person asking the question in the TechNet link above “Since wuauclt has been depreciated in windows 10, I was googling what has replaced it.

I found that usoclient is what has replaced this command for windows update in the command line. ”

What authoritative source is there for this claim, well I found this

“The wuauclt.exe /detectnow command has been removed and is no longer supported. To trigger a scan for updates, do either of the following:

  • Run these PowerShell commands:
    $AutoUpdates = New-Object -ComObject "Microsoft.Update.AutoUpdate"
    $AutoUpdates.DetectNow()
    
  • Alternately, use this VBScript:
    Set automaticUpdates = CreateObject("Microsoft.Update.AutoUpdate")
    automaticUpdates.DetectNow()

Funny thing about this is I found that wuauclt /reportnow still works in Server 2016, as noted in my other blog posts. I generally didn’t use /detectnow. However what I found was that the new commands did work for me.

Such as these as mentioned from Spiceworks:

“Start checking for updates: UsoClient StartScan

Start downloading Updates: UsoClient StartDownload

Start installing the downloaded updates: UsoClient StartInstall

Restart your device after installing the updates: UsoClient RestartDevice

Check, Download and Install Updates: UsoClient ScanInstallWait”

Then of course these as mentioned in the TechNet post:

“RefreshSettings – used to quickly enact any settings changes
RestartDevice – as the name implies, it restarts the device. Can be used in a script to allow updates to finish installing on next boot.
ResumeUpdate – used to tell the tool to resume updating after a reboot.
StartDownload – initiates a full download (from Microsoft) of existing updates
StartInstall – kicks-off the installation of the downloaded updates
ScanInstallWait – Combined Scan Download Install
StartInteractiveScan – we’ve yet to get this one to work, but it suggests that the process may work in a GUI
StartScan – kicks-off a regular scan”

While it is nice to see something available, it would be nice if MS made a more formal announcement of the deprecation and the replacements.

Hope this helps someone.

SharePoint – Can’t Delete ContentType

The Story

You know every post has to start with a story. So it’s story time, It all started with a site that needed to be templated and used to create new sites. Now when the user went to actually deploy the new template via “create new site” link under the site contents area, it errored out stating that there was an error.

Create SharePoint Site Template

I wouldn’t have blog posts if everything work via the happy path, there’s other people to blog about that…

This of course required jumping through some hoops to even make the site savable as a template, in my case it was just the property to be set using PowerShell:

$web = Get-SPWeb http://your_ site
$web.AllProperties["SaveSiteAsTemplateEnabled"] = "true"
$web.Update()
$web.Dispose()

Else you’ll get the following error:

So once this is done, you can finally create a template.

However, now we have to actually deploy it.

*NOTE* when you create a template of a site, you are secretly creating and activating a “Solution” to the main site. So if you need to manage, or delete a template you first have to deactivate the solution then you can delete it.

Deploy New SharePoint Site Template!

Would you expect anything else form my blog post? 😛 OK this should be easy enough let’s just delete this old content type as it was a legacy one left behind from a migration.

So first since this template is trash, you’d figure these type of checks would take place at creation of the template.

Ahhh SharePoint never cease to piss me off… OK, let’s google this…

The first source is dead on the solution… However it required making a direct Database change. To keep SharePoint “supported” state, although obviously broken. The alternative solution is to either find the original feature package, and re-install it either via command line (stsadm.exe), or Powershell or the front end. Of course if this is a third party feature, and you only have installation for older SharePoint, then this would have to be cleaned up on the old environment before migration. If I find the link (didn’t save at the moment) 🙁 there apparently is a way to map this ContentType to “Dummy” features, delete the content type, then delete the dummy feature. This is the only alternative way while doing it via the front end to stay “supported”.

In the mean time, you can also spin up the site in a test environment, and do the needful on the content type in the database backend (connected to the instance, and Database for the site content (WSS_Content by default):

Update DBO.ContentTypes
Set IsFromFeature = 0
Where ContentTypeID = 0xIDNum

The content type ID can be extract from the address bar via the front end as it is known by the web parameter ctypeID:

Now you’d figure there be no problem delete the content type, until another error shows up with a different reason. (OK I remember it being different but until I run through these test again maybe they were the same, as the second source explains…)

[Insert Picture of error after DB change]

Googling I came across this guys very nice blog post about the same issue!

Really short version… the content type is still used/referenced by another SharePoint object within the environment. He does show and reference some really nice C# code to help track the issuing objects. However I have no interest in building an App, just to find these… there has to be another way!

Ohh stackexchange how beautiful you are

$site = Get-SPSite("your-site-url");
foreach ($web in $site.AllWebs) {
   $ctype = $web.ContentTypes["Your Content Type"]
   $usages = [Microsoft.Sharepoint.SPContentTypeUsage]::GetUsages($ctype)
   foreach ($usage in $usages) {
      Write-Host $usage.Url
   }
}

Which helped me track the objects, in my case Lists…

Turned out to be the list in all subsites called “Tasks” now this is a SharePoint created list object, however they were created after this particular feature was enabled on the site, thus all subsites inherited the issue.

Now there are some nice online references to delete content types, or lists and other objects via PowerShell.

However if you know the object model well enough you can pull one liners to do wonders…

$spsite = Get-SPSite http://yoursite
$webs = $spsite.AllWebs
($webs.Lists | ?{$_.Title -eq "Tasks"}).Delete()
$webs.Dispose()
$spsite.Dispose()

And just like that hundreds of old SharePoint lists that were no longer used were gone. If the lists you have contain data that is to be kept, you are going to have to migrate the data to a new list, then delete the offending list and migrate the data back.

OK, NOW you can create a template from the site, and deploy it and it should succeed without issue. You can now navigate to the site content area where the solution packages are stored and copy it out, and then upload it to your production environment and create new clean sites. However note that this won’t fix the issue in your production side.

So you’ll have trade offs to consider in which way you decide to handle the issue.

Summary

SharePoint is a beast of a designed machine, and can often include some bugs that were not expected. I hope to extent this blogs and provide more SharePoint related content in the future. Cheers, I hoped this helped someone out there.

Apple Fun Times

An Apple Story

Well another day, more fun with big Tech. Today… Apple.

People love em, People hate em. Me I tend to swing to the latter, why, cause I’ve always been one who doesn’t believe in giving up freedoms for security. With Apple that’s exactly how it works.

In the Apple world (that beautiful utopia), all Apple software is run only on authorized Apple hardware. Apple even has the audacity to take it so far as to secretly  “pair” modular parts of the phone (such as the camera) to be “locked/usable” to only the board it came from, watch this YouTube video on the matter that has a crazy 10 mil views.

I won’t go too much into this, insane design choices. Instead I’ll simply attempt to cover some “things” I discovered simply attempting to deploy a couple new iPhones.

Thing 1 (How it began)

The first thing to note is… How did we get here? In this case I mean, let’s face it, Apple primary stakeholders were individuals, selling products to, well, people. The thing is they became so popular that people, being people, started to use these devices for a lot. Overtime they crept there way into the business world, which of course left a freaking huge gap as to how to…. you guessed it manage these devices.

Now if you are even slightly familiar with teh Apple ecosystem, at least in terms of their mobile phones and the iOS operating system. You maybe aware of a change to the factory restore process, roughly around iOS ~7 saga. That change was that if you happened to put a device into DFU mode, and attempt to set it as a new phone, you will be prompted to authorize that from the account (AppleID) that was previously associated with that device. This is a problem if a corporation owns the device, but a user uses a personal AppleID, tied to a personal Email address. In this case before you’d have to jump through some hoops with Apple. Such as provide proof of purchase and all this other fun jazz that can take a fair amount of time.

Bring in MDM (Mobile Device Management), Now if this was simple, I would have no issues with this. The fact is, there are far more hoops you have to jump through then you realize to make this a reality.

Thing 2 (Apple Business Manager)

Apple’s take on MDM is Apple Business Manager (ABM). Now as far as I know, ABM is not in itself MDM, it is merely a prerequisite required to actually starting use an MDM (from another provider) to manage iPhones, and other iDevices.

I’ll do my best to cover the processes here, but please note the entire process was not actually gone through in its entirety. So there will be more questions than answers through most of this blog post. I do apologies for this and if you want to stop reading you certainly can at this point.

So here’s the simplified overview PDF of ABM…

Overview of Managed Apple IDs for Business

And this is what the login page looks like:

Once you are logged in, it’s a super simplified Web UI that looks like a 4 year old designed it (in California of course). Now before you can even do anything at all, the very first step is to “authorize your domain”.

How do you do this? OK let me take one quick step back here. Reading this More detailed guide to ABM, the first part is signing up for ABM, in more cases than not, this will be handled by a Value Added Reseller. Once you are signed up, and have defined the “administrators”, they will have to be the one to “Validate the domain”, which (from experience) is nothing more than  a specially generated string you have to create a TXT record for on your external DNS provider for said domain. (Pissssst AKA DNS Validation).

Funny enough, even though I know (again from experience) that this is a required step, it was not anywhere in the Get started guide PDF I just referenced. Here’s how to do it though (according to Apple).

Thing 3 (Federation)

This part, honestly has me so confused. Throughout the history of Apple, they don’t integrate with anything else, unless it is Apple. Yet there’s this….

  1. Is federation a requirement to use ABM and MDM for Apple devices?
    I don’t know… lets ask someone on the Apple IRC channel…
    Apparently it is, and apparently MS Azure AD is the only auth provider to federate with? Uhhh ok, Not sure what suited donkeys sucked whos dick to make that deal….*Note* The IRC user that helped me above was a really cool guy.
  2. MS Azure AD is the only listed auth provider for federating.

Well that sure sounds like a bag of ass.

This is sort of where the road dies for me. As there is no Azure AD for us to use. So great, not sure where this requirement is listed. So anyway… normally…

Thing 4 (Mobile Providers and Reseller IDs)

If you did manage to federate, the next thing you need to do is “authorize” resellers and cell providers. You do this by taking the resellers ID (usually given to you by the VAR), then in the ABM click settings (lower left), then Device Management Settings, then edit Customer Numbers, and add them.

Simple.

Thing 5 (MDM)

I’d love to cover this in more details, however since Thing 3 didn’t fly, I’m not exactly sure how this part works. When I clicked the “Add MDM server” it seemed to have wanted to simply generate a key pair, then I fully assume here, you use the public key by adding to the MDM server of choice.

When I figure out which MDM servers actually are available to use, and how to make them work, I plan to extend this blog to help cover those steps.

What now?

Well I guess if you don’t have Azure AD then the options available to you to manage Apple devices seems rather limited. There are limited control and auditing one can do with ActiveSync, but that’s only through MS Exchange servers if you have them.

New Phones!

So got some new Apple phones to deploy. Just note I’m not a fan of Apples hardcore stance on hardware lockdown (“for security”), including this now even swapping good working parts from another good working phone. Unreal…

First User, First Issue
Restore/Update UI Wizard Logic

First users transfer, the main thing was the user was excited that they stated their phone was fully up-to-date. While normally I would love to hear this, it actually caused me grief in when I went to upload the back up profile to the phone and got this nice alert from iTunes “iOS on  phone to old”…

Now, I would have assumed by this day and age computers would be more intuitive then this. So instead of iTunes having a nice prompt “Would you like me to update to the latest version for you and load your saved profile?” it gives the ugly prompt above, and expects you to jump through all the OOBE prompts of the phone, connect it to a network and update it before you can load your profile. Redic.

*Note* I managed to click on “setup a new phone” in iTunes, then I could click the phone icon on the upper bar area, then click on  update device. So it is possible in iTunes, it’s just not as intuitive as one would like.

Second User, Second Issue
Backup Encryption Logic

Now you figure without ABM/MDM issues would be less, but I digress.

With the second user, created a backup in iTunes. Now this is where I really got my knackers in a twist. After successfully creating a backup then go to restore it onto the new phone randomly get a prompt.. “Please enter the password for the backup.”

Like wtf you talking about.. I didn’t set a password when I created the backup, where did this password come from? Off to Google!

First result!

“Upon first turning on “Encrypt iPhone backup” in iTunes, a password must be set for your encrypted backups. This applies automatically to all future backups, without needing to enter it again.

If you later wish to turn off backup encryption or to restore from an encrypted iTunes backup, it is necessary to enter the current backup encryption password.

If you have forgotten or do not know the password, then encryption cannot be turned off and the iTunes backup cannot be used. There is no way around this feature.

An alternative solution for transferring the content and settings to your new iPhone is to back up your old iPhone using iCloud and then to restore your new iPhone from the iCloud backup. (iCloud backups are encrypted as standard, so will also include your sensitive data).”

Like, yeah ok, I could use iCloud but that will encrypt it with the same password I’d imagine, either way wanted to do it via iTunes to save a bit of time. Keep looking.

Second Result!

Wow there’s everything from try the iTunes password, to 0000, to windows admin password, to “your first ever iTunes password”. Get outta here, sure enough none of them worked, even though the comments appears the answer is all over the place. If you get lucky with any of these, congrats. Moving on.

Third Source!

“You can’t restore an encrypted backup without its password. With iOS 11 or later, you can make a new encrypted backup of your device by resetting the password. Here’s what to do:

On your iOS device, go to Settings > General > Reset.

Tap Reset All Settings and enter your iOS passcode.

Follow the steps to reset your settings. This won’t affect your user data or passwords, but it will reset settings like display brightness, Home screen layout, and wallpaper. It also removes your encrypted backup password.

Connect your device to iTunes again and create a new encrypted backup.

You won’t be able to use previous encrypted backups, but you can back up your current data using iTunes and setting a new backup password. If you have a device with iOS 10 or earlier, you can’t reset the password.”

Which lucky for me was the case this time. So this method actually worked, who’d thought that the encryption password is just part of the system settings. Oh Apple.

Anyway so this is why this really puts a knack in my gears, If someone made a backup on just iTunes, thinking there were good, and only when they restore do they get the password prompt, and they had lost their old phone, they’d be pissed. Frankly so would I, iTunes should state that fact as soon as a backup is being created to prompt for the password to ensure the user is 1) aware that there phones data is encrypted with a password and 2) that they know what that password is.

Third User, Third Issue
Storage Check Logic

Yeah, or should I say lack there of. When I started to complete the transfer of this user device, the storage used was much higher then the rest. However low and be hold I started the iTunes backup process not even thinking about that, cause, well why would I?

Now, I’ve been called out on being a “resource monitor” and by that I mean I spent a fair amount of time ensuring a system is working as intended by validating theory’s and deployments by, non other than looking at the data. In this case Windows Task Manager. If your really want to get in the weeds you can use Sysinteral’s Tools. Anyway, I noticed the hard drive space getting low, and the process bar and iTunes not being anywhere near the end (you’ve figure it show some stats, but just a bar).

It wasn’t long till it ran out, now I’m kind of shocked there’s not some simple validation logic coded here. It simply just took up all the data it could, and prompted an error telling me to “clear space”. I’ll not I did a whole bunch of that till there was nothing left to clear and it still sucked the hard disk dry. Now I wanted to see if I could simply point the iTunes install to another path like an external USB hard drive and use that, might lose some speed on the slower bus but should still be decent. Funny it was asked by this person and it was the answer by Doug which worked for me, the only reason the person asking the question didn’t understand the answer was cause of how iTunes for windows is hardcoded to use the Windows Users AppData location (Windows environment Variable %appdata%) which will always by default be the windows install drive. So even though they installed iTunes on the external drive it won’t use it to say the backup data. I would have expected there to be an option, which one was stated is available under the “advanced” area of the settings, but that clearly didn’t work cause after setting it and running the iTunes backup I could see it still using the windows install drive and the users appdata location, via resource monitor. Only after I created a syslink for that folder to the external drive was it using it to create a backup of the phone.

What a pain…

 

Noted Changes

  1. 6 Pin default for unlock code, can be changed to 4, but depends on if you use it to store pay stuff then, it might not be available.
  2. This one really pissed me off… It’s not a “power button” no that’s just to universally known, and far to single purpose driven. So instead let’s call it a “Side Button” and completely change the normal operation of powering on n off a phone. User comes up and the phone is stuck in a process “signing in to cloud” or some rubbish…. I look at the phone and go… well… did you reboot it? They laugh, I laugh, they look at the phone… try a bit and say… How do I do it? They laugh, I laugh, and then I press down the button and Siri listens to me swearing at it cause they made the power button the siri button…. brilliant!You can read here for more details. So from universal “Press and hold the power button” is now
    “One method is to press and hold either the Volume Up button or the Volume Down button and the Side button simultaneously until you see the Slide to Power Off slider.NOTE: Quickly pressing the Volume Up button and the Side button takes a screenshot instead of showing the Slide to Power Off slider.

    Alternatively, you can quickly press the Volume Up button followed by the Volume Down button and then press and hold the Side button to access the Slide to Power Off slider. When using this method, the SOS slider doesn’t display. If you’re nervous about accidentally triggering a 911 call, you can use this method”

    Brilliant, I’d almost consider this to be “genius” level.

  3. Zoom vs Standard This isn’t new technically, but is worth nothing about some issues around the type you pick. (Entertainingly enough, this blogger also notes of some of the un-intuitiveness of Apple’s design choices)
  4.  I’m sure there’s more I’ll save this as a place marker. Publish for now and write a summary once I’m done with this nightmare. I don’t even have a proper category for this content, and by gawd I don’t want to make an Apple category….  *shutters*

UniFi Shows MAC address instead of Hostname

I noticed this recently, that the UniFi management interface would show some clients as just their mac addresses instead of the host names like most other devices.

Searching I found this one, but it was after an update, I did not update the software.

Then I found this thread which was more what I was looking for, which tells me how the name is retrieved … “DHCP Snooping”.

Alright, so taking a look at the DHCP server, I noticed it was indeed empty names on the IPs that were given out.

Didn’t take me long to determine that it was Android devices. When I wanted to configure a hostname to the device I found out with the latest version.. I can’t?

“Hostname is used to easily identify and remember hosts connected to a network. It’s set on boot, e.g. from /etc/hostname on Linux based systems. Hostname is also a part of DHCPREQUEST (standardized as code 12 by IETF) which a DHCP client (Android device in our case) makes to DHCP server (WiFi router) to get an IP address assigned. DHCP server stores the hostnames to offer services like DNS. See details in How to ping a local network host by hostname?.

Android – instead of using Linux kernel’s hostname service – used property net.hostname (since Android 2.2) to set a unique host name for every device which was based on android_id. This hostname property was used for DHCP handshake (as added in Android 2.2 and 4.0). In Android 6 net.hostname continued to be used (1, 2, 3, 4) in new Java DHCP client when native dhcpcd was abandoned and later service was removed in Android 7. Since Android 8 – when android_id became unique to apps – net.hostname is no more set, so a null is sent in DHCPREQUEST. See Android 8 Privacy Changes and Security Enhancements:

net.hostname is now empty and the dhcp client no longer sends a hostname

So the WiFi routers show no host names for Android 8+, neither we can set / unset / change it.

However on rooted devices you can set net.hostname manually using setprop command or add in some init’s .rc file to set on every boot. Or use a third party client like busybox udhcpc to send desired hostname and other options to router. See Connecting to WiFi via ADB Shell.”

Well then… Now I have to manually set Aliases and use DHCP reservations just to be able to track these devices… cause “privacy

Summary…. Thumbs up… man!

Palo Alto Networks – Service Routes

The Story

You can read about Service routes from PAN directly here.

Basically … “The firewall uses the management (MGT) interface by default to access external services, such as DNS servers, external authentication servers, Palo Alto Networks services such as software, URL updates, licenses and AutoFocus. An alternative to using the MGT interface is to configure a data port (a regular interface) to access these services. The path from the interface to the service on a server is known as a service route. The service packets exit the firewall on the port assigned for the external service and the server sends its response to the configured source interface and source IP address.”

This is generally used if you configure the firewall, but don’t actually happen to physically plug anything into the MGMT port of the Firewall (MGMT on Physical or VNIC0 on VMs). However the device does have a internet connection, or has some interface on the dataplane that has access to a specific service. Whatever the need may be they can be useful to know they exist and can be utilized for certain situations.

When I discussed this with a friend who deploys many of these devices, it was opted to use the MGMT interface for most things. I did note one case such as Email, where you could configure the service route for that via the gateway interface for the mail server, thus only require one IP in the ACLs of the mail relay/server.

He did note that you could not test email from the passive firewall, as the interface won’t be active. Which could be problematic for other monitoring services such as SNMP, if utilized. Which was noted. Luckily many different services (SNMP/Email/LDAP) can be configured independently and all  default to the MGMT interface.

Summary

The main reason I even noticed this was due to email not working  on the alternative firewall after it took over from a failover, even though the dashboard on both firewall stated the running configs are both the same. Well it turns out that service routes I guess are not tested for synchronization between peers.

So yeah… not that if you are using Service Routes with PAN firewalls.

MacOS as a VMware VM

The Story

Requirement: MacOS, or something like it to play with the OS to support people.

Problem: Don’t own a Mac.

Idea: Just VM it like everything else.

Reality: See below…

Well… Yeah this is a thing. Need to play with this so here we go?

Sources: How to Install macOS on VMware in Windows PC [Mojave] | by BuildSomeTech | Medium

First Problem – Making/Getting Image File

No ISO is usually available directly from Apple, So you have to create it… with a Mac. Well I don’t got one of those, so nopers, on that.

Solution… Trust someone else to upload a clean version of the ISO, source blogger above did that, but again have to trust the ISO.

Second Problem – Download Quota

Too many people want this ISO now… have to bypass the download quota. Spoiler requires a Google account. I wasn’t exactly sure but after logging in I was able to create a shortcut of the source folder into myDrive, then create a shortcut to that, then download the file directly.

Once I got the file I extracted it with 7zip and entered the password provided from the source blog post. I then proceeded to create a new VM on my ESXi 6.7 hosts managed by vCenter 6.7.

Create VM

Only main thing was much like the source was to pick the MacOS under other:

I was not as nice with the hardware specs and left them default minimum:Ahh crap, I have to up the memory, to “min 4 GB”, fine, my host not gonna like me haha. So right click the VM, edit settings, bump up the memory to 4GB, while we are here, click VM options -> Advanced -> Edit Configuration

Then add a config: smc.version with value of 0

Also ensure to force boot into BIOS/EUFI menu so you can mount the ISO from above. Little trick (the Disc icon is greyed out till the VM is powered on in the VMRC (VMware Remote Console). So you can mount it and reset the VM to boot the ISO.

but when on..

Third Problem – Need the Unlocker

I got an error telling me that there are no compatible hypervisors, wait what?

Ohhhh!!!! Classic Apples, that’s why all these other guides are using an “unlocker

“MacOS is only able to be installed on Apple-branded, official hardware. Apple does not license you to install it otherwise. By using that “unlocker” program–which is actually a circumvention or “crack” device–you would be violating the EULAs of both Apple and VMware. So, the only way you are legally able to run MacOS in a virtual machine is on official Apple hardware. Any other installation type is illegal.” – daphnissov

Yeap… Apple, Apple, Apple… Ugh

So I went created a temp account on the unlocker site, downloaded the 208 version and ran it on a test ESXi host I had:

and rebooted it… Server rebooted, create new VM, set guest to MacOS and…

man… FFS… after even more research, managed to find newer version of the unlocker from better open sources (Git) 🙂

Like this one, and this one and one other one my colleague pointed me to, but I don’t have the link. Either way, downloading the package and running the command to install….

(┛ಠ_ಠ)┛彡┻━┻

for reals…. this project is pissing me off.

So after Googling this, I found this reddit thread...

“the unlocker.tgz is not included in the master commit , dunno why, do this…” -KciNicKGX

Well **** off…. my nice colleague managed to check ou the build python script and noted a view things, and managed to create a unlocker package for me with the required unlocker.tgz file with in it… using it…

Finally! I can pick GuestOS MacOS 10.14 and the VM actually Boots!

Mount the ISO

OK… Now if you power on the VM you should see the disc icon available…

(I wonder how many people are just trippin cause my VM name is Majove and not Mojave… lol)

don’t let the greyed out icon fool you, just right click it and you’ll be able to mount the ISO from your local workstation without having to upload it to a datastore.

Let’s mount the ISO we had such a hard time getting above.

Now boot it by picking the virtual CD-Rom in the list…

so I finally got an Apple logo wooo..


Result Failure!

and does it work…. Nope.. Boot loop…

I found one other reference to a boot loop, all the comments state is to try alternative ESXi host version, which right now I don’t have.

You know what… Fuck this stupid ass fucking dumb shit.

I’m out.