Free Hypervisor Backup
Before Part 3

The Story

I’m currently in the quest to fulfill my needs for a free VM backup solution, my hypervisor of choice right now is ESXi, while there are great alternative free hypervisors ( Citrix’s Xen, Microsoft’s Hyper-V, Linux’s KVM) I personally always felt comfortable with VMwares user interface (I’m talking the old Windows Phat Client).  I totally understood the need for a web based client, however I felt it a sham to drop the phat client as to provided multiple benefits that the Web UI just doesn’t. Although the latest 6.7 Web UI has been pretty decent, besides their placement of the Hostname location…. :@

Anyway… if you’ve been following my posts you’ll see I decided to give Veeam Free a shot. I love these guys, great software, so I figured best to get better antiquated with their software, to my dis-may their simply relied on VMwares APIs solely. Which meant no SSH backdoor tricks for use ESXi free users.

However as I also mentioned in my previous post, an awesome dude who runs virtuallyGhetto.com, William Lam; wrote a script to complete the task we wanted via the hosts local CLI, which can be connected to via SSH. The script is called GhettoVCB. I took a quick look at the source code and did find a couple instances of zombie code and other anomalies but for the most part looked decent enough to give it a shot. Now I will get to this stuff in my next post using the same example VM I will specify here where I came across a couple issues and interesting facts I discovered during this adventure.

My Discoveries

The first thing I noticed about the script was the lack of certain dependency checks, in this case there’s no actual source code dependencies that his script relies on, it’s all meant to run on ESXi, and sure enough there’s some line of code to check for that…

Nice, however there doesn’t seem to be a check to validate if the datastore specified in the very first variable has enough space to complete it’s task.

In this case the script would simply error out once the destination ran out of space stating the source VMDK was the issue, this lead me briefly down the wrong rabbit whole. After validating I had no issues with my source VMDK (booted the VM and checked all services and FileSystem integrity) I noticed a couple things.

1) Even though I specified Thin disc for my destination, which had enough space to store the VM data (60~ GBs), the thin disc was attempting to create the full provisioned size of the disc, cause…..

2) I forgot the source disc was set to Thick Provisioned Eagered Zero

So there was a couple things about this VM….

1)  It’s my ZoneMinder VM which holds my IP camera footage on motion detection

2) This data is mostly useless due to no events having taken place

Alright so now my goal was the following:

1)  Remove all the un-needed data

A) Open ZoneMinder Web UI
B) Click on Camera
C) Delete All Events
D) wait a while before all the MySQL queries to kick off to clear the data
E) Used “df -h” to watch usage drop

2) Once I had all the data removed, I had to re-claim the space. I decided to dig up my old blog post on the subject matter… Well that was a bit underwhelming and simply provided my links instead of any valid examples… So I hope to provide a bit better details here:

A) I’m running Linux not Windows so sdelete is out.

B) My first attempts I decided to follow this DD example “dd if=/dev/zero of=/zeroes && rm -f /zeroes” (DO NOT DO THIS, from my tests it caused mySQL service to not come up properlly). I then found people stating to use secure-delete or zerofree, however I had some issues with these against a live system and wanted a simple live system, general technique… if there so many references stating dd can do it… how.. then I found this
“dd if=/dev/zero of=/tmp/empty.dd bs=1048576; rm /tmp/empty.dd”
I’m assuming maybe cause I used /tmp instead of /… only diff I can think of.

C) At this point I made a backup using ghettoVCB which I had to use an alternate Datastore to save the VMDK that I would then finally hole punch, in this case the ghettoVCB script converts the VMDK from thick to Thin however will still be the size of the provisioned Disc.

D) Now this is where I started to get a bit… annoyed… However I did learn a few things… one is that & to bring a process to the background doesn’t disconnect that process from the current terminal session. So while I was SSH’d in and running the clone operation, it wasn’t fully completing when my SSH session timed out, even though I used & to run it in the background. Read this for more info but the gist take away is this: “nohup and disown both can be said to suppress SIGHUP, but in different ways. nohup makes the program ignore the signal initially (the program may change this). nohup also tries to arrange for the program not to have a controlling terminal, so that it won’t be sent SIGHUP by the kernel when the terminal is closed. disown is purely internal to the shell; it causes the shell not to send SIGHUP when it terminates.” –Gilles

E) OK so after a few annoying hours of failed transfers due to my own ignorance. I finally had a legit copy of my VMDK in thin format but sucking up a lot of space (See pic above) *Note I technically did the DD trick after making a backup, but the size shown on the datastore would still be teh same* the final part… hole punching: So quick re-cap, we deleted unused data, then zeroed out the unused blocks, we see a low size in the guest system (E.G. df -h), but we still see a large size used by the thin provisioned disc. K let’s hole punch
“vmkfstools -K /vmfs/volume/Datastore/VM/VM.vmdk”

Host Status During Hole Punch

CPU ESXi Host

CPU Storage Unit (FreeNAS)

Disk Usage Host

Disk Usage Storage Unit (FreeNAS)

Datastore Latency Host

Network Usage Host

As you can tell from the source images we can tell a couple things:

1)  HolePunching a VMDK does a high read I/O on the datastore

2) If the Datastore is iSCSI based it’ll saturate the iSCSI NIC (this can cause a performance degradation for other VMs utilizing the same Datastore

3) The latency increases due to the high Read I/O on the Datastore, again this directly affects performance of VMs running on the same Datastore

Once done the VMDK looked liked this:

Results / Summary

*NOTE* You can use the following command to convert a Thick Disk to thin manually, if you wish to Hole Punch a VMDK without the script.

vmkfstools -i Source-Thick.vmdk -d thin Destination-thin.vmdk

So why did I go through all this pain? Well I didn’t like the fact I was backing up useless data, weather it be pointless old images, or zeros. In the end the same VM backup went from taking almost an hour down to 5 minutes!

Free Hypervisor Backup
Part 2 – The VMware Screw

Veeam

Run Veeam by clicking the icon on the desktop or in the start menu, for Veeam Backup and Replication.

First Run

At first you will get this:

click apply.

Click Veeam, Zip, haha I expected this.. 😛

Click ok, and the add host wizard pops up.

Infrastructure Wizard

In my case I’m using ESXi.

Credentials

In the next section you will need to specify the credentials, you could specify the root account, however in my case even with one host, and only me, I decided to create a Veeam account on my ESXi host to use for this case. On 5.5 using the phat client it is really easy and intuitive, highlight the host, click the  local User and Groups tab, right click the open space, select new user, then click the permissions tab, click add user, select the newly created user, select the admin role. Done! Click here for 6.5/6.7 or the Web UI, not as intuitive. Click the add button, and add the account details that you specified when you created them on the hosts.

Then click OK, then next.

You will get this alert if you use self-signed certificates, even though I did write a blog post on setting up my own PKI, I did not use it in the case, as my Veeam server and ESXi host are not part of my AD domain, this also does simplify some aspects of the installation/deployment. Click Connect.

Click Finish, congrats you’ve added your free ESXi host. 😀

The dis-appointment

Next! Storage, Veeam needs to know where to save your data. Alright, seems there was no requirement here besides having local storage or a USB drive already attached, or in my case I used an SMB share. However I was very soon disappointed to see this error…

So…. so much for this being a free option, which I don’t think is fair, anyway. As usual its not even Veeam fault, this is cause VMware doesn’t allow the APIs for this, check this Veeam blog post out for more details.

If you use VMware a lot you you might have come across a blog site called virtuallyghetto run by William, this guy is great and my colleague just happened to find a script that was written by him to use the VMware CLI directly to create snapshots of VMs and copy their delta files to another disk, completely free.

In Part 3 I hope to install and try out this script, see how it handles my needs. Stay tuned!

Free Hypervisor Backup
Part 1 – Installing Veeam Backup

Intro

A little while back I had blogged about how you can get ESXi for free (you can also choose to use Hyper-V free with any version of Windows Server 2016/10, or using the stand alone core image).

However now that I have a couple nice hypervisor test beds, (I use FreeNAS for my storage needs, I hope to write a couple FreeNAS posts soon) how do we go about making backups, now we could manually backup the VM files manually, but that takes a lot of work, and I’d generally don’t like dealing with the file directly as soon as snapshots get involved, then I prefer to stick with the providers APIs. As you can guess I don’t have time to learn ever providers huge list of APIs, let alone the time to build any type of application for it (be it direct .NET, ASP.NET (w/ whatever front end (bootstrap/angular/etc)), JAVA (shutters), and whatever… so I could go on here but I’ll stop.

I’m personally not going to test a whole bunch of different solutions, but instead pull a bit of a fan boy and cover just Veeam. I came from using Backup Exec (which is now the hot potato of Backup Software, since it almost destroyed Symantec)… anyway, to using Veeam, and it was a breath of fresh air, not only do they have amazing support staff you know what they are doing (usually if you get in the higher tiers), but they also have a great form site with a good following and replies by the developers themselves. You also don’t need to sign up to read them if you need to find a solution to a problem in a pinch, they don’t mind airing out any dirty laundry cause more often then not it’s not directly their fault but the APIs they rely on. Anyway moving on.

Getting the Installation Media

To start go here to grab Veeam Free Backup. This requires a login, I can only assume to avoid Captcha, or other mechanism to prevent DDOS or annoyances, as well as information gathering. Feel free to use fake information for this.

Now Veeam can only be installed on Windows, see here for all the detailed specs.

I’ll choose Windows Server 2016 Datacenter as I have it available with my MSDN for all my educational needs. 😀

So at this point we have:

  1. A supported OS installed physical or virtual (i prefer virtual specially for labs)
  2. A Copy of the latest version of Veeam free
  3. A hypervisor (Hyper-V or ESXi) with VMs

*If you are looking to backup physical machines liek desktops and laptops look at Veeam’s agent options, Veeam Windows agent and Linux agent allow to backup physical machines.

Running the Installation Media

After updates it’s finally time to mount that ISO! In my case I had downloaded it on my workstation machine running Windows with the vSphere phat client, so I mounted it via the vSphere option to mount a local ISO to the VM. After mounting, and double clicking the installation executable, you are presented with this:

The EULA

Ooo, ahhhhh, click install…. and accept the EULA

Licensing (Free)

You will be present with this license part of the wizard, but as the text at the bottom indicates, click next without this to use free mode… wow how intuitive, no radio buttons, or check boxes… just simple intuitive wizard design…. would you just look at that… a thing of beauty. Click Next.

I was good with an all-in-one so I left the defaults, click next,

Dependencies

What is this? A clear, concise dependency check! And here I thought I could trick them by not installing things and see how it go, they seem to have done a good job covering their bases… and what is this?! and install button… you mean… I don’t have a vague link to a KB with some random technical blabber that links me to an executable to install before having to re run the wizard…. well lets see if it even works… Click Install… (Assuming internet connection; which this server does have, as how I got it updated)

Kool…

What is this?! no way…. it installed everything for me… and I didn’t have to reboot or re-run the wizard. Get out of town!; and click next.

Install location and verification

Again I’m OK with the defaults, click Install.

Let it install (it will use MS SQL Express (which is free up to 10 GB DB’s).

There’s a saying that goes “waiting is the hardest part”, thankfully with Veeam, this seems to be the case. Be patient while the installation completes, you’ll be glad you did. 🙂

Alright finally…

Click Finish, Now that was easy.

Click Restart.

Summary

That’s it! That’s all there is to it, the smoothest installation I’ve ever done, so smooth it doesn’t actually warrant it’s own blog post. But what the heck…

In Part 2 I’ll cover some basic configurations, and backup our first VM!

Creating and Managing Local Users ESXi 6.5/6.7

The Story

I recently started playing around with the later ESXi hypervisor (OK I’ve tried the Web UI before, and simply stayed away). Now it has been far more polished with the release of 6.7. I have been enjoying  the experience a far amount. However, then I needed to create another account on my free host (since I do not have vCenter to play around with in my home lab). While most things a seemed pretty intuitive at first..

Creating a User

Host -> Manage -> Security & Users -> Add a User (Specify Username and Password)

If you actually tried to login at this point I’d laugh a bit, but it could happen, you just created a user account, right? Well first thing you should have noticed is that there was no options to define what permissions this newly created user should have, read-only?, administrator?, etc.

So you click on Roles, there are all the nice pre-created roles… mhmm nice… alright… so… how do I map a user to a role?

You can look all under Security and users (where it should be), heck you can even look all under all the Manage options… you won’t find it there either… I had to find this out by googling… and if I have to google it… it’s not intuitive…

Assigning the Roles

So click on the main host icon in the left nav area, then when the right pane has loaded, select the Action menu, you should see it on the list of options above the host, right next to refresh.

Then select “Permissions”.

When the Host’s permissions modal box appears, click add user.

Marvel at how you can now assign users to roles, from here instead of the logical place where you easily found creating the user. Even if they wanted to keep the actions menu, and the modal box, just create a dang link under Security and Users… Arrrrggg.

Get your Free ESXi here!

Get ’em while they’re hot. Fresh from the bunnums of the internet!

Now I love my ESXi, and I recently converted my old gaming rig into a hypervisor with non other than my favorite beast ESXi! I first played with 6.5, and don’t get me wrong the fact it was a direct login to the host right from a fresh install is such a thing of beauty. With a plugin available for a smoother console experience from the web driven one.  While the HTML5 based web interface is very slick, the console isn’t exactly 100% real time. With the plugins it’s a nice way around that, however the host management tasks are all locked down to the hosts HTML 5 web interface. So long goes any chance of using the old phat (.Net based) client. I have to say thats sad cause I LOVE the phat client, it is by far the smoothest of all management interfaces, in my experience.

Anyway, logging into my personal host… I see this

This of course doesn’t surprise me. However believe it or not you can continue to run ESXi completely free. It’s generally enough for most peoples needs, there however some limitations.

  • No support
  • Free ESXi cannot be added to a vCenter Server
  • 2 physical CPUs
  • Unlimited cores per CPU
  • Unlimited physical Memory
  • max. 8 vCPU per VM

I won’t go over the details too much but the basics steps are as follows:

  1. Sign up to VMware.com
  2. Goto the vSphere Download Center 6.5 or 5.5.
  3. Register, Download your ISO and grab your Key.

Free ESXi 6.5 – How to Download and get License Keys

 

USB 3.0 Support on Windows 7 Guest VM

In Short, it’s not supported. If you’re running Workstation 9 or above, there’s this trick.

Now this guy goes into the real nitty gritty, and I love that! I however was working with ESXi 5.5 u3b. Now VMware did the same thing with the ESXi hypervisor and introduced USB 3.0 support via the xHCI controller. However the exact same limitation apply.

1) Drivers of USB 3.0 Host Controller are not provided by VMware Tools.

2) VMware USB 3.0 Host Controller will work only if your Virtual Machine OS has Native USB 3.o Support. Examples of such OS are – Windows 8, Windows Server 2012 and Linux Kernel 2.6.31 and above.

He goes on to say he’s screwed, but I’ve found the older EHCI +UHCI controller works for USB 1.1 and 2 devices I haven’t fully tested all case scenarios however. .For a Windows Server 2016 VM, on a HP Gen9  server with ESXi 5.5. My findings were as follows:

  1. Installed xHCI usb controller, via VM settings.
  2. Guest OS picked up hardware change and installed driver without issue.
  3. Plugged in USB 2.0 device, showed up in Host, as USB device became available to add to VM via VM settings, so added device.
  4. Guest OS didn’t see the USB device connected.
  5. Removed device via VM settings, then disconnected from host.
  6. Connected USB 3.0 Stick into host, added to VM via VM settings.
  7. Device was seen on Guest VM, and performance was equal to that of the sticks specs. (18~20  MB/s write, 100+MB/s Read)

I wasn’t sure why the USB 2.0 Device didn’t show up, so I simply removed the xHCI USB controller, and instead installed the EHCI +UHCI. Re-Connected the USB 2.0 devices and added it to the VM, this time the device did show up. I can’t remember the exact performance counters. I’ll update this post when I do some better analysis. My plan is to script some I/O tests using diskspd and PowerShell. Stay tuned. 😀

I’m also going to see if I can connect the same USB device via hardware pass-through instead of utilizing the USB controllers and Devices VM settings options.  I’ve manly done this with RDM’s and storage controllers with storage type VM’s (FreeNas mostly).

As for the main point of this post… I figured the main link I posted and this one here as well form the VMware forms that I’d be able to get a way to make the xCHI controller work on the Windows 7 VM guest.  The answer is basically grab the Intel xCHI drivers for Windows 7/2008R2 from Intel and install it manually, not via the setup.exe.

To my dismay I couldn’t get it to work, the wizard simply couldn’t locate the device (since the hardware IDs didn’t match) and installing the otherwise the device wouldn’t start.

I even decided to try and use double driver (extracts drivers) against a newer guest OS. This also failed. I simply couldn’t get it to work.

Lame mans V2V

This one will be short n sweet.
V2V a VMDK to a VHDX

Get this.

DO this:

Import-Module ‘C:\Program Files\Microsoft Virtual Machine Converter\MvmcCmdlet.psd1’

ConvertTo-MvmcVirtualHardDisk -SourceLiteralPath (Drive):\VM-disk1.vmdk -VhdType DynamicHardDisk -VhdFormat vhdx -destination (Drive):\vm-disk1

ESXi 6.0 on Windows 10 Hyper-V

You might ask why, I recently completed the VMware training module for installation and management of VMware vSphere components, to start to play around I don’t exactly have a bunch of hardware kicking around. I do however, have my awesome gamming Rig which is massively over powered in terms of CPU, Memory however… err not so much, disk I/O… also meh, these will need to be expanded on, but I do at least have Windows 10 running on SSD and a 3 TB spindle disc for more regular storage needs, but everyone knows a 7200 RPM disk provides mediocre performance.
Anyway I’m choosing Hyper-v Since I already have windows, and it comes free with windows, there are other options such as Oracles Virtual Box, and VMware player (can only run one VM at a time though for the free version :S)

Besides that here’s the steps so far.

1) Activate your Windows 10 Pro (1607), as mentioned installed mine on a 120 GB SSD.

2) Ensure VT-x and VT-e and probably VT-d is enabled, and that you have a motherboard and CPU capable of doing virtualization.

3) make sure all your hardware drivers are up-to-date.

4) Install Hyper-v.

5) Configure server settings such as HDD location, CPU allocation, and networks. In my case, I want my ESXi hosts to be isolated from the internet, so I pick internal.

6) Grab the ESXi ISO installation media from VMware (login and subscription required).

7) Create VM, I had to pick Gen 1 with BIOS, Gen 2 with EUFI didn’t boot the ISO for me.

8) I noticed at first attempt at the ESXi VM, it was sitting at loading kernel for an awfully long time, sure enough a simple Google search and discovered this gem.

When ESXi installer runs hit tab; add ignoreHeadless=TRUE

9) Before I could go any further I came across the dreaded, there are no network adapters available. You can Google this, but you will probably get blog posts about people attempting to load the nested VM with only 2 GB of ram when ESXi minimum requires 4 GB of RAM, so you have to be very specific in your search. In this case it’s amazing the power of the open community these days:
Turns out (as usual) it’s a driver related issue (don’t worry I’ll talk about this a couple times throughout this guide).
Lucky enough some lad was genius enough to figure out a solution, not only that but also provide the direct VIB to inject into the ISO file.
I followed the instructions, discovering that the latest supported release was for 5.x including 5.5 for ESXi installation customizer.
A double whammy it didn’t run on my Windows 10 x64 box… dismay not, we’re playing with VMs here.
I quickly created another VM and install my old Windows XP ISO with custom Dark Vista theme imbedded.
The great part was getting the files into the VM was a breeze. Simply shutdown the VM, navigate to the VMs HDD folder, Right click the VHDX file, and select the mount context menu.
This mounts the system and C:\ as separate disks on my windows host, copied the files in, booted the VM, and followed the instructions using the provided VIB and the ESXi 6.0 installer from VMware (login required). Bam sure enough I got a new custom ESXi 6.0 installer ISO file. Moved it out in the same fashion. Mounted it as the ESXi VM’s disc, and booted it up!
Finally the installation moves on! (Make sure you choose a “Legacy Network adapter”)
*Note I do not discuss storage choice when setting up this test host, I simply chose to create a VHDX file of 1 TB for the Nested VMs and for ESXi to be installed on.

10) Once the installation completes, and reboots make sure to hit SHIFT + O and add “ignoreHeadless=TRUE”. Let ESXi boot in DCUI

11) At DCUI, navigate to “Troubleshooting Options”, Then “Enable Shell”

12) Press ALT + F1 (Not F2 as the source states, F2 is the DCUI, F1 is the console). Then Login with root.

13) Type in this command and you won’t have type the headless part of the boot.
“esxcfg-advcfg –set-kernel “TRUE” ignoreHeadless”
(Copy command then select the Menu item “Clipboard”, then “Type Clipboard Text”) (classic Ctrl + V works too)

14) I was finally able to manage set an IP address for managing the host, the virtualized ESXi host hahaha. Sadly the vSphere client failed to connect on my XP VM.
So I setup a Windows 7 x64 bit VM instead. I set this up on Hyper-V on my Windows ten machine, alongside my ESXi hosts to mimic having a laptop running Windows 7.
The vSphere phat client can be downloaded from VMware (login required). Creating my first test VM on my nested ESXi host seemed to have an issue, reading further in the communities shows others with the exact same issue.

Turns out one can simply add a line to the VM’s VMX file “vmx.allowNested = TRUE”. This can be done via SSH (if enabled) or direct console (ALT+F1) using vi.

15) Another thing I noticed was when I was using the Hyper-V Manager’s console to manage my Windows VM running vSphere, and then having it open up vSphere’s console that the Hyper-v console would hang.
My only option at this point was to change my Windows 7 mgmt VM’s network setup. Instead of it only being in the locked down management network, I added another NIC to the VM after creating an external vSwitch in Hyper-V.
Since I have a DHCP server in my local LAN, having the Windows 7 NIC setup to DHCP provided it from my DHCP pool. Using ipconfig (in VM) or checking my DHCP server’s pool I was able to find the IP to remote into.
This of course required setting up remote desktop permissions on the Windows 7 VM. This also allowed me to work in full screen mode, and didn’t crash when opening up vSphere consoles, including of course copy and paste abilities. :D.

16) Next sort of problem was kind of expected. No x64 VM’s in my Nested enviro. There’s topics on this. So I decided to grab the latest 32 bit version of windows that’s available… you guessed it; Server 2008 (Not R2).
Grabbing a couple different versions available from MSDN, gave me a tad bit of issues. First off, don’t use the Checked/Debug, I played with the standard and the SP2 versions. I found the issue was it was hanging at completing installation.
Checking the VM stats via vSphere phat client VM’s performance tab, showed MAX CPU (not always a sign of being hung as it could still just be processing, but definitely a sign on the less), then the big give away, Disk I/O and consumed memory.
Disk I/O was none, and the consumed memory was on a steady decline till it plateaued neared nothing, all signed of stuck or looped process. Since I felt like giving it a little benefit of the doubt, and I had two virtual ESXi hosts to play with,
I decided to bump the CPU on one from 2 to 4. This allowed me to create a VM with 4 virtual CPUs instead of 2. Not sure why this would make a diff, and not sure if it exactly was. So I mounted the same 2008 with SP2 ISO and load the full desktop standard.
This time it finally got into the desktop… guess I’ll try the Standard core now on my other host after upping the CPU as well… let’s see. Yay Server Core installed using the standard 2008 32bit ISO with 4 core CPU.

17) Next issue I came across was not being able to have the VM’s inside the nested ESXi servers communicate with any other device in the same flat layer 2 network. I was sure I had configured everything correctly.
If one Googles this they will find lots n lots of articles on it stating the importance of promiscuous mode. I was up super late trying to figure out this problem and was starting to get a bit crazy. Setting all forms of the settings I could possibly find.
Including attempting to set mirror ports on the ESXi’s VM NICs on Hyper-v hahaha. AS I mentioned you’ll find many references to it, but googling promiscuous mode hyper-v and you discover most people stating to add a line to the VM’s XML config file.
Well it probably won’t take you long before you discover you VM config location doesn’t contain XML files but rather vmcx files. Yeeeeapppp, good luck opening them up… they are now binary…. Wooooo! No admins playing around in here! Take that you tweakers!
This was a change in Hyper-V starting with Server 2016 / Windows 10. I spent a couple hours tumbling down this rabbit hole. To help other I’ll make this part as clear as mud!
IN Hyper-V, ON THE ESXi VMs NETWORK SETTING THAT IS THE LEGACY NETOWRK ADAPTER (the one used as the “physical” adapter in the ESXi vSwitch) EXPAND THE SETTING AND UNDER ADVANCED FEATURES SELECT “Enable MAC address spoofing”.
That’s it! That is Server 2016/Windows 10 Hyper-V’s work around for nested hypervisors. Although as usual support people on TechNet instead of giving an answer or a technical work around would rather dust their hands the classical “not supported” instead of “It’s possible, here’s how, but if something doesn’t work with these settings that’s all we can help with” which I feel would have been a far better response. Maybe these support people just aren’t aware, who knows here’s where I found my answer.

18) So now that I got my hosted ESXi servers up and running and communicating the next step is vCenter. vCenter will setup its own SSO domain, we can add a MS AD domain later and change the default SSO domain to be our active direct domain. However the default SSO domain created at vCenter deployment is the local configuration domain for all vCenter services. Grab vCenter Appliance from VMware. You might be wondering what gives when you discover under the download list for vCenter that there’s an ISO and an IMG file, but no OVA/OVF. This is cause in vSphere 6.0 the vCenter appliance is deployed via a client system using some weird system to communicate to the host to deploy via some web stuff… even I don’t know the exact details of what’s up, either way, if you attempt to create an VM and mount the ISO, you’ll find it’s not bootable. So mount it to the management VM. In my case my Windows 7 VM with vSphere installed. Since Windows 7 doesn’t have native ISO mounting features I had to install virtual clone drive. Then mount the ISO and navigate inside.

Oddly enough it almost seems as if you need a windows system to deploy a Linux appliance. Under the VCSA folder you should find an integration plugin exe installable… run the installable exe file. There seems to be a set that states installing certificates and service, this might be the start of the certs of the built in SSO domain. Not sure though. Once it’s done it sort of leaves you in the dark… as every just closes and there’s not complete window in the wizard…
Guess I’ll just run vsa-setup.html now… Since I have a native version of Windows 7 setup… looks like I’ll need IE 10/11 as the default IE 8 won’t suffice. Lucky for me the Windows 7 machine still had access to the internet, so I Googled the IE 11 installer and ran it, this may be a pre-requirement for the normal installer. As it seems to download and install required updates. You may need to find an offline installer file for IE 11 if you are in a test enviro where you Windows machine doesn’t have access to the internet.

Click Allow. Another pop-up will appear, click Allow.

Now we can finally click install: S

Accept the user agreement, then enter one of the hosts IP address. Member I installed and run this one the Windows 7 machine that can already access the hosts via SSH or the vSphere phat client. So I will enter the IP address here as I haven’t setup DNS at all yet in my environment, and one wouldn’t technically yet if the plan was to have nested DNS servers (The DNS the hosts point to are VMs it hosts).

I made a wrong IP entry, it alerted me as it couldn’t connect to the host, then corrected the IP, and got a cert warning.

Setup the Virtual Appliance OS’s Root password (I believe it uses openSUSE, so this would set the underlying openSUSE root password).

Now under the deployment type if one were never to connect any other vCenter server into the SSO domain for enhanced link mode, you can pick embedded, however for scalability, and the fact I need to setup a Windows Server vCenter to run Update Manger, I’ll create an external Platform Services Controller (PSE). This will require me to run through the wizard separately to actually deploy the vCenter server, in this case I’m actually just setting up the PSE Virtual Appliance (VA). Hence the options all make sense.

As I mentioned this will create the SSO domain for all vCenter services, do not make this the same as your AD domain, this will cause confusion between domains when you add and set your AD domain as the primary SSO identity source. I stick with VMware default vsphere.local, then add a site name (generally this would be some sort of regional reference). Also set the SSO admin password.

It complained about DNS a requirement and a System name, I’m assuming this is hostname, even though it requested it be either FQDN or IP like it was required for some sort of looked, I specified simply the hostname, and a DNS server IP that is not yet even setup for DNS (That which will be my PDC in my test AD setup) This allowed me to continue the setup. I’m thinking this might be what it enters as a common name or SAN for its cert. is my guess.
I went back and changed it to an IP address as I figured my first couple attempts to access it will be through its IP address and I didn’t want to deal with cert warnings. It did however warn me that FQDN is more preferred and this makes sense when a proper DNS system is already implemented.
Hahaha Sure enough had to go through all that IE 11 setup, and plugin installation for it simply to deploy an OVF file hahaha.

Cool, I guess it’s more that based on how you want to set vCenter up with the new PSE instead of having a bunch of Documentation to read through (While this is technically always best to do anyway) it sort of automates the templates to deploy and how to configure them using a questionnaire type setup. I believe in 6.5 this is maybe easier with some sort of HTML 5 based deployment system. Not sure though.
So I hit a couple snags on deployment. First off I thought I was stuck on not being able to do nest x64 virtualization on my nested ESXi hosts. Until the great lads in Freenodes #vmware told me to enable virtualization extensions to the ESXi VM.
“17:18 < genec> Zew: then did you forget to pass VT-x to the ESXi VM?” – Oh Neat! Thanks genec.
Since I was running all my stuff on Hyper-V I had to Google this. Did take long till I found my answer.
Set-VMProcessor -VMName -ExposeVirtualizationExtensions $true
The VM name being my ESXi hypervisor.

This however I only discovered after I enabled the whole vmx.allowNested = TRUE bit on the deployed VM after I saw that it failed with that usual error message. Luckily enough a bit of googling again and I was able to find my answer.
“You can add vmx.allowNested = “TRUE” to /etc/vmware/config in the ESXi VM to avoid having to put it in every nested VM’s configuration file.” –Thanks Matt
I’ll delete the existing vApp and try my deployment again.
Once I managed to mount the VSCA ISO and install the client plugin, and attempt to deploy the PSC/VCSA I got hung up. It appears all my x64 VMs within my Nested ESXi hosts failed to properly boot. All the different VCSA\PSC versions all went into a boot loop. Windows 7 x64 gave a fault screen after loading the installer files and attempting to boot the setup.exe. Server 2016 just showed a black screen. Looking into this I discovered this guy’s blog… looks like I may have to resort to VMware Workstation Pro!
I’ll post this blog post for now as it has become rather long. I will post my success or failure in the upcoming weeks. Stay tuned!

Jan 2018 Update

I remember this being extremely painful, but was so way easier on Workstation Pro.

vCenter Network Partitioned

Have you ever experienced a Network Partitioned warning in vSphere 5? Hopefully not, but if you find yourself with this warning in vSphere. Don’t panic its not as bad as it could have been in 4.x.

This literally just means that the host can not communicate with any of the designated VMK’s checked off for management traffic. In my case it happened after making network changes to my infrastructure. In this case I still had bonded links at my switches, but somehow the VMK load balancing algo had switched to “route based on originating port ID”, this load balance algo doesn’t work with bonded NICs, and needs to be “route based on IP hash”. My end goal was to get off bonded links for my host and use the default load balance algo that VMware uses, as this can be down with non stacked switches and can be done with minimal switching knowledge (in case others need to manage the system in the future).

It took me a little bit to catch the issue, cause the symptoms were that each host could ping any device in their respected management subnets but NOT the other host, flat /24 subnet too, really had me baffled. As I couldn’t vMotion in this state either, but lucky the VMs on each host remained active (as they have separate communication VMPGs on dedicated physical connections).

Once I caught the error, I was able to verify vMotion worked again. That’s all there is to it!

To Paraphrase to solution:

1) Check which VMKs have management checked off.
2) Check those vSwitches physical connections.
3) If multiple ports check configs on physical switch and load balance algo.
4) Google any errors along the way.
5) Check host to host communication by consoling into host and using vmkping.

Jan 2018 Update

I remember this…

Changing Network Location to Domain

Have you ever restored a VM? Have you done your DR testing by actually doing a full recovery with AD? Did you find you had a couple odd things occur after restore, such as not being able to RDP into your recovered server? Chances are your network profile has changed to public, instead of Domain. This in turn causes certain firewall rules to trigger.

I remember coming across this issue multiple times, especially when people usually want private instead of public and vice versa. So chances are you’ve come across this, telling you to use PowerShell cmdlet to change its setting, which to my guess makes a registry change. The other option they specified was to use the GUI.

Well I find changing local security policies and all that other stuff rather annoying. Soo after a bit more googling I found a really nice answer, which worked and was very simple to implement. Very nicely written and easy to follow by a Evan A Barr. You can view his site here.

To Paraphrase to solution Using Network Connection Properties:

0) by adding a DNS suffix so that NLA can properly locate the domain controller.
1) Go to Network Connections.
2) Go to the properties of of the network adapter in the wrong location.
3) Go to the properties for IPv4.
4) Click the "Advanced..." button.
5) Select the DNS tab.
6) Enter your domain name into the text box for "DNS suffix for this connection:".
7) Disable and then enable the connection to get NLA to re-identify the location.