A general system error occurred: Launch failure

Failure to Launch

Sound the Alarm! Sound the Alarm!

*ARRRRRRREEEEEEEEERRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
ARRRRRRREEEEEEEEERRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR

and one heck of a day sweating bullets, am-I-right?!

Also sorry about the lack of updates, with spring and work, it’s been hard to find time to blog. Deepest apologies.

The Story

Well it’s Tuesday so it’s clearly a day for a story, and boy do I have a story for you. I was going about my usual way of working.. endlessly to meet my goal… that never able to acquire goal of perfection…. anyway… I had completed a couple major tasks of going from vCenter 5.5 to 6.5u2, and while I have yet to blog about that fun bag (cause I had used my own PKI and certificates so the migration scripts VMware provides pooped the bed till I replaced them with self signed) but I digress this has nothing to do with that… well sort of.

Where was I… oh right… I was vMotioning a couple VMs to some new 6.5U2 hosts I had setup on new hardware (yeah buddy this was a whole new world (Why did I just say that in the voice of Princess Jasmine?)) but alas the final VM was not meant to move for it had stalled @ 20 percent (I had this happen once before during my deployment and discovered some interesting things about multi-NIC vmotions) I probably should blog about that toooooo but alas my time is running short. (I still have many other things to tackle and blog about). and then……..

“A general system error occurred: Launch failure”

ugh…. I could have jumped right to the logs but I jumped on google instead…

CloudSpark apparently recently came across this issue and posted on all the many places he apparently could think… reddit, and VMware forums 

the VMware post got a tad long, but it was more the post on reddit that got me wondering….

“Haven’t seen this personally, but there’s a few things via google on the “Failed to connect to peer process” error that suggest it could be due to running out of storage. Specifically in one scenario, the /tmp/vmware-root folder on the ESXi host fills up with logs.” – Astat1ne

When I SSH into the host and ran “df -h” I was surprised to be reported back with an error:

esxcli returned an error: 1

Something to that extent anyway. I ended up Moving all the VMs back off the host, and swapped out the SD card the ESXi software was installed on (I had a copy of the SD card with a clone of the ESXi software and host configure that was created right after the host was installed and configured).

Sure enough after reboot powering it back on, “df -h” returned clean. I was then able to vMotion all the VMs back onto the set of new hosts.

Def a generic error message I’ve never seen, and thinking about it now is rather comical. (I know when your in the middle of the problem it’s not so funny)…

BUT it sure is now! :D…. this is so going to come back to bite me….

(await updates here post April 30th)

VMware ESXi 5.5
D-Link DGE-530T RevC

The Story

Are you guys ready for a story? This one is actually not so bad. A couple days ago I post on Facebook if anyone happened to have a spare PCI/PCIe Network Interface Card (NIC), since it was going to be used for interest access I was ok with it being 100, but was aiming for 1000 (now that Shaw provide over 300mbps internet, clearly 100 doesn’t cut it).

After a day of no luck, and a bunch of funny remarks (as almost none of my friends had any idea of what I was talking about), I decided to take another look through my old computer hardware to see what I could scrounge up…

PCI NIC Found!

well, well, not even dusty, a PCI NIC, exactly what I needed in my hypervisor to play with OPNsense. I originally was going to try layer 2 trunking via VLANs, however the main vSwitch already had VMkernel Nics bound to the physical adapter @ layer 3, and the same interface on my firewall (Palo Alto) wouldn’t allow me to create a layer 2 sub-interface is the main interface was already bound to layer 3. Since I wanted my OPNsense VM to get an actual public IP address, this required my device to get a connection from my VM, directly to my modem at layer 2… yeah another NIC. So here we are, and it didn’t take long for me to shut down my VMs and install the card, and boot my hypervisor back up (I hope to one day have multiple hypervisor to not have to shut down my VMs, but even then, if you don’t pay chances are you won’t get access to the APIs that migrate the memory states of the VMs for you, so it’s a hassle either way…. anyway back to the story.

PCI NIC Found … NOT

Oh Borat, who brought you in?!?! So as you may have guessed I went to add a new vSwitch for my new VM to get it’s direct Public IP, and to my dismay there was no physical NIC to pick… what the….

So to Google! and hopefully either VMware support, or usually always better personal blogs! We all loves these right… ahem… anyway…

You can probably guess where the official answer went, but I’ll enlighten you as I did follow along for … pain? OK I don’t know why I did, I was really hopeful it wasn’t going to be the answer I knew it was going to be….

Hey! some of the command they provided helped, or did they? All this was, was some BS data chasing to tell you, IT’s Not supported, SOWWY!

Clearly, there must be some answers by the community forums right??

Community’s great! VMwares…. :S

So what do we get… One…  unanswered and crying about a badly referenced link to source two... also unanswered crying about the same stuff we already know…. it’s officially not supported. Well I’m running ESXi 5.5 Free and using GhettoVCB’s scripts, also unsupported, so not really an issue… the issue is teh lack of help right now.

But bring me down, I don’t thikn so, the internet has many sites, and many people sharing their knowledge, how?!?! BLOGS! Ahem…

Blogs to the Rescue!

Yes believe it or not it is the power of the real untethered, unfiltered beauty that is blogging that we actually get some meat and potatoes. My first source showed signs of light! One problem, it’s literally 9 years old and using ESXi 4. OK well it also wanted a fair amount of direct file placing and special manipulation. Most of this works fairly differently in ESXi 5.x, and vibs or precompiled binaries that work with esxcli are the more preferred method. I avoid saying supported here, cause I use these methods to install unsupported packages :D.

Alright, so now what, well the Holy Grail!  This King managed to not only blog about getting this working but shared the drivers/vibs packages required to get it to work too! Epic! Let’s get this dang NIC working…

1) Grab the VIB files

2) Change your support level on ESXi5+:

~ # esxcli software acceptance set –level=CommunitySupported
Host acceptance level changed to ‘CommunitySupported’.

3) Install the driver with: “esxcli software vib install -v /DLink-528T-1.x86_64.vib“

4) Reboot

Sounds simple enough lets give it a shot… and I hit some errors, classic…

I won’t show the erros just yet as I have it one long snippet, but basically I had a bit of problems cause of the GhettoVCB scripts I had pushed on to my host, but the error results weren’t exactly clear… I attempted a couple things first, like copying the VIB to the path it kept complaining about and specifying the fully qualified path to the VIB.. nothing till I stumbled across this...

esxcli software vib install -v /full/path/to/.vib -f

which finally gave me a driver install successful!

Alright, and after reboot…..


OMG! No way, there it is with the proper name and everything. Considering the blog post I followed was for a different NIC model I wasn’t sure if it would work, but there it is… so lets not get to ahead of ourselfs and see if it comes up and is able to transmit packets…

I was having some issues initially so I decided to give my lil netbook a simple /24 IP and give my OPNsense a simple /24 IP just to validate the card wasn’t the issue, or the drivers I just installed.

Plug them together, lights come up, that’s good… checking ESXI vSphere…

That’s good, and finally can we transmit?!?!

Hey!!!! we have communication! Now it’ll be figuring out getting the Public IP configured properly. But we’ll save that for another post. 😀 Cheers!

How to Shrink a VMDK

Hey all,

Not often you have to shrink a VMDK file, expanding one is super easy, even on a live Virtual Machine. Shrinking one however, isn’t as straight forward.

This guy does a decent job giving a step by step tutorial, but you can soon realize you can do it even faster, and without cloning…

1)  Use his math to get the disk size you need to edit inside the vmdk:

The number highlighted above, under the heading #Extent description, after the letters RW, defines the size of the VMware virtual disk (VMDK).

this number – 83886080, and it’s calculated as follows:

40 GB = 40 * 1024 * 1024 * 1024 / 512 = 83886080

2) Only shrink VMDKs in which you know the end of the disk contains allocated blocks, do this in a test only, make sure you have backups.

Now instead of cloning, simply remove the disk from the vm, and re-attach it. watch it’s reattached size be smaller, and it matches, much like the source guys post.

Free Hypervisor Backup
Before Part 3

The Story

I’m currently in the quest to fulfill my needs for a free VM backup solution, my hypervisor of choice right now is ESXi, while there are great alternative free hypervisors ( Citrix’s Xen, Microsoft’s Hyper-V, Linux’s KVM) I personally always felt comfortable with VMwares user interface (I’m talking the old Windows Phat Client).  I totally understood the need for a web based client, however I felt it a sham to drop the phat client as to provided multiple benefits that the Web UI just doesn’t. Although the latest 6.7 Web UI has been pretty decent, besides their placement of the Hostname location…. :@

Anyway… if you’ve been following my posts you’ll see I decided to give Veeam Free a shot. I love these guys, great software, so I figured best to get better antiquated with their software, to my dis-may their simply relied on VMwares APIs solely. Which meant no SSH backdoor tricks for use ESXi free users.

However as I also mentioned in my previous post, an awesome dude who runs virtuallyGhetto.com, William Lam; wrote a script to complete the task we wanted via the hosts local CLI, which can be connected to via SSH. The script is called GhettoVCB. I took a quick look at the source code and did find a couple instances of zombie code and other anomalies but for the most part looked decent enough to give it a shot. Now I will get to this stuff in my next post using the same example VM I will specify here where I came across a couple issues and interesting facts I discovered during this adventure.

My Discoveries

The first thing I noticed about the script was the lack of certain dependency checks, in this case there’s no actual source code dependencies that his script relies on, it’s all meant to run on ESXi, and sure enough there’s some line of code to check for that…

Nice, however there doesn’t seem to be a check to validate if the datastore specified in the very first variable has enough space to complete it’s task.

In this case the script would simply error out once the destination ran out of space stating the source VMDK was the issue, this lead me briefly down the wrong rabbit whole. After validating I had no issues with my source VMDK (booted the VM and checked all services and FileSystem integrity) I noticed a couple things.

1) Even though I specified Thin disc for my destination, which had enough space to store the VM data (60~ GBs), the thin disc was attempting to create the full provisioned size of the disc, cause…..

2) I forgot the source disc was set to Thick Provisioned Eagered Zero

So there was a couple things about this VM….

1)  It’s my ZoneMinder VM which holds my IP camera footage on motion detection

2) This data is mostly useless due to no events having taken place

Alright so now my goal was the following:

1)  Remove all the un-needed data

A) Open ZoneMinder Web UI
B) Click on Camera
C) Delete All Events
D) wait a while before all the MySQL queries to kick off to clear the data
E) Used “df -h” to watch usage drop

2) Once I had all the data removed, I had to re-claim the space. I decided to dig up my old blog post on the subject matter… Well that was a bit underwhelming and simply provided my links instead of any valid examples… So I hope to provide a bit better details here:

A) I’m running Linux not Windows so sdelete is out.

B) My first attempts I decided to follow this DD example “dd if=/dev/zero of=/zeroes && rm -f /zeroes” (DO NOT DO THIS, from my tests it caused mySQL service to not come up properlly). I then found people stating to use secure-delete or zerofree, however I had some issues with these against a live system and wanted a simple live system, general technique… if there so many references stating dd can do it… how.. then I found this
“dd if=/dev/zero of=/tmp/empty.dd bs=1048576; rm /tmp/empty.dd”
I’m assuming maybe cause I used /tmp instead of /… only diff I can think of.

C) At this point I made a backup using ghettoVCB which I had to use an alternate Datastore to save the VMDK that I would then finally hole punch, in this case the ghettoVCB script converts the VMDK from thick to Thin however will still be the size of the provisioned Disc.

D) Now this is where I started to get a bit… annoyed… However I did learn a few things… one is that & to bring a process to the background doesn’t disconnect that process from the current terminal session. So while I was SSH’d in and running the clone operation, it wasn’t fully completing when my SSH session timed out, even though I used & to run it in the background. Read this for more info but the gist take away is this: “nohup and disown both can be said to suppress SIGHUP, but in different ways. nohup makes the program ignore the signal initially (the program may change this). nohup also tries to arrange for the program not to have a controlling terminal, so that it won’t be sent SIGHUP by the kernel when the terminal is closed. disown is purely internal to the shell; it causes the shell not to send SIGHUP when it terminates.” –Gilles

E) OK so after a few annoying hours of failed transfers due to my own ignorance. I finally had a legit copy of my VMDK in thin format but sucking up a lot of space (See pic above) *Note I technically did the DD trick after making a backup, but the size shown on the datastore would still be teh same* the final part… hole punching: So quick re-cap, we deleted unused data, then zeroed out the unused blocks, we see a low size in the guest system (E.G. df -h), but we still see a large size used by the thin provisioned disc. K let’s hole punch
“vmkfstools -K /vmfs/volume/Datastore/VM/VM.vmdk”

Host Status During Hole Punch

CPU ESXi Host

CPU Storage Unit (FreeNAS)

Disk Usage Host

Disk Usage Storage Unit (FreeNAS)

Datastore Latency Host

Network Usage Host

As you can tell from the source images we can tell a couple things:

1)  HolePunching a VMDK does a high read I/O on the datastore

2) If the Datastore is iSCSI based it’ll saturate the iSCSI NIC (this can cause a performance degradation for other VMs utilizing the same Datastore

3) The latency increases due to the high Read I/O on the Datastore, again this directly affects performance of VMs running on the same Datastore

Once done the VMDK looked liked this:

Results / Summary

*NOTE* You can use the following command to convert a Thick Disk to thin manually, if you wish to Hole Punch a VMDK without the script.

vmkfstools -i Source-Thick.vmdk -d thin Destination-thin.vmdk

So why did I go through all this pain? Well I didn’t like the fact I was backing up useless data, weather it be pointless old images, or zeros. In the end the same VM backup went from taking almost an hour down to 5 minutes!

Free Hypervisor Backup
Part 2 – The VMware Screw

Veeam

Run Veeam by clicking the icon on the desktop or in the start menu, for Veeam Backup and Replication.

First Run

At first you will get this:

click apply.

Click Veeam, Zip, haha I expected this.. 😛

Click ok, and the add host wizard pops up.

Infrastructure Wizard

In my case I’m using ESXi.

Credentials

In the next section you will need to specify the credentials, you could specify the root account, however in my case even with one host, and only me, I decided to create a Veeam account on my ESXi host to use for this case. On 5.5 using the phat client it is really easy and intuitive, highlight the host, click the  local User and Groups tab, right click the open space, select new user, then click the permissions tab, click add user, select the newly created user, select the admin role. Done! Click here for 6.5/6.7 or the Web UI, not as intuitive. Click the add button, and add the account details that you specified when you created them on the hosts.

Then click OK, then next.

You will get this alert if you use self-signed certificates, even though I did write a blog post on setting up my own PKI, I did not use it in the case, as my Veeam server and ESXi host are not part of my AD domain, this also does simplify some aspects of the installation/deployment. Click Connect.

Click Finish, congrats you’ve added your free ESXi host. 😀

The dis-appointment

Next! Storage, Veeam needs to know where to save your data. Alright, seems there was no requirement here besides having local storage or a USB drive already attached, or in my case I used an SMB share. However I was very soon disappointed to see this error…

So…. so much for this being a free option, which I don’t think is fair, anyway. As usual its not even Veeam fault, this is cause VMware doesn’t allow the APIs for this, check this Veeam blog post out for more details.

If you use VMware a lot you you might have come across a blog site called virtuallyghetto run by William, this guy is great and my colleague just happened to find a script that was written by him to use the VMware CLI directly to create snapshots of VMs and copy their delta files to another disk, completely free.

In Part 3 I hope to install and try out this script, see how it handles my needs. Stay tuned!

Free Hypervisor Backup
Part 1 – Installing Veeam Backup

Intro

A little while back I had blogged about how you can get ESXi for free (you can also choose to use Hyper-V free with any version of Windows Server 2016/10, or using the stand alone core image).

However now that I have a couple nice hypervisor test beds, (I use FreeNAS for my storage needs, I hope to write a couple FreeNAS posts soon) how do we go about making backups, now we could manually backup the VM files manually, but that takes a lot of work, and I’d generally don’t like dealing with the file directly as soon as snapshots get involved, then I prefer to stick with the providers APIs. As you can guess I don’t have time to learn ever providers huge list of APIs, let alone the time to build any type of application for it (be it direct .NET, ASP.NET (w/ whatever front end (bootstrap/angular/etc)), JAVA (shutters), and whatever… so I could go on here but I’ll stop.

I’m personally not going to test a whole bunch of different solutions, but instead pull a bit of a fan boy and cover just Veeam. I came from using Backup Exec (which is now the hot potato of Backup Software, since it almost destroyed Symantec)… anyway, to using Veeam, and it was a breath of fresh air, not only do they have amazing support staff you know what they are doing (usually if you get in the higher tiers), but they also have a great form site with a good following and replies by the developers themselves. You also don’t need to sign up to read them if you need to find a solution to a problem in a pinch, they don’t mind airing out any dirty laundry cause more often then not it’s not directly their fault but the APIs they rely on. Anyway moving on.

Getting the Installation Media

To start go here to grab Veeam Free Backup. This requires a login, I can only assume to avoid Captcha, or other mechanism to prevent DDOS or annoyances, as well as information gathering. Feel free to use fake information for this.

Now Veeam can only be installed on Windows, see here for all the detailed specs.

I’ll choose Windows Server 2016 Datacenter as I have it available with my MSDN for all my educational needs. 😀

So at this point we have:

  1. A supported OS installed physical or virtual (i prefer virtual specially for labs)
  2. A Copy of the latest version of Veeam free
  3. A hypervisor (Hyper-V or ESXi) with VMs

*If you are looking to backup physical machines liek desktops and laptops look at Veeam’s agent options, Veeam Windows agent and Linux agent allow to backup physical machines.

Running the Installation Media

After updates it’s finally time to mount that ISO! In my case I had downloaded it on my workstation machine running Windows with the vSphere phat client, so I mounted it via the vSphere option to mount a local ISO to the VM. After mounting, and double clicking the installation executable, you are presented with this:

The EULA

Ooo, ahhhhh, click install…. and accept the EULA

Licensing (Free)

You will be present with this license part of the wizard, but as the text at the bottom indicates, click next without this to use free mode… wow how intuitive, no radio buttons, or check boxes… just simple intuitive wizard design…. would you just look at that… a thing of beauty. Click Next.

I was good with an all-in-one so I left the defaults, click next,

Dependencies

What is this? A clear, concise dependency check! And here I thought I could trick them by not installing things and see how it go, they seem to have done a good job covering their bases… and what is this?! and install button… you mean… I don’t have a vague link to a KB with some random technical blabber that links me to an executable to install before having to re run the wizard…. well lets see if it even works… Click Install… (Assuming internet connection; which this server does have, as how I got it updated)

Kool…

What is this?! no way…. it installed everything for me… and I didn’t have to reboot or re-run the wizard. Get out of town!; and click next.

Install location and verification

Again I’m OK with the defaults, click Install.

Let it install (it will use MS SQL Express (which is free up to 10 GB DB’s).

There’s a saying that goes “waiting is the hardest part”, thankfully with Veeam, this seems to be the case. Be patient while the installation completes, you’ll be glad you did. 🙂

Alright finally…

Click Finish, Now that was easy.

Click Restart.

Summary

That’s it! That’s all there is to it, the smoothest installation I’ve ever done, so smooth it doesn’t actually warrant it’s own blog post. But what the heck…

In Part 2 I’ll cover some basic configurations, and backup our first VM!

Creating and Managing Local Users ESXi 6.5/6.7

The Story

I recently started playing around with the later ESXi hypervisor (OK I’ve tried the Web UI before, and simply stayed away). Now it has been far more polished with the release of 6.7. I have been enjoying  the experience a far amount. However, then I needed to create another account on my free host (since I do not have vCenter to play around with in my home lab). While most things a seemed pretty intuitive at first..

Creating a User

Host -> Manage -> Security & Users -> Add a User (Specify Username and Password)

If you actually tried to login at this point I’d laugh a bit, but it could happen, you just created a user account, right? Well first thing you should have noticed is that there was no options to define what permissions this newly created user should have, read-only?, administrator?, etc.

So you click on Roles, there are all the nice pre-created roles… mhmm nice… alright… so… how do I map a user to a role?

You can look all under Security and users (where it should be), heck you can even look all under all the Manage options… you won’t find it there either… I had to find this out by googling… and if I have to google it… it’s not intuitive…

Assigning the Roles

So click on the main host icon in the left nav area, then when the right pane has loaded, select the Action menu, you should see it on the list of options above the host, right next to refresh.

Then select “Permissions”.

When the Host’s permissions modal box appears, click add user.

Marvel at how you can now assign users to roles, from here instead of the logical place where you easily found creating the user. Even if they wanted to keep the actions menu, and the modal box, just create a dang link under Security and Users… Arrrrggg.

Get your Free ESXi here!

Get ’em while they’re hot. Fresh from the bunnums of the internet!

Now I love my ESXi, and I recently converted my old gaming rig into a hypervisor with non other than my favorite beast ESXi! I first played with 6.5, and don’t get me wrong the fact it was a direct login to the host right from a fresh install is such a thing of beauty. With a plugin available for a smoother console experience from the web driven one.  While the HTML5 based web interface is very slick, the console isn’t exactly 100% real time. With the plugins it’s a nice way around that, however the host management tasks are all locked down to the hosts HTML 5 web interface. So long goes any chance of using the old phat (.Net based) client. I have to say thats sad cause I LOVE the phat client, it is by far the smoothest of all management interfaces, in my experience.

Anyway, logging into my personal host… I see this

This of course doesn’t surprise me. However believe it or not you can continue to run ESXi completely free. It’s generally enough for most peoples needs, there however some limitations.

  • No support
  • Free ESXi cannot be added to a vCenter Server
  • 2 physical CPUs
  • Unlimited cores per CPU
  • Unlimited physical Memory
  • max. 8 vCPU per VM

I won’t go over the details too much but the basics steps are as follows:

  1. Sign up to VMware.com
  2. Goto the vSphere Download Center 6.5 or 5.5.
  3. Register, Download your ISO and grab your Key.

Free ESXi 6.5 – How to Download and get License Keys

 

USB 3.0 Support on Windows 7 Guest VM

In Short, it’s not supported. If you’re running Workstation 9 or above, there’s this trick.

Now this guy goes into the real nitty gritty, and I love that! I however was working with ESXi 5.5 u3b. Now VMware did the same thing with the ESXi hypervisor and introduced USB 3.0 support via the xHCI controller. However the exact same limitation apply.

1) Drivers of USB 3.0 Host Controller are not provided by VMware Tools.

2) VMware USB 3.0 Host Controller will work only if your Virtual Machine OS has Native USB 3.o Support. Examples of such OS are – Windows 8, Windows Server 2012 and Linux Kernel 2.6.31 and above.

He goes on to say he’s screwed, but I’ve found the older EHCI +UHCI controller works for USB 1.1 and 2 devices I haven’t fully tested all case scenarios however. .For a Windows Server 2016 VM, on a HP Gen9  server with ESXi 5.5. My findings were as follows:

  1. Installed xHCI usb controller, via VM settings.
  2. Guest OS picked up hardware change and installed driver without issue.
  3. Plugged in USB 2.0 device, showed up in Host, as USB device became available to add to VM via VM settings, so added device.
  4. Guest OS didn’t see the USB device connected.
  5. Removed device via VM settings, then disconnected from host.
  6. Connected USB 3.0 Stick into host, added to VM via VM settings.
  7. Device was seen on Guest VM, and performance was equal to that of the sticks specs. (18~20  MB/s write, 100+MB/s Read)

I wasn’t sure why the USB 2.0 Device didn’t show up, so I simply removed the xHCI USB controller, and instead installed the EHCI +UHCI. Re-Connected the USB 2.0 devices and added it to the VM, this time the device did show up. I can’t remember the exact performance counters. I’ll update this post when I do some better analysis. My plan is to script some I/O tests using diskspd and PowerShell. Stay tuned. 😀

I’m also going to see if I can connect the same USB device via hardware pass-through instead of utilizing the USB controllers and Devices VM settings options.  I’ve manly done this with RDM’s and storage controllers with storage type VM’s (FreeNas mostly).

As for the main point of this post… I figured the main link I posted and this one here as well form the VMware forms that I’d be able to get a way to make the xCHI controller work on the Windows 7 VM guest.  The answer is basically grab the Intel xCHI drivers for Windows 7/2008R2 from Intel and install it manually, not via the setup.exe.

To my dismay I couldn’t get it to work, the wizard simply couldn’t locate the device (since the hardware IDs didn’t match) and installing the otherwise the device wouldn’t start.

I even decided to try and use double driver (extracts drivers) against a newer guest OS. This also failed. I simply couldn’t get it to work.