Free Hypervisor Backup
Part 2 – The VMware Screw

Veeam

Run Veeam by clicking the icon on the desktop or in the start menu, for Veeam Backup and Replication.

First Run

At first you will get this:

click apply.

Click Veeam, Zip, haha I expected this.. 😛

Click ok, and the add host wizard pops up.

Infrastructure Wizard

In my case I’m using ESXi.

Credentials

In the next section you will need to specify the credentials, you could specify the root account, however in my case even with one host, and only me, I decided to create a Veeam account on my ESXi host to use for this case. On 5.5 using the phat client it is really easy and intuitive, highlight the host, click the  local User and Groups tab, right click the open space, select new user, then click the permissions tab, click add user, select the newly created user, select the admin role. Done! Click here for 6.5/6.7 or the Web UI, not as intuitive. Click the add button, and add the account details that you specified when you created them on the hosts.

Then click OK, then next.

You will get this alert if you use self-signed certificates, even though I did write a blog post on setting up my own PKI, I did not use it in the case, as my Veeam server and ESXi host are not part of my AD domain, this also does simplify some aspects of the installation/deployment. Click Connect.

Click Finish, congrats you’ve added your free ESXi host. 😀

The dis-appointment

Next! Storage, Veeam needs to know where to save your data. Alright, seems there was no requirement here besides having local storage or a USB drive already attached, or in my case I used an SMB share. However I was very soon disappointed to see this error…

So…. so much for this being a free option, which I don’t think is fair, anyway. As usual its not even Veeam fault, this is cause VMware doesn’t allow the APIs for this, check this Veeam blog post out for more details.

If you use VMware a lot you you might have come across a blog site called virtuallyghetto run by William, this guy is great and my colleague just happened to find a script that was written by him to use the VMware CLI directly to create snapshots of VMs and copy their delta files to another disk, completely free.

In Part 3 I hope to install and try out this script, see how it handles my needs. Stay tuned!

Free Hypervisor Backup
Part 1 – Installing Veeam Backup

Intro

A little while back I had blogged about how you can get ESXi for free (you can also choose to use Hyper-V free with any version of Windows Server 2016/10, or using the stand alone core image).

However now that I have a couple nice hypervisor test beds, (I use FreeNAS for my storage needs, I hope to write a couple FreeNAS posts soon) how do we go about making backups, now we could manually backup the VM files manually, but that takes a lot of work, and I’d generally don’t like dealing with the file directly as soon as snapshots get involved, then I prefer to stick with the providers APIs. As you can guess I don’t have time to learn ever providers huge list of APIs, let alone the time to build any type of application for it (be it direct .NET, ASP.NET (w/ whatever front end (bootstrap/angular/etc)), JAVA (shutters), and whatever… so I could go on here but I’ll stop.

I’m personally not going to test a whole bunch of different solutions, but instead pull a bit of a fan boy and cover just Veeam. I came from using Backup Exec (which is now the hot potato of Backup Software, since it almost destroyed Symantec)… anyway, to using Veeam, and it was a breath of fresh air, not only do they have amazing support staff you know what they are doing (usually if you get in the higher tiers), but they also have a great form site with a good following and replies by the developers themselves. You also don’t need to sign up to read them if you need to find a solution to a problem in a pinch, they don’t mind airing out any dirty laundry cause more often then not it’s not directly their fault but the APIs they rely on. Anyway moving on.

Getting the Installation Media

To start go here to grab Veeam Free Backup. This requires a login, I can only assume to avoid Captcha, or other mechanism to prevent DDOS or annoyances, as well as information gathering. Feel free to use fake information for this.

Now Veeam can only be installed on Windows, see here for all the detailed specs.

I’ll choose Windows Server 2016 Datacenter as I have it available with my MSDN for all my educational needs. 😀

So at this point we have:

  1. A supported OS installed physical or virtual (i prefer virtual specially for labs)
  2. A Copy of the latest version of Veeam free
  3. A hypervisor (Hyper-V or ESXi) with VMs

*If you are looking to backup physical machines liek desktops and laptops look at Veeam’s agent options, Veeam Windows agent and Linux agent allow to backup physical machines.

Running the Installation Media

After updates it’s finally time to mount that ISO! In my case I had downloaded it on my workstation machine running Windows with the vSphere phat client, so I mounted it via the vSphere option to mount a local ISO to the VM. After mounting, and double clicking the installation executable, you are presented with this:

The EULA

Ooo, ahhhhh, click install…. and accept the EULA

Licensing (Free)

You will be present with this license part of the wizard, but as the text at the bottom indicates, click next without this to use free mode… wow how intuitive, no radio buttons, or check boxes… just simple intuitive wizard design…. would you just look at that… a thing of beauty. Click Next.

I was good with an all-in-one so I left the defaults, click next,

Dependencies

What is this? A clear, concise dependency check! And here I thought I could trick them by not installing things and see how it go, they seem to have done a good job covering their bases… and what is this?! and install button… you mean… I don’t have a vague link to a KB with some random technical blabber that links me to an executable to install before having to re run the wizard…. well lets see if it even works… Click Install… (Assuming internet connection; which this server does have, as how I got it updated)

Kool…

What is this?! no way…. it installed everything for me… and I didn’t have to reboot or re-run the wizard. Get out of town!; and click next.

Install location and verification

Again I’m OK with the defaults, click Install.

Let it install (it will use MS SQL Express (which is free up to 10 GB DB’s).

There’s a saying that goes “waiting is the hardest part”, thankfully with Veeam, this seems to be the case. Be patient while the installation completes, you’ll be glad you did. 🙂

Alright finally…

Click Finish, Now that was easy.

Click Restart.

Summary

That’s it! That’s all there is to it, the smoothest installation I’ve ever done, so smooth it doesn’t actually warrant it’s own blog post. But what the heck…

In Part 2 I’ll cover some basic configurations, and backup our first VM!

USB 3.0 Support on Windows 7 Guest VM

In Short, it’s not supported. If you’re running Workstation 9 or above, there’s this trick.

Now this guy goes into the real nitty gritty, and I love that! I however was working with ESXi 5.5 u3b. Now VMware did the same thing with the ESXi hypervisor and introduced USB 3.0 support via the xHCI controller. However the exact same limitation apply.

1) Drivers of USB 3.0 Host Controller are not provided by VMware Tools.

2) VMware USB 3.0 Host Controller will work only if your Virtual Machine OS has Native USB 3.o Support. Examples of such OS are – Windows 8, Windows Server 2012 and Linux Kernel 2.6.31 and above.

He goes on to say he’s screwed, but I’ve found the older EHCI +UHCI controller works for USB 1.1 and 2 devices I haven’t fully tested all case scenarios however. .For a Windows Server 2016 VM, on a HP Gen9  server with ESXi 5.5. My findings were as follows:

  1. Installed xHCI usb controller, via VM settings.
  2. Guest OS picked up hardware change and installed driver without issue.
  3. Plugged in USB 2.0 device, showed up in Host, as USB device became available to add to VM via VM settings, so added device.
  4. Guest OS didn’t see the USB device connected.
  5. Removed device via VM settings, then disconnected from host.
  6. Connected USB 3.0 Stick into host, added to VM via VM settings.
  7. Device was seen on Guest VM, and performance was equal to that of the sticks specs. (18~20  MB/s write, 100+MB/s Read)

I wasn’t sure why the USB 2.0 Device didn’t show up, so I simply removed the xHCI USB controller, and instead installed the EHCI +UHCI. Re-Connected the USB 2.0 devices and added it to the VM, this time the device did show up. I can’t remember the exact performance counters. I’ll update this post when I do some better analysis. My plan is to script some I/O tests using diskspd and PowerShell. Stay tuned. 😀

I’m also going to see if I can connect the same USB device via hardware pass-through instead of utilizing the USB controllers and Devices VM settings options.  I’ve manly done this with RDM’s and storage controllers with storage type VM’s (FreeNas mostly).

As for the main point of this post… I figured the main link I posted and this one here as well form the VMware forms that I’d be able to get a way to make the xCHI controller work on the Windows 7 VM guest.  The answer is basically grab the Intel xCHI drivers for Windows 7/2008R2 from Intel and install it manually, not via the setup.exe.

To my dismay I couldn’t get it to work, the wizard simply couldn’t locate the device (since the hardware IDs didn’t match) and installing the otherwise the device wouldn’t start.

I even decided to try and use double driver (extracts drivers) against a newer guest OS. This also failed. I simply couldn’t get it to work.

Reclaim unused space from VMDK

Let’s say you have a bunch of servers *Cough Server 2008 R2* that have been fairly well maintained and all running on VMware’s ESXi hypervisor system. As a regular server admin you’ve come to terms with updates and keeping systems for the most part on the latest n greatest. Now lets also say you happen to be the storage admin as well and you find you are running out of space on your SAN. What do you do? Usually buy more space. But lets get to the real heart of the matter… Systems, if not properly set up, get messy (don’t get me started on Windows registry.) we’re sticking with storage as the topic of the day. Well good news is I’m here to help you reorganize and re-claim all that space. Lets get started!

*NOTE* if you are running thick provisioned discs you’ll have to svmotion them to another datastore to convert them to thin first.

First and foremost you’re going to want to clean up your WINSXS folder. Don’t believe me, run windirstat to find out just how big it has become from all those updates. 

How do you clean up your WINSXS? You may ask well, first ensure your server has Windows6.1-KB2852386-v2-x64 installed. Note these steps work for Windows 7 as well if anyone happens to need to save space on a client machine. You might be able to find cleanmgr.exe online, but your safer to copy it from another server. or try this. run cleanmgr.exe make sure you run it as an administrator and clean system files. Clean up the old update files. Reboot (Your HAVE to reboot to complete the update removals before moving to the next step!)

For the next part you may or may not want to do depending on what the app reports. Run Disk Defrag. In this case my servers were about %40 fragged; meaning that over time as files were added, used,and then deleted they were placed randomly throughout the disk depending on where the FAT (File Allocation Table) generally in this case NTFS telling which sections were free to overwrite. Yup when you delete a file it’s not actually deleted from the sections just from the table. So Defragging pretty much “shoves” all the actually still in use data nice and organized at the “front” of the disc.  This is generally only required on spindle discs, if your system is using SSD, or a logical unit based on RAID this won’t matter.

Now if you’re simply clearing space on a phsyical device, barebone device. You’re pretty much good to go. However for the rest of us virtulaized guys who want to reclaim space on our SAN’s we still have a ways to go.

This is where I find the “fun” begins. if you attempt to look it up you’ll find some old articles from VMware about using vmware tools. Well #1) The GUI options are gone,if you attempt to find vmware tools under control panel, you won’t find it. #2) If you go ahead and try to use the cmdlets you’ll probably find it simply returns the disc can’t be shrunk. I personally say don’t waste your time attempting to do anything here with VMware tools. For Linux users you can accomplish this via dd very easily. For the rest of us Windows users we can thank the Great Mark Russinovic for sysinternals, in particular this time for sdelete. Grab it and run sdelete -z (important in v1.56 it was -c, in 1.61 use -z) If you don’t specify a drive it will use the drive you run the cmd from, I’m assuming.

Time for the last and final fun part. Read this and this. Once you’ve done that I’ll provide my findings:

1) You have to svmotion between datastore of different blocksizes (I found the 2 MB block size was the one that worked for me)

2) you can’t use the vmkstools holepunch option against a VMDK stored on a NFS datastore

To Paraphrase to solution:

1) Remove and delete temp files, unused profiles, and old update files.
2) Defrag to organize all the blockson the guest file system.
3) Use sdelete or dd to zero dirty blocks.
4) Hole punch or svmotion the VMDK to shrink used size.
5) Enjoy a beer and a bunch of recovered space.
6) You might even notice a performance increase from all the organized guest file systems

Jan 2018 Updates

2016 didn’t have many posts, but they sure are good ones, I forgot all about this stuff. haha.