Blog

Posted on

How to vMotion a VM without vCenter

Well here I am… again…

In short, you figure… “Ummm just vMotion the VM in vCenter” and for the most part I would agree, however what do you do if you need to move a VM, for example vCenter, and it just so happens to be on an ESXi host that is not within a cluster with other similar ESXi hosts, or in a cluster without EVC? (In most cases rare, sure) However I happened to be just in that situation recently.

First thing I thought I’d just copy the files via ESXi console, using the CP command, and it for the most part it seemed to work for one smaller VM. However when I went to do it against vCenter. It seemed to be going longer then I had expected. After nearly an hour… I decided to see what was going on… but since I was just using CP command how do I find out the processes time?

“Yes, by running stat on target file and local file, and get a file size,

i.e stat -c “%s” /bin/ls”

Oh neat, so when I went to check the source was 28.5 Gigs…. and the target was 94 Gigs… wait wait what??? I can only assume something messed up with the copying cause the files were thin provisioned… not sure stopped the process and deleted the files…

Now I began to Google search and I wasn’t searching properly and found useless results such as this: Moving virtual machines with Storage vMotion (1005544) (vmware.com) then I got my act together and found exactly what I was looking for from here: How to move VMware ESXi VM to new datastore using vmkfstools | Alessandro Arrichiello (alezzandro.com)

So basically I liked this, and it was what I needed, I was just slightly annoyed that 1) there wasn’t a nice way to do multiple VMDKs via his examples, just for all the other files, so I took the one liner from the other files trick, and found out how to get the path I need from the files in question.

Low and behold here’s how to do the magic!

1) This assumes a shared datastore between hosts (if you need to move files between hosts without a shared datastore, follow this guide from VMware arena.) (I’m not sure but I think you can leave the VM’s registered, but they have to be powered down, and that there are no snapshots.)

2) Ensure you make the directory you wish to move the VM files to.

mkdir /vmfs/volumes/DatastoreTarget/VMData

3) Copy/Clone VMDK files to target.

find "/vmfs/volumes/DatastoreSource/VMData" -maxdepth 1 -type f | grep ".vmdk" | grep -v "flat" | while read file; do vmkfstools -i $file -d thin /vmfs/volumes/DatastoreTarget/VMData/${file##*/}; done

4) Copy remain files to target.

find "/vmfs/volumes/DatastoreSource/VMData/" -maxdepth 1 -type f | grep -v ".vmdk" | while read file; do cp "$file" "/vmfs/volumes/DatastoreTarget/VMData/"; done

Once done cloning and copying all necessary files, add the VM from the new datastore back to inventory.

In the vSphere client go to: Configuration->Storage->Data Browser, right click the destination datastore which you moved your VM to and click “Browse datastore”.

Browse to your VM and right click the .vmx file, then click “Add to inventory”

Boot up the VM to see if it works, when asked whether you copied or moved it, just answer that you moved it. In this case it all depends on if you want the VM DI to stay the same as it is known within vCenter. As long as you properly delete the old files and removed it from the host inventory, this will complete the VM migration. If you don’t plan on deleting the old VM, or do not care about VM IDs or backups, then select “I copied it”.

Hope this helps someone.

Posted on

Failed to create VMFS datastore. Cannot change host configuration.

Quick one here. Create a new logical disk via RAID5, after an old logical unit failed from only a single bad disk.

No issues deleting the old logical disk, and creating a new one via HP storage controller commands.

However was greeted with this nice error.

From here, by Cookies04: ”

I had the same problem and in order to fix it I had to run three commands through an SSH connection. From what I have seen and found this error comes from having disks that were part of different arrays and contain some data on them. When I ran the commands I was then able to connect the data stores with no issues.

1. Show connected disks.

ls -lha /vmfs/devices/disks/

(Verify the disk is seen. You will probably see your disk ID then :1. This is a partition on the disk. We only need to work about the main disk ID.)

2. Show the error on disk.

partedUtil getptbl /vmfs/devices/disks/(disk ID)

(It will probably indicate that the GPT is located beyond the end of the disk.)

3. Wipe disk and rewrite with a basic MSDOS partion.

partedUtil setptbl /vmfs/devices/disks/(disk ID) msdos

(The output from this should be similar to msdos and the next line will be o o o o)

I hope this helps you out.”

Looks like it worked… Thanks Cookie04!

Posted on

Disable Automatic Detect Settings via GPO

Hello everyone,

If you found this blog post, chances are you are trying to disable this setting:

Well let me tell you, it was not as easy as I thought.

*Expectations* go into GPMC, create a GPO, find a predefined option to deploy and done.

*Reality*… Try Again.

First off, a huge shout out to the IT Bros for some help in understanding some nitty gritty’s

In short:

  • They use GPP or IEAK11 to set the setting, and define the properties.
    -In this blog post I do too, but I do it differently, for reasons you’ll see.
  • The Proxy Setting is usually a user defined setting, but there is a GPO option to change it to machine based setting.
    -Computer Configuration > Administrative Templates > Windows Components > Internet Explorer. Enable the policy Make proxy settings per-machine (rather than per user).
    -It was not described how to set the proxy setting, or define the proxy server address after setting this option. (If you know the answer, leave a comment.)
  • The green underscore for the IE parameter means this setting is enabled and will be applied through Group Policy. Red underlining means the setting is configured, but disabled. To enable all settings on the current tab, press F5. To disable all policies on this tab use the F8 key.
    -This is relevant when making IE option changes via the built in GPP for IE options.
    -I found the F5/F8 to enable/disable options was global, all or nothing, and only worked on some of the tabs, not all of them.
    -Defining IE options this way felt more like a profile or multiple options, and not granular enough to define just a single option. (This is the main reason for this blog post.)

All super helpful, but I didn’t want to do it this way as I only wanted to make a change to the one and only setting, I was hoping to do it without having to figure out the complexity of the IE options GPP “profile”.

I eventually stumbled upon the TechNet thread that ultimately had the answer I needed. A couple things to note from this thread, which is also covered by the IT Bros.

  • The initial “Marked as Answer” is actually just the option to lock down the changing of the IE LAN Settings, Automatically Detect Settings. It does not disable it.
  • The setting is enabled by default ON a non configured machine, or non-hardened domain joined machine.
  • The actual answer is simple a Reg Key that defines the setting. (Thanks Mon Laq)
  • The Reg Key in question is volatile (It disappears after setting it, there seems to be no official answer as to why, if you know please a comment).

Which leads to why do this in the first place if it appears to be such a hassle to set? Well for that it came down to answer by “raphidae” on this TechNet thread, which lead to this POC of a possible attack vectore, which apparently allows credential stealing even from any locked machine.

I unfortunately haven’t been able to test it, I don’t have the devices mentioned in the blog, but maybe any laptop can do the same just less conspicuous.

Anyway…. Long story short, to achieve the goal will have to be done in two parts.

  1. Deploy a User based GPO (GPP to be exact) that will push the required registry key.
  2. Deploy a GPO to lock down the changing of that setting.
    *NOTE* From testing the end user has the ability to write/change the keys that the GPP pushes down to the end machine user settings to. The GPO simply greys out the options under the IE options area. It does not prevent the changing or creating of the registry DWORD. (I wonder if changing to machine settings could lock this down? Leave a comment if you know.)

So creating the GPO (Assuming a pre-created GPO, or create a new dedicated one):

In the GPO navigate to: User Configuration -> Preferences -> Windows Settings -> Registry.

Right click and Add new Registry Item, Ensure you pick the HKCU class.
Ensure Path = SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings
Type = DWORD
Value Name = AutoDetect
Value Data = 0

In the end it should look like this:

Given the GPO is configured in an OU that contains all your users, it should apply to the machine and you should see the checkbox for “Automatically Detect Settings” be turned off.

The second step now requires making another setting change, since this one is machine based I deploy it (link it) to an OU that contains all the end users workstations. (Again if I could figure out how to change this setting making it a machine based setting instead of user, you could simply the deployment to be all targeted at machines and not both users and machines.)

Anyway the second GPO:

This time drill down to: Computer Configuration -> Policies -> Administrative Templates -> Windows Components -> Internet Explorer -> Disable changing Automatic Configuration settings. Enable

Ensure machine are in the OU in question, and gpupdate /force on the end machines. The final result will be like the first picture in this blog post. Again this option really only greys out the UI, it does not in fact prevent users from adding the required key in regedit and having the option change anyway.

Hope this post helps someone.

*FOLLOW UP UPDATE* This alone did not stop the WPAD DNS queries from the machine. Another mention was to stop/disable the WinHTTPProxySrv. When checking this service via Services.msc it appeared to be enabled by default and greyed out to change the startup type or even to stop it. I found this spiceworks post with a workaround.

To test on a single workstation edit the following registry key:

HKLM\System\CurrentControlSet\Services\WinHttpAutoProxySvc

“Start” DWORD

Value = 4 (Disabled)

Sure enough rebooting the machine the service shows to be off and not running. So far checking packets via Wireshark shows the WPAD queries have indeed stopped.

*Another Update* I am unaware if these changes actually prevents the exploit from working as I’m unsure if option 252 for DHCP still allows for the exploit to run. This requires further follow-up, validation.

Posted on

vMotion – Not Allowed in the Current State

First things first, I vMotioned the vm to another host and that worked fine, so the issue appeared to be target related. I also found this post, which states to restart the mgmt and vpxa services:

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

doing this on the source ESXi did nothing, again seeming the issue is on the target. Did the same tasks on the target and it still failed.

I then disconnected the target esxi, put it in maintenance mode, rebooted it, took it out of maintenance mode, reconnected to vCenter, and this time the vMotino worked.

Hope this helps someone.

Posted on

ESXi 6.x Datastore Not Mounted

Quick post here, I had to recover from a flooded basement. Sorry for the day outage. I had to put my disc in another server and load FreeNAS, and import my ZFS volumes, recreate the iSCSI targets, and then I added them to my ESXi hosts, and rescanning the HBAs shows the disks…

but the datastores were not visible…

so I googled and found this VMware thread with some helpful commands to try. (I do kind of agree with the OP, that its annoying they removed the front end UI for import that could handle this)

esxcli storage vmfs snapshot list

esxcfg-volume -M UUID

Ehh it worked!

Hope this helps someone. If this doesn’t work you might have some other underling issue?

Posted on

Veeam As Built Report

Source: HomeLab – Veeam VBR Documentation with AsBuiltReport – A Technology Blog (zenprsolutions.net)

Prereqs:

Ughhh…

Google!! … Let’s try this…

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Nice it worked…

You can use the commands like in the source guide, I however simply downloaded the source files and extracted to Veeam server c:\temp.

Don’t forget to use recursive… lol

As you might be able to tell, my Veeam instance is non domain joined. Using the local Admin account to run everything.

Let’s GO!

Little missing field validation, but not important for what will just be text field in a report.

and failed cause bad credentials, right I forgot to put in my password when making the snippet, lets build the credentials again properlly and run through again…

Alright some file checking here, nice….

and success short lived… Enterprise Edition… like what??

Well I tried… I’ll update this blog once there’s an update from the Dev.

Posted on

Using StarTech USB3HDMI in OBS

Startech USB3HDMI

This is a quick post. I’ve lately been streaming and I picked up this sick capture device the “StarTech USB3HDMI” I got this after EposVox, check out his twitch channel here, did a review of the product and posted it on youtube here.

This video really helped me out in making this purchase and configuring it to work with OBS. I haven’t tested this card with other capturing based software, and technically OBS is a streaming based software, but can be used to record the canvas of multiple input sources, so in reality way more powerful then basic video capture alone.

Anyway here’s the pictures of it working on OBS, as a device capture input and the settings to make it work for:

Composite AKA the Red/White/Yellow RCA plugs

If you don’t see all these options try scrolling, the scroll bar is easy to miss. It’s important to pick the audio device here for the Red/White audio input to be picked up. There’s also addition settings when you click “Configure Video” which oddly enough includes some audio options…?

As you can see Audio input set to embedded. With video selected as 6/Composite. That’s it for that one. Sometimes I find you have to restart OBS or reseat the capture device for things to play nice.

This is the capture I do the most and I found I could actually play games from this source directly without a low latency splitter. For near zero latency game play though you might want to get a near zero latency splitter.

VGA

Time to test VGA, to start this test I’m actually going to take my computer and power it on a standard VGA monitor, after the output is good, I will then switch it over to the capture card.

Oddly it didn’t show up at first, after I restarted OBS the source showed fine.

Now for this setup you’d almost assume that the device has a 3.5mm headphone jack or maybe RCA plugs for Audio in… but the only one is the ones available for composite, now you possible could go from a 3.5mm audio out to RCA in on the composite input of the capture card and use the same audio settings as the above composite, however I personally have not tested that.

For now I use the Line-In on my mainboard on the computer running OBS to capture to audio in this setup. If you’re capture computer doesn’t have a dedicated Line-IN or Aux in, then attempt the trick I mentioned in the previous paragraph.

DVI

The VGA is actually physciall done via a  DVI-I to VGA adapter that comes with the kit, and the fact it has a DVI-I capture input means it can grab analog VGA and digital signal on the same interface, and probably why this card costed as much as it did. So if you need to capture DVI, simply unplug the DVI-I to VGA adapter and plug the DVI cable directly into the capture card and pick 2/DVI from the device settings/properties as shown in the previous snippets from the RCA and VGA input options discussed above.

I do not have snips of this as (at the time of this writing) I do not have a DVI based device to capture its source.

HDMI/DP

Since Display port and HDMI are (for the most part) interchangeable (won’t discuss the technical details about features of each, and which version) but for the most part, it’s like USB, backwards compatible.

That being said it has a limitation of only 1080p, so don’t expect 4k capturing here. Also while the box did not define it, I’m also assuming 30 FPS or so, and not from my testing but from EposVox apparently decent latency (a couple hundred ms?) nothing worse than what a bad TV would give.

Posted on

Email Stuck in Exchange Transport in 2022

Happy New Year!

If you are an exchange admin you may want to check out the notice from Microsoft. But you probably already have considering it started in the beginning of the new year: Email Stuck in Exchange On-premises Transport Queues – Microsoft Tech Community

So you probably already implemented this fix.

We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS), as illustrated below:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

Using the Automated Solution

  • Download the script here: https://aka.ms/ResetScanEngineVersion
  • Before running the script, change the execution policy for PowerShell scripts by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
  • Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell).

Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked           : 01/01/2022 08:58:22 PM -08:00
LastUpdated           : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001 (note: higher version number starting with 211233... is also OK)
UpdateStatus          : UpdateAttemptSuccessful

Using the Manual Solution

In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange mailbox server in your organization that downloads antimalware updates. Edge Transport servers are unaffected by this issue.

Verify the impacted version is installed
Run Get-EngineUpdateInformation and check the UpdateVersion information. If it starts with “22…” then proceed. If the installed version starts with “21…” you do not need to take action.

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 (or higher)

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

If you want to know why this happened here’s a answer from the comments:

John_C_Kirk – “This wasn’t due to a change on 31st Dec. The problem is caused by an integer overflow error: the anti-malware component is converting the date/time into “YYMMDDHHMM” format and storing it as a signed 32-bit number (max value 2147483648). So, in Dec 2021, the number would start with “2112…” (below the threshold). In Jan 2022, the number would start with “2201…” (above the threshold).”

Two Thumbs up on implementation.

Posted on

ESXi /tmp is Full

I’ll keep this post short and to the point. Gott errors in the alerts.

I was like huh, interesting… go to validate it on the host by logging in via SSH then typing the command:

vdf -h

At the bottom you can see /tmp space usage:

I then found out about this cool command from this thread:

find /tmp/ -exec ls -larth '{}' \;

This will list all the files and their sizes to gander at, when I noticed a really large file:

I decided to look up this file and found this lovely VMware KB:

The Workaround:

echo > /tmp/ams-bbUsg.txt

The solution:

To fix the issue, upgrade to VMware AMS to version 11.4.5 (included in the HPE Offline Bundle for ESXi version 3.4.5), available at the following URLs:

HPE Offline Bundle for ESXi 6.7 Version 3.4.5

https://www.hpe.com/global/swpublishing/MTX-a38161c3e8674777a8c664e05a

HPE Offline Bundle for ESXi 6.5 Version 3.4.5

https://www.hpe.com/global/swpublishing/MTX-7d214544a7e5457e9bb48e49af

HPE Offline Bundle for ESXi 6.0 Version 3.4.5

https://www.hpe.com/global/swpublishing/MTX-98c6268c29b3435e8d285bcfcc

Procedure

  1. Power off any virtual machines that are running on the host and place the host into maintenance mode.
  2. Transfer the offline bundle onto the ESXi host local path, or extract it onto an online depot.
  3. Install the bundle on the ESXi host.
    1. Install remotely from client, with offline bundle contents on a online depot: 
      esxcli -s <server> -u root -p mypassword software vib install -d <depotURL/bundle-index.xml>
    2. Install remotely from client, with offline bundle on ESXi host: 
      esxcli -s <server> -u root -p mypassword software vib install -d <ESXi local path><bundle.zip>
    3. Install from ESXi host, with offline bundle on ESXi host: 
      esxcli software vib install -d <ESXi local path><bundle.zip>
  4. After the bundle is installed, reboot the ESXi host for the updates to take effect.
  5. (Optional) Verify that the vibs on the bundle are installed on your ESXi host. 
    esxcli software vib list
  6. (Optional) Remove individual vibs. <vib name> can be identified by listing the vibs as shown in #5. 
    esxcli software vib remove -n <vib name>

    Summary

    Use the commands shown to trace the source of the usage, your case may not be as easy. Once found hopefully find a solution. In my case I got super lucky and other people already found the problem and solution.

Legal Information Proudly powered by WordPress