Veeam – Can’t get service content. Soap fault.

So the other day I added a new Windows managed server to Veeam and as usual I came across some errors and issues that had to be resolved, and some tips on what too look out for to resolve them. Besides the one error being used for two different issues (network vs authorization), it’s generally not that bad and easy to decypher exactly which of the two is the cause. However sometimes you come across an error that seems to have multiple causes and knowing which one it is can be sometimes difficult to diagnose.

Today was one of those things, after adding the newly added managed server as a Veeam vSphere Proxy I was hit with this error when attempting to complete any replication jobs…

Processing configuration Error: Client error: Cannot get service content.
Soap fault. No DataDetail: 'get host by name failed in tcp_connect()', endpoint: 'https://vcenter.domain.local:443/sdk'

Googling this I found one post on the Veeam forms that was a basic dead end.

And this nice thread on Spiceworks.

The only thing different between this Proxy and my other one was that it was not domain joined, which I didn’t see as a pre-req… and sure enough it’s not, but in my case it was phlights response that nailed it for me:

“I attempted to connect to vcenter from my remote proxy and found that it didn’t have an entry for vcenter in DNS. Remoted into vcenter and performed ipconfig /registerdns. Remote proxy could then connect to vcenter. I did a test replication job successfully. Yeah!”

In my case the error showed the vcenter server by the hostname that was not fully qualified, domain joined machines will auto add the domain suffix on a DNS request, but in this case a standalone system, even pointing to the same DNS servers, won’t. As soon as I saw this I had two options:

Add a domain suffix in the DNS settings of the Proxy as to make the vcenter server lookup succeed OR
Just add a static record in the Proxy host file.

since I didn’t need this system to do any other particular domain looks up I simply did #2. Then my Replication job worked. Why it didn’t fall back to another proxy that did work is beyond me…..

Also why the proxy needs to communicate with vCenter is also beyond me…

September 8, 2020August 17, 2021

Veeam – Adding a Windows Managed Server

Unlike most other blog posts that seem to love to follow the “happy path”, that never happens with me so I’m going to go over this cause something WILL go wrong…

Pre-required reading.

Now I got this as my first error attempting to add the server:

Things to check here:

Network and services:
In my case first issue was DNS, and DNS cache, since I added a newly created hostname the Veeam server was attempting to query it’s local DNS cache, I had to ensure all DNS servers had a valid record (nslookup/dig) then validate those on the local system (ping) which failed and required a local DNS cache flush (ipconfig /flushdns).Also make sure you didn’t click “No” when connected to the network, else it would have set the firewall zone to “Public”, change it back to Private or open the firewall accordingly.
File and Print Services on target:
Next I had to create a temp share folder to ensure share services were started (since I was using Windows 10, and not Server 2016/2019), otherwise much like others have mentioned… somewhere (I’ll link if I find the Veeam thread again).
This can also show up if the user account is incorrectly entered or if used as “.\user”. While this was stated as a solution to an alternative issue (to be mentioned below), I got the error above using the account in that syntax. I had to use “HOSTNAME\USERNAME”.

The second error I got was:

Things to check here:

Are you using local accounts? (Managed Server being added not part of domain) More than likely yes (otherwise you haven’t granted the domain account local administrative rights on the server being added).In this case as covered in this Veeam thread.

This issue is not Veeam specific rather MS specific, which has been the case since the inception of Windows Vista.

If you are in this boat you have 3 options:

Join the host to the same domain as Veeam. Created a dedicated domain account and place into the managed server local admins groups (preferably via GPO).*Most recommend
If domain joining is out of the question these are the other 2 options…
Enable and use the built in local administrator account “HOSTNAME\Administrator)*Recommend if domain join not possible (It’s less likely that this account would be directly compromised vs the alternative solution). This is also mentioned by Gostev directly in the Veeam thread shared above.
Disable UAC for local account to utilize remote calls:

cmd /c reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\system /v LocalAccountTokenFilterPolicy /t REG_DWORD /d 1 /f

This adds a reg key to disable UAC. as Mentioned by Gostev why this isn’t done automatically as it’s a security risk. No solution seems good here (besides domain joining). In this case it’s better to just use to local admin account… ughhh.

and sure enough using the local administrator account worked and the wizard moved on…

The rest of it’s a wizard, if you got to this point there should be no other major issues moving on…

*UPDATE* Veeam 11, if you can’t get option 2 to work, you’ll have to update to Veeam 11a, whenever that’s set to be released. See this Veeam Forum post for more details. Only option for V11 is to disable UAC… :S

September 25, 2019

Fixing Veeam (Veeam Service won’t Start)

Veeam Won’t Start

Yeap, the one thing you don’t want can happen at the worst time. For me I was testing a hypervisor upgrade scenario, and my host sure enough failed to come up successfully. Well…. shit.

While I was going crazy trying to bring my host back up (the stock ESXi images wasn’t good enough cause…. RealTek, yeah… this Mobo I picked was an overall bad choice, sad cause it’s ASUS… anyway…

I went to go restore some VMs from backup onto other hosts till I could recover my main host (find that custom ESXi install image) and to my dismay… Veeam console failed to connect…

Failed to connect to the Veeam Backup & Replication server:
No connection could be made because the target machine actively refused it :9392

ughhhh, what? this is a standalone server, not domain joined, no special services account or MSAs, or separate servers, like what gives?

Event viewer is literally useless… as nothing shows anywhere for any hints.

First Fix Attempt

OK so, the usual, google, and let’s see here…

Like other symptoms not much help and a generic console error, so this fix was worth a shot, what I took away from it was how to do a manual DB backup (assuming this is all the settings and configurations if re-install required) and some registry keys used by Veeam and that this was not the problem (not the droids you are after). I thought maybe I had updated and not tested, as I do tend to do shutdown instead of reboot, with my limited resources and well windows is heavy on resources.

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\SqlServerName (This is the server name where SQL is running)
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\SqlInstanceName (This is the instance name needed for the connection, which is in the format Servername\InstanceName)
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\SqlDatabaseName (This is the database name in the Databases folder once you connect)

But sadly no good, as I guess my issue is not related to any lock files on the SQL DB… ok so what else is there…

Second Fix

So I started reading this one and at first I was thinking, yup same problem, and reading along, I like Foggy but them not sharing the answer was rather annoying… then after some others reported the solution and my jaw literally dropped (probably why they tell you call support, cause this is some dirty laundry…)

as Tommy stated

“It is very likely to caused by the changing of the host name, do refer to the following link, i managed to my Veeam service started again.”

What….

sure enough running the req query command and hostname showed I had indeed changed the hostname to something more suitable AFTER installation.

Why they’d rely on a reg key vs a simply enviroment variable is really beyond me, cause the problem with using a reg key for this is pretty clear here….

So let’s try to fix this, thanks to the second guys reply by spacecrab:

“I know this is an old post, but thank you for replying with this information. I installed Veeam Backup and Replication before changing the default generated hostname, and it was really throwing me through a loop. The fix noted at that url worked perfectly after I rebooted to reset the services. I’ll relay the content here in case that sources goes away.

In my case I had renamed the computer from a default WIN234dfasd type name to a ‘much’ better alternative. Veeam refers to the local computer name in a couple of registry entries and promptly stopped working – which we didn’t notice until later.

The keys are:

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\SqlServerName
HKLM\SOFTWARE\Veeam\Veeam Backup Catalog\CatalogSharedFolderPath

Backup of the site’s Virtual Machines is now running again.”

alright let’s update some keys to be Veeam…. I just used reg edit to do this vs figuring out the exact query (although I probably should figure out a query in-case other keys but meh….

and after a reboot… Woah! all the Veeam services are running, sure enough I can connect to my standalone Veeam Server! Wooo thanks Spacecrab!