So The other day I posted about upgrading vCenter to 7.0.x while everything went fine during the upgrade. For some odd reason a couple days later when I went to navigate to the vCenter login page I was greeted with:
 An error occurred while fetching identity providers.
Kind of wished I had read this reddit post right off the hop, cause the first reply was is going to be my answer at the end of this post.
I did however first hit this KB about it as well I was a bit thrown off has it indicated to only do it if you see the following in the logs:
2021-03-10T09:27:03.474Z [tomcat-exec-14 INFO com.vmware.identity.token.impl.X509TrustChainKeySelector opId=] Failed to find trusted path to signing certificate <STS Certificate Subject, example - C=US,CN=ssoserverSign\,dc\=vsphere\,dc\=local> java.security.cert.CertPathBuilderException: Unable to find certificate chain.
Which I could not see, so I wasn’t sure if this was the issue or not. What I did see in my logs was the following:
2021-09-17T23:58:03.945Z [tomcat-exec-14 WARN com.vmware.vcenter.trustmanagement.impl.VcIdentityProviders opId=] com.vmware.sso.interop.ldap.NoSuchObjectLdapException: No such object LDAP error [code: 32] and 2021-09-18T01:19:01.322Z [tomcat-exec-26 INFO com.vmware.vapi.security.AuthenticationFilter opId=] Not successful authentication java.lang.RuntimeException: Authentication data not found Caused by: com.vmware.vapi.dsig.json.SignatureException: Cannot verify the signature over the provided data
So it wasn’t matching. Looking at my firewall I couldn’t see any LDAP connections from vCenter to my LDAP server since the upgrade. So I decided instead to try a reboot. This simply made things worse.
No Healthy Upsteam
Now when I’d try access vCEnter Web UI I was greeted with a blank white web page with simple text stating “No Healthy Upstream”, now looking into this, people reached this problem for several different reasons. As mentioned here and here and for some odd reason this guy just changed his IP address?! Weird.
For me I checked the local Hosts file and it was fine, and couple other mentioned fixes and they all didn’t work for me.
For some reason at this point I decided to double the mentioned work around in the initial VMware KB I found as the main login symptom was exactly the same even though I couldn’t validate the same log entries within the logs.
How to Copy Files to VCSA via WinSCP
Now a couple real quick things to note here. You need to copy a script to the VCSA. If you get unable to agree on a cipher suite, you’ll need to update your copy of WinSCP to a newer version. Also instead of doing what VMware says to change the shell on the VCSA, do what this guy suggests instead:
“In the new connection dialog, specify the Host name, User name and then click the Advanced button,
Choose the Environment/SFTP option
Specify for SFTP server: shell /usr/libexec/sftp-server”
so much easier.
I decided to take a look at the script after copying it to the VCSA, and it had this line which had me hopeful it would actually work to resolve my issue:
/opt/likewise/bin/ldapmodify -x -h localhost -p 389 -D "cn=administrator,cn=users,$DOMAINCN" -w "$DOMAINPASSWORD" -f sso-sts.ldif | tee -a $LOGFILE
So I followed along with the workaround specified in the KB…
1) Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
2) If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:
chsh -s /bin/bash
3) Connect to the PSC or vCenter Server with an SSH session if you have not already per Step 2.
4) Navigate to the /tmp directory:
5) Run chmod +x fixsts.sh to make the file executable.
chmod +x ./fixsts.sh
6) Run ./fixsts.sh.
Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:
service-control --stop --all service-control --start --all
To my Amazement it actually worked, and I was able to login into the vCenter server!! Wooo!
Kinda funny that 7.0 is stated as 6.8 in the scripts.. mhmm