{"id":1233,"date":"2021-09-20T14:00:45","date_gmt":"2021-09-20T19:00:45","guid":{"rendered":"http:\/\/zewwy.ca\/?p=1233"},"modified":"2021-11-09T19:23:47","modified_gmt":"2021-11-10T01:23:47","slug":"fixing-vcenter-500-an-error-occurred-while-fetching-identity-providers","status":"publish","type":"post","link":"https:\/\/zewwy.ca\/index.php\/2021\/09\/20\/fixing-vcenter-500-an-error-occurred-while-fetching-identity-providers\/","title":{"rendered":"Fixing vCenter [500] An error occurred while fetching identity providers."},"content":{"rendered":"

<\/span>Story<\/span><\/h1>\n

So The other day I posted about upgrading vCenter to 7.0.x<\/a> while everything went fine during the upgrade. For some odd reason a couple days later when I went to navigate to the vCenter login page I was greeted with:<\/p>\n

[500] An error occurred while fetching identity providers.<\/pre>\n
Kind of wished I had read this reddit post<\/a> right off the hop, cause the first reply was is going to be my answer at the end of this post.<\/p>\n
I did however first hit this KB about it as well<\/a> I was a bit thrown off has it indicated to only do it if you see the following in the logs:<\/p>\n
(\/var\/log\/vmware\/trustmanagement\/trustmanagement-svcs.log)<\/p>\n
2021-03-10T09:27:03.474Z [tomcat-exec-14 \u00a0INFO \u00a0com.vmware.identity.token.impl.X509TrustChainKeySelector \u00a0opId=]\u00a0Failed to find trusted path to signing certificate<\/b>\u00a0<STS Certificate Subject, example - C=US,CN=ssoserverSign\\,dc\\=vsphere\\,dc\\=local<\/i>>\r\njava.security.cert.CertPathBuilderException:\u00a0Unable to find certificate chain<\/b>.<\/pre>\n
Which I could not see, so I wasn’t sure if this was the issue or not. What I did see in my logs was the following:<\/p>\n
2021-09-17T23:58:03.945Z [tomcat-exec-14 WARN com.vmware.vcenter.trustmanagement.impl.VcIdentityProviders opId=] com.vmware.sso.interop.ldap.NoSuchObjectLdapException: No such object\r\nLDAP error [code: 32]\r\n\r\nand\r\n\r\n2021-09-18T01:19:01.322Z [tomcat-exec-26 INFO com.vmware.vapi.security.AuthenticationFilter opId=] Not successful authentication\r\njava.lang.RuntimeException: Authentication data not found\r\nCaused by: com.vmware.vapi.dsig.json.SignatureException: Cannot verify the signature over the provided data<\/pre>\n
So it wasn’t matching. Looking at my firewall I couldn’t see any LDAP connections from vCenter to my LDAP server since the upgrade. So I decided instead to try a reboot. This simply made things worse.<\/p>\n
<\/span>No Healthy Upsteam<\/span><\/h2>\n
Now when I’d try access vCEnter Web UI I was greeted with a blank white web page with simple text stating “No Healthy Upstream”, now looking into this, people reached this problem for several different reasons. As mentioned here<\/a> and here<\/a> and for some odd reason this guy just changed his IP address<\/a>?! Weird.<\/p>\n
For me I checked the local Hosts file and it was fine, and couple other mentioned fixes and they all didn’t work for me.<\/p>\n
<\/span>Try Anyway<\/span><\/h2>\n
For some reason at this point I decided to double the mentioned work around in the initial VMware KB<\/a> I found as the main login symptom was exactly the same even though I couldn’t validate the same log entries within the logs.<\/p>\n
<\/span>How to Copy Files to VCSA via WinSCP<\/span><\/h3>\n
Now a couple real quick things to note here. You need to copy a script to the VCSA. If you get unable to agree on a cipher suite, you’ll need to update your copy of WinSCP to a newer version. Also instead of doing what VMware says to change the shell on the VCSA, do what this guy suggests<\/a> instead:<\/p>\n
“In the new connection dialog, specify the Host name, User name and then click the Advanced button,<\/p>\n
(VCSA 6.5)<\/p>\n
Choose the Environment\/SFTP option<\/p>\n
Specify for SFTP server: shell \/usr\/libexec\/sftp-server”<\/p>\n
so much easier.<\/p>\n
I decided to take a look at the script after copying it to the VCSA, and it had this line which had me hopeful it would actually work to resolve my issue:<\/p>\n
\/opt\/likewise\/bin\/ldapmodify -x -h localhost -p 389 -D \"cn=administrator,cn=users,$DOMAINCN\" -w \"$DOMAINPASSWORD\" -f sso-sts.ldif | tee -a $LOGFILE<\/pre>\n
So I followed along with the workaround specified in the KB…<\/p>\n
1) Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the \/tmp folder.<\/p>\n
2) If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter:<\/p>\n
chsh -s \/bin\/bash<\/pre>\n
3) Connect to the PSC or vCenter Server with an SSH session if you have not already per Step 2.<\/p>\n
4) Navigate to the \/tmp directory:<\/p>\n
cd \/tmp<\/pre>\n
5) Run chmod +x fixsts.sh to make the file executable.<\/p>\n
chmod +x .\/fixsts.sh<\/pre>\n
6) Run .\/fixsts.sh.<\/p>\n
.\/fixsts.sh<\/pre>\n
Restart services on all vCenters and\/or PSCs in your SSO domain by using below commands:<\/p>\n
service-control --stop --all\r\nservice-control --start --all<\/pre>\n
my results:<\/p>\n
\"\"<\/a><\/p>\n
\"\"<\/a><\/p>\n
To my Amazement it actually worked, and I was able to login into the vCenter server!! Wooo!<\/p>\n
*Update* Here’s a great blog post covering managing or creating custom certificates with vCenter 7<\/a><\/p>\n
Kinda funny that 7.0 is stated as 6.8 in the scripts.. mhmm<\/p>\n","protected":false},"excerpt":{"rendered":"
Story So The other day I posted about upgrading vCenter to 7.0.x while everything went fine during the upgrade. For some odd reason a couple days later when I went to navigate to the vCenter login page I was greeted with: [500] An error occurred while fetching identity providers. Kind of wished I had read … <\/p>\n
Continue reading “Fixing vCenter [500] An error occurred while fetching identity providers.”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"sfsi_plus_gutenberg_text_before_share":"","sfsi_plus_gutenberg_show_text_before_share":"","sfsi_plus_gutenberg_icon_type":"","sfsi_plus_gutenberg_icon_alignemt":"","sfsi_plus_gutenburg_max_per_row":"","footnotes":""},"categories":[5,8],"tags":[105,375,16],"class_list":["post-1233","post","type-post","status-publish","format-standard","hentry","category-hypervisors","category-server-administration","tag-error","tag-identity-provider","tag-vcenter"],"_links":{"self":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1233","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/comments?post=1233"}],"version-history":[{"count":5,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1233\/revisions"}],"predecessor-version":[{"id":1287,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1233\/revisions\/1287"}],"wp:attachment":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/media?parent=1233"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/categories?post=1233"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/tags?post=1233"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}