{"id":1576,"date":"2024-06-16T23:39:47","date_gmt":"2024-06-17T04:39:47","guid":{"rendered":"https:\/\/zewwy.ca\/?p=1576"},"modified":"2024-06-18T23:05:51","modified_gmt":"2024-06-19T04:05:51","slug":"remove-orphaned-datastore-in-vcenter-again","status":"publish","type":"post","link":"https:\/\/zewwy.ca\/index.php\/2024\/06\/16\/remove-orphaned-datastore-in-vcenter-again\/","title":{"rendered":"Remove Orphaned Datastore in vCenter Again"},"content":{"rendered":"

<\/span>Story<\/strong><\/span><\/h1>\n

I did this once before, but that time was due to rebuilding a ESXi host and not removing the old<\/a> datastore. This time however it’s due to the storage server failing.<\/p>\n

<\/span><\/span>The Precursor<\/span><\/h2>\n

I did NOT want to write this blog post. This post comes from the fact that VMware is not perfect and I\u2019m here to air some dirty laundry\u2026. Let\u2019s get started.<\/p>\n

<\/span>What Happened?<\/span><\/h2>\n

The short answer is I don’t fully know, all I know is that the backend storage server (FreeNas 11.1u7) running iSCSI started showing weird signs of problems (Reporting Graphs not rendering). Since I wanted to possibly do some Frankenstein surgery on the unit (iOmega px12-350r). I started to vMotion the primary VMs I needed on to local ESXi storage.<\/p>\n

Even though I checked the logs, I can’t determine what is causing all the services to not start.\u00a0 Trying to manually start it, just showed gibberish in the system log.<\/p>\n

\"\"<\/a><\/p>\n

<\/span>The Problem<\/span><\/h2>\n

Since I couldn’t get it back up they show as inaccessible in vCenter:<\/p>\n

\"\"<\/a><\/p>\n

Attempting to unmount them results in an error:<\/p>\n

\"\"<\/a><\/p>\n

Not sure what that means, I even put the host in maintenance mode and gives the same error. Attempting to remove the iSCSI configuration to which hosts those datastores, also errors out with:<\/p>\n

\"\"<\/a><\/p>\n

Strange how can there be active sessions when it literally dead?<\/p>\n

I tried following my old blog post on a similar case, but I was only able to successfully unmount the datastore via esxcli but the Web GUI would still show them…<\/p>\n

esxcli storage filesystem list\r\nesxcli storage filesystem unmount -u UID<\/pre>\n
\"\"<\/a><\/p>\n
Any attempt to set them as offline failed as they were status as dead anyway…<\/p>\n
\"\"<\/a><\/p>\n
As you can see no diff:<\/p>\n
\"\"<\/a><\/p>\n
<\/span>Solutions?<\/span><\/h2>\n
I attempted to look up solutions, I found one post of a similar nature here:<\/p>\n
How to remove unmounted\/inaccessible datastore from ESXi Host (tomaskalabis.com)<\/a><\/p>\n
When I attempted to run the command,<\/p>\n
esxcli storage core device detached remove -d naa.ID<\/pre>\n
it sadly failed for me:<\/p>\n
\"\"<\/a><\/p>\n
I was at a dead end… I could see the dead devices with no files or I\/O bound to them, but I can’t seem to removed them.. they show as detached…<\/p>\n
esxcli storage core device detached list<\/pre>\n
\"\"<\/a><\/p>\n
as a last ditch effort I rescanned one last time and then ran the command to check for devices.<\/p>\n
esxcli storage core adapter rescan --all\r\nesxcli storage core device list<\/pre>\n
\"\"<\/a><\/p>\n
checking the Web Gui I could see the Datastores gone but the iSCSI config was still there, attempting to remove it would result in the same error as above. Then I realized there were still static records defined, once I deleted them, everything was finally clean on the host.<\/p>\n
\"\"<\/a><\/p>\n
<\/span>Do It Again!<\/span><\/h2>\n
Since this seem to be a per host thing lets see if we can fix it without maintenance mode, or moving VMs. Test host.. this broken datastores check:<\/p>\n
\"\"<\/a><\/p>\n
Turns out its even easier… just remove static iSCSI targets, remove dynamic target, rescan storage and adapters:<\/p>\n
\"\"<\/a><\/p>\n
I guess sometimes you just overthink things and get lead down rabbit holes when a simple solution already easily exists. I followed these simple steps on the final host and oddly one datastore lingered:<\/p>\n
\"\"<\/a>Well let’s enable SSH and see what’s going on here…<\/p>\n
esxcli storage filesystem list\r\nesxcli storage filesystem unmount -u 643e34da-56b15cb2-0373-288023d8f36f<\/pre>\n
\"\"<\/a><\/p>\n
esxcli storage core device list\r\nesxcli storage core device set -d naa.6589cfc0000005e95e5e4104f101a307 --state=off<\/pre>\n
“Unable to set device’s status. Error was: Unable to change device state, the device is marked as ‘busy’ by the VMkernel.: Busy”<\/p>\n
Mhmmm different then last time, which might explain why it wasn’t auto removed.<\/p>\n
esxcli storage core device world list -d naa.6589cfc0000005e95e5e4104f101a307<\/pre>\n
\"\"<\/a><\/p>\n
hostd-worker and if I run the command to get process VMs it doesn’t show makes me think the old scratch\/core dump…<\/p>\n
\"\"<\/a><\/p>\n
I’m not sure what restarting HostD does so I’ll move critical VMs off just to be save and then test restarting that service to see if it released it’s strangle hold…<\/p>\n
\/etc\/init.d\/hostd restart<\/pre>\n
After this it did show disconnected from vCenter for a short while, then came back, and the old Datastore was done.<\/p>\n
\"\"<\/a><\/p>\n
Although the datastore was gone.. the disk remained, and I couldn’t get rid of it.<\/p>\n
\"\"<\/a><\/p>\n
I don’t get it… do I have to reboot this host….<\/p>\n
ughh reboot worked… what a pain though.<\/p>\n
If you want to know what datastore\/UUID is linked to what disk run<\/p>\n
esxcli storage vmfs extent list<\/pre>\n
Now for G9-SSD2, I tried to remove it since it showed signs of on the way out. and I couldn’t… seem like an on going story here. I could only unmount it from the CLI.<\/p>\n
\"\"<\/a><\/p>\n
Weird, I deleted The G9-SSD3 normally, then I detached the disk containing G9-SSD2. Then when I recreated G9-SSD3, the G9-SSD2 just disappeared. The drive still shows as unconsumed and detached.<\/p>\n
Now I have to go rebuilt my shared storage server…<\/p>\n","protected":false},"excerpt":{"rendered":"
Story I did this once before, but that time was due to rebuilding a ESXi host and not removing the old datastore. This time however it’s due to the storage server failing. The Precursor I did NOT want to write this blog post. This post comes from the fact that VMware is not perfect and … <\/p>\n
Continue reading “Remove Orphaned Datastore in vCenter Again”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"sfsi_plus_gutenberg_text_before_share":"","sfsi_plus_gutenberg_show_text_before_share":"","sfsi_plus_gutenberg_icon_type":"","sfsi_plus_gutenberg_icon_alignemt":"","sfsi_plus_gutenburg_max_per_row":"","footnotes":""},"categories":[5,8],"tags":[234,233],"class_list":["post-1576","post","type-post","status-publish","format-standard","hentry","category-hypervisors","category-server-administration","tag-datastore","tag-remove"],"_links":{"self":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1576","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/comments?post=1576"}],"version-history":[{"count":4,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1576\/revisions"}],"predecessor-version":[{"id":1580,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/posts\/1576\/revisions\/1580"}],"wp:attachment":[{"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/media?parent=1576"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/categories?post=1576"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zewwy.ca\/index.php\/wp-json\/wp\/v2\/tags?post=1576"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}