Creating and Managing Local Users ESXi 6.5/6.7

The Story

I recently started playing around with the later ESXi hypervisor (OK I’ve tried the Web UI before, and simply stayed away). Now it has been far more polished with the release of 6.7. I have been enjoying the experience a far amount. However, then I needed to create another account on my free host (since I do not have vCenter to play around with in my home lab). While most things a seemed pretty intuitive at first..

Creating a User

Host -> Manage -> Security & Users -> Add a User (Specify Username and Password)

If you actually tried to login at this point I’d laugh a bit, but it could happen, you just created a user account, right? Well first thing you should have noticed is that there was no options to define what permissions this newly created user should have, read-only?, administrator?, etc.

So you click on Roles, there are all the nice pre-created roles… mhmm nice… alright… so… how do I map a user to a role?

You can look all under Security and users (where it should be), heck you can even look all under all the Manage options… you won’t find it there either… I had to find this out by googling… and if I have to google it… it’s not intuitive…

Assigning the Roles

So click on the main host icon in the left nav area, then when the right pane has loaded, select the Action menu, you should see it on the list of options above the host, right next to refresh.

Then select “Permissions”.

When the Host’s permissions modal box appears, click add user.

Marvel at how you can now assign users to roles, from here instead of the logical place where you easily found creating the user. Even if they wanted to keep the actions menu, and the modal box, just create a dang link under Security and Users… Arrrrggg.

Resetting Local Admin Password in Windows
Bypassing Windows Login

I’m getting ready soon to do a presentation on hacking my works laptops. I was giving the green light on a spare laptop we had purchased for corporate use. So in this case my test bed was a HP Folio 9470m, decent little guy for most basic office work. Like most places we run Windows, staying with the latest updates it was configured for Windows 10.

I won’t get into to much technical details as I’ll save that for the Long Con coming up this November. However, like most security and hacks there are many layers involved and the windows login just happens to be one of them.

Now I am already an admin on these systems, however I assume the role of a perpetrator and choose to find ways to break in as if I was not a current admin. This is where my mind got blown!

Alright, so the basic thing you need to break into windows is…. Windows! lol, well ok, you can actually pull this trick off with other… (Yup, just tested with lubuntu 17 against my Windows 7 imaged laptop)… OSes, but in this case we’ll stick with using your windows install media, as you likely already have this on hand in whatever form you may need it (CD/DVD, USB… or ughhhh unno whatever).

Once in the System32 Directory simply run the following:

move Utilman.exe Utilman.exe.bak

copy cmd.exe Utilman.exe

That is pretty much it, reboot. When you are prompted in login, press the Windows Key + U and marvel at how you get a elevated command prompt, then type lusrmgr in the command window to get the local users and groups windows and alter account, create accounts, change passwords to your hearts content.

Remember the bare minimum you need is any OS that can read NTFS (Usually for most Windows installations) and physical access to the system you are attempting to get into (given the boot options are not locked down by the UEFI/BIOS) which is the next layer I’ll talk about in my next blog post.

If you need a bit more “hold my hand” guidance in pulling this off here is a good source. This of course was using the Windows 7 installation media, against a Windows 7 machine, but the general concept of the trick, replacing UtilMan.exe with the cmd.exe can be done in many ways, then when the Windows image boots and you are at the login screen literally clicking the accessibility icon, or CTRL + U will open an elevated command prompt.

Util next time… Keep fit and have fun. 😛

*NOTE* this trick still works even with Core edition,

As stated by Tiji from here:

“Or in short (a little bit more simple):
– Boot From Recovery CD
– Press “F10″
– Type ‘copy D:\Windows\System32\cmd.exe D:\Windows\System32\utilman.exe” (replace driveletter)
– Reboot and press the utility manager symbol (Or press Windows Key + U)
– Type ‘net user Administrator *’
– Enter the password
Done”

App Pool Crash on First Load

I’ll keep this on brief; for real this time.

So you created a MSA/gMSA for your Dev to use on ASP.NET.

You granted it Logon as a service rights, as well as batch logon right via group nesting in IIS_USRS group. You granted it all proper permissions on the physical path that IIS is using for the Site/App Pool, as well as any Database permissions if applicable. Yet every time you attempt to navigate the site you get a “503; Service unavailable” and when you go to check the app pool you find it is down. Right click it, select start and it comes right back up without issue, wash, rinse, repeat.

Turns out this happens cause you didn’t fully qualify the MSA/gMSA under the App Pool’s Identity settings. Even though you enter “gMSAAcct$” under the identity field and leave password fields blank, and IIS accepts this… without fault, what I believe is happening here is even though the check IIS has in place, does validate this to a be a real domain account, or service account, it doesn’t prepend or append (depending on which user construct you want to refer to) where ever it stores this user account. This is only a guess.

So you have to fully qualify it; “Domain\gMSAAcct$” You’ll notice it (IIS) will accept it just like it did before. Then watch in amazement as the page loads and doesn’t crash when you attempt to load it in a browser….

Just another Mon…. Tuesday

The Start

Nothing new or exciting to the start of my day, clean house! Now I was actually cleaning my office, not making system changes 😉 Then…. Monday happened, I mean it’s really Tuesday but it was my first day back after the weekend (The first weekend I finally didn’t have to do any system changes)… life’s good right?

I made my first talks with our DBA, and then followed up with our Developer. Our current Developer is one really hard working and amazing dev, so he needed a couple things from my to move his current project forward; a CNAME record, along with a gMSA. This is one of the many reasons I like this guy (not only understands security posture, but what is needed for it to all work!). So the first one took seconds, and the second a couple more (besides the fact I need to reboot the server, cause I choose to do IDLGA instead of granting the computer account direct permissions to retrieve the gMSA’s password)… Yes… yes I’m aware of the special klist purge command to clear kerberos tickets, but I wanted to be 100% sure. Then came the problem…

The Problem

I don’t want to get too into the nitty gritty, but the gist of it was we had an authoritative source of data that resided on an older SQL server, our Devs new project was in a whole new data center, utilizing a whole new database server.

Since we didn’t want to alter any firewall rules between the datacenters (while they are all in house and owned by the same company, the two data centers are still wall gardened off, with a 2 way trust created for most authentication purposes) this would mean either:

A) I have to allow the old SQL server to do LDAP queries against my new Datacenters DC’s. (I wasn’t in the mood for architecture changes, which I already stated so this was last resort) Then grant the new datacenters gMSA account permissions on the database.

B) Figure out a way to utilize two different accounts, to make two different source data calls, from the same App/code.

Now I like the sound of B cause lets face it, it puts all the work on the Dev and not me. (If this sounds Dilbertish…. cause it is :P) At this point I was pretty confident that this was possible… I mean… why not? Well a couple seconds later my Dev comes back and tells me that it is in fact not possible…. well sort of not possible… it’s not possible for our exact case…. for reals… let me explain, first off I’m talking about ASP.NET, second of all I’m talking about 2 different connection strings to a Database. For some references we both found, like this and this, and this, and this and even this … ok that’s a fair amount of reading (sadly I still couldn’t find one of the original sources haha) but in each case you are probably wondering (How do I specify the user name and password for an alternative connection string when using Windows Auth instead of SQL auth)…. Drum Rolll…………..

YOU CAN’T …… TADA

So…..

The Second Problem

This lead us to our second problem, while the new SQL instance was already configured for mixed auth (This means it allows Windows as well as local SQL server authentication to be permitted), our old SQL server instance…. not so much. As much as I wanted to avoid infrastructure changes, seems it was inevitable…. so I asked my DBA if this would be a problem, since you can change the auth mode on any given instance and not all instances on the server, figured this was a quick a easy solution; Enable Mixed Auth mode, re-start the instance, create a local SQL account to be hard-coded and used by the App until the source data can be properly relocated (Thus removing any hard coded garbage in the app). Alright! until….. Ughhhhhh…

The Third Problem

When my DBA went to restart the instance, he decided to use SSMS remotely (now there is nothing wrong with this, I didn’t know it was even possible and was excited to learn something new… until…) the service failed to come up successfully (Ohh boy here we go), so sure enough we get into fix mode. My DBA jumps right into Event Viewer (good man) and discovers the first error stating the service was unable to bind to the port as it is in use (DBA opens SQL Config Manager, and Services.msc and sees both services and not running). This however instantly told me one thing… the service didn’t stop properly, event though Services.msc showed no signs of running, Task Manager and Tasklist, showed otherwise. Here’s the kickers, every attempt to force stop the process (service instance sqlserver.exe) reported back “Access Denied” even running psexec (Love you Mark!!) as SYSTEM still reported “Access Denied”

The Fix

At this point I basically figured that we had to reboot the server (I also assumed it would get stuck shutting down at the “stopping service” stage, but amazingly it did not!) Sure enough after the reboot everything came up without a hitch and the new Mixed Auth mode was enabled for our Dev’s alternate ASP.NET connection string! OK I know this sounds pretty crappy for a solution but honestly it was the only thing we had left in our toolbox, and it fixed both problem one (Mixed Auth mode is now enabled for old instance) and the fact the instance came up without a problem.

Until…..

While we (DBA, Dev, and Myself(SysAdmin)) continued to test our other applications that were built via other means, it seemed a couple things were broken, this ones a little bit funny cause we assumed there was an issue for everyone, however turned out only to be an issue for the DBA and Dev, not myself (but I wasn’t on my local machine to do any front end testing from my account) so let me explain, The Dev kept digging into the real nitty gritty of the code, jumping all the way into the backend of the SQL’s stored procedures and views, to discover there was empty values being returned (Now I have no clue if this was always an issue (based on the fix) or if it actually was due to something else….anyway) turns out one of the built in views it used as a source to create a temp table was returning null thus throwing the error when calling one of the stored procedures. When the Dev and I went to talk to the DBA in the lunch room, we had discussed some of the permission changes we had just implemented on the Security Logins of the instance, and made some assumptions, so I went into the back-end AD groups to validate somethings, sure enough it was a little funny in that due to the fact their stand accounts had direct logins for the instance (generally not a fan of this, as I love scalable design and prefer to utilize IDGLA) so my DBA told me he had fixed this, and then he told me something I never would have expected and is a huge learning experience for me:

WHEN YOU GRANT AN ACCOUNT THE “SYSADMIN” ROLE/PERMISSION THE OTHER ROLES IN WHICH THE ACCOUNT IS A MEMBER DOES NOT APPLY PROPERLY (IS BYPASSED OR SOMETHING).

Literally, so what happened was there was a group we have defined to be granted sysadmin rights on the server (to manage them, not manipulate data) normally this contains admin based accounts (we all do standard account and admin accounts for least privileged best practice right :). However their admin accounts and their standard accounts were in there, which I removed, and once that was corrected and the proper nested grounds their standard accounts where suppose to get based on other roles, then applied properly and the issue was fixed.

Party in the House… Until….

Yes… believe it or not my day does not end here…. there was simply more information the great world of IT had to shove down my tiny brain that’s already overloaded and overwhelmed at the pure magnitude of knowledge you need to manage systems!! WHYYYYYY! GOD WHYYY!!!!

Anyway…. so to end the day we get a unique error message from one of our workflows, and sure enough another email from an external user providing a snipping of an error (How nice of them). Is this a coincidence…. not a chance 100% related… So again most of the heavy lifting is done by our Dev (this guy….. he’s a super star!) He managed to break it down to an assembly problem… but we were shocked as to how this could be (we checked everything was working after all our above fixes)… well until our DBA made a confession (He wasn’t happy to have found out his own account was the DB owner of a fair share of DB’s within the instance) so he secretly clean it up… well after some trial and error (reverting the change a couple times) it turned out that the error “error the server may be running out of resources or the assembly may not be trusted with permission_set external_access or unsafe” was simply due to a single missing permission needing to be granted to the new DB owner account:

In SSMS -> Instance -> Logins -> Account -> Properties -> Securables -> Check Grant for Unsafe Assemblies

Sometimes… you just gotta run unsafe code 😛

Alright! Home Time!

I Spike You!

Ahhh the internet….

Publish my own personal mx record in hopes to get my own email going….
I decided to see why my email outbound wasn’t working (sigh even following Paul Cunningham’s post seems I’m missing something) seems all my out-bound based SMTP connections to external mail servers seems to be failing. According to my firewall (Palo Alto) The rule is allowing it out but the application shows incomplete… like it’s never establishing a connection. So from my previous posts, I use telnet to attempt a connection on external known IPs for SMTP mail server, and sure enough no connections can be established (I know I’ll eventually have to create a receive connector from outbound sources and create a security rule to allow email from outside in, but I wanted to tackle email going out first).

I decided to attempt the same port 25 connection to the new record I created (I have multiple internet connection to utilize to actually test connections from “outside” instead of having to rely on a loop back NAT rule or anything). to my dismay it showed failed to connect (I already expected this as I created a NAT rule but I never created a security rule to allow the connection). I decided to go to my Monitor tab to see if I could see the attempted connection, I indeed did see it. However what surprised me more was the failed attempts from others in the short time I created this record (considering I had the IP for a long time and pretty much all ports were blocked forever, I didn’t expect there to be much attempts) these were either crawlers or something else…. but guess who the every first was….

141.212.122.227
University of Michigan (AS36375)

Not once, but twice from two sequential IP addresses…. Mhmmm what are those Michigans up to?

185.35.62.150… unknown, someone remaining anonymous, Michigian Hookup? occurred 3 minutes after.

Then Hours Later….

107.170.227.216
Digital Ocean, Inc. (AS14061)

Not sure who they are, might have to check em out..

Couple hours later…

46.29.161.101…. Anonymous

I guess it only makes sense after Americans, and Anonymous it be nothing other than the Russians right…. To be fair I don’t actually known wtf thi sis lol, Japanese mixed with Russian or something pile of who knows what.

95.181.178.182
FOP ILIUSHENKO VOLODYMYR OLEXANDROVUCH (AS57311)

They are least tried three times in a row from same IP (Good thought idea, if it doesn’t work once, heck try again a couple times)

Then my attempt… pretty funny what you can hear if you just listen…

This isn’t actually I Spike You! Like from the old school GoldenEye movie, but this is what you’d actually do if you wanted to “Spike” someone online, this is my actual server I plan to use of course, but if I actually wanted to find out what people are up to I’d create a honeypot. Maybe now that I post this, they’ll think my mx record is a honeypot, but it’ll secretly become in use… sometime…. lol

Multi BIN games on Lakka

I figured I may as well start a series on what I have to do to get Lakka setup (Lakka is a skin pretty much for retroArch, compiled for many different small board architectures including the Raspberry Pi). Today I got a game that happened to have multi bins, now I hope to post soon on how I managed to get multi disc games to play nice, which basically involved using a PsX2PsP conversion tool, which I read from another source, but that is for another post. This time I had multi bins up to 12 separate ones, and that other tool although supporting bin files only supported up-to 5 for multi-disc purposes not multi track purposes… so what to do… Google! and I found this ahh blogs make this world go round. So I basically did the same thing but in a VM to be safe, used DAEMON tools to mount the .cue, then used ImgBurn to make a single bin/cue file. Then used that in the PsX2PsP tool for a bit of compression… and it worked 😀

Configuring an Anonymous Receive Connector on Exchange 2016

The Story

Well in my previous post I discussed the issue I faced resolving an email problem with one of our development applications in which it was unable to send emails after a recent Exchange upgrade/migration. So initially we were going to simply rebuild our own workflow in-house using ASP .NET Core. Until we noticed that even our own workflows were failing… in this case the answer from the old post which was super vague “reconfigure the receive connector”. Then I somehow stumbled upon my answer through one of my hundreds of google searches… I founds this gem!

OK before I link the gem which will be the source to my answer. I also wanted to point something out real quick here in hopes maybe someone can comment below the answer to this one:

When using Exchange 2016 as an email SMTP relay, and you use a no-reply from address, with an external email address for the destination, how do you query to find out if its gone through, or stuck in que? All I could see in the ECP it always required me to select a mailbox… there’s no mailbox associated with these relayed email messages, so how does one check this?

OK, now for the gem. This guy “Paul Cunningham” He’s… uhhhhhh… He’s uhhhhh… he’s uhhhh a good guy. So I always knew you could use telnet to check certain ports and services… but this was so concise… it nailed the problem…

From my K2 Server or my in house workflow server:

1) Ensure Telnet Client feature is enabled

2) Open cmd prompt or PowerShell:

telnet exchangeServer 25
helo
mail from: user@corp.ca
rcpt to: ExternalUser@gmail.ca

220 EXSERVER.exchange2016demo.com Microsoft ESMTP MAIL Service ready at Thu, 22
Jun 2018 12:04:45 +1000
helo
250 EXSERVER.exchange2016demo.com Hello [192.168.0.30]
mail from: adam.wally@exchange2016demo.com
250 2.1.0 Sender OK
rcpt to: exchangeserverpro@gmail.com
550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

huh, just like the source blog, now why would I be getting that error… I allowed Anonymous users via the check box under the receive connectors security tab… yet Paul does a lil extra step that doesn’t seem to be mentioned elsewhere, and that check box I mentioned is his first line, but then look at the interesting second line….

[PS] C:\>Set-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" -PermissionGroups AnonymousUsers
[PS] C:\>Get-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

Since I was using anonymous settings on the Application server side (K2 in this case) I gave the second PowerShell cmdlet a run from my new exchange server.

Amazingly enough just like the source blog after running the second line (edited to fit my environment obviously) then the rcpt to succeeded!

20 EXSERVER.exchange2016demo.com Microsoft ESMTP MAIL Service ready at Thu, 22
Jun 2018 12:59:39 +1000
helo
250 EXSERVER.exchange2016demo.com Hello [192.168.0.30]
mail from: test@test.com
250 2.1.0 Sender OK
rcpt to: exchangeserverpro@gmail.com
250 2.1.5 Recipient OK

Part 2 – The Solution

If K2 is configured to use EWS, check that stuff out elsewhere, if you landed here from my previous post looking for the answer to the “There is no connection string for the destination email address ‘Email Address'” and wanted to know how that person altered his receive connector:

[PS] C:\>Get-ReceiveConnector "EXSERVER\Anon Relay EXSERVER" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

K2 SMTP Configurations

Intro

I’m going to keep this post short, just in hopes that I don’t go off the rails on this product; K2 Blackpearl 4.7. I have plenty of awesome SharePoint 2010 to 2016 migration content yet to post on my site. I’m sorry I wish I could get all the awesome things I do on here, there are many awesome things I keep thinking about; iOmega NAS conversion I did replacing the ix12 OS with FreeNAS, My test enviroment, ISCSI MPIO (VMware, Microsoft, linux configs)… anyway… K2…. ugh

The Problem

I’ve never posted about this product on my site before, cause to be frank…. I tried to stay away from it as much as possible, so all I did was update the base OS and pray that the developers or users didn’t complain about errors in any of the “K2 apps”. Trust me there are lots I can’t tell you how many times I had o hear about K2 issues… anyway, I digress. Ya’d figure it’s this simple eh… well for running the setup manager, for first time config sure… but you have to run it again if you want to ever change this value… *ahem*… alright so lets say you did this… it’ll just work right… you set the email server destination and port, it’s gotta work for everything in the server (We’ere talking standalone, not clustered). right?…. Nope… So you eventually find that this is the most common, and generic error you will find if you ever have any email issues with K2… “There is no connection string for the destination email address ‘Email Address'” and you will get this for a lot of different things. Which oddly enough some of it gets covered here. But it’s a mess, and you have no clue which problem is the cause for the error. So much like the shared link there:

Check 1 – Environment Variables

Check the Environment Variables (If you are not sure what Environment Variables are you can read this *Ahem* Awesome… WhitePaper)

Alright… so we re-ran the config manager, updated the email settings there, updated out environment variables, we gotta be good now!…..

“There is no connection string for the destination email address ‘Email Address'”

Check 2 – SMTP Config Strings

You got to be…… ok, ok…. we got this… there’s got to be something else we must have missed… let’s see… mhmmm as Mikhal says from here…

“Email configuration is externalized from process and K2 server relies on connection strings in configuration file, processes look at Environment Library, but you should also keep in mind String Table. I saw cases when people did update of Mail Server field in environment library, but their workflow was deployed from other environment with old/incorrect email settings which were written into String Table during deployment time – so you should also make sure that you have correct settings there.”

soooo… From K2 terrible support page you can either dig in and manually edit “k2hostserver.exe.config” Really….. really….. .exe.config ….. anyway load the terrible Windows application that they use to edit this XML type config file. Now you mind find your self wondering “Do I have to create a new SMTP connection string for every from and destination address? That’s…. just…… unmanageable!” And yup it sort of is, what my awesome colleague and I discovered (He’s a K2 Master by the way) is that any internal “spoofed” mail (since we had decided we didn’t utilize any of K2’s EWS integration) would only work when we had that particular user with a SMTP string in this tool; ConnectionStringEditor.exe

It took my colleague a really long time before magically discovering what the syntax was in the SMTP connection string to be a wild card E.G. *@Zewwy.ca …. Drum Roll…….. NOTHING

That’s right nothing to make a wildcard SMTP connection string simply leave the field blank in the first step of the wizard. Alright… so now we had validated workflows could send on behalf of over SMTP based email (even to exchange without any EWS integration)… However we also utilized another workflow to send external email address emails…

“There is no connection string for the destination email address ‘Email Address'”

Check 3 – SMTP Receive Connector

Are you Kidding ME?!?!?! Alright…. Jesus what else did I forget/miss…

At this point you may find yourself a lil bit stuck as every other post points you to the same solutions above, or like this jerk-off goes over everything, then laughs in your face and says “‘There is no connection string for the destination email address testsmtp4velocity@gmail.com’. That is a pity. I thought we just added it. Ok, I admit, I knew this was going to be the result, but I thought I’d keep you intrigued. Now you will have to wait for my next article to see how to fix it. You can find the answer in part 2 of this article, along with a few other tips on how to resolve some other issues when trying to send an email through a SMTP server.” only to find that there is no Part 2….

So I was kind of stuck with the initial share I gave…

“Thanks for your help. We found the root reason is that the Exchange Server config the receive policy so we changed the policy then resolved the problem.”

If dealing with issues isn’t bad enough the internet is littered with useless help. “Don’t worry guys, I figured out my problem, if you have this problem too, well I figured out mine, good luck with yours.” Anyway, again I digress and I am not such a jerk and I will tell you how I finally managed to resolve this issue for good.

So I double checked my receive connector on my Exchange server… may there’s just something I missed… well… I covered everything, No TLS…. using Port 25, listening from only my K2 Server, Anonymous Users Checked off under security… What the heck I have them all covered…

Check 4 – Part 2

I was at my wits end until this! (Part 2 LOLOLOLOL, for real though… it’s coming soon. in like 20-30 mins, maybe an hour shouldn’t take me long to write up)

Setting Mailbox Subitem Permissions

I had to do this for Resource Items, to allow all staff to view the Calendar of the resource so they could schedule items accordingly. As one of the comments mentions from this Spiceworks post

Set-MailboxFolderPermission "roomname:\Calendar" -User Default -AccessRights LimitedDetails

This can however be expanded on knowing powershell being object oriented, roomname is actually any mailbox, :\ is the delimiter, Calendar is the SubFolder. -User $Group or $User (Groups are usually best practice per IDGLA) and finally -AccessRights, you can specify any of the access rights you see in the pull down option of Outlook.

That’s it. that’s all there is to mangling Mailbox sub-item permissions.

Splitting WordPress Titles
Post Headers

I wanted to do this since I noticed my one header being really long and was unsightly… I decided to google this, like I google everything…

First one I found, I didn’t want to dink with code, I like coding but mostly PowerShell (If you haven’t noticed based on my categories)… Sorry you’re out!

Second one I found, I didn’t like cause it used a plugin… However there’s always something great to learn form the comments, specially “pessimists” *Cough* realists such as this great comment by “KRZYSIEK DRÓŻDŻ”

“Wow, you really need a plugin for that?

Why don’t you just insert tag? Installing million plugins, that aren’t doing anything really isn’t a good idea… Especially, if such plugin is not popular, so very few people have looked at/controlled it’s code (this plugin had 30 active installs).”

Made me gooo, waaaaaa that’s it? So sure enough I add <br> in my Title, and Bam! The Title is split on 2 lines, now that was easy. Thanks Krzysiek!

Looks like my third source basically does the same thing.