SSH Banners

I love SSH… like I really love it. It is pretty surprising that Windows only first had the ability to naively ssh only in the recent Windows 10 1803 build. That’s pretty sad. Just teste don my 1803 build.. nope… well I know I have done it before… anyway teh point I wanted to get here was more about SSH servers.

Now normally allowing SSH in to a system basically enables an SSH services on that system, making it the SSH server. Then you usually utilize a workstation, like the computer you usually use to navigate websites, like this one, to connect to that server with a piece of software (with Windows that’s usually Putty)… if I can get that dang native ssh to work (shakes fist)… anyway…

When you enable this service it is pretty powerful, depending on how you configure it, and what application is running the service. There are plenty of flavors to choose from (this is pretty common with Linux and open source). This is usually a good thing cause each one is scoped for a certain target audience. In my case I wanted to bring some old life back to my old Asus Router. I’ve been running DDWRT on it for a long time, and utilizing the simple command line interface (embedded linux) on a decent lil system only working as an AP otherwise, is a fun lil place to use IRC. 🙂 Find me on #Freenode (#Windows-Server, VMware, Cisco, Skullspace, FreeNAS) .

Now I figured if I was going to use this again, why not have some fun and re-do my loggin banner. Now in this case there are two things to consider:

1) The Message of the Day (MOTD) – This displays as soon as a client connects before it asks for a username. In most cases this is a great place to place your unauthorized message. (In my case the MOTD was tied to a Read-Only FileSystem file, and I had no intentions of compiling my own build, so I decided to utilize the option to not display this).

2) The Login Banner – This message displays after you have specified a user name.

Now there can be many ways to customize your login banner, you may need to google the based on the SSH server you are using. In my case my router was utilizing dropbear lucky for me they have decent documentation.

In my case I simply had to create a simple text file pointing anywhere using the -b option: dropbear -b /somepath/banner.file

After I created my file I configured my startup script to point to my new banner file. Sure enough now when I log on I see this:

Boo yeah! Now that’s sweet.

Zewwy Joins GitHub!

Actually I created my account apparently over 6 years ago :O…..

Sadly in those 6 years all I contributed was a couple really lousy DDWRT scripts…. and that OS was painful as getting help with it was near impossible.

I’ve updated my blog social links to include my GitHub Link. I moved my PowerShell Center Write-Host function there. I hope to move my SharePoint script I had written for the SharePoint migration at work. As well as some other contributions coming up soon. Maybe I’ll fork GhettoVCB and finally do part 3 of my Free Hypervisor Backup series. I’m sure everyone’s been waiting so eager for. 🙂

This was a short one. I hope to provide some more meaningful content in the short while. Just been a rough couple days recently.

BitLocker Can’t find the file

This was an interesting one, created a new Windows image to deploy recently. Then after deployment went to enable bitlocker and was prompted with the error “The system cannot not find the file specified”. Since this was new to new, what other than to do a web search to see if anyone else had experienced this, and sure enough, yup.

Short answer: rename REAgent.xml file (in C:\Windows\System32\Recovery) to REAgent.xml.old (or dlete it but I haven’t tested that).

and it worked, apparently….

“Sooooo, what we have found is that when we captured the image, since we had already opened the Bitlocker console (even though we hadn’t actually Bitlocked the unit), the REAgent.xml file (in C:\Windows\System32\Recovery) had been populated with the specific GUIDs for both WinreBCD and WinreLocation path.” – Borch25

I like borch, can’t wait for more.

Free Hypervisor Backup
Before Part 3

The Story

I’m currently in the quest to fulfill my needs for a free VM backup solution, my hypervisor of choice right now is ESXi, while there are great alternative free hypervisors ( Citrix’s Xen, Microsoft’s Hyper-V, Linux’s KVM) I personally always felt comfortable with VMwares user interface (I’m talking the old Windows Phat Client). I totally understood the need for a web based client, however I felt it a sham to drop the phat client as to provided multiple benefits that the Web UI just doesn’t. Although the latest 6.7 Web UI has been pretty decent, besides their placement of the Hostname location…. :@

Anyway… if you’ve been following my posts you’ll see I decided to give Veeam Free a shot. I love these guys, great software, so I figured best to get better antiquated with their software, to my dis-may their simply relied on VMwares APIs solely. Which meant no SSH backdoor tricks for use ESXi free users.

However as I also mentioned in my previous post, an awesome dude who runs virtuallyGhetto.com, William Lam; wrote a script to complete the task we wanted via the hosts local CLI, which can be connected to via SSH. The script is called GhettoVCB. I took a quick look at the source code and did find a couple instances of zombie code and other anomalies but for the most part looked decent enough to give it a shot. Now I will get to this stuff in my next post using the same example VM I will specify here where I came across a couple issues and interesting facts I discovered during this adventure.

My Discoveries

The first thing I noticed about the script was the lack of certain dependency checks, in this case there’s no actual source code dependencies that his script relies on, it’s all meant to run on ESXi, and sure enough there’s some line of code to check for that…

Nice, however there doesn’t seem to be a check to validate if the datastore specified in the very first variable has enough space to complete it’s task.

In this case the script would simply error out once the destination ran out of space stating the source VMDK was the issue, this lead me briefly down the wrong rabbit whole. After validating I had no issues with my source VMDK (booted the VM and checked all services and FileSystem integrity) I noticed a couple things.

1) Even though I specified Thin disc for my destination, which had enough space to store the VM data (60~ GBs), the thin disc was attempting to create the full provisioned size of the disc, cause…..

2) I forgot the source disc was set to Thick Provisioned Eagered Zero

So there was a couple things about this VM….

1) It’s my ZoneMinder VM which holds my IP camera footage on motion detection

2) This data is mostly useless due to no events having taken place

Alright so now my goal was the following:

1) Remove all the un-needed data

A) Open ZoneMinder Web UI
B) Click on Camera
C) Delete All Events
D) wait a while before all the MySQL queries to kick off to clear the data
E) Used “df -h” to watch usage drop

2) Once I had all the data removed, I had to re-claim the space. I decided to dig up my old blog post on the subject matter… Well that was a bit underwhelming and simply provided my links instead of any valid examples… So I hope to provide a bit better details here:

A) I’m running Linux not Windows so sdelete is out.

B) My first attempts I decided to follow this DD exampledd if=/dev/zero of=/zeroes && rm -f /zeroes” (DO NOT DO THIS, from my tests it caused mySQL service to not come up properlly). I then found people stating to use secure-delete or zerofree, however I had some issues with these against a live system and wanted a simple live system, general technique… if there so many references stating dd can do it… how.. then I found this
“dd if=/dev/zero of=/tmp/empty.dd bs=1048576; rm /tmp/empty.dd”
I’m assuming maybe cause I used /tmp instead of /… only diff I can think of.

C) At this point I made a backup using ghettoVCB which I had to use an alternate Datastore to save the VMDK that I would then finally hole punch, in this case the ghettoVCB script converts the VMDK from thick to Thin however will still be the size of the provisioned Disc.

D) Now this is where I started to get a bit… annoyed… However I did learn a few things… one is that & to bring a process to the background doesn’t disconnect that process from the current terminal session. So while I was SSH’d in and running the clone operation, it wasn’t fully completing when my SSH session timed out, even though I used & to run it in the background. Read this for more info but the gist take away is this: “nohup and disown both can be said to suppress SIGHUP, but in different ways. nohup makes the program ignore the signal initially (the program may change this). nohup also tries to arrange for the program not to have a controlling terminal, so that it won’t be sent SIGHUP by the kernel when the terminal is closed. disown is purely internal to the shell; it causes the shell not to send SIGHUP when it terminates.” –Gilles

E) OK so after a few annoying hours of failed transfers due to my own ignorance. I finally had a legit copy of my VMDK in thin format but sucking up a lot of space (See pic above) *Note I technically did the DD trick after making a backup, but the size shown on the datastore would still be teh same* the final part… hole punching: So quick re-cap, we deleted unused data, then zeroed out the unused blocks, we see a low size in the guest system (E.G. df -h), but we still see a large size used by the thin provisioned disc. K let’s hole punch
“vmkfstools -K /vmfs/volume/Datastore/VM/VM.vmdk”

Host Status During Hole Punch

CPU ESXi Host

CPU Storage Unit (FreeNAS)

Disk Usage Host

Disk Usage Storage Unit (FreeNAS)

Datastore Latency Host

Network Usage Host

As you can tell from the source images we can tell a couple things:

1) HolePunching a VMDK does a high read I/O on the datastore

2) If the Datastore is iSCSI based it’ll saturate the iSCSI NIC (this can cause a performance degradation for other VMs utilizing the same Datastore

3) The latency increases due to the high Read I/O on the Datastore, again this directly affects performance of VMs running on the same Datastore

Once done the VMDK looked liked this:

Results / Summary

*NOTE* You can use the following command to convert a Thick Disk to thin manually, if you wish to Hole Punch a VMDK without the script.

vmkfstools -i Source-Thick.vmdk -d thin Destination-thin.vmdk

So why did I go through all this pain? Well I didn’t like the fact I was backing up useless data, weather it be pointless old images, or zeros. In the end the same VM backup went from taking almost an hour down to 5 minutes!

Free Hypervisor Backup
Part 2 – The VMware Screw

Veeam

Run Veeam by clicking the icon on the desktop or in the start menu, for Veeam Backup and Replication.

First Run

At first you will get this:

click apply.

Click Veeam, Zip, haha I expected this.. 😛

Click ok, and the add host wizard pops up.

Infrastructure Wizard

In my case I’m using ESXi.

Credentials

In the next section you will need to specify the credentials, you could specify the root account, however in my case even with one host, and only me, I decided to create a Veeam account on my ESXi host to use for this case. On 5.5 using the phat client it is really easy and intuitive, highlight the host, click the local User and Groups tab, right click the open space, select new user, then click the permissions tab, click add user, select the newly created user, select the admin role. Done! Click here for 6.5/6.7 or the Web UI, not as intuitive. Click the add button, and add the account details that you specified when you created them on the hosts.

Then click OK, then next.

You will get this alert if you use self-signed certificates, even though I did write a blog post on setting up my own PKI, I did not use it in the case, as my Veeam server and ESXi host are not part of my AD domain, this also does simplify some aspects of the installation/deployment. Click Connect.

Click Finish, congrats you’ve added your free ESXi host. 😀

The dis-appointment

Next! Storage, Veeam needs to know where to save your data. Alright, seems there was no requirement here besides having local storage or a USB drive already attached, or in my case I used an SMB share. However I was very soon disappointed to see this error…

So…. so much for this being a free option, which I don’t think is fair, anyway. As usual its not even Veeam fault, this is cause VMware doesn’t allow the APIs for this, check this Veeam blog post out for more details.

If you use VMware a lot you you might have come across a blog site called virtuallyghetto run by William, this guy is great and my colleague just happened to find a script that was written by him to use the VMware CLI directly to create snapshots of VMs and copy their delta files to another disk, completely free.

In Part 3 I hope to install and try out this script, see how it handles my needs. Stay tuned!

Free Hypervisor Backup
Part 1 – Installing Veeam Backup

Intro

A little while back I had blogged about how you can get ESXi for free (you can also choose to use Hyper-V free with any version of Windows Server 2016/10, or using the stand alone core image).

However now that I have a couple nice hypervisor test beds, (I use FreeNAS for my storage needs, I hope to write a couple FreeNAS posts soon) how do we go about making backups, now we could manually backup the VM files manually, but that takes a lot of work, and I’d generally don’t like dealing with the file directly as soon as snapshots get involved, then I prefer to stick with the providers APIs. As you can guess I don’t have time to learn ever providers huge list of APIs, let alone the time to build any type of application for it (be it direct .NET, ASP.NET (w/ whatever front end (bootstrap/angular/etc)), JAVA (shutters), and whatever… so I could go on here but I’ll stop.

I’m personally not going to test a whole bunch of different solutions, but instead pull a bit of a fan boy and cover just Veeam. I came from using Backup Exec (which is now the hot potato of Backup Software, since it almost destroyed Symantec)… anyway, to using Veeam, and it was a breath of fresh air, not only do they have amazing support staff you know what they are doing (usually if you get in the higher tiers), but they also have a great form site with a good following and replies by the developers themselves. You also don’t need to sign up to read them if you need to find a solution to a problem in a pinch, they don’t mind airing out any dirty laundry cause more often then not it’s not directly their fault but the APIs they rely on. Anyway moving on.

Getting the Installation Media

To start go here to grab Veeam Free Backup. This requires a login, I can only assume to avoid Captcha, or other mechanism to prevent DDOS or annoyances, as well as information gathering. Feel free to use fake information for this.

Now Veeam can only be installed on Windows, see here for all the detailed specs.

I’ll choose Windows Server 2016 Datacenter as I have it available with my MSDN for all my educational needs. 😀

So at this point we have:

  1. A supported OS installed physical or virtual (i prefer virtual specially for labs)
  2. A Copy of the latest version of Veeam free
  3. A hypervisor (Hyper-V or ESXi) with VMs

*If you are looking to backup physical machines liek desktops and laptops look at Veeam’s agent options, Veeam Windows agent and Linux agent allow to backup physical machines.

Running the Installation Media

After updates it’s finally time to mount that ISO! In my case I had downloaded it on my workstation machine running Windows with the vSphere phat client, so I mounted it via the vSphere option to mount a local ISO to the VM. After mounting, and double clicking the installation executable, you are presented with this:

The EULA

Ooo, ahhhhh, click install…. and accept the EULA

Licensing (Free)

You will be present with this license part of the wizard, but as the text at the bottom indicates, click next without this to use free mode… wow how intuitive, no radio buttons, or check boxes… just simple intuitive wizard design…. would you just look at that… a thing of beauty. Click Next.

I was good with an all-in-one so I left the defaults, click next,

Dependencies

What is this? A clear, concise dependency check! And here I thought I could trick them by not installing things and see how it go, they seem to have done a good job covering their bases… and what is this?! and install button… you mean… I don’t have a vague link to a KB with some random technical blabber that links me to an executable to install before having to re run the wizard…. well lets see if it even works… Click Install… (Assuming internet connection; which this server does have, as how I got it updated)

Kool…

What is this?! no way…. it installed everything for me… and I didn’t have to reboot or re-run the wizard. Get out of town!; and click next.

Install location and verification

Again I’m OK with the defaults, click Install.

Let it install (it will use MS SQL Express (which is free up to 10 GB DB’s).

There’s a saying that goes “waiting is the hardest part”, thankfully with Veeam, this seems to be the case. Be patient while the installation completes, you’ll be glad you did. 🙂

Alright finally…

Click Finish, Now that was easy.

Click Restart.

Summary

That’s it! That’s all there is to it, the smoothest installation I’ve ever done, so smooth it doesn’t actually warrant it’s own blog post. But what the heck…

In Part 2 I’ll cover some basic configurations, and backup our first VM!

Creating and Managing Local Users ESXi 6.5/6.7

The Story

I recently started playing around with the later ESXi hypervisor (OK I’ve tried the Web UI before, and simply stayed away). Now it has been far more polished with the release of 6.7. I have been enjoying the experience a far amount. However, then I needed to create another account on my free host (since I do not have vCenter to play around with in my home lab). While most things a seemed pretty intuitive at first..

Creating a User

Host -> Manage -> Security & Users -> Add a User (Specify Username and Password)

If you actually tried to login at this point I’d laugh a bit, but it could happen, you just created a user account, right? Well first thing you should have noticed is that there was no options to define what permissions this newly created user should have, read-only?, administrator?, etc.

So you click on Roles, there are all the nice pre-created roles… mhmm nice… alright… so… how do I map a user to a role?

You can look all under Security and users (where it should be), heck you can even look all under all the Manage options… you won’t find it there either… I had to find this out by googling… and if I have to google it… it’s not intuitive…

Assigning the Roles

So click on the main host icon in the left nav area, then when the right pane has loaded, select the Action menu, you should see it on the list of options above the host, right next to refresh.

Then select “Permissions”.

When the Host’s permissions modal box appears, click add user.

Marvel at how you can now assign users to roles, from here instead of the logical place where you easily found creating the user. Even if they wanted to keep the actions menu, and the modal box, just create a dang link under Security and Users… Arrrrggg.

Resetting Local Admin Password in Windows
Bypassing Windows Login

I’m getting ready soon to do a presentation on hacking my works laptops. I was giving the green light on a spare laptop we had purchased for corporate use. So in this case my test bed was a HP Folio 9470m, decent little guy for most basic office work. Like most places we run Windows, staying with the latest updates it was configured for Windows 10.

I won’t get into to much technical details as I’ll save that for the Long Con coming up this November. However, like most security and hacks there are many layers involved and the windows login just happens to be one of them.

Now I am already an admin on these systems, however I assume the role of a perpetrator and choose to find ways to break in as if I was not a current admin. This is where my mind got blown!

Alright, so the basic thing you need to break into windows is…. Windows! lol, well ok, you can actually pull this trick off with other… (Yup, just tested with lubuntu 17 against my Windows 7 imaged laptop)… OSes, but in this case we’ll stick with using your windows install media, as you likely already have this on hand in whatever form you may need it (CD/DVD, USB… or ughhhh unno whatever).

Once in the System32 Directory simply run the following:

move Utilman.exe Utilman.exe.bak

copy cmd.exe Utilman.exe

That is pretty much it, reboot. When you are prompted in login, press the Windows Key + U and marvel at how you get a elevated command prompt, then type lusrmgr in the command window to get the local users and groups windows and alter account, create accounts, change passwords to your hearts content.

Remember the bare minimum you need is any OS that can read NTFS (Usually for most Windows installations) and physical access to the system you are attempting to get into (given the boot options are not locked down by the UEFI/BIOS) which is the next layer I’ll talk about in my next blog post.

If you need a bit more “hold my hand” guidance in pulling this off here is a good source. This of course was using the Windows 7 installation media, against a Windows 7 machine, but the general concept of the trick, replacing UtilMan.exe with the cmd.exe can be done in many ways, then when the Windows image boots and you are at the login screen literally clicking the accessibility icon, or CTRL + U will open an elevated command prompt.

Util next time… Keep fit and have fun. 😛

*NOTE* this trick still works even with Core edition,

As stated by Tiji from here:

“Or in short (a little bit more simple):
– Boot From Recovery CD
– Press “F10″
– Type ‘copy D:\Windows\System32\cmd.exe D:\Windows\System32\utilman.exe” (replace driveletter)
– Reboot and press the utility manager symbol (Or press Windows Key + U)
– Type ‘net user Administrator *’
– Enter the password
Done”

App Pool Crash on First Load

I’ll keep this on brief; for real this time.

So you created a MSA/gMSA for your Dev to use on ASP.NET.

You granted it Logon as a service rights, as well as batch logon right via group nesting in IIS_USRS group. You granted it all proper permissions on the physical path that IIS is using for the Site/App Pool, as well as any Database permissions if applicable. Yet every time you attempt to navigate the site you get a “503; Service unavailable” and when you go to check the app pool you find it is down. Right click it, select start and it comes right back up without issue, wash, rinse, repeat.

Turns out this happens cause you didn’t fully qualify the MSA/gMSA under the App Pool’s Identity settings. Even though you enter “gMSAAcct$” under the identity field and leave password fields blank, and IIS accepts this… without fault, what I believe is happening here is even though the check IIS has in place, does validate this to a be a real domain account, or service account, it doesn’t prepend or append (depending on which user construct you want to refer to) where ever it stores this user account. This is only a guess.

So you have to fully qualify it; “Domain\gMSAAcct$” You’ll notice it (IIS) will accept it just like it did before. Then watch in amazement as the page loads and doesn’t crash when you attempt to load it in a browser….

Just another Mon…. Tuesday

The Start

Nothing new or exciting to the start of my day, clean house! Now I was actually cleaning my office, not making system changes 😉 Then…. Monday happened, I mean it’s really Tuesday but it was my first day back after the weekend (The first weekend I finally didn’t have to do any system changes)… life’s good right?

I made my first talks with our DBA, and then followed up with our Developer. Our current Developer is one really hard working and amazing dev, so he needed a couple things from my to move his current project forward; a CNAME record, along with a gMSA. This is one of the many reasons I like this guy (not only understands security posture, but what is needed for it to all work!). So the first one took seconds, and the second a couple more (besides the fact I need to reboot the server, cause I choose to do IDLGA instead of granting the computer account direct permissions to retrieve the gMSA’s password)… Yes… yes I’m aware of the special klist purge command to clear kerberos tickets, but I wanted to be 100% sure. Then came the problem…

The Problem

I don’t want to get too into the nitty gritty, but the gist of it was we had an authoritative source of data that resided on an older SQL server, our Devs new project was in a whole new data center, utilizing a whole new database server.

Since we didn’t want to alter any firewall rules between the datacenters (while they are all in house and owned by the same company, the two data centers are still wall gardened off, with a 2 way trust created for most authentication purposes) this would mean either:

A) I have to allow the old SQL server to do LDAP queries against my new Datacenters DC’s. (I wasn’t in the mood for architecture changes, which I already stated so this was last resort) Then grant the new datacenters gMSA account permissions on the database.

B) Figure out a way to utilize two different accounts, to make two different source data calls, from the same App/code.

Now I like the sound of B cause lets face it, it puts all the work on the Dev and not me. (If this sounds Dilbertish…. cause it is :P) At this point I was pretty confident that this was possible… I mean… why not? Well a couple seconds later my Dev comes back and tells me that it is in fact not possible…. well sort of not possible… it’s not possible for our exact case…. for reals… let me explain, first off I’m talking about ASP.NET, second of all I’m talking about 2 different connection strings to a Database. For some references we both found, like this and this, and this, and this and even this … ok that’s a fair amount of reading (sadly I still couldn’t find one of the original sources haha) but in each case you are probably wondering (How do I specify the user name and password for an alternative connection string when using Windows Auth instead of SQL auth)…. Drum Rolll…………..

YOU CAN’T …… TADA

So…..

The Second Problem

This lead us to our second problem, while the new SQL instance was already configured for mixed auth (This means it allows Windows as well as local SQL server authentication to be permitted), our old SQL server instance…. not so much. As much as I wanted to avoid infrastructure changes, seems it was inevitable…. so I asked my DBA if this would be a problem, since you can change the auth mode on any given instance and not all instances on the server, figured this was a quick a easy solution; Enable Mixed Auth mode, re-start the instance, create a local SQL account to be hard-coded and used by the App until the source data can be properly relocated (Thus removing any hard coded garbage in the app). Alright! until….. Ughhhhhh…

The Third Problem

When my DBA went to restart the instance, he decided to use SSMS remotely (now there is nothing wrong with this, I didn’t know it was even possible and was excited to learn something new… until…) the service failed to come up successfully (Ohh boy here we go), so sure enough we get into fix mode. My DBA jumps right into Event Viewer (good man) and discovers the first error stating the service was unable to bind to the port as it is in use (DBA opens SQL Config Manager, and Services.msc and sees both services and not running). This however instantly told me one thing… the service didn’t stop properly, event though Services.msc showed no signs of running, Task Manager and Tasklist, showed otherwise. Here’s the kickers, every attempt to force stop the process (service instance sqlserver.exe) reported back “Access Denied” even running psexec (Love you Mark!!) as SYSTEM still reported “Access Denied”

The Fix

At this point I basically figured that we had to reboot the server (I also assumed it would get stuck shutting down at the “stopping service” stage, but amazingly it did not!) Sure enough after the reboot everything came up without a hitch and the new Mixed Auth mode was enabled for our Dev’s alternate ASP.NET connection string! OK I know this sounds pretty crappy for a solution but honestly it was the only thing we had left in our toolbox, and it fixed both problem one (Mixed Auth mode is now enabled for old instance) and the fact the instance came up without a problem.

Until…..

While we (DBA, Dev, and Myself(SysAdmin)) continued to test our other applications that were built via other means, it seemed a couple things were broken, this ones a little bit funny cause we assumed there was an issue for everyone, however turned out only to be an issue for the DBA and Dev, not myself (but I wasn’t on my local machine to do any front end testing from my account) so let me explain, The Dev kept digging into the real nitty gritty of the code, jumping all the way into the backend of the SQL’s stored procedures and views, to discover there was empty values being returned (Now I have no clue if this was always an issue (based on the fix) or if it actually was due to something else….anyway) turns out one of the built in views it used as a source to create a temp table was returning null thus throwing the error when calling one of the stored procedures. When the Dev and I went to talk to the DBA in the lunch room, we had discussed some of the permission changes we had just implemented on the Security Logins of the instance, and made some assumptions, so I went into the back-end AD groups to validate somethings, sure enough it was a little funny in that due to the fact their stand accounts had direct logins for the instance (generally not a fan of this, as I love scalable design and prefer to utilize IDGLA) so my DBA told me he had fixed this, and then he told me something I never would have expected and is a huge learning experience for me:

WHEN YOU GRANT AN ACCOUNT THE “SYSADMIN” ROLE/PERMISSION THE OTHER ROLES IN WHICH THE ACCOUNT IS A MEMBER DOES NOT APPLY PROPERLY (IS BYPASSED OR SOMETHING).

Literally, so what happened was there was a group we have defined to be granted sysadmin rights on the server (to manage them, not manipulate data) normally this contains admin based accounts (we all do standard account and admin accounts for least privileged best practice right :). However their admin accounts and their standard accounts were in there, which I removed, and once that was corrected and the proper nested grounds their standard accounts where suppose to get based on other roles, then applied properly and the issue was fixed.

Party in the House… Until….

Yes… believe it or not my day does not end here…. there was simply more information the great world of IT had to shove down my tiny brain that’s already overloaded and overwhelmed at the pure magnitude of knowledge you need to manage systems!! WHYYYYYY! GOD WHYYY!!!!

Anyway…. so to end the day we get a unique error message from one of our workflows, and sure enough another email from an external user providing a snipping of an error (How nice of them). Is this a coincidence…. not a chance 100% related… So again most of the heavy lifting is done by our Dev (this guy….. he’s a super star!) He managed to break it down to an assembly problem… but we were shocked as to how this could be (we checked everything was working after all our above fixes)… well until our DBA made a confession (He wasn’t happy to have found out his own account was the DB owner of a fair share of DB’s within the instance) so he secretly clean it up… well after some trial and error (reverting the change a couple times) it turned out that the error “error the server may be running out of resources or the assembly may not be trusted with permission_set external_access or unsafe” was simply due to a single missing permission needing to be granted to the new DB owner account:

In SSMS -> Instance -> Logins -> Account -> Properties -> Securables -> Check Grant for Unsafe Assemblies

Sometimes… you just gotta run unsafe code 😛

Alright! Home Time!