Category: Server Administration

Posted on

WordPress: Error Establishing a Connection to the Database

This will be a short one, as I didn’t take screen shots, and I didn’t have to do much to resolve it. Just wanted to make note of it.

The other day I wanted to check my own site and instead of loading I got the message “Error Establishing a Connection to the Database”.

I applied my usual fix first; reboot. While the VM was rebooting it appeared there had been some disk corruption? the automatic fsck failed stating a manual fsck was required.

so…

fsck /dev/sda1

and a bunch of “errors” and fix?<y>

after a bunch of answering yes, it stated the disk was repaired successfully.

after this I typed “exit” or “return”, and the system rebooted like normal. Lucky for me the WordPress site came up clean after that. However even had this had failed, we all have backups right?

Posted on

Exchange: Something Went Wrong

Fixing Exchange

Now, I’ve taken a couple Exchange courses. They cover all the bases… expect when things go wrong. That’s why it’s nice to have labs… today in my Lab I discovered I was unable to get email from my exchange server, neither from activeSync nor Outlook Web App (OWA).

Something went wrong alright… first thing I noticed was my disk had run out of space… whoops. Hahaha. Expand the drive, reboot and… Something Went Wrong…

Sigh…. alright event viewer… what ya got for me…

Unable to mount…. I guess it didn’t like what happened to the DB after the disk ran out of space… some quick googling (1 and 2 copy cats… and can’t even tell you the DB file locations…) and one more, more personal blog post.

Exchange Default DB File Locations

If you are using Exchange Server 2000 & 2003, you can locate your EDB files at:

C:\Program Files\Exchsrvr\MDBDATA\Priv1.edb
C:\Program Files\Exchsrvr\MDBDATA\Pub1.edb

If you are using Exchange Server 2007, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Mailbox Database.edb
C:\Program\Files\Microsoft\Exchange Server\Mailbox\First Storage Group\Public Folder Database.edb

If you are using Exchange Server 2010, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V14\Mailbox Database\Mailbox Database.edb
C:\Program Files\Microsoft\Exchange Server\V14\Public Folder Database\Public Folder Database.edb

If you are using Exchange Server 2013, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox database Name\Mailbox database Name.edb

If you are using Exchange Server 2016, you can locate your EDB files at:

C:\Program Files\Microsoft\Exchange Server\V15\Mailbox\Mailbox Database Name.edb

Repairing the Exchange Mailbox Database

Which they all tell you to use a cool old “tool” eseutil. which seemed straight forward, ensure you run an elevated cmd or you won’t have access to the directory path of the exchange DB file. In my case I used the Exchange 2013 path which was the version used in my lab.

I also moved the log files:

move *.log c:\temp

Yeah… that took a lil while.

Mounting the Exchange Mailbox Database

Once it’s repaired used Exchange Mgmt Shell to mount it:

Whoops, silly me, since everyone said to stop the information store service I did… so after starting the service, and rerunning the command it succeeded.

Which resulted in:

Much better!

Summary

  1. Check the source of the Database Corruption. (Mine was Disk Space)
  2. Stop the Information Store Service
  3. Check the Validity of the Mailbox Database (eseutil /mh)
  4. Repair if required (eseutil /p)
  5. Restart the Exchange Information Store Service
  6. Mount the Mailbox Database

Hope this helps someone.

Posted on

Windows MCS and MPIO

I was configuring some iSCSI disk on a Windows server and noticed there were two different options available that seem to provide similar functionality and I had to know… What’s the difference?

Source

THE DIFFERENCE BETWEEN MCS AND MPIO (IN A NUTSHELL):

First we agree upon the common features within both technologies: both serve a multipathing for (iSCSI) I/O-operations utilizing multiple hardware (or OSI Level 1) components, such as Ethernet NICs or iSCSI HBAs. The purpose of multipathing is redundancy and aggregation – how this is implemented depends on the above depicted figures, i.e. through the decision which paths are active and which are passive (or standby, using Microsoft parlance). For the exact definition of the policies, such as round robin, weighted path, fail over only, etc. please refer to “MS iSCSI UG”, p. 41.

Finally here come the condensed definitions for both technologies:

MCS allows the initiator to establish multiple TCP/IP connections to the same target within the same iSCSI session.

MPIO in contrast allows the initiator to establish multiple iSCSI sessions (each having single TCP/IP connection) to the same target, effectively aggregating the duplicate devices into a single device.

If you are not familiar with the terminology (initiator, target, session, connection, initiator port and network portal) please refer to “Multivendor Post” which provides very informative sketches to the iSCSI network architecture.

Now that we know that MCS means effectively several connections within a session and MPIO means multiple sessions the question is when to use what. Mainly you will have to concentrate on two perspectives – vendor support and load balance policy inheritance. The question – or rather schools of thought – about the speed and performance differences are factored out here, because in the author’s opinion these are almost equal and you will probably never get to the point of fully utilizing them. With this said consider the following simple rule of thumb: you can use MCS only when it is supported from the vendor’s SAN and you are not using hardware iSCSI HBAs. In any other case use MPIO. The second thought is – if considering the above conditions you are able to use MCS, but want to apply different load balancing policies to different targets (and effectively LUNs or groups of LUNs) you will still be better off using MPIO. This is because load balancing policies are session adherent. In other words when you are applying policy to MCS it is for the whole session, no matter how many connections are aggregated “beneath” it. On the other side when using MPIO you can set different policies for different LUNs, because the multipathing is using different iSCSI sessions.

 

I’m still trying to wrap my hear around exactly what the source is getting at. But will update the blog when I do some more testing.

Posted on

HPE SSD Firmware Bug (Critical)

I’m just gonne leave this right here…..

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00092491en_us

I wonder who they outsourced the firmware code out to back in 2015….

”

IMPORTANT: This HPD8 firmware is considered a critical fix and is required to address the issue detailed below. HPE strongly recommends immediate application of this critical fix. Neglecting to update to SSD Firmware Version HPD8 will result in drive failure and data loss at 32,768 hours of operation and require restoration of data from backup in non-fault tolerance, such as RAID 0 and in fault tolerance RAID mode if more drives fail than what is supported by the fault tolerance RAID mode logical drive. By disregarding this notification and not performing the recommended resolution, the customer accepts the risk of incurring future related errors.

HPE was notified by a Solid State Drive (SSD) manufacturer of a firmware defect affecting certain SAS SSD models (reference the table below) used in a number of HPE server and storage products (i.e., HPE ProLiant, Synergy, Apollo, JBOD D3xxx, D6xxx, D8xxx, MSA, StoreVirtual 4335 and StoreVirtual 3200 are affected).

The issue affects SSDs with an HPE firmware version prior to HPD8 that results in SSD failure at 32,768 hours of operation (i.e., 3 years, 270 days 8 hours). After the SSD failure occurs, neither the SSD nor the data can be recovered. In addition, SSDs which were put into service at the same time will likely fail nearly simultaneously.

To determine total Power-on Hours via Smart Storage Administrator, refer to the link below:

Smart Storage Administrator (SSA) – Quick Guide to Determine SSD Uptime

”

Yeah you read that right, drive failure after a specific number of run hours. Yeah total drive failure, if anyone running a storage unit with these disks, it can all implode at once with full data loss. Everyone has backups on alternative disks right?

Lesson and review of today is. Double check your disks and any storage units you are using for age, and accept risks accordingly. Also ensure you have backups, as well as TEST them.

Another lesson I discovered is depending on the VM version created will depend which ESXi host it can technically be created on. While this is a “DUH” thing to say, it’s not so obvious when you restore a VM using Veeam and Veeam doesn’t code to tell you the most duh thing ever. Instead the recovery wizard will walk right through to the end and then give you a generic error message “Processing configuration error: The operation is not allowed in the current state.” which didn’t help much until I stumbled across this veeam form post…

and the great Gostev himself finishes the post with…

by Gostev » Aug 23, 2019 5:52 pm

“According to the last post, the solution seems to be to ensure that the target ESXi host version supports virtual hardware version of the source VM.”

That’s kool…. or about… why doesn’t Veeam check this for you?!?!?!
Once I realized what the problem was, I simply restored the VM with a new name on the same host it was backed up from (Which was on a 6.5 ESXi host) and I was attempting to restore the VM on a 5.5 ESXi host. Again, after I realized I had created the VM under the options that I picked a higher VM level allowing it only to be used with higher versions of ESXi it was like again… “DUHHH” but then it made me think, why isn’t the software coded to check for such an obvious pre-requisite?
Whatever nothings perfect
Posted on

Getting A+ Qualys Report

As some of you may know you can validate the security strength of your HTTPS secured website using https://www.ssllabs.com/ssltest/index.html

A good read on Perfect Forward secrecy

I use HA Proxy with Lets Encrypt for my sites security. While setting up those to plugins to work together apparently by default it’s not using the most secure suites ok the dev shows how you can adjust accordingly… but which ones? This what I get by default:

Phhh only a B, lets get secure here.

Little more searching I find the base ssl suites from mozilla config generator

which gave me this for the string of suites

ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

But then ssllab report still complained about weak DH… so had to remove the final two options in the list leaving me with this

ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305

Now after applying the setting on the listener I get this!

Mhmmm yeah! A+ baby but looks like some poor saps may not be able to see my site:

Too bad so sad for IE on older OS’s, same with iOS (Macs) running older Safari.

Now let’s tackle DNS CAA well I was going to discuss how to set this up, but the linked site covers it well. Since my external DNS provider was listed in the supported providers, I logged into my providers portal to manage my DNS, and sure enough the wizard was straight forward to grant Lets Encrypt authority to sign my certificates! Finally one that was actually really easy! Wooo!

Now I suppose I can eventually play with experimental TLS1.3 but I’ll save that for another post! Cheers!

 

Posted on

Mitigating from CVE-2018-3646 on ESXi 6.7

To keep this short, new VCSA 6.7 has VUM built in. No more Flash needed. Yay finally.

So I upload the latest 6.7u3 image, create my baseline, and test remedy one of my simple laptop hosts. After system reboots and comes back on VCSA dashbaord… uhhh what’s with this yellow warning icon…. Summary…

OK great, so after years of Intel being ahead of AMD, looks like at the cost of some pretty shitty shortcuts. and these shortcuts have caused Intel a huge problem, and pretty everyone else. Since it affected everyone, everyone has some form of write up on it. In this case VMware has coded the above warning, with a reference to this KB, so you know read that if you want a dry overview.

As you can see I have what shows as 4 logical processors, but after applying the mitigation (setting VMkernel.Boot.hyperthreadingMitigation to true on the host advanced settings) and rebooting…

Yay the warnings gone, but apparently so are half my logical processors?

If your wondering why they didn’t enable this by default is due system resource management, which of course, is exactly what vSphere is. Since it affects the available resource of the host it may not be able to accommodate the workload it was originally designed for. In my case it’s a lab and my work load is obviously very light, and this isn’t an issue for me.

Was it worth the mitigation? I don’t really know at this point as I’m unaware of any easy simple tactics any attacker could use to attack my footprint. At the same type CPU resource is not my major constraint, it’s usually memory.

For now better safe than sorry. In my next post I hope to cover the vCenter upgrade path and an error that happened to me along the way, luckily it wasn’t that hard to recover from. 🙂

Cheers!

Posted on

ESXi 6.5 Stuck on Initializing Scheduler
PSOD PCPU1 could not start

I’m making this post short to note this odd experience with this host build.

First weird thing was when trying to install ESXi I couldn’t get past vmkusb not sure what it was about but only found this decent reddit post with the same problem.

In short he noticed it would only get past this if a second USB was plugged into the USB2 ports, and sure enough that worked for me too. strange….

Then a couple days later while doing some more test boots, I get a Purple screen of death, complaining about the PCPU1 not starting or some shit, ugh again lets see what others have to say… well I found this vmware thread on it, he basically stated that resetting the BIOS settings worked, after farting around with some bios settings, I had other failed boots and my CPU and system was rather hot. I let everything cool down, added more powerful fans and tried again after resetting the BIOS to factory and much like the post it worked.

After finalizing the build a little more, I switched to an old 2.5″ HDD.. same problem but I noticed it gets stuck on initializing scheduler before PSODing, while I searched for what might be up with that I found this…

It did help shit, same problem next boot, instead of just resetting the BIOS I played around with a couple more settings like I enabled intel AES-NI which helps for CPU offloading of AES computations. and another one which I sadly forgot, and then my next boot was fine. saving this at this point in case it comes back again.

Posted on

Upgrading my ASUS RT-N16

The ASUS RT-N16

I love this thing, I remember when I first read my first blog posts about the specs, and what it could all do…

Wireless

Wireless Frequency Bands 2.4 GHz
Number of Antennas 3
WLAN Mode 802.11n
Transmit Power 15.5 to 19.5 dBm
Antenna Placement External

Interface

Ports 1 x Ethernet (RJ45) (Uplink)
4 x 10/100/1000 Mb/s Gigabit Ethernet (RJ45)
2 x 480 Mb/s USB Type-A

Performance

Throughput 300 Mb/s
CPU 480 MHz Broadcom SoC
128 MB RAM
32 MB Flash

Security

Wireless Security WEP, WPA, WPA2

Those are some good specs for 2010, pretty much a decade ago. and most of the blogs touted DD-WRT, which I joined the form site way back in 2012… looking back at my old posts didn’t seem to get much of any help… but sure had oddities I was recently running a KongMod of DDWRT (build 22000M) Circa 2014, looking it up found out he stopped to made modded firmware for OpenWRT. I grabbed the latest DDWRT for my router using the DDWRT database factory reset settings, cleared NVRam and used IE with a windows laptop with static IP bound to port 1 on the router….

Soft Brick?

Gave the system enough time to boot, but noticed the pings were not coming back up, the Power light would flicker during boot and then stay off, while the wireless LED said lit.

I thought I may have soft bricked it, so I grabbed the stock firmware and flashing tool from ASUS website to my dismay even though I could press the restore button and have the power LED blink slowly indicating it ready for TFTP file, even the flashing tool would fail either that its not in flashing mode, or faiiled to flash. I thought I was hooped in this case and was in a soft lock loop, and thought I would have to JTAG flash it…

Then for shits (since the WiFi LED was on) I wondered if it was broadcasting… and when I checked for a available WiFi on my phone I was shocked to see it was, I connected, shocked again, and could ping the router… wait what??

Solution?

Sure enough I could see the DDWRT web interface.. I was stumped and started to Google, but only found one post that was dead on… but as I figured the solution provided did not work for me, well the vlan1 check setting BS.

There was another bunch of posts stating to add commands “swconfig dev eth0 set enable_vlan 1” or some crap, yeah that didn’t work either. Even though people said don’t do it, I decided to use the DDWRT web interface Firmware Update section over WiFi (either was would have to JTAG flash if it failed) So at first I simply used the K2.6 Mini build instead of the mega, after the flash the exact same shit, but the power light at least stayed on. but again could only connect via WiFi. Since the only other answer was “I flashed a newer firmware” which is a timeless statement lol which exact version who knows, and I sadly didn’t have the old Kong build if I simply wanted to go back.

AdvancedTomato FTW

I was about to try OpenWRT when I decided to look at Tomato again… so flashed it via the DDWRT firmware update section (Fuck you DDWRT) and to my amazement it came up perfectly, Wifi was fine, and I could ping it on a physical LAN port again. Woooo!

Since I wasn’t used to the interface I did need a bit of a hand getting it setup as s simple AP again, guess it makes sense DHCP is set at the bridge so if you want to setup different NICs for different subnets and still have their own DHCP, but in my case I wanted none.

Then I read this nice post by How-To-Geek on configuring traffic monitoring, something I never had on DDWRT, so not only is the new UI a fresh change, so are some of the features. I really hope also a lot less bugs. Cause DDWRT with OTRW was buggy and a HUGE PITA.

Optware?

Well googling did show there was the possibility… and installation seemed straight forwarded enough… of course both guides being 8+ years old, wasn’t too compelling, so I checked if the source referenced script was still accessible… and it was! Nice, checking the script out I see another external reference source and check it out too, amazingly it’s still reference-able too.

So I followed along, starting by first attempting to create a partition (512 MB, labeled “optware” as ext2) I did this by USB pass-through of my USB stick to a Mint Linux VM. Then simply using gparted created my partition, I also created a 1 Gig, and 2.5 Gig ext 2 partition labeled whatever with the spare space. (I tried a 4 Gig partition, but… it failed to mount so stuck with the recommendations).

Ran the installation as suggested…

wget http://tomatousb.org/local--files/tut:optware-installation/optware-install.sh -O - | tr -d '\r' > /tmp/optware-install.sh
chmod +x /tmp/optware-install.sh
sh /tmp/optware-install.sh

I did this of course after verifying that indeed my partition was mounted as /opt, and the script ran without issue.. .amazing…

after that, I first installed htop, cause lets face it, normal top sucks…

ipkg install htop

I followed this up with the main packages I actually used, screen and irssi. This allows me to have a persistent IRC chat client (given the AP/Router doesn’t reboot)

ipkg install screen
ipkg install irssi

Add User?

Now I remember specifically having issues with DDWRT, and adding standard users with limited permissions. Specifically with the name showing up weird…

So I searched quickly to see if it was possible, and if so how people were doing it…

much like the guy in the first link, I didn’t quite follow what was going on and then after checking each line, eventually it made sense. (Basically defining specific environment variables, and special actual files with embedded lines that are all saved to NVRAM, then a script (3 lines) is run to populate the linux user list)

I did add “adduser” but much like mentioned elsewhere it would complain about not having “passwd”, there was no packages for “mkpasswd”, or “makepasswd”. I wasn’t in the mood to change my root system password and run a single stupid line to set one users password … :S (

sed -n -e "s,^root:,$UNAM:,p" < /etc/shadow >> /etc/shadow.custom

)

Instead much like the alternative suggestion on the page itself “You can also cut & paste passwd and shadow entries from another linux box.” which is exactly what I did, using my Linux Mint VM, I used openssl passwd with a salt to generate a MD5 hashed password.

Now I was able to SSH in with my new non root account, YAY!

Now according to the source “These commands need only be done once for each custom username. Thereafter, the user will always be created every time the router boots up. To delete a user, edit /etc/passwd.custom and /etc/group.custom and delete the line with that username, then save them to nvram.”

OK…. I’m going to reboot now…

mhmmm is it going to work….? Oooo… account is there after reboot in passwd file… and line exists for account in shadow file, and home dir exists… lets log in… well shit… the password didn’t save… still same as root even though they differ in the shadow file… k… let’s make a new one… save in shadow file, relogin, yup password changed. and now change in shadow.custom and save, and reboot…. arrrrggggg C’MON

Maybe you have to run that setfile commands when changing a file set to nvram? second try… there we go! success.

Screen, Irssi and the fun Stuff

It’s been a while since I had to reconfigure this stuff so someone’s blog to help me along the way, and it’s rather old now… but still good stuff and this simple one

and then run irssi 😀 (by typing irssi and hitting enter)

Silly Rabbit Trix are for… I mean Irssi I’m following a guide already…

/network list

to make adding to Freenode easier instead of having to type /connect irc.freenode.net I’m going to setup a reference much like the existing reference from the above command:

The above shows names, but not there DNS lookups which is the server list

To add our reference:

/server add -auto -network Freenode irc.freenode.net

Now that we have an auto connecting server, we’d like to specify the user and login details:

/network add -nick Zew -autosendcmd "/msg nickserv IDENTIFY *******" Freenode

Now that my usual helpful sources are added (you can always catch me in one of these places) let’s test it all out… run /quit and then irssi again. Which worked! I was joined to my server, authed and joined to my channels 😀

Use Ctrl + X to switch connected servers, Esc + left or Right to move channels.

and finally “Ctrl + A then Ctrl + D . Doing this will detach you from the screen session which you can later resume by doing screen -r .”

See you on IRC!

 

Posted on

VMware Host Oddities

I’ll keep this post short as I find it rather interesting turn of events…

I was ensuring config files were being backed up as good Sysadmin Posture would suggest. I noticed an issue on one of the hosts when I went to run the configuration backup. When I was hit with a “general system error”. I Wish I would have saved the actual error message but when I went to research it, it gave me more people complaining about the error when trying to vMotion (In this case I grabbed the most specific line from the call back in hopes to get something) most results simply pointed to saying restarted the ESXi hosts management agents. I decided to see if I could vMotion or if any other symptoms from the host, but everything showed green in vCenter, and vMotions worked without an issue.

I left it for the night (before) and decided to update the host since it was running 6.5-u2 and needed to be updated to 6.5-u3. I was hoping it would resolve the config backup issue (I had an older copy on hand but wanted most up to date) So I decided to do the update via a ISO image and a host reboot, moved VMs, no problem, placed host into Maintenance mode, no issues, send host for shutdown (I wasn’t using iLO or iDRAC, so needed to manually mount the USB hosting the installation media), while I was at the console waiting for the host to show “shutting down” it simply …. didn’t… after 10 minutes which is far more then generous I decided to press F2 to get into the console to have to tell me … uggghhh I wish I would have taken a picture of the error, something along the lines of you can’t use the console cause you been locked out…. it was weird, after waiting another 10 minutes and nothing happening (I’m assuming there must have maybe been some actual underlying issue with the datastore holding the scratch?) I decided to just do a hard shutdown (hold power for 5 seconds). I could rebuild this server faster if need be. Powering it back up was perfectly fine through all POST checks, booted the ESXi 6.5u3 installer, booted fined, checked for logical drives, found all of them perfectly fine including the SD card holding the OS (scratch changed to a persistent location, another datastore on the local host with multiple disks used for a datastore) anyway I selected it and it saw there was an existing installation and selected to update it, successful, reboot. reboots fine, but host doesn’t come back to vCenter…..

Log into host directly, and notice it states it has 18 VMs, all with a status of error, now I had moved them and they are even still running on another host (thankfully it didn’t do anything stupid to try and hijack them) at this point I had called VMware support and placed an SR. Once I had a tech on the line, I discussed my symptoms and issues. At this point I asked what was the best course of action as, I didn’t want to rebuild the host (I could, but time). In this case I simply un-registered all the VMs that were in error as they were not associated with the host), then attempted to re-connect the host. It first failed complaining about bad username and password (I’m not sure if this was the root account used to add it, or the VPX user it uses to manage the host) but it prompted the wizard like adding a new host, since all other settings were still fine on the host (network and vswitches w/ VMPGs and VMKs, and all Datatstores) after this wizard the host was re-added back into the cluster with the latest updates, and running the backup config command:

esxcli storage vmfs extent list

worked without error. So yay! all fixed, but one thing still bothered me… what was the root cause…

I decided to check my scratch settings real quick, and sure enough it hold good and is on redundant datastore on the local ESXi host, so not sure what the root cause was, but it was fixed. I’m posting this for my own reference. Just incase this issue re-arises.

 

Legal Information Proudly powered by WordPress