VMware ESXi boot and the Config

Sadly this post will be really short as again, lots going on. Recovering a host that failed after a regular reboot, which had a superblock corruption on it’s main OS drive. Also, the BELK series will be done, I just need a bit more time. Sorry for the delays.

“Failed to load /sb.v00” [Inconsistent Data]

Since this drive was not on the main datastore on the host all the VMs were unaffected.

Now loading linux showed the drive data was till accessible, but I also had a feeling this USB drive was on it’s way out. I created a copy using DD, *sadly I didn’t do it the smart way and place it on a drive big enough to save it as a image file, but instead directly to another drive of the same size.

I tried to install the same image of ESXi on top of the current one in hopes it would fix the boot partition files along the way. This only made the host get past /sb.v00 and vault randomly past it with “Fatal Error: 6 [Buffer Too Small]”

I was pretty tired at this point since the server boot times are rather long and attempts were becoming tedious. I did another DD operation of the drive, to the same drive (still not having learned my lesson) and when I awoke to my dismay, it failed only transferring 5 gigs with an I/O error. This really made me sure the drive was on the way out, but it was still mountable (the boot partitions 5, 6 and 8)

At this point you might be wondering, why doesn’t he just re-install and reload a backup config? Which is fair question, however one was not on hand, but surely it must be somewhere on the drive. I know how to create and recover on a working host but a one that can’t boot? Then I found this gem.

Now through out my attempts I did discover the boot partitions to be 5 and 6 and I did even copy them from a new install to my copied version I made about and it did boot but was a stock config. I was stumped till I read the section from the above blog post on “How to recover config from a system that doesn’t boot”. Line 7 was what nailed it on the head for me:

“mount /dev/sda5 /mnt/sda5

7. In the /mnt/sda5 directory, you can find the state.tgz file that contains ESXi configuration. This directory (in which state.tgz is stored) is called /bootblank/ when an ESXi host is booted.”

I was just like … wat? That’s it. Grabbed the bad main drive mounted on a linux system, saw the state.tgz file and made a copy of it, connected the new drive that had a base ESXi config, replaced the state.tgz file with the one I copied, booted it and there was the host in full working state with all network configs and registered VMs and everything.

Not sure why the config is stored in the boot partition, but there you go. Huge Shout out to Michael Bose for his write I suggest you check it out. I have saved it case it disappears from the internet and I can re-publish it. For now just visit the link. 🙂

BELK Stack on Docker
(Part 2 – Docker Compose)

The Story Continues

Following on from the last post, today we cover docker-compose to allow for easier deployment of docker images and configurations. As from my previous post you may want to indulge in the same reading I did here.

Past those nice formalities, I find myself missing something… I’m not sure what it could be… oh yeah…. dependencies!

Installing Docker-Compose

Can I use apt-get?

Would seem like it… but

IT’s outdated… 😀

Other way is via pip or the intended way

Working with Docker-Compose

  • docker-compose ps lists all the services in a network. This is especially helpful when troubleshooting a service as it will give you the container ID and you can then run docker -it exec <ID> bash to enter the container and debug as needed.
  • docker-compose buildgenerates any needed images from custom Dockerfiles. It will not pull images from the Docker hub, only generate custom images.
  • docker-compose up brings up the network for the services to run in
  • docker-compose stopstops the network and saves the state of all the services
  • docker-compose start restarts the services and brings them back up with the state they had when they were stopped
  • docker-compose downburns the entire Docker network with fire. The network and all the services contained within are totally destroyed.

How to Docker-Compose?

The last big question is: how to write a docker-compose.yml, and it’s actually very easy and follows a standard formula.

Here is a template of what any docker-compose.yml will look like.

  • Sample Docker Compose Template
version: "2"
  services:
    <name_of_service>:
      build: <path_to_dockerfile>
OR
      image: <name_of_image:version>
      enviroment:
        - "ConfVar:value"
        - "homeDir:/home/dir"
      ports:
        - "[HostPort]:[ContainerPort]"
        - "80:80"
      volumes:
        - /path/container/will/use

Every docker-compose file will start with a minimum of version: "2", if you’re doing a Docker Swarm file it will need version: "3", but for a single docker-compose.yml, you’ll need v2.

See here for more on the use of volumes

I’m gonna keep this post short and use examples of these first two blogs it part 3. Where I setup and configure the first container in the BELK Stack; Elasticsearch.

See you all at part 3! 😀

BELK Stack on Docker
(Part 1 – Docker)

The Story

This time our goal to setup a SEIM (Security Event & Information Monitoring) which will gather data via the BELK Stak (Beat, Elasticsearch, Logstash and Kibana). This is going to take (I’m assuming, as I’ve just started) about 4-5 separate blog posts to get this off the ground.

It has taken me a couple weeks of smashing my head into a wall simply due to my own ignorance, so in this blog series I’m going to cover more step-by-step exactly what needs to be done for my particular setup. There are many ways you can configure services these days, which still includes bare metal. If I so chose I could run Docker on a bare metal Ubuntu server, or even a bare metal windows server, but in this case I’m going to install docker on a Ubuntu server which will happen to be itself a VM (Virtual Machine).

Now with that in mind, here’s some basic reading you probably should do before continuing on. Now before we go on let’s be clear on one thing, docker itself doesn’t run on magic, or fluffly rainbow clouds, as I mentioned in the paragraph above it runs on some system, whether that’s again bare metal or some VM of some kind [Think IaaS (Infrastructure as a Service)], in this blog it will be a Ubuntu VM. The specs of this machine should suffice for the application and workloads that are going to be created on it.

Dockerfile Commands

Below, are the commands that will be used 90% of the time when you’re writing Dockerfiles, and what they mean.

FROM — this initializes a new build stage and sets the Base Image for subsequent instructions. As such, a valid Dockerfile must start with a FROM instruction.

RUN — will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the Dockerfile.

ENV — sets the environment variable <key> to the value <value>. This value will be in the environment for all subsequent instructions in the build stage and can be replaced inline in many as well.

EXPOSE — informs Docker that the container listens on the specified network ports at runtime. You can specify whether the port listens on TCP or UDP, and the default is TCP if the protocol is not specified. This makes it possible for the host and the outside world to access the isolated Docker Container

VOLUME — creates a mount point with the specified name and marks it as holding externally mounted volumes from the native host or other containers.

You do not have to use every command. In fact, I am going to demonstrate a Dockerfile using only FROM, MAINTAINER, and RUN.

Images vs. Containers

The terms Docker image and Docker container are sometimes used interchangeably, but they shouldn’t be, they mean two different things.
Docker images are executable packages that include everything needed to run an application — the code, a runtime, libraries, environment variables, and configuration files.
Docker containers are a runtime instance of an image — what the image becomes in memory when executed (that is, an image with state, or a user process).

Examples of Docker containers. Each one comes from a specific Docker image.
In short, Docker images hold the snapshot of the Dockerfile, and the Docker container is a running implementation of a Docker image based on the instructions contained within that image.

This is true, however this image is a bit misleading as it’s missing the versioning which will become apparent a bit later on in this blog post.

Docker Engine Commands

Once the Dockerfile has been written the Docker image can be built and the Docker container can be run. All of this is taken care of by the Docker Engine that I covered briefly earlier.

A user can interact with the Docker Engine through the Docker CLI, which talks to the Docker REST API, which talks to the long-running Docker daemon process (the heart of the Docker Engine). Here’s an illustration below.

The CLI uses the Docker REST API to control or interact with the Docker daemon through scripting or direct CLI commands. Many other Docker applications use the underlying API and CLI as well.

Here are the commands you’ll be running from the command line the vast majority of the time you’re using individual Dockerfiles.

  • docker build — builds an image from a Dockerfile
  • docker images — displays all Docker images on that machine
  • docker run — starts container and runs any commands in that container
  • there’s multiple options that go along with docker run including
  • -p — allows you to specify ports in host and Docker container
  • -it—opens up an interactive terminal after the container starts running
  • -v — bind mount a volume to the container
  • -e — set environmental variables
  • -d — starts the container in daemon mode (it runs in a background process)
  • docker rmi — removes one or more images
  • docker rm — removes one or more containers
  • docker kill — kills one or more running containers
  • docker ps — displays a list of running containers
  • docker tag — tags the image with an alias that can be referenced later (good for versioning)
  • docker login — login to Docker registry

A big thank you to: Paige Niedringhaus for her contributions you can see most of this theory content was a direct copy paste, but not all the content just the basic relevant ones (in case the source material ever goes down).

Now that we got the theory out of the way, let’s get down to the practical fun!

Installing Docker

https://docs.docker.com/install/linux/docker-ce/ubuntu/

Uninstall old versions🔗

Older versions of Docker were called docker, docker.io, or docker-engine. If these are installed, uninstall them:

$ sudo apt-get remove docker docker-engine docker.io containerd runc

It’s OK if apt-get reports that none of these packages are installed.

Installing Dependencies

Don’t got non moving on…

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

apt-key fingerprint 0EBFCD88

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

Install Docker Engine – Community

Update the apt package index.

sudo apt-get update

Install the latest version of Docker Engine – Community and containerd, or go to the next step to install a specific version:

sudo apt-get install docker-ce docker-ce-cli containerd.io

Got multiple Docker repositories?

If you have multiple Docker repositories enabled, installing or updating without specifying a version in the apt-get install or apt-get update command always installs the highest possible version, which may not be appropriate for your stability needs.

To install a specific version of Docker Engine – Community, list the available versions in the repo, then select and install:

List the versions available in your repo:

apt-cache madison docker-ce
docker-ce | 5:18.09.1~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
docker-ce | 5:18.09.0~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
docker-ce | 18.06.1~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
docker-ce | 18.06.0~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
...
b. Install a specific version using the version string from the second column, for example, 5:18.09.1~3-0~ubuntu-xenial.
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

Verify that Docker Engine – Community is installed correctly by running the hello-world image.

sudo docker run hello-world

Woooo, what a lot of fun…. Just note one thing here…

Executing the Docker Command Without Sudo (Optional)

By default, the docker command can only be run the root user or by a user in the docker group, which is automatically created during Docker’s installation process. If you attempt to run the docker command without prefixing it with sudo or without being in the docker group, you’ll get an output like this:

docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
See 'docker run --help'.

If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group:

sudo usermod -aG docker ${USER}

To apply the new group membership, log out of the server and back in, or type the following:

su - ${USER}

You will be prompted to enter your user’s password to continue.

Confirm that your user is now added to the docker group by typing:

id -nG

If you need to add a user to the docker group that you’re not logged in as, declare that username explicitly using:

sudo usermod -aG docker username

The rest of this article assumes you are running the docker command as a user in the docker group. If you choose not to, please prepend the commands with sudo.

Let’s explore the docker command next. Thanks Brian

Creating your Dockerfile

The first thing we’re going to do is create a new directory to work within; so open a terminal window and issue the command as root…

 mkdir /dockerfiles
chown dadocker:docker /dockerfiles

Change into that newly created directory with the command

 cd /dockerfiles

Now we create our Dockerfile with the command nano Dockerfile and add the following contents:

FROM ubuntu:latest
MAINTAINER NAME EMAIL

RUN apt-get  update && apt-get -y upgrade && apt-get install -y nginx

Where NAME is the name to be used as the maintainer and EMAIL is the maintainer’s email address.

Save and close that file. (In my case i called it dockerfile; with a lowercase d)

Building the Image

Now we build an image from our Dockerfile. This is run with the command (by a user in the docker group):

docker build -t "NAME:Dockerfile" .

Where NAME is the name of the image to be built.

in this case . simply represents the local directory, else specify the path of the file…

Listing Images

docker images

Deleting Images

docker rmi image:tag

Running Images (Creating Containers)

docker run image

well poop, after running and stopping a container I was unable to delete the images… Internets to the rescue! since a force seemed rather harsh way to do it.

By default docker ps will only show running containers. You can show the stopped ones using docker ps --all.

You can then remove the container first with docker rm <CONTAINER_ID>

If you want to remove all of the containers, stopped or not, you can achieve this from a bash prompt with

$ docker rm $(docker ps --all -q)

The -q switch returns only the IDs

yay it worked!

Summary

Most of the time you won’t be directly installing docker, or building your own images, but if you do you at least now know the basics. These will become import in the future blog posts. I hope this helps with the basic understanding.

In the next blog post I’ll cover Docker-Compose, which will allow use to spin up multiple images into a single working container which will be the bases of our ELK stack. 🙂

FreeSWITCH – IVRs and Fun Stuff

In my previous post I covered how to setup FreeSWITCH behind a PAN firewall, connect with a gateway (the ITSP) and configure a very basic default and public dialplans to get simple inbound and bound calls working.

For the best reading I suggest this book. 🙂

FreeSwitch

IVRs

The path when working with IVRs is:

/etc/freeswitch/ivr_menus/

inside we can see the following:

root@FreeSwitch:~# ls /etc/freeswitch/ivr_menus/
demo_ivr.xml  my_ivr.xml  new_demo_ivr.xml

not a whole lot. First thing I wanted to figure out was how calling ext 5000 would lead to this IVR.

So looking at the default internal dial-plan, we can see it

nano /etc/freeswitch/dialplan/default.xml
    <!-- a sample IVR  -->
    <extension name="ivr_demo">
      <condition field="destination_number" expression="^5000$">
        <action application="answer"/>
        <action application="sleep" data="2000"/>
        <action application="ivr" data="demo_ivr"/>
      </condition>
    </extension>

Looks like it’s the application IVR data=demo_ivr. Quick way to find out let’s rename the file and see if the dialplan still works..ok, it’s not the filename, but the name value in the xml files located within the folder specified above.

Now as you can see I managed to get the screaming monkeys to work by simply recording a stream of screaming monkeys and exporting it with audacity in compressed ULaw wav format, uploaded it to freeSWITCH via WinSCP. then changed the action type and param value.

Now if you get into the nitty gritty, you’ll notice the default IVR uses phrases which are pieced together pieces of smaller recordings. These you may notice by default are also relatively referenced, instead of fully “ivr/ivr-that-was-invalid-entry.wav” which you may notice from searching exists only in the language folders of the sounds…

for the time being I won’t get into making that custom of an IVR, instead start off simple. I’m gonna create an audio recording of my options (1 for sales, 2 for work, 3 for support, 4 for other). Then used WinSCP to copy to the Freeswitch server, then copied to location.

root@FreeSwitch:/etc/freeswitch/ivr_menus# cp /home/zewwy/ZewwyCA.wav /usr/share/freeswitch/sounds/
root@FreeSwitch:/etc/freeswitch/ivr_menus# ls /usr/share/freeswitch/sounds/
en  es  fr  music  pt  ru  ScreamingMonkey.wav  ZewwyCA.wav

ok now, let’s make the IVR do stuff with these options….

first we’ll set the caller ID name and number in vars.xml so we can use the default variables in our directories, and hopefully for outbound calls.

So with my recording in place, i created my IVR as follows:

nano /etc/freeswitch/ivr_menus/zewwy_ivr.xml

then created a new extension to reach in my default dialplan

nano /etc/freeswitch/dialplan/default.xml

Now I just need to change my public dialplan to call this extension instead of my SPAPhone directly.

Change 1002 to 5006. That’s it. 😀 now got a IVR.

Time of Day Call Routing

I used FreeSwitch’s own post on this as a reference.

For me, I created option one in the options for “Sales”. I don’t want to be bothered about items for sales when I’m sleeping, or off work. Currently as above you can see calls are going straight to my voice mail. Well let’s change that…

<!-- My Cell -->
<extension name="mycell">
  <condition field="destination_number" expression="^mycell$">
    <action application="bridge" data="sofia/gateway/${default_gateway}/1#######"/>
  </condition>
</extension>

First, let’s create a new extension for Sales; 2222: in default dialplan

<extension name="Sales-x2222">
  <condition field="destination_number" expression="^2222$">
    <action application="transfer" data="Sales"/>
  </condition>
</extension>

just below that:

<extension name="Sales" continue="true">
<condition field="destination_number" expression="^Sales$"/>
  <condition wday="2-6" hour="9-18">
    <action application="set" data="Sales_open=true"/>
    <action application="transfer" data="xfer-to-sales"/>    
    <anti-action application="set" data="Sales_open=false"/>
    <anti-action application="transfer" data="xfer-to-sales"/>
  </condition>
</extension>
<extension name="xfer-to-sales">
  <condition field="destination_number" expression="^xfer-to-sales$"/>
  <condition field="${Sales_open}" expression="^true$">
    <action application="transfer" data="mycell"/>
    <action application="answer"/>
    <action application="sleep" data="2000"/>
    <action application="voicemail" data="default ${domain_name} 1002"/>
    <anti-action application="voicemail" data="default ${domain_name} 1002"/>
  </condition>
</extension>

If you can’t tell what is happening here, we are creating a dial number named Sales-x2222 when you dial 2222. Then we define the sales work hours Weekdays from 9 till 6, which defines the normal action lines, in this case it sets a variable “Sales_open” to true, otherwise if not within this time, set the variable to false. In the third area we use this flag to either call mycell or leave a voicemail on ext 1002.

Now I simply changed the first line in my IVR instead of bridging a call to my cell, I send it to ext 2222 which will only call my cell during working hours. 🙂

*NOTE* I could have also simply done this directly under the Public based dialplan such as this:

<include>
	<expression name="public_did">
		<condition field="destination_number" expression="1#######$">
			<action application="answer"/>
			<condition wday="2-6" hour="9-18">
				<action application="transfer" data="callfwd">
				<anti-action application="ivr" data="zewwy_ivr">
			</condition>
		</condition>
	</expression>
	<expression name="callfwd">
		<condition field="destination_number" expression="^callfwd$">
			<action application="answer"/>
			<action application="speak" data="flite|rms|Calling someone in reguards to a item for sale. Hold please."/>
			<action application="set" data="effective_caller_id_name=SALE(${caller_id_name})"/>
			<action application="set" data="effective_caller_id_number=${caller_id_number}"/>
			<action application="bridge" data="sofia/gateway/${default_gateway}/1#######"/>
		</condition>
	</expression>
</include>

However I wanted my IVR to stay the same no matter when it was called. So I placed my time of day routing at the internal dial-plan on the specific use case/department. It was simply a more scalable example to use.

Blast Group

here’s an example of a 2 phone blast group via ext 511 using the default dial plan, blasting ext’s 1002 and 1003

this took me a bit…

Hunt Group

ext 512 dials Ext 1001, if not available call 1002 (ring only for 20 second), if not available ring ext 1003 (for 30 seconds), if no answer drop to 1002’s voice mailbox.

some Music on Hold would be nice while the call is being transferred…

    <extension name="a_blast_group">
      <condition field="destination_number" expression="^511$">
        <action application="set" data="ringback=${us-ring}"/>
        <action application="set" data="transfer_ringback=$${hold_music}"/>
        <action application="answer"/>
        <action application="sleep" data="2000"/>
        <action application="bridge" data="sofia/$${domain}/1002,sofia/$${domain}/1003"/>
      </condition>
    </extension>

    <extension name="a_hunt_group">
      <condition field="destination_number" expression="^512$">
        <action application="set" data="ringback=${us-ring}"/>
        <action application="set" data="transfer_ringback=$${hold_music}"/>
        <action application="answer"/>
        <action application="sleep" data="2000"/>
        <action application="set" data="continue_on_fail=true"/>
        <action application="set" data="call_timeout=20"/>
        <action application="bridge" data="sofia/$${domain}/1001|sofia/$${domain}/1002"/>
        <action application="set" data="call_timeout=30"/>
        <action application="bridge" data="sofia/$${domain}/1003"/>
        <action application="voicemail" data="default ${domain_name} 1002"/>
      </condition>
    </extension>

Hope someone found this guide useful. 🙂

FreeSWITCH

The Story

My buddy Troy did a presentation, I wanna try it out. So this is going to be a shit show… let’s go…

Sources: Specs

Minimum/Recommended System Requirements
32-bit OS (64-bit recommended)
512MB RAM (1GB recommended)
50MB of Disk Space
System requirements depend on your deployment needs. We recommend you plan for 50% duty cycle.

Install Source for Debian 10

Buddy Troys Presentation

Install Debian 10

So I’ll setup a VM with those nice minimum requirements, could def use the memory savings, most servers these days are redic.

LimbooooooooOOOOOOOOOooo! How low can you go?!

Alright let’s install Debian 10.

Install Source Info and Install Source Image I’ll use the netinstall image.

Mount image to VM… booot er up (I’m gonna try EFI instead of BIOS)

Nice, it booted, Install Graphical or Install, just install, we want to keep it CLI only as it has bare resource allocations.

set root password, create alternative user, guided use entire disk, or set however you like, or however you deploy your AC3 AWS nodes or whatever cloud based instance you might be using. Whatever floats your digital boats.

*Digitized Voice* All your base are belong to us…

ohhh boy…. anyway.

SSH and Standard system packages… this installer keeps going…

Wooo never thought I’d see the day… OK so now that we finally have a clean Debian server, we can move to the next step. 😀

FreeSwitch Install

From Source: “Debian 10 “Buster” is the reference platform for FreeSWITCH™ as of version 1.10

Dependencies are available from FreeSWITCH repository via the “apt-get build-dep freeswitch” command.”

ok let’s try that?

Not sure why that’s at the top of the document when it doesn’t work out of the box, let’s follow along with the “easy way” then…

apt-get update && apt-get install -y gnupg2 wget lsb-release

Moving on…

wget -O - https://files.freeswitch.org/repo/deb/debian-release/fsstretch-archive-keyring.asc | apt-key add -

# you may want to populate /etc/freeswitch at this point.
# if /etc/freeswitch does not exist, the standard vanilla configuration is deployed
apt-get update && apt-get install -y freeswitch-meta-all

Uhhh ok, I don’t have a config in mind per say so I guess I’ll use the predefined one without creating that directory or file… let’s go!

Off she goes 200+ already! That didn’t take too long. Let’s see if we can get into the freeswitch cli…

fs_cli -rRS

[ERROR] fs_cli.c:1565 main() Error Connecting [Socket Connection Error]

OK dokie then, let’s give er a good old reboot. After reboot, haza!

The Presentation in a Nut Shell

SIP (Session Initiation Protocol) -> Initiates the connection for the task
SDP (Session Description Protocol) -> Connection for what
RT(C)P (Real Time (Control) Protocol) -> RTP: Audio Packets RTCP: Metadata

Now slide 25 while very simple topology layout isn’t crazy it was the mentioning of alternative NAT tricks which kind of boggled my mind a bit. The other day I had issues with my Signal app using mobile data excessively even though I was on WiFi. Took me a little while to figure out but it was my firewall that was blocking the traffic and it appears Signal secretly uses any alternative networks on the device to establish the required connection. During the research for a solution, I found a PaloAltoNetworks thread on the issue

Creating a rule with the three main applications (Signal, SSL, STUN) allowing any service ports, and then turning off my mobile data. Still resulted in failed Signal calls. I have to open the rule up completely and even disable server response inspection. I had talked to my local PAN technical rep, I might just make a separate blog post about that entirely. Anyway just making note of that as a possible infrastructure to hurdle while I go through this endeavor…

Check out this Wiki Page on more details on STUN if you have the basic understanding of the difference between TCP and UDP the contents should be fairly easy to digest. However, I digress and move on.

Well it’s going to be harder than I thought to put all this info into a Nut Shell, so instead I’ll try to cover each piece of the puzzle one at a time. First thing on any server is to have a static IP (at least if your behind a NAT which is mentioned many times in his presentation, and I’ll discuss my setup and how that flies when we get to that step). For now let’s just set our internal static private IP address.

OK strange, coming back to this VM from yesterday I was still int the freeswitch CLI, yet typing /exit would bring up the same freeswitch CLI… so hard reboot… and… ok so the initial Debian install guide said to do fs_cli with some options. Read here for a PDF of details options truns out the -R is reconnect when disconnect, and /exit, /bye, /quit are all disconnects. So just use fs_cli without -R, and the /exit works without issue.

Set Host Static IP

Now with that annoyance out of the way, well use this Debian guide to set out IP as root.

nano /etc/network/interfaces

from:

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

to:

auto eth0
iface eth0 inet static
        address 192.0.2.22/24
        gateway 192.0.2.1

This leaves us so far with this very, very basic network diagram:

Super simple, but doesn’t cover the SIP connections coming in, and the following STUN and NAT traversal attempts, while most home routers may allow these connections the PA I’m behind, not so much. So we’ll again cover those details when we get to them.

*NOTE* I had to make this a Static IP NAT else I wouldn’t get Audio when doing direct sip calls not out to the regular PSTN. See this Post for more details. I also created the custom Apps and app override rules

Managing Users

Slide 33 starts off with:

sudo chmod g+w -R /etc/freeswitch

I’m starting to realize the slide out of context (being there for the presentation) is rather hard to follow along with, but since I’m doing everything as root for now (Since I didn’t add my standard user to the FreeSwitch group) I’ll just ignore this line for now.

  • cd /etc/freeswitch/directory/default
  • Remove everything but 1000.xml, 1001.xml
  • Edit for our user accounts
  • We can reload this configuration without restarting freeswitch

Ok dokie….

Neat! I had no idea you could do that! so do…

rm -v !("1000.xml"|"1001.xml")

That was so easy! 🙂 Since I’m again reading this out of context I’m not sure what edits were made to what files for the line “edit for our user accounts”. Looks like just the XML files we left behind, these appear to be template, as usual XML based, so knowing which fields to edit can be a bit trick to point out.

OK… a bit more details from the main source

That makes sense… so my final config:

Connecting a Phone

Before we can make any calls we are going to need a phone, now this can be almost any device, a laptop with softphone software , or can be a physical phone as long as it supports the SIP protocol. Lucky for me I have 2 different devices at my disposal for testing. Two older Cisco phones: 1) a Cisco SPA525g and 2) a Cisco WiFi 7925.

The desk phone I feel still looks nice and modern despite it’s age, the 7925 looks like an old brick Nokia and you can imagine the software is just as bad. So let’s see how we can get this into the mix. Lucky for me both support SIP.

Accordingly to the slides on slide 32 looks like we have to define the server listening address and port (which we will leave at the default 5060) that will be the unencrypted default port.

nano /etc/freeswitch/vars.xml

Oh yeah… that default password thing mentioned in slide 31, not sure why this would be clear text in a clear text xml file on a open config path, but… *Smiles n Nods* changing cleartext password.

Since I’m not making any changes at this time, we will just exit and cover applying config changes once we get there. I need to double check some sources to get my head around all this stuff right now, so please bare with me on this blog post, I’m literally learning everything as I go.

7925 – Settings locked (Press **# to unlock settings)
Setup all the WiFi settings then connected to my local network. Use this Cisco Doc for help configuring a Cisco 7925

This is were things kind of went sour. the Cisco 7925g is SCCP or Cisco protocol only (lame) so no SIP. literally Nothing. There maybe a way to use Skippy mod for FreeSwitch but well see about that in a future post.

As for the SPA525g It took me a good while of digging before I figured it out.

  1. Step IP
  2. Log into Web interface http://IP
  3. Click on “Admin Login”
  4. You should now see a SIP tab, leave it, click on Ext 1
  5. Fill in the Proxy Address: (IP of freeswitch)
    User ID: 1001
    Password: (AS set in XML)

I finally managed to get a successful registration after this (but all the soft buttons were lit up and displaying 1001)

So the phone doesn’t look nice (yet) and we only have one, with our 7925 out of the picture for now, I guess I may have to rely on a softphone after all. :S

To adjust the alternative buttons click the “phone” tab and set each line to disabled, or name them as alternative lines as Louis does in this youtube video.

Now I just need to setup user 1000 since the 7925 was so nice to not use SIP, like at all SCCP only, no thanks for now. So, in today’s more modern times, I’ll just use a softphone. I decided to play with linphone on my andriod phone.

Open it up and four options appeared create account, use linphone account, use sip account, or fetch remote configuration… Use SIP account.

username: 1000
password: AsSetInXLM
Domain: IPofFreeSwitch
Name: Optional

And it connected!

OK so now out setup looks like this:

Now we have the very basics to make out first call: 2 Users and 2 Registered phones. So from the SPA525g, I dialed 1000, and sure enough my linphone rang, picked it up and had my first self configured SIP call. It was the usual self mocking type comments back and forth. After hanging up there was some feeling of accomplishment. But no time to stop here… there’s more fun to be had!

Unfortunately I was unable to make a call from linphone to the SPA phone cause as others have mentioned in the comments for some reason it auto adding +1 in front of all numbers dialed and it won’t simply ring ext 1001.

Important Tid Bits

  • Log directory – /var/log/freeswitch
  • Configuration directory – /etc/freeswitch
  • Database directory – /var/lib/freeswitch/db
    • Hosts SQLite databases
    • SQLite is the default database, many are supported
  • Daemon is configured via systemd
 sudo systemctl start freeswitch [or] service freeswitch start|stop|status
  • Administration – make yourself part of the freeswitch group
sudo usermod -aG freeswitch useraccount

Invaluable tools for administration – fs_cli (included in freeswitch)

let’s also install sngrep (this will come in handy later)

sudo apt install sngrep

FreeSwitch Configs

  • /etc/freeswitch/freeswitch.xml
    • This is the “point of entry” for configuration
    • It includes /etc/freeswitch/vars.xml, and does fileglob-includes for other important bits
      • autoload_configs/*.xml
        • This is where module configurations live (e.g. database connectivity, SIP stack, more)
      • dialplan/*.xml
        • This is where dialplans live (e.g. how do you dial out, IVRs, etc)
      • directory/*.xml
        • This is where user provisioning lives by default
  • /etc/freeswitch/vars.xml
    • This is where the “preprocessor variables” and generally very important variables live
    • You can think of it as “settings that you can’t change at runtime”
    • Includes all your favourite hits, such as:
      • <X-PRE-PROCESS cmd=”set” data=”default_password=1234″/>
        • Change this ASAP!
      • <X-PRE-PROCESS cmd=”stun-set” data=”external_rtp_ip=stun:stun.freeswitch.org”/>
      • <X-PRE-PROCESS cmd=”stun-set” data=”external_sip_ip=stun:stun.freeswitch.org”/>

*The “stun” entries are for NAT traversal; if you’re not behind a NAT device, you can change these to “host:your.domain” or your IP address.

Since our FreeSwitch is behind a NAT as shown in the first topology picture, I left these fields defaulted.

  • /etc/freeswitch/vars.xml
    • Let’s change our domain:
      • <X-PRE-PROCESS cmd=”set” data=”domain=$${local_ip_v4}”/>
    • Other notable entries:
      • <X-PRE-PROCESS cmd=”set” data=”internal_sip_port=5060″/>
        • SIP phones will register to your server on this port
      • <X-PRE-PROCESS cmd=”set” data=”external_sip_port=5080″/>
        • Calls will come from your ITSP on this port

At this point in his slides he goes on about making an external call, while I do plan on getting to that, I needed a VoIP provider so I’m currently working on getting a VoIP provider setup. In the meantime…

Voice MaiL

I sure enough left a phone ringing, for a good amount a rings it seemed and I was automatically transferred to a user voice mail, amazingly no configuration was required.

The softphone (Linphone) also didn’t seem to have an indicator for such a thing and after a bit of da googling, I found you simply dial *98.

On the SPA525g first time pressing the mail icon will ask you to enter the voicemail number, which I entered incorrectly and had to find this guide to help me figure how to change it.

Setup -> User Preferences -> Call Preferences -> VoiceMail

Although I was able to listen to the message I found I would always get cut off at 30 seconds.

Nope Any call gets cut off after 30 seconds… I’m about to give up on this shit…

Troubleshooting, Yay!

I did manage to get a bit of help from my buddy Troy and a nice user on the FreeSwitch channel on IRC in #FreeNode

We used sngrep and realized that I was not getting a ACK message from the phone.

As you can see no ACK….

Cyrillax from IRC mentioned enabling advanced debug…

sofia loglevel all 9
sofia global siptrace on

This will output a lot to the screen, so if you need to backscroll and are using putty ensure you add plenty of backscroll lines the default is 200, and that is not enough..

Checking the debug logs we can see the contact info is not what we wanted, the phone is trying to connect to the FreeSwitch via the public IP address:

Now on the SPA525g we entered Advanced config area opening up additional configurations and told the phone to use the outbound proxy after defining it (with the IP address of the FreeSwitch) which worked and we had calls with the IVR last as long as required. I’m not sure if this will suffice when it comes times for outbound calls, but well cover that when we get there. lol I’ve been saying that a lot.

I still wasn’t sure if the additional proxy configs was the right solution to the problem, although it did resolve the problem and acks were sent from the phone directly back to freeswitch. However every softphone I setup even after setting freeswitch to the proxy IP it wouldn’t work and I’d see the SDP sent with the Public contact in the field every time…

no matter how I configured the FreeSwitch XML config files I couldn’t seem to get it to provide the contact of the private IP not the public one, which I kept reading and hearing that’s normally what you want. I couldn’t see these requests for traffic in my Monitor tab of the Palo Alto firewall, so I thought it was a dud or wasn’t happening, but decided to create a U-Turn NAT rule anyway.

after committing I finally got an ACK! hahah from the firewall itself, kind of as expected since it TCP based, in this case and required to completed TCP’s 3 way handshake.

The diagram looks like this now:

Now things work, except for some reason I can’t call the softphone from the spa525g. But the Softphone can ring the SPA525g just fine…. ughhh my ignorance is causing gremlins! OK everyone can call 5000, and voicemail, but noone can call the softphone @ 1000. I’ll figure this one out tomorrow.

I decided to see if this was the problem, and reverted the outbound proxy settings I had added to the SPA525g. and sure enough go multiple SDP with no ACK, this time it was cause they were attempting to negotiate via UDP not TCP has my rule above I created for TCP only… OK let’s duplicate the rule and also allow UDP. Since it now is using UDP for the SDP and I did not define that port in my UTurn NAT rule, I”ll create another one for UDP but without source NAT translation… so it’ll look like this:

The rule looks like this now:

and after committing we get an ACK from the phone directly, without configuring an outbound proxy setting on the phone. 😀

Sure enough, on the SPA525g, everything works, calls to the 5000 built in IVR, VoiceMail, the works. Now lets try the softphone again… nope….

OK well I’m not sure if it’s the VIA field causing me grief or the fact that calls being made when routing from the FreeSwitch keep saying from 1001@freeswitch instead of the users making the call 1000. I removed the CIDR from users 1001.xml and copied it, changed the password, updated the config with:

fs_cli -x "reloadxml"

And oddly enough I was finally able to call the softphone on ext 1000. There was a long delay before the ringing started but it worked this time?! like what?

Sure enough I can call both ways now, but when I call 1001 from the softphone (1000) it rings right away, if I call 1000 from the SPAphone (1001) there’s a delay before the ringing starts. I’m not sure if this is some limitation of the app I’m using. I also have no idea how the heck making that change made the calls start working…

after creating two new users (copied 1001.xml and changed all 1001 to 1002 and 1003 respectively). Now calls going both ways are instant and all phones soft and SPA are working 100%.

I stand corrected… calling ext 5000 give me now a busy signal… this is starting to really annoy me…

Oh wait… right I changed 1001.xml with random 1005 numbers….

once I reverted this back to default as pictured at the top of this blog post, ext 5000 started working again… Not sure why this is but I guess it might be time to check out the dial plans?

30 Second Cut Off

Check to make sure the FreeSwitch Server is getting the required ACK. See Above for example.

Call Connects but No Audio with Direct SIP Routing

Check your internet connection NAT rule for the FreeSwitch server, ensure it is a Static NAT, not Dynamic IP and Port.

10 Second Delay In Call Answer

I searched this one up the other day, and I’ve heard it could be DNS (check you /etc/resolve.conf) mine was good. Heard it was due to STUN people set there STUN servers blank, this however will have consequences on the SDP contact information, so I wouldn’t recommend this, but it has been mentioned. In my case it was all of a sudden deliberate sleep execution due to not having changed the default password in vars.xml.

So yeah…. make sure you change the default password. then reloadxml in fs_cli.

Dial Plans and Phone Numbers

Different Dial Plan Directories

You may have noticed we have (used for internal phones)

 /etc/freeswitch/dialplan/default/

as well as (used by les.net dialing in)

 /etc/freeswitch/dialplan/public/

OK…. now we finally got past all the lower layer technical hurdles we can finally get to configuring the application itself. However we need to … collaborate with external sources. Now for me I’m lucky and have a local VoIP provider that is small in size but very technically aware, and much like Troy’s slide I use the same provider. Les.net

I tried to setup an account with them anonymously but that didn’t work as I had to call in as my account got suspended with fake info… Whomp wommmm womomomo.

So after I got my account verified, clicking on Order DID, pick the area, the area-code and any other information and the order details pop up (slide 37):

hahahaha, it’s cheaper for me to order a number for Fargo, ND then it is for me to order a local Winnipeg number… hahahah ahhhh… btw I am not using that number, I’ll still with the free DID for now, anyway…

The point is now we should have the basics in place to get FreeSwitch server registered with an external VoIP provider so we can make calls to the, sweet, sweet, candy… I mean public phone system. Sooo we are working on this:

As you can see the SIP/SDP/RTC/RTCP arrow is both ways, so if the les.net proxy send UDP based packets at me, the Palo Alto Firewalls will not know what to do with them, and drop em like they’re hot, drop em like they’re hot… When the Bi…. whoops going off track anyway, let’s create some rules to allow connection from our Internet telephony service provider (ITSP).

These details should be provided to you by your ITSP.

I thought about it a bit and did not create and open bi-directional NAT rule cause I’m sure my ITSP doesn’t want DNS and alternative requests from my freeswitch, so instead I created an open one way NAT rule that says anything from LesNet SIP proxy’s send it to my Freeswitch, in hopes those proxy’s are also setup to send only what they need to the right place. I still need a security rule though to make this work. So again I’ll leave it open, monitor the traffic and restrict the application or service ports accordingly.

Now that we got the firewall out of the way let’s go configure the FreeSwitch server.

/etc/freeswitch/sip_profiles/external/

This directory is for integrating with upstream providers. You can have multiple ITSP gateways. These handles incoming SIP traffic on port 5080
(Which we have our NAT and SEC rule so this should be good to go now)
Example: you could register a DID for multiple provinces, and have each trunk as it’s own gateway.
/etc/freeswitch/sip_profiles/internal.xml
This configures your internal profile (port 5060) for accepting connections from SIP phones (Which we already went through the nitty gritty above)

Let’s create a file in this directory, lesnet.xml

cp /etc/freeswitch/sip_profiles/external/example.xml /etc/freeswitch/sip_profiles/external/lesnet.xml

Use the username, password, and proxy provided by lesnet’s login page(To do this on LesNet have to create a new SIP Peer / Trunk, then click the edit button on it, this will present the required details to enter into the xml file.)

New profiles can be loaded at runtime

fs_cli
sofia profile external rescan
sofia status gateway proxy.sip.les.net

Note – “sofia” is the name of the SIP stack used by freeswitch.

WOW! it worked!

Since this was a connection from freeswitch to lesnet I didn’t see it hit my newly created rule instead it used my default home network outbound rule which was allowed.

My excitement was again short lived as I hit another road block (story of my life). Turns out I kept seeing repeated Registrations and 401 responses. I wasn’t sure of this and made a change to my external gateway…

nano /etc/freeswitch/sip_profiles/external/lesnet.xml

sofia profile external restart reloadxml

So to get out bound to even show up on the lesnet side some changes were required.

nano /etc/freeswitch/vars.xml
 <X-PRE-PROCESS cmd="set" data="default_provider=proxy3.sip.les.net"/>

then again some reloadxml

fs_cli -x "reloadxml"

now when we make calls it’s bust but at least they show on the call logs on the ITSP portal.

Incoming Calls

Now for incoming calls, after you verify a stable connection with the ITSP Gateway/proxy, and see it their online portal, you may have to map a number to a DID Peer/Trunk, In this case I saw my registered FreeSwitch as SIP Peer 79908, then under “Your DIDs” have to click on the number you wish to route, and select the end SIP peer to route those calls to, in my case SIP Peer 79908.

At this point you should be able to see the calls come in on the ITSP call logs and the FreeSwitch via sngrep, but it won’t be routed anywhere according to FreeSwitch’s dial plan so…

nano /etc/freeswitch/dialplan/public/1204666xxxx.xml

Now I don’t think you have to name it this way, pretty sure you can name it differently but this is for simplicity for now. and fill it with:

This should be all that’s required, just do another reloadxml, and dial the number.

Outbound Calls

Now with the current Dial-Plan that’s defaulted 01_example.com.xml it’s using the gateway variable we defined in vars.xml so our only outbound proxy at this point. Since I was able to see the calls hitting the les.net portal but getting denied I decided to give les.net a call to see if maybe they had an idea why.

When checking my SIP peer trunk on the portal which was my FreeSwitch it was registering every 20-30 minutes, it was suggested to drop it to between 60-90 seconds.

So in the gateway settings:

nano /etc/freeswitch/sip-profiles/external/lesnet.xml

thx

sofia profile external restart reloadxml

Turns out that wasn’t the case, I had a hunch the problem was the fact the source was 000000000 as your can see:

so I quickly googled this to see if I could find something.. I found this

“dial and bgdial

If the caller id values are not set, the variables in conference.conf.xml will be used. Specifically, the value for caller-id-number will be used for the number and the value for caller-id-name will be used for the name.

If the conference will be dynamically created as a result of this api call (ie this will be the first participant in the conference) – and the caller id name and number is not provided in the api call – the number and name will be “00000000” and “FreeSWITCH”. This appears to be unaffected by the variables in conference.conf.xml.”

Ohhhhhhhh… ok so if I set the call outbound number in the user file…

This works for the single user, to make it more of a NAT like you do with a single public IP address and want to share the internet, you set this variable in the vars.xml file.

and sure enough :D…

Yes!! hahah finally.

That’s it for now! Next round I’ll cover IVRs and all the other fun stuff. This is just the basics. and even then doesn’t cover it very well, just enough to get it all to work. I also noticed that I didn’t have to the NAT rules or the security rules so just the basic NAT is required for FreeNAS and the phones I guess… hahaha

 

Even More PowerShell Fun

The Story

It’s another day, and we all know what that means… yes, another blog post, and even more PowerShell! Can you feel all the power!?!?!

This time it came down to the storage size of my Exchange servers C:\ which turns out to be due to Logs. Logs are great, and best practice is to only clear them if you have a backup copy. Often is the case that logs can be truncated after a backup via VSS by many backup solutions however in my case I could and probably should get that validated with Veeam (as I can’t seem to get that working ‘out of the box’) at the moment. So instead I wanted to know what was “usually” done server side even if someone was not implementing a backup solution.

Source: https://social.technet.microsoft.com/wiki/contents/articles/31117.exchange-201320162019-logging-clear-out-the-log-files.aspx

Neat, but the script is just alright, good for them doing what they want and that’s running it as a scheduled task. Not my goal, but a great source and starting point… let’s have some fun and give this script some roids, much like my last one… I’ll give this a home on GitHub.

Things learned…

  1. Working with the Registry
  2. Determining if Elevated (This is great and I may have a solution to the conundrum in my previous PowerShell post)
  3. Getting a Number, and validating it
  4. Validating Objects by Type
  5. Getting Folder Sizes

Check out my script for all the fun coding bits. I’m a bit tired now as it’s getting late so not much blogging, all more coding. 🙂

ErrorAction Stop Not Stopping Script

Quick Educational note (Source)

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{Write-Host "No Key"}
Write-Host "This should not hit"

Produces:

Well poop… The catch block was triggered but the script did not stop…

Oddly, changing to Throw, which is ugly does make the script stop…

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{throw "No Key"}
Write-Host "This should not hit"

Nice it worked this time, but it’s ugly…

Write-Error is just as ugly, but doesn’t stop the script?

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{Write-Error "No Key"}
Write-Host "This should not hit"

Produces:

Yet if I follow Sages answer in the source, and do a script variable for the stop action it then works???!?!

$ErrorActionPreference = [System.Management.Automation.ActionPreference]::Stop
$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2}
catch{Write-Error "No Key"}
Write-Host "This should not hit"

Those are really weird results, but all still ugly… so it seems even though -ErrorAction Stop causes a non-terminating error to be treated as a terminating error, but depending on what you do in the catch block determines if there’s a break/exit event being done. In my case to have things look nice and actually stop the script I have to do the follow.

$ETLLogKey2 = 'HKLM:\SOFTWARE\Microsoft\Search Foundation for Exchange\Diagnostics'
try{Get-ItemProperty -Path $ETLLogKey2 -ErrorAction Stop}
catch{Write-host "No Key";break}
Write-Host "This should not hit"

Which finally produced the output I wanted. (I could have also used exit in place of break)

Finally!

Validating URLs:

Source:

Good but broken due to the match on the scheme extension:

function isURI($address) {
	($address -as [System.URI]).AbsoluteURI -ne $null
}

function isURIWeb($address) {
	$uri = $address -as [System.URI]
	$uri.AbsoluteURI -ne $null -and $uri.Scheme -match '[http|https]'
}


isURI('http://www.powershell.com')
isURI('test')
isURI($null)
isURI('zzz://zumsel.zum')

"-" * 50

isURIWeb('http://www.powershell.com')
isURIWeb('test')
isURIWeb($null)
isURIWeb('zzz://zumsel.zum')
isURIWeb('hp:') #Return True

 

Better results with:

function isURI($address) {
($address -as [System.URI]).AbsoluteURI -ne $null
}

function isURIWeb($address) {
$uri = $address -as [System.URI]
$uri.AbsoluteURI -ne $null -and $uri.Scheme -match "http|https"
}


isURI('http://www.powershell.com')
isURI('test')
isURI($null)
isURI('zzz://zumsel.zum')

"-" * 50

isURIWeb('http://www.powershell.com')
isURIWeb('test')
isURIWeb($null)
isURIWeb('zzz://zumsel.zum')
isURIWeb('hp:') #Return True

Outlook and the Cache Mode

The Story

Wouldn’t be another post without another day of annoyances… yup, just another day. So recently it’s been reported that when users are working remotely their outlook decides not to open….

Now since they are remote, I have noticed this only happens if the Outlook client is configured in “Online mode”, instead of “Exchange Cached Mode”… see this MS Docs for more information on the different types and when to utilize each mode.

Originally I configured Online Mode, as most users are locally available in the work network and this is not a major problem. Also when using “Cache mode” not all emails show up in Outlook right away, specially if using folders, there generally shows a link “Click here to view more on Microsoft Exchange” this means items that were not cached (depending on the cache time slider this may vary, I choose 12 months), view this support if you can’t see the link in cache mode enabled.

I used to have an issue with this setting, user reported items wouldn’t load even after clicking the link, however I haven’t seen this to be an issue anymore, so I recommend to enable Outlook Cache mode unless you fall under the other points in MS’s Doc linked above.

Finding Users That Are Not Using Cache Mode

So I was now on a new mission… “How do I know who’s not using Cache mode on outlook?” And this is were the rabbit hole began….

First result, same question

Craig Harts reply…

his or response seem like it’s rather easy until…

wtf is this…? Depreciated?!?! C’mon, but there’s an alternative now that’s better right? No…. Thanks MS… seriously… thanks…

So this older post goes over alternatives, not because they didn’t have access to the cmdlet above, but simply cause they didn’t trust the results.. huh…

Problem is this seem to be reg keys used by older outlook, and new outlook seems to use alternative keys… This sure is fun! However, thanks to others that blog and code in their spare time as well, in this case thanks to Jose Espitia much like in my last blog post this is a great start, but usual me, I don’t like expecting anyone to change the source code. In this case you have to provide a file with a computer list, output path, but it’s hard coded…. OK you know what that means! Yeah I usually would create a new GitHub Repo but first…

Much like the older post mentioned they choose to target end user directly to get the most “accurate data”. This however assumes four things:

  1. That Exchange admins are workstation admins, and have elevated rights on all machines.
  2. That all firewalls are configured and permissions to allow remote querying.
  3. That the user’s are in fact online at the time the query is made.
  4. That RemoteRegistry Service is started and running on end machines.

In my Case it was not and disabled on all machines, I didn’t feel it was beneficial enough to introduce a risk for the simple sake of determining a users connection mode on Outlook.

So I decided to go back to Option A from the old original post, parsing all the RPC Access logs… Using this an alternative reference.

Well I gave it a try and looking through all th elogs there was no reference to “Classic” so I guess that’s no longer valid option either.

Looks like I’m stuck on this one, I can’t seem to find a valid way to find this information out server side with MS removal of the Get-LoggingStatistics cmdlet. And attempting to query all ends users has to many restrictions/hoops that I do not wish to implement. In this case I have to simply go around to all users machines, or wait for the to complain when they work remotely.

Thanks Microsoft I really love what your doing for SysAdmins. Taking away mark and putting him on as COO of Azure so all our beautiful tools from SysInternals are now just the way they are, and new tools, don’t need em right, just buy your cloud subscriptions and who needs SysAdmins… 😛

Anyway… that’s it for today. Sorry no advanced scripts form this post, just use Jose’s script if you wish to query end users machines. Just ensure you have RemoteRegistry service running on all end users machines, and not blocking it in the firewall as well. Hoops I have no interest in jumping through.

Cheers!

WMIC Fun!

I’ve blogged about WMI before, more for setting up dedicated accounts for monitoring purposes.

Today we are going to have some fun with WMIC, the command line interface for simple and quick query data.

I got these ideas after reading this source blog… and I was curious at what level these worked (admin or not)

Using WMI

Most WMIC commands are issued in the following format:

wmic [Object Class] [Action] [Parameters]

For example, you can collect a list of groups or users on the local system and domain using the following commands:

wmic group list brief
wmic useraccount get name,sid

Yup, SIDs are no secret and you can pretty much query the whole domain if there’s been no hardening done. I haven’t tested this on a hardened domain but out of the box all users login name and SID are open for any standard user to query.

You can also perform the same data collection over the network without ever logging into the remote machine provided you know have some administrative credentials that the remote system will accept.

The same command issued against a remote system in another domain looks like this:

wmic /user:"FOREIGN_DOMAIN\Admin" /password:"Password" /node:192.168.33.25 group list brief

I can’t test this in my lab as I don’t have an alternative domain to play with (yet), but let’s see if I can query a member server using a standard domain account:

wmic /node:subca.zewwy.ca group list brief

nope well that’s good…

Processes
WMIC can collect a list of the currently running processes similar to what you’d see in “Task Manager” using the following command:

wmic process list
wmic process get name

Note that some of the WMIC built-ins can also be used in “brief” mode to display a less verbose output. The process built-in is one of these, so you could collect more refined output using the command:

wmic process list brief

Yup, those all work, even as standard user.

Some examples

Start an Application

wmic process call create "calc.exe"

Yeah… that worked…

I decided to see if I could somehow exploit these to get elevated rights, so far no dice.. but I did find this randomly while searching for a possible way…

sure enough, if you add start cmd.exe /k “net use” and name it net use.bat it will go into and endless loop. Mhmm interesting and easiest way to do a Denial Of Service attack.

anyway moving on…

System Information and Settings

You can collect a listing of the environment variables (including the PATH) with this command: (standard User works)

wmic environment list

OS/System Report HTML Formatted

wmic /output:c:os.html os get /format:hform

This was literally cause my standard account didn’t have access to C:\temp cause I created the folder using my admin account at some earlier point in time.

Products/Programs Installed Report HTML Formatted

wmic /output:c:product.html product get /format:hform

Turn on Remoted Desktop Remotely

Wmic /node:"servername" /user:"user@domain" /password: "password" RDToggle where ServerName="server name" call SetAllowTSConnections 1

Get Server Drive Space Usage Remotely (any node commands require elevated permissions, standard user fails at these generally)

WMIC /Node:%%A LogicalDisk Where DriveType="3" Get DeviceID,FileSystem,FreeSpace,Size /Format:csv MORE /E +2 >> SRVSPACE.CSV

Get PC Serial Number (works as standard user)

wmic bios get serialnumber

Get PC Product Number (works as standard user)

wmic baseboard get product

Find stuff that starts on boot (works as standard user)

wmic STARTUP GET Caption, Command, User

Reboot or Shutdown (works as standard user)

wmic os get buildnumber
wmic os where buildnumber="2600" call reboot

Get Startup List (works as standard user)

wmic startup list full

Information About Harddrives (works as standard user)

wmic logicaldisk where drivetype=3 get name, freespace, systemname, filesystem, size, volumeserialnumber

Information about OS (works as standard user)

wmic os get bootdevice, buildnumber, caption, freespaceinpagingfiles, installdate, name, systemdrive, windowsdirectory /format:htable > c:osinfo.htm

User and Groups

Local user and group information can be obtained using these commands:

wmic useraccount list
wmic group list
wmic sysaccount list

For domain controllers, this should provide a listing of all user accounts and groups in the domain. The “sysaccount” version provides you with system accounts built-in and otherwise,which is useful for any extra accounts that may have been added by rootkits.

Identify any local system accounts that are enabled (guest, etc.)

wmic USERACCOUNT WHERE "Disabled=0 AND LocalAccount=1" GET Name

Number of Logons Per USERID

wmic netlogin where (name like "%skodo") get numberoflogons

Get Domain Names And When Account PWD set to Expire

WMIC UserAccount GET name,PasswordExpires /Value

Patch Management

Need to know if there are any missing patches on the system? WMIC can help you find out with this command:

wmic qfe list

The QFE here stands for “Quick Fix Engineering”.
The results also include the dates of install should that be needed from an auditing standpoint.

Shares

Enumeration of all of the local shares can be collected using the command:

wmic share list

The result will also include hidden shares (named with a $ at the end).

Find user-created shares (usually not hidden)

wmic SHARE WHERE "NOT Name LIKE '%$'" GET Name, Path

so far all these are working as standard user, but that doesn’t mean anything.

Networking

Use the following command to extract a list of network adapters and IP address information:

wmic nicconfig list

Get Mac Address:

wmic nic get macaddress

Update static IP address:

wmic nicconfig get description, index
wmic nicconfig where index=9 call enablestatic("192.168.16.4"), ("255.255.255.0")

Yup got to be an admin for that one

Change network gateway:

wmic nicconfig where index=9 call setgateways("192.168.16.4", "192.168.16.5"),(1,2)

Enable DHCP:

wmic nicconfig where index=9 call enabledhcp

Get List of IP Interfaces

wmic nicconfig where IPEnabled='true'

Services

WMIC can list all of the installed services and their configurations using this command:

wmic service list

The output will include the full command used for starting the service and its verbose description.

Other examples

Service Management

 wmic service where caption="DHCP Client" call changestartmode "Disabled"

Look at services that are set to start automatically

wmic SERVICE WHERE StartMode="Auto" GET Name, State

Services Report on a Remote Machine HTML Formatted:

wmic /output:c:services.htm /node:server1 service list full / format:htable

Get Startmode of Services

Wmic service get caption, name, startmode, state

Change Start Mode of Service:

wmic service where (name like "Fax" OR name like "Alerter") CALL ChangeStartMode Disabled

Get Running Services Information

Wmic service where (state="running") get caption, name, startmode, state

Another interesting feature of WMIC is its ability to record the run-time command executed and runtime configuration all in one XML file. A recorded session might look something like this:

wmic /record:users_list.xml useraccount list

Of course, since WMIC wasn’t designed as a recording device, there are some caveats to using the XML. First, you can only use XML output, there are no other formats defined.

Event logs

Obtain a Certain Kind of Event from Eventlog

wmic ntevent where (message like "%logon%") list brief

Clear the Eventlog

wmic nteventlog where (description like "%secevent%") call cleareventlog

Retrieve list of warning and error events not from system or security logs

WMIC NTEVENT WHERE “EventType < 3 AND LogFile != ‘System’ AND LogFile != ‘Security’” GET LogFile, SourceName, EventType, Message, TimeGenerated /FORMAT:”htable.xsl”:” datatype = number”:” sortby = EventType” > c:appevent.htm

Thanks Andrea!

Requesting, Signing, and Applying internal PKI certificates on VCSA 6.7

The Story

Everyone loves a good story. Well today it begins with something I wanted to do for a while but haven’t got around to. I remember adjusting the certificates on 5.5 vCenter and it caused a lot of grief. Now it may have been my ignorance it also may have been due to poor documentation and guides, who knows. Now with VMware now going full linux (Photon OS) for the vCenter deployments (much more light weight) it’s still nice to see a green icon in your web browser when you navigate the nice new HTML5 based management interface. Funny that the guide I followed, even after applying their own certificate still had a “not secure” notification in their browser.

This might be because he didn’t install his Root CA certs into the computers trusted CA store on the machine he was navigating the web interface from. However I’m still going to thank RAJESH RADHAKRISHNAN for his post in VMArena. it helped. I will cover some alternatives however.

Not often I do this but I’m lazy and don’t feel like paraphrasing…

VCSA Certificate Overview

Before starting the procedure just a quick intro for managing vSphere Certificates, vSphere Certificates can manage in two different modes

VMCA Default Certificates

VMCA provides all the certificates for vCenter Server and ESXi hosts on the Virtual Infrastructure and it can manage the certificate lifecycle for vCenter Server and ESXi hosts. Using VMCA default the certificates is the simplest method and less overhead.

VMCA Default Certificates with External SSL Certificates (Hybrid Mode)
This method will replace the Platform Services Controller and vCenter Server Appliance SSL certificates, and allow VMCA to manage certificates for solution users and ESXi hosts. Also for high-security conscious deployments, you can replace the ESXi host SSL certificates as well. This method is Simple, VMCA manages the internal certificates and by using the method, you get the benefit of using your corporate-approved SSL certificates and these certificates trusted by your browsers.

Here we are discussing about the Hybrid mode, this the VMware’s recommended deployment model for certificates as it procures a good level of security. In this model only the Machine SSL certificate signed by the CA and replaced on the vCenter server and the solution user and ESXi host certificates are distributed by the VMCA.

I guess before I did the whole thing, were today I’m just going to be changing the cert that handles the web interface, which is all I really care about in this case.

Requirements

  • Working PKI based on Active directory Certificate Server.
  • Certificate Server should have a valid Template for vSphere environment
    Note : He uses a custom template he creates. I simply use the Web Server template built in to ADCS.
  • vCenter Server Appliance with root Access

Requesting the Certificate

Now requesting the certificate requires shell access, I recommend to enable SSH for ease of copying data to and from the VCSA as well as commands.

To do this log into the physical Console of the VCSA, in my case it’s a VM so I opened up the console from the VCSA web interface. Press F2 to login.

Enable both SSH and BASH Shell

OK, now we can SSH into the host to make life easier (I used putty):

Run

 /usr/lib/vmware-vmca/bin/certificate-manager

and select the operation option 1

Specify the following options:

  • Output directory path: path where will be generated the private key and the request
  • Country : your country in two letters
  • Name : The FQDN of your vCSA
  • Organization : an organization name
  • OrgUnit : type the name of your unit
  • State : country name
  • Locality : your city
  • IPAddess : provide the vCSA IP address
  • Email : provide your E-mail address
  • Hostname : the FQDN of your vCSA
  • VMCA Name: the FQDN where is located your VMCA. Usually the vCSA FQDN

Once the private key and the request is generated select Option 2 to exit

Next we have to export the Request and key from the location.

There are several options on how to compete this. Option 1 is how our source did it…

Option 1 (WinSCP)

using WinSCP for this operation .

To perform export we need additional permission on VCSA , type the following command for same

chsh -s /bin/bash root

Once connected to vCSA from winscp tool navigate the path you have mentioned on the request and download the vmca_issued_csr.csr file.

Option 2 (cat)

Simple Cat the CSR file, and use the mouse to highlight the contents. Then paste it into ADCS Request textbox field.

Signing The Request

Now you simply Navigate to your signing certificate authorizes web interface. usually you hope that the PKI admin has secured this with TLS and is not just using http like our source, but instead uses HTTPS://FQDN/certsrv or just HTTPS://hostname/certsrv.

Now we want to request a certificate, an advanced certificated…

Now simply, submit and from the next page select the Base 64 encoded option and Download the Certificate and Certificate Chain.

Note :- You have to export the Chain certificate to .cer extension , by default it will be PKCS#7

Open Chain file by right click or double click navigate the certificate -> right click -> All Tasks -> export and save it as filename.cer

Now that we have our signed certificate and chains lets get to importing them back into the VCSA.

Importing the Certificates

Again there are two options here:

Option 1 (WinSCP)

using WinSCP for this operation .

To perform export we need additional permission on VCSA , type the following command for same

chsh -s /bin/bash root

Once connected to vCSA from winscp tool navigate the path you have mentioned on the request and upload the certnew.cer file. Along with any chain CA certs.

Option 2 (cat)

Simply open the CER file in notepad, and use the mouse to highlight the contents. Then paste it into any file on the VCSA over the putty session.

E.G

vim /tmp/certnew.cer

Press I for insert mode. Right click to paste. ESC to change modes, :wq to save.

Run

 /usr/lib/vmware-vmca/bin/certificate-manager

and select the operation option 1

Enter administrator credentials and enter option number 2

Add the exported certificate and generated key path from previous steps and Press Y to confirm the change

Custom certificate for machine SSL: Path to the chain of certificate (srv.cer here)
Valid custom key for machine SSL: Path to the .key file generated earlier.
Signing certificate of the machine SSL certificate: Path to the certificate of the Root CA (root.cer , generated base64 encoded certificate).

Piss what did I miss…

That doesn’t mean shit to me.. “PC Load letter, wtf does that mean!?”

Googling, the answer was rather clear! Thanks Digicert!

Since I have an intermediate CA, and I was trying either the Intermediate or the offline it would fail.. I needed them both in one file. So opened each .cer and pasted them into one file “signedca.cer”

Now this did take a while, mostly around 70% and 85% but then it did complete!

Checking out the web interface…

Look at that green lock, seeing even IP listed in the SAN.. mhm does that mean…

Awwww yeah!!! even navigating the VCSA by IP and it still secure! Woop!

Conclusion

Changing the certificate in vCenter 6.7 is much more flexable and easier using the hybird approach and I say thumbs up. 😀 Thanks VMware.

Ohhh yea! Make sure you update your inventory hosts in your backup software with the new certificate else you may get error attempting backup and restore operations, as I did with Veeam. It was super easy to fix just validate the host under the inventory area, by going through the wizard for host configuration.