Category: Uncategorized

Hello everyone!

Been a little while since I posted a blog entry. I’m at the OpenStack summit in Boston this week, and I am doing a demo of Mellanox’s SwitchDev running on Ubuntu 17.04, Zesty Zapus.

By default, we at Ubuntu enable SwitchDev in the Kernel. Also, because Zesty is running the latest stable kernel (4.10) and has the latest updated SystemD, SwitchDev runs out of the box. No need to configure or apply any patches, it just works.

I am demoing in the Mellanox booth at OpenStack Summit Ubuntu, running on a Mellanox SN2100 switch, with SwitchDev enabled. This demo also shows that because it is running Standard Ubuntu Server, you can install other Canonical technologies on it, such as Metal-as-a-Service (MAAS) and Juju. The Demo is running MAAS on the switch, and it is managing 10 servers running in an Ubuntu Orange Box and with Juju, we are deploying OpenStack on to the servers in the Orange Box.

You can view the Demo by clicking here.

Let me know in the comments what you think. There is no audio.

Hello everyone! Not sure how helpful this article will be, but I found it quite helpful for myself, and I just want to really just write down what I did so that if I have to do this in the future, which I have now done this about 16 times in the last 4 years, I have a reference.

The premise of this article is mainly how to create a Linux router in a Virtual machine so that you have direct access to your VM network from any machine on your network.

Many of us that have virtual home labs, usually will use network segmentation to separate our VM’s. For example, you may want to build an OpenStack lab, but not want it to be impacted by your home DHCP server or impact that network so that your kids or guests don’t mess around with it, so you’ll put it on a private network that only those VM’s can access, and perhaps use NATing for Internet access. While this does work, sometimes if you want to work on the systems, like if you spin up a Horizon server, you need a jump box on both your regular network and your internal network, which can be a hassle. Or, if you want to have some people have access to your environment, but don’t want them on the full network, this method works really well.

Basically, I came up with this need about 4 years ago when I worked for a company that had very strict networking policies. I was testing OpenStack in our Hyper-V environment, but it had no access to the Internet. To get around this, I created a VM on the Hyper-V host that had 3 NIC’s, one that used the Hosts Network adapter that had access to the Internet for updates as the main egress port, another NIC that was used to manage VM’s from my workstation, and the last was an internal network that was going to be used for the intercommunications of the OpenStack nodes. ‘

This VM I decided was going to run CentOS, since the company was a Red Hat shop, and I am quite a bit more familiar with Red Hat (even though as I write this, I found that as I have worked with Canonical for over a year now, I have forgotten some of the slight differences between the two). I managed to build a CentOS router, and it worked. I was able to get my machines in the private network out to the internet without having to NAT each one out the internet port that would have caused bottlenecks with the other VM’s, and the best part, I was able to connect directly to the VM’s from my workstation without needing a jump box, so I could share the OpenStack environment with my co-workers and they could test it.

So, in my house, I am doing something quite similar. I have a KVM host that has 4 networks, my external network with my private IP addresses, my internal network on the 10.1.10.0/24 subnet with its own DHCP and DNS servers, and my private internal KVM network that is not NAT’d (192.168..2.0/24) and my KVM NAT’d network (192.168.122.0/24).

Now, I know what your thinking, why didn’t I just use the NAT’d address range and all my machines would have access to the internet and I could download files and not have to do all this. Your correct on one part. The machines would have access to the Internet, and they have access to everything on my internal network, however, its one way only. I cannot on my workstation connect to those servers unless I use a jump box, which i do not want to do. Of course, I could have adjusted the settings in KVM network or even added the NAT’d routers IP address to my Routing table on my core router point the KVM host as the next hop for resolution. That I can do in my home lab, but what if I’m not running KVM? What if I’m running Hyper-V or VMware ESXi? While it is possible to do the same thing on the other Hypervisors, if you are not familiar with Powershell or the esx-cli command, you could spend hours on this, and potentially break the core networking on those hosts. This method is quick and somewhat painless.

First thing you need to do is build a VM, with NIC’s on each network segment you want it to manage. In this example, I just put two, one on my Internal 10 network, and one on the non-NAT’d network. I installed CentOS 7 on this, minimal install, and I gave it a static IP on my 10 network, the gateway and DNS servers on that network, as well as the 192.168.2.1/24 IP address on the other interface but no gateway or DNS. After it was installed, I ran yum update to update the server and rebooted it. After the reboot, I enabled IPv4 forwarding in the /etc/sysctl.conf file by adding net.ipv4.ip_forward = 1 to it. Then run sysctl -p to make the changes take effect. Now we are ready to setup the firewall rules to allow IP masquerading and forwarding. Run ip a to see the devices and what networks they are connected to. Then, run:

firewall-cmd --direct --add-rule ipv4 nat POSTROUTING 0 -o ext-eth -j MASQUERADE
firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i int-eth -o ext-eth -j ACCEPT
firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i ext-eth -o int-eth -m state --state RELATED,ESTABLISHED -j ACCEPT
firewall-cmd --zone=trusted --add-source=192.168.2.0/24

That is it on the server. Now, on your router, the main one, you need to add the static route to it so that it knows how to forward packets to your 192.168.2.0/24 network to it. Most home routers have this capability in the Advanced section usually labeled “Static Routes.” Here, enter in the network, 192.168.2.0 and the netmask or 255.255.255.0, and the next hop or source IP depending on how its labeled will be the IP address of your Virtual Router you just built, on the 10.1.10.0 network. Give it the static IP address you gave the router, and save the configuration.

Now test that you can get to a Virtual Machine that is attached to the 192.168.2.0 network and is using your virtual router as its gateway.

ping 192.168.2.2

You should get a reply. Try to SSH to that machine and if you get it, your done. Last thing you need to do if everything test right, is make the firewall rules permanent, but typing the following:

firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -o eno16777984 -j MASQUERADE
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i eno33557248 -o eno16777984 -j ACCEPT
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i eno16777984 -o eno33557248 -m state --state RELATED,ESTABLISHED -j ACCEPT
firewall-cmd --permanent --zone=trusted --add-source=192.168.2.0/24

And thats it. You can do this for any other network you build in your VM environment if you want to be able to access those machines from any other client.

If you have any questions, or just want to leave a comment on if this helped you, leave ’em on the bottom.

Thanks!

[ayssocial_buttons id=”2″]

Hello everyone! Hope everyone is having a good start to summer. I’ve been extremely busy as usual, but I had a moment of time to start this new HOWTO, How to Install OpenVPN in Ubuntu 16.04 so that you can connect to your home machines or browse the Internet safely from anywhere in the world. If you don’t know what a VPN, or Virtual Private Network is, this is a simple answer. Its a Network that allows encrypted information between the VPN server and your machine so that it appears like it is on the same network as the rest of your home equipment, but are over the internet. This is useful if you are working remote and need access to your servers at home, but don’t have them connected directly to the Internet with their own IP address.

The main reason I am writing this, is because I had to setup a VPN connection to my home lab so that my co-workers could connect to the various network equipment I have in my lab and test on this equipment. So I setup a VPN so that they can connect into my lab, get on the switches, get on the console concentrator, and power up, power down, and work on the switches remotely. It’s extremely secure since I have to give the user a certificate to connect to my VPN server and I control them so that if they don’t need access anymore, I kill that certificate in my Certificate Authority and they can no longer login on my network.

This HowTo is going to show how I setup OpenVPN on Ubuntu 16.04, and secured the system using UFW so that only 2 ports are exposed to the world to limit the attack surface of my VPN server.First thing I did was install Ubuntu Server 16.04. I used Virtual Machines quite extensively, so that is how this started. I created a VM, made sure to set it’s network interface to my external IP pool, gave it 1GB of RAM and 1 vCPU, 16GB of storage and installed Ubuntu on it. The only other software I installed was OpenSSH-Server and that was completed. I then modified the /etc/network/interfaces file so that it had a static IP address, gateway and DNS server information, subnet range, and what the device was called. This is important since it will come into play when you are setting up the the VPN server so that it knows what to tunnel through for the firewall rules. In this example, the device is ens160, but it will be whatever your system calls it, typically this is eth0.

After the server was installed, I ran the following to make sure it was all up to date and had the latest repositories:

sudo apt update && sudo apt upgrade -y

I reboot the server after this so that it used the new IP address, and was running with the latest updates.

I than ran sudo apt install openvpn easy-rsa to install the required binaries I needed.

I than ran make-cadir ~/openvpn-ca. This command creates the minimum config files and sources so that you can build a Certificate Authority (CA) on the system. This is required to create the certificates that will be used by the server and the clients to connect and verify the systems so that they trust each other.

Once that completes, change directory to the CA folder cd ~/openvpn-ca, and modify the vars file vi vars. Go to the section that looks like this:

export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="SanFrancisco"
export KEY_ORG="Fort-Funston"
export KEY_EMAIL="me@myhost.mydomain"
export KEY_OU="MyOrganizationalUnit"

Modify these variables for your needs. Also, find the variable KEY_NAME and change it to the name of your server.

export KEY_NAME="server"

Now, you are ready to build the CA. Run source vars and you should get the following output:

NOTE: If you run ./clean-all, I will be doing a rm -rf on /home/wililupy/openvpn-ca/keys

Go ahead and run ./clean-all to make sure that the environment is good to go. Now we are ready to build the CA. Run the command ./build-ca.

You will be given a bunch of options, most of which you already set in the vars file, so just hit enter to accept them.

We now are ready to create the server certificate, the key and encryption files. This is done with the command ./build-key-server server where server is the name of your VPN server. Once again, it looks at the vars file and uses those for the defaults, and then it will have two prompts you need to answer. The first one is:

Certificate is to be certified until June 13 15:26:11 2026 GMT (3650 days)
Sign the certificate? [y/n]:y

The second one is:

1 out of 1 certificate requests certified, commit? [y/n]y

It will update the database and now we are ready to generate the encryption key. Use the command ./build-dh to do this. It takes about 2 minutes for this command to complete. You will see …. and * while it randomizes. Lastly, we need to generate the HMAC signature. To do this use the following command:

openvpn --genkey --secret keys/ta.key

Now we are ready to build the client certificate so that you can connect to your VPN server. While still in the ~/openvpn-ca directory, and while you are still sourced to vars, run ./build-key client where client is the hostname of the client machine/username. Make sure you say Y at the prompts to sign the certificate and commit the certificate.

You are now ready to copy the required files to the /etc/openvpn directory so that we can configure openvpn to run.

Go into the keys directory:

cd ~/openvpn-ca/keys and copy the certificates and keys to /etc/openvpn

sudo cp ca.crt ca.key server.crt server.key ta.key dh2048.pem /etc/openvpn

We are now ready to copy the example server.conf file to the /etc/openvpn directory so that we can configure the server. You have to uncompress it first:

gunzip -c /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz | sudo tee /etc/openvpn/server.conf

Now we have to modify the file so that it works with our environment.

sudo vi /etc/openvpn/server.conf

Search for redirect-gateway and remove the ; to uncomment the setting so that it looks like this:

push "redirect-gateway def1 bypass-dhcp"

Then below that is the “dhcp-option DNS” settings. Uncomment them and set them to your DNS servers or leave them as the defaults. I changed them to my internal DNS so that users can use my internal names of my systems and get to them easier than searching around for IP addresses. Next, uncomment the HMAC section by searching for tls-auth and just under that variable, add key-direction 0. Last, search for user and uncomment user nobody and group nogroup so that the service knows who to run as.

Now we have to allow the system to do IP Forwarding and modify the Firewall to secure the system. First, modify /etc/sysctl.conf and uncomment net.ipv4.ip_forward=1 and then save the file and run sudo sysctl -p to make the changes take effect.

Next, modify the /etc/ufw/before.rules so we can setup Masquerading for the VPN server. Right after the #  ufw-before-forward option, enter the following:

*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.8.0.0/8 -o ens160 -j MASQUERADE
COMMIT

Remember when I said to remember your network device from when we were setting up the static IP of the server? After the -o option in the before-rules file, that is where the name of your device goes. Save the file. Now we have to set UFW to forward by default. Modify the /etc/default/ufw file and find the DEFAULT_FORWARD_POLICY and set it to "ACCEPT". Save this file and now all we have to do is allow ufw the openvpn port and protocol and enable the ssh variable:

sudo ufw allow 1194/udp
sudo ufw allow 22/tcp

Now we need to disable and re-enable ufw so that it will read the changes in the files we modified:

sudo ufw disable
sudo ufw enable

Now we are ready to start OpenVPN. Since our configuration is called server.conf, when we start openvpn, we will tell it @server so that it will use that configuration. Nice this about openvpn, is that we can have multiple configuration, and multiple instances of the VPN server running, all we have to do is trail @configname after it and it will run that config. To start openvpn, run the following command:

sudo systemctl start openvpn@server

Check that it is running by running sudo systemctl status openvpn@server and look for the Active: active (running). If everything looks good, set it to run at startup by running sudo systemctl enable openvpn@server.

Now we are ready to setup the clients. First thing I did was create a new directory for the client files so that I could scp them to my colleagues and my different machines and devices (OpenVPN works on Windows, MacOSX, Linux, iPhone, and Android)

mkdir -p ~/client-configs/files

Also, because there will be multiple keys in this folder for different machines, I locked it down so that only I had access to that folder: chmod 700 ~/client-configs/files.

Next, I copied the example configuration for clients to this location:

cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf ~/client-configs/base.conf and then edited the file to meet my client needs.

First thing is to search for remote in the file and change the server_IP_address to the public IP address of your VPN server. Next uncomment the user and group variables by deleting the leading ‘;’.

Next, search for the ca.crt and client.crt sections and comment them out with the ‘#’, and finally, add the key-direction 1 in the file somewhere so that it knows how to use the keys. Save the file and you’re done.

Now, I found this really cool script at https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-16-04.

#!/bin/bash

# First argument: Client identifier

KEY_DIR=~/openvpn-ca/keys
OUTPUT_DIR=~/client-configs/files
BASE_CONFIG=~/client-configs/base.conf

cat ${BASE_CONFIG} \
    <(echo -e '') \
    ${KEY_DIR}/ca.crt \
    <(echo -e '\n') \
    ${KEY_DIR}/${1}.crt \
    <(echo -e '\n') \
    ${KEY_DIR}/${1}.key \
    <(echo -e '\n') \
    ${KEY_DIR}/ta.key \
    <(echo -e '') \
    > ${OUTPUT_DIR}/${1}.ovpn

Create a file called make_config.sh and paste the script into that file. Save the file, then make it executable by running chmod 700 ~/client-configs/make_config.sh.

If you remember, we created a client certificate and key previously, using the build-key client command. This created a client.key file in the ~/openvpn-ca/keys directory. We are now going to build a configuration for the VPN that uses these keys. Make sure you are in the ~/client-configs directory and run ./make_config.sh client where client is the name of the client configuration you are creating. The name should match what you entered in the build-key command previously. This will generate a file called client.ovpn which needs to be copied to the client. I use SCP or SFTP to transfer the files between Linux and MacOSX, but for Windows or IOS or Android, getting the certificate file on the system may be a little trickier. For Windows, I use FileZilla or WinSCP. Just login to the VPN server and copy the ovpn file to your home directory on the system.

In Ubuntu Desktop 16.04, make sure you have OpenVPN installed, (sudo apt install network-manager-openvpn-gnome) open up Network Manager, go to VPN Connections, Configure VPN, and click Add. From the drop down, select Import a saved VPN configuration… and browse to your .ovpn file. Select Open and verify that everything looks right, the vpn server’s IP address, the name of the certificates, and click Save. Now you are ready to test. Connect your new VPN and verify that you connect successfully. Check your network devices for the new tun0 device and IP address of 10.8.0.x (ifconfig tun0). Try to connect to a server in your internal network and verify that everything is working as normal.

And thats it. It really isn’t that difficult to setup. If you have any questions, or if this blog helped you in anyway, let me know. I like to think that I’m helping someone out there.

Thanks!

[ayssocial_buttons id=”2″]

Funniest thing I heard tonight in IRC (yes, I know I’m old but we use this for work, way better then private chat programs, it allows for us to interact with customers directly and collaborate and come up with solutions quicker) there was an argument happening about solving a problem and one of the put downs was “Your parents met on Everquest!” And he replied back, “Yeah, they did and and I met my wife on Xbox live!” Think about that for a second. Video games and interactions online are creating the future humans on this planet. While some people might say that’s messed up, is it really? We used to meet people in bars, or by totally chance, but now people can meet doing the thing we like, and what we like to do, play a video game with death match killing someone. Go online, shoot them in the face with a gun, tea bag their corpse, and then ask if they want to be friends after the times up. Next thing you know, you have something in common with them besides that you like to kill zombies and kick field goals, but you like hiking and fishing and surfing.
People say that I don’t want my kids playing video games because they are not interacting with kids. I say, sit down, and watch your kids play. My son, who has Aspergers, is a hero online. People look up to him online. He gets other kids asking him how to do things. He’s “normal” online. No, he’s normal all the time, he’s just judged differently online.
So, before you say that the Internet is bad, and online is bad, take a step back, and realize, your parents could have met killing each other in Doom. ????

[ayssocial_buttons id=”2″]

Wow! Talk about a crazy week. And it couldn’t have been better. After many days trying to get a working kernel in Snappy Ubuntu Core with customized Kernel Modules built against the kernel into a single snap bundle was easy is like saying giving a cat a bath is easy (I did that today as well, waiting for the kernel to finish building…)

Anyways, this blog post is about how I managed to get it working using snapcraft 2.8.4, running on Ubuntu 16.04, with kernel 4.4.0-21-generic from the Ubuntu kernel repository, and building the modules and then depmod’ing them and making sure they get put into the final snap.

First thing I did, download snapcraft. Easy to do now that it is in the main repository for Ubuntu 16.04. All you do is sudo apt update then sudo apt install snapcraft -y. Once that is complete, then you can run snapcraft init and you have a blank snapcraft.yaml file waiting for you to tell it what you are going to build. Below is an example that I used for my kernel snap:

name: custom-kernel
 version: 4.4.6
 summary:  custom kernel
 description:  custom kernel for Snappy
 type: kernel
 parts:
    kernel:
      plugin: kernel
      source: .
      kconfigfile: 4.4-config
      kconfigs:
        - CONFIG_SQUASHFS=m
      kernel-initrd-modules:
        - squashfs
        - ahci

I then ran sudo apt source linux-image-`uname -r` to get the latest kernel source that is running on Xenial. After that, I copied my snapcraft.yaml file to the location I downloaded the kernel source files to and copied the default kernel config file from /boot/config-4.4.0-21-generic to the kernel source directory and renamed it to 4.4-config. I then ran snapcraft build and after two hours, and giving my cat a bath, it was done. I was then able to build my custom kernel modules against these headers so that I knew they would have the same symbols for it. All I did was modify my makefile so that instead of it looking in /lib/modules/`uname -r`/build for my modules, I pointed it to /home/wililupy/linux-4.4.0/parts/kernel/build and modified the kernel version to 4.4.6. I then ran make all and it built my *.ko files I needed.

I then copied the *ko files to /home/wililupy/linux-4.4.0/parts/install/lib/4.4.6/extra (I had to create the extra directory mkdir -p extra) and then ran depmod -b ~wililupy/linux-4.4.0/parts/install 4.4.6 to update the modules symbols for the compiled kernel. I then ran  snapcraft snap. It then built my kernel.snap file. I then ran ubuntu-device-flash, which you will need to download from here. Run sudo ./ubuntu-device-flash core 16 --channel=edge --kernel=custom-kernel_4.4.6_amd64.snap --gadget=canonical-pc --os=ubuntu-core -o custom-snappy.imgand you will have your custom image that you are now ready to install on your device.

Since I work heavily in the Whitebox switching area, I use ONIE to install my software on whitebox switches. Read up on how this works from their Github or if you want, ask me in the comments below.

Happy Hacking!!

[ayssocial_buttons id=”2″]

Hello,

Don’t think too many of you noticed this site being down, but I had to rebuild it. I had a firewall issue that had to be corrected, and during the initial configurations, all the moving pieces wouldn’t play nice, so it was easier to rip it all down and start over from scratch.

So, now that I am all back, be looking forward to some randomness of information, mostly for my own need, but if you find it helpful, let me know in the comments and I’ll keep up with it.